The Next Chatbot Is Growing on This Year's Compost
Chatbots learned to talk from what people wrote on sites like Reddit. Now marketers fill those sites with fake posts written for the chatbots to read, and every fake that gets through becomes part of what the next chatbot believes.
Earlier this year somebody on Reddit asked a skincare group whether anyone had tried a particular hypochlorous acid spray, a face spray people use for acne. Dozens of people answered. One of them, posting as Primary-Taro4254, said they had been using a similar spray from a company called Honeydew Labs and had "honestly really liked it."
It reads like a stranger trying to help, which is exactly why a post like that gets believed.
Before long, a member of r/SkincareAddiction, a Reddit group with more than a million visitors a week, warned everyone to "be aware of honeydew labs" and its "marketing bots." A moderator took action against the account and set the group to hold back every post that mentions Honeydew Labs until a moderator has read it.
The fake posts have changed shape. Moderators like Maya Adivi used to look for a review that was a little too glowing, a product link with tracking codes in it, or before-and-after pictures. Now a suspected spammer asks a question, such as how everyone keeps track of their skincare routine, and a wave of replies comes back mentioning AI skincare apps. The questions are often open-ended, worded the way a person types a question to a chatbot.
A post worded like that is aimed at the AI tools that read Reddit as much as at the people in the thread.
Where chatbots learned to talk
Everything a chatbot can do, it learned from things people wrote. The patience, the hedging, the way an explanation arrives in about the order a person would give it: all of it came out of a vast pile of human writing, scraped off the open internet and fed in. The chatbot is a very good imitation of all that writing.
Reddit is one of the richest parts of that pile, because it is where people go to tell strangers what actually happened to them. People got so tired of search results written by marketers that they started typing the word "reddit" onto the end of every search, to get to pages written by actual people. Chatbots ended up relying on it too. In May, according to the analytics company Semrush, Reddit was the website cited most often by ChatGPT, Perplexity and Google's AI tools.
A place that people trust because it is full of real people is exactly the place worth filling with fake ones.
Faking what the chatbots read
The faking has a name, and there are agencies that sell it. In late 2023 a group of researchers, most of them at Princeton, measured what makes a chatbot more likely to quote a piece of text, and called the practice Generative Engine Optimization. Citing outside sources helped a great deal. Adding statistics helped. A line that sounded like an expert helped.
Public-relations agencies now quote the exact figures from that study to their clients. One agency's field guide tells clients that being found online used to mean ranking high in search results, and now means being quoted by an AI. Another, summarising more than a million citations from ChatGPT, Gemini and Claude, reports that 82 percent of what those answers cite is press coverage and other public mentions, the kind of attention PR agencies are hired to get for their clients.
Marketing companies teach the Reddit part openly. This is a lesson from Surfer, a company that sells search-marketing software:
How little it takes was measured last May by three researchers at Cornell Tech, a university campus in New York. They wanted to know whether the "deep research" tools that write long, sourced reports could be steered by editing the pages those tools read. Adding about thirteen words to a community web page was enough to get an entirely invented company recommended by name in the finished report. When a research tool read the edited page, it named the invented company 38 to 51 percent of the time, and up to 62 percent when the same words sat on more than one page. They ran it in a simulation so as not to vandalise the real internet. They also measured where the tools went for information, and Reddit was between half and three-quarters of the community pages they read.
Thirteen words, on a page anybody can edit.
When chatbots learn from chatbots
Scientists already know what happens when a machine is fed on machine writing. In 2024 a team led by Ilia Shumailov published a study in Nature showing that AI systems trained on text made by earlier AI systems get worse with each generation. The unusual things go first, the specific details and the rare thing somebody actually said, until what comes back is bland, average text, delivered with the same confidence as before. In one of the study's examples, a model starts with a passage about English church towers. By the ninth generation it is listing jackrabbits.
That study described an accident. The internet filled up with machine writing, the scrapers swept it up, and the quality fell. What is happening on the skincare forum is the same kind of damage, done on purpose, with a product name slipped in.
What the next chatbot learns
Everything posted in public eventually gets scraped, and what gets scraped is what the next chatbot is built out of. A planted thread can fool a chatbot that looks things up to answer somebody's question today, and it also becomes a tiny part of the writing used to train whichever chatbot comes next.
What is being sold to marketers has changed to match. The promise is that a brand mentioned often enough in that writing will be recommended by a chatbot without the chatbot looking anything up, because it learned the brand during training and treats it as something it already knows. A billboard can be taken down. A planted thread that has already been used to train a chatbot cannot be taken back out of it. What is left inside is a slight preference for one brand name over another, spread through a system nobody can open up and inspect.
Reddit says it has been fending off bots for twenty-one years and now uses AI to catch them: about 23 million spam views blocked a day, about 25,000 new spam posts and comments caught a day, nearly two million fake votes removed every day. The tech news site TechCrunch described the arrangement as the company using AI to solve a problem AI largely created.
So the loop now keeps going by itself. People wrote, and chatbots were trained on what they wrote. Machines wrote posts pretending to be people, and other machines tried to catch and remove them. Whatever gets past them is scraped, and the next chatbot is trained on that.
The skincare group's filter will catch the next Honeydew Labs post. The account gets banned and the thread comes down. None of that touches the copies that were already collected. They sit in scraped data somewhere, waiting to be used to train the next chatbot, which will sound patient and reasonable and have no idea that it recommends a face spray because somebody slipped the suggestion into what it learned from.
The fake posting is spread across thousands of small accounts, and no one person controls it. At the other extreme, one man owns nearly every step between what people write and what an AI tells them.