Reddit’s CEO says blocking AI bots has been a “pain in the ass.” Search engines and AI developers, other than Google and OpenAI, can no longer access Reddit posts. Without agreements like the ones it has with those companies, Reddit says it can’t control how its data is used or appears, and it has to block search engines along with AI because the lines between them are blurring.
Other websites also have data scraping complaints. AI search engine Perplexity has been accused of ignoring robots.txt, a website file that tells search engines and AI data scrapers what they have permission to access, and signed a revenue-sharing deal with news outlets this week to smooth things over. Several websites were unable to stop Anthropic because it kept changing what its bot was called. Trying to head off similar complaints, OpenAI worked with publishers on how their content appears in its new AI search engine.
But this is also a fight for survival. Aside from plagiarism, publishers are concerned that AI search engines — which summarize data from websites, instead of just providing a list of links to them — will hoard the web traffic and revenue that keep the lights on.