<
https://www.404media.co/it-is-trivially-easy-to-use-reddit-to-manipulate-ai-search-research-suggests/>
"A tiny snippet of user-generated text as short as 13 words long is often
enough to manipulate the AI agents that power tools like ChatGPT and Google’s
AI search, new research shows. The study suggests that it is trivially easy for
brands to inject promotional content on sites like Reddit, Quora, and Wikipedia
with the end goal of poisoning or manipulating the output of AI tools.
The preprint research, done by Hal Triedman, Tingwei Zhang, and Vitaly
Shmatikov of Cornell University, is called “Deep-research agents can be
poisoned via user-generated content” and provides a mechanism and research
basis for a problem that has been noticed by Reddit moderators and Wikipedia
editors, namely that their websites are getting flooded with promotional
content from brands trying to do AEO, or AI-engine optimization.
404 Media
has repeatedly reported on this booming industry, in which brands try to
promote their product by seeding the websites that AI tools most often cite and
scrape from with inauthentic and spammy content.
The Cornell research finds that deep research agents, which are the real-time
scrapers that tools like Google AI search and ChatGPT use to retrieve web
content with citations in response to user queries, cite user-generated content
from sites like Reddit or Wikipedia in roughly half of all queries, and that
nearly a quarter of all citations come from user-generated websites. The paper
suggests that what we have been seeing is basically "Redditor suggests you put
glue on your pizza" as a service, or an end-to-end attack against the systems
that increasingly dominate the ways that people access information online. The
researchers found that “a single poisoned Reddit comment can influence
generated outputs for an entire cluster of related [AI] queries,” the paper
said.
“We show that a tiny snippet—just 13 words—of retrieved text on a UGC website
like Reddit, Wikipedia, Quora, Facebook, etc. can change AI agents to output
spam / scam content pretty consistently,” Triedman told
404 Media.
The fact that such small snippets of texts in even single comments can be used
to ultimately trick LLMs raises questions about whether Reddit’s volunteer
moderators or Wikipedia’s volunteer editors are going to be able to durably
protect the communities they moderate and edit from AI manipulation over time."
Cheers,
*** Xanni ***
--
mailto:xanni@xanadu.net Andrew Pam
http://xanadu.com.au/ Chief Scientist, Xanadu
https://glasswings.com.au/ Partner, Glass Wings
https://sericyb.com.au/ Manager, Serious Cybernetics
