<
https://techcrunch.com/2025/03/27/open-source-devs-are-fighting-ai-crawlers-with-cleverness-and-vengeance/>
"AI web-crawling bots are the cockroaches of the internet, many software
developers believe. Some devs have started fighting back in ingenuous, often
humorous ways.
While any website might be targeted by bad crawler behavior — sometimes taking
down the site — open source developers are “disproportionately” impacted,
writes Niccolò Venerandi, developer of a Linux desktop known as Plasma and
owner of the blog
LibreNews.
By their nature, sites hosting free and open source (FOSS) projects share more
of their infrastructure publicly, and they also tend to have fewer resources
than commercial products.
The issue is that many AI bots don’t honor the Robots Exclusion Protocol
robot.txt file, the tool that tells bots what not to crawl, originally created
for search engine bots.
In a “cry for help” blog post in January, FOSS developer Xe Iaso described how
AmazonBot relentlessly pounded on a Git server website to the point of causing
DDoS outages. Git servers host FOSS projects so that anyone who wants can
download the code or contribute to it.
But this bot ignored Iaso’s robot.txt, hid behind other IP addresses, and
pretended to be other users, Iaso said.
“It’s futile to block AI crawler bots because they lie, change their user
agent, use residential IP addresses as proxies, and more,” Iaso lamented.
“They will scrape your site until it falls over, and then they will scrape it
some more. They will click every link on every link on every link, viewing the
same pages over and over and over and over. Some of them will even click on the
same link multiple times in the same second,” the developer wrote in the post."
Cheers,
*** Xanni ***
--
mailto:xanni@xanadu.net Andrew Pam
http://xanadu.com.au/ Chief Scientist, Xanadu
https://glasswings.com.au/ Partner, Glass Wings
https://sericyb.com.au/ Manager, Serious Cybernetics
