<
https://www.techdirt.com/2025/07/14/tackling-the-ai-bots-that-threaten-to-overwhelm-the-open-web/>
"It is a measure of how fast the field of AI has developed in the three years
since
Walled Culture the book (free digital versions available) was published
that the issue of using copyright material for training AI systems, briefly
mentioned in the book, has become one of the hottest topics in the copyright
world, as numerous posts on this blog attest.
The current situation sees the copyright industry pitted against the generative
AI companies. The former wants to limit how copyright material can be used,
while the latter want a free for all. But that crude characterization does not
mean that the AI companies can be regarded as on the side of the angels when it
comes to broadening access to online material. They may want unfettered access
for themselves, but it is becoming increasingly clear that as more companies
rush to harvest key online resources for AI training purposes, they risk
hobbling access for everyone else, and even threaten the very nature of the
open Web.
The problem is particularly acute for non-commercial sites offering access to
material for free, because they tend to be run on a shoestring, and are thus
unable to cope easily with the extra demand placed on their servers by AI
companies downloading holdings en masse. Even huge sites like the Wikimedia
Projects, which describes itself as “the largest collection of open knowledge
in the world”, are struggling with the rise of AI bots:
We are observing a significant increase in request volume, with most of this
traffic being driven by scraping bots collecting training data for large
language models (LLMs) and other use cases. Automated requests for our
content have grown exponentially, alongside the broader technology economy,
via mechanisms including scraping, APIs, and bulk downloads. This expansion
happened largely without sufficient attribution, which is key to drive new
users to participate in the movement, and is causing a significant load on
the underlying infrastructure that keeps our sites available for everyone.
Specifically:
Since January 2024, we have seen the bandwidth used for downloading
multimedia content grow by 50%. This increase is not coming from human
readers, but largely from automated programs that scrape the Wikimedia
Commons image catalog of openly licensed images to feed images to AI models.
Our infrastructure is built to sustain sudden traffic spikes from humans
during high-interest events, but the amount of traffic generated by scraper
bots is unprecedented and presents growing risks and costs."
Cheers,
*** Xanni ***
--
mailto:xanni@xanadu.net Andrew Pam
http://xanadu.com.au/ Chief Scientist, Xanadu
https://glasswings.com.au/ Partner, Glass Wings
https://sericyb.com.au/ Manager, Serious Cybernetics
