<
https://www.techdirt.com/2025/04/10/ai-crawlers-are-harming-wikimedia-bringing-open-source-sites-to-their-knees-and-putting-the-open-web-at-risk/>
"The current rapid advances in generative AI are built on three things.
Computing power, some clever coding, and vast amounts of training data. Lots of
money can buy you more of the first two, but finding the necessary training
material is increasingly hard. Anyone seeking to bolster their competitive
advantage through training needs to find fresh sources. This has led to the
widespread deployment of AI crawlers, which scour the Internet for more data
that can be downloaded and used to train AI systems. Some of the prime targets
for these AI scraping bots are Wikimedia projects, which claim to be “the
largest collection of open knowledge in the world”. This has now become a
serious problem for them:
We are observing a significant increase in request volume, with most of this
traffic being driven by scraping bots collecting training data for large
language models (LLMs) and other use cases. Automated requests for our
content have grown exponentially, alongside the broader technology economy,
via mechanisms including scraping, APIs, and bulk downloads. This expansion
happened largely without sufficient attribution, which is key to drive new
users to participate in the movement, and is causing a significant load on
the underlying infrastructure that keeps our sites available for everyone.
Specifically:
Since January 2024, we have seen the bandwidth used for downloading
multimedia content grow by 50%. This increase is not coming from human
readers, but largely from automated programs that scrape the Wikimedia
Commons image catalog of openly licensed images to feed images to AI models.
Our infrastructure is built to sustain sudden traffic spikes from humans
during high-interest events, but the amount of traffic generated by scraper
bots is unprecedented and presents growing risks and costs.
AI crawlers seek to download as much material as possible, including the most
obscure, so Wikimedia projects that are optimized for human use incur extra
costs:
While human readers tend to focus on specific – often similar – topics,
crawler bots tend to “bulk read” larger numbers of pages and visit also the
less popular pages. This means these types of requests are more likely to
get forwarded to the core datacenter, which makes it much more expensive in
terms of consumption of our resources.
Wikimedia’s analysis shows that 65% of this resource-consuming traffic is
coming from bots, whereas the overall pageviews from bots are about 35% of the
total. As the
Diff news story notes, this is becoming a widespread problem
not just for Wikimedia, but across the Internet. Some companies are responding
with lawsuits, but for another important class of sites this is not a practical
option."
Cheers,
*** Xanni ***
--
mailto:xanni@xanadu.net Andrew Pam
http://xanadu.com.au/ Chief Scientist, Xanadu
https://glasswings.com.au/ Partner, Glass Wings
https://sericyb.com.au/ Manager, Serious Cybernetics
