<
https://www.techdirt.com/2026/06/25/the-free-and-open-web-is-under-attack-at-the-ietf/>
"The ability to access publicly available information using automated tools is
a central value and benefit of a free and open internet. Automated access—often
called crawling or scraping—powers important, useful tools for locating,
preserving, and analyzing online information. For example, crawling and
scraping helps journalists, researchers, and watchdog organizations report the
news, find security flaws, and investigate discrimination. Crawling the web
allows non-profits like the Internet Archive to preserve historical copies of
websites. Tools for automated comparison shopping allow consumers to find the
best deals on items they want to buy. And so on.
Yet the open internet access is increasingly under threat from publishers and
Big Tech companies alike. Fearing lost advertising and licensing revenues,
website operators increasingly claim that they need to lock down their sites
from bots that crawl public web content to train or operate AI models. Some
companies are even trying to embed their business models into internet
standards by changing Internet Engineering Task Force (IETF) technical
standards that shape much of the internet.
Many of their economic anxieties are understandable. AI bots can strain
websites’ infrastructure, in some cases, degrading site performance or taking
them offline altogether. Upgrading systems costs money that some sites may not
have. And AI is likely to disrupt the business models many publishers adopted
in response to the rise of the internet, if users rely on AI overviews instead
of visiting source websites.
However reasonable these fears may be, the answer is not to change the IETF
standards from neutral protocols that encourage openness to restrictive
requirements designed to monetize internet access."
Cheers,
*** Xanni ***
--
mailto:xanni@xanadu.net Andrew Pam
http://xanadu.com.au/ Chief Scientist, Xanadu
https://glasswings.com.au/ Partner, Glass Wings
https://sericyb.com.au/ Manager, Serious Cybernetics
