The website owners whose content is harvested by various AI crawlers have finally received a unified solution. According to news on November 1st, OpenAI, Google, Microsoft, and Anthropic announced that they will jointly adopt a new standard, allowing content publishers to declare “reject AI scraping” at once, without having to negotiate with each site individually.
A switch, used by multiple devices
This mechanism was submitted by Anthropic’s technology leader Kevin Kelley to the Internet Engineering Task Force (IETF), and is directly embedded in the existing robots.txt file of the website. In the past, robots.txt only told search engines which pages could be crawled and which couldn’t. Now, its scope has been extended to include AI training. The difference is that previously, publishers had to register their preferences separately if they wanted to block a particular AI; with the new standard, one entry can notify multiple entities simultaneously. For those media, forums, and independent websites that are troubled by having material quietly fed into models, this means saving a bunch of repetitive negotiations.
From fighting each other to a unified signal
The background behind promoting this is very practical—AI crawlers grab web pages everywhere and stuff content into training sets, causing significant trouble for Baidu. The old method of “each company setting its own rules” can no longer keep up. Anthropic has taken the lead in adopting these standards, OpenAI says it will implement them within the next few months, and Google and Microsoft expect to follow next year. Even Meta’s spokesperson has expressed support for establishing universal signals, though the implementation date is still undecided. When the four giants with the most computing power rarely reach an agreement on “how to be rejected,” the content copyright dispute has finally moved from chaos to a negotiable tabletop.