OpenAI, Google, Microsoft and Anthropic back shared AI scrape opt-out
Original title: The biggest names in AI are backing a new way to opt out of scraping
Anthropic already honors the robots.txt token; Google and Microsoft plan support next year, and OpenAI says coming months.
OpenAI, Google, Microsoft, and Anthropic say they will honor a shared “policy token” in robots.txt so publishers can set AI-training preferences once instead of per company. Anthropic technical lead Kevin Kelley proposed the signal through the IETF. Microsoft says it will also separate opt-outs for AI training and grounding, while Bing search listings stay unchanged. Anthropic has adopted the signal; Google and Microsoft plan support next year, and OpenAI says the coming months.
Unconfirmed: Adoption timelines are company statements, and the IETF signal is still a proposal.
- Sites now must signal AI training preferences separately to each company.
- The token would live in a website’s existing robots.txt file.
- Microsoft says Bing results are unaffected by the preference.
- RSL licensing and Cloudflare’s default AI-crawler block are separate efforts.
- Products
- Bing
- Organizations
- Internet Engineering Task Force
From the original article
1 min read at theverge.com
OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI, as reported earlier by Search Engine Roundtable. With this control, publishers can apply their AI training preferences across multiple AI companies, instead of setting them individually.
By Emma Roth · theverge.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.