Skip to content
NewA calmer, faster ZeroHour
StoryAI industryinfo

Major AI firms back a shared robots.txt training opt-out

2 sources · 2 articlesFirst reported Updated ()1
What's new

First merged summary.

Merged summary · x-ai/grok-4.7 · rewritten as coverage arrivesSummaries by Vani AI

OpenAI, Google, Microsoft, and Anthropic say they will honor one shared robots.txt signal for AI-training preferences, on staggered timelines.

OpenAI, Google, Microsoft, and Anthropic say they will honor a voluntary shared “policy token” in robots.txt so site operators can set one preference covering use of website content for foundational generative AI training, instead of separate rules for each provider. Anthropic technical lead Kevin Kelley proposed the signal through the IETF AI Preferences working group; Anthropic has already adopted it. Google and Microsoft plan support next year, while OpenAI says implementation is still months away and that existing OpenAI crawler controls will still apply. Microsoft says it will also separate opt-outs for AI training and grounding, and that Bing search listings are unaffected. The Verge notes RSL licensing and Cloudflare’s default AI-crawler block are separate efforts.

Key facts
  • The preference is a voluntary policy token in a site’s existing robots.txt file.
  • Anthropic has adopted the signal; Google and Microsoft plan support next year.
  • OpenAI aims to implement it in the coming months and will publish details later.
  • OpenAI says the signal will not replace its existing robots.txt crawler controls.
  • Microsoft says Bing search listings stay unchanged by the training preference.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

Timeline

dated events in the reporting · oldest first
  1. Oct 30, 2026OpenAI and peers announce the shared robots.txt AI-training signal. · OpenAI News

What each outlet adds

beyond the shared facts
  • OpenAI NewsOpenAI timeline, existing crawler controls remain, and training-use scope.
  • The VergeNames Google, Microsoft, Anthropic, proposer Kevin Kelley, and Bing carve-out.
Vendors
OpenAIGoogleMicrosoftAnthropic
Products
Bing
Organizations
Internet Engineering Task ForceOpenAI

Coverage

oldest first · one row per article · UTC
  1. · 13h ago OpenAI NewsFirst report

    OpenAI will honor a shared robots.txt AI training preference signal

    Site operators can state one robots.txt preference for multiple providers; OpenAI says implementation is still months away.

  2. · 2h ago The Verge · AI

    OpenAI, Google, Microsoft and Anthropic back shared AI scrape opt-out

    Anthropic already honors the robots.txt token; Google and Microsoft plan support next year, and OpenAI says coming months.