OpenAI will honor a shared robots.txt AI training preference signal
Original title: Making it easier for websites to express AI training preferences
Site operators can state one robots.txt preference for multiple providers; OpenAI says implementation is still months away.
OpenAI said on October 30, 2026 that it will honor a common robots.txt signal covering use of website content for foundational generative AI training. The voluntary signal, announced with other participants in the IETF AI Preferences working group, lets operators give one preference to multiple providers instead of separate crawler rules. Existing OpenAI robots.txt crawler controls will still apply. The company aims to implement the signal in the coming months and will publish details later.
Unconfirmed: OpenAI has not published how or exactly when it will implement the signal.
- Operators previously identified each crawler and maintained separate provider rules.
- The new signal will not replace OpenAI’s existing robots.txt crawler controls.
- OpenAI is joining other IETF working-group participants in the announcement.
- The company encourages other AI providers to honor the same signal.
- Organizations
- OpenAIInternet Engineering Task Force
From the original article
3 min read at openai.comOpenAI will honor a shared robots. txt signal to help website operators communicate their AI training preferences to participating providers.
openai.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.