Have it both ways: stay discoverable in search while disallowing AI training
Cloudflare launches Disallow AI Training setting letting sites block AI training via robots.txt while staying indexed in search; Apple, Google, and Microsoft honor it.
Cloudflare announced a 'Disallow AI Training' setting that publishes a no-training preference in robots.txt so sites can block AI training (including by mixed-use crawlers) without losing search indexing. Apple, Google, and Microsoft meet Cloudflare's new 'Accountable' designation, which requires training/summary opt-out mechanisms, URL-level training visibility, and assurance that opt-outs don't affect search rankings. Cloudflare cites that under 1% of its sites block search bots while 17% block AI training, and its Block settings now apply to mixed-use crawlers as well. Granular controls over how much content appears in AI summaries are planned for early next year.
- Disallow AI Training keeps search indexing while refusing training by mixed-use crawlers
- Apple, Google, and Microsoft earn Cloudflare's Accountable designation
- Block and 'Block on pages with ads' now apply to mixed-use crawlers
- 17% of Cloudflare sites block AI training; under 1% block search bots
- AI summary quantity controls planned for early next year
From the original article
10 min read at blog.cloudflare.com
Cloudflare is giving site owners a way to stay discoverable while disallowing AI training. New controls and an Accountable designation establish a shared model with Apple, Google, and Microsoft.
By Bryan Becker · blog.cloudflare.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.