Say it once: Introducing Bot Preference Sync

Cloudflare launched Bot Preference Sync, which automatically syncs robots.txt files with AI bot policies covering search, agent, and training crawlers.

Cloudflare's new Bot Preference Sync feature automatically aligns a site's robots.txt with its configured AI bot policies for Search, Agent, and Training bot categories. The goal is to let site owners manage which automated and AI crawlers access their content without manually maintaining static robots.txt files.

• Automatic synchronization of robots.txt with Cloudflare bot policies
• Covers Search, Agent, and Training AI bot categories
• Removes need to hand-maintain static robots.txt files