Continuous accountability for Verified bots
Cloudflare will score Verified bots on whether they honor robots.txt and rate-limit signals over time.
5
Stories per day · 30 days
2 RSS
3 stories, newest first · one card per story
Stories tagged #web-bot-auth
The Internet has a second audience
Cloudflare says non-human traffic now exceeds half the Internet as AI agent requests surged more than 1,700%.
3
terms.txt: A Consent and Compensation Protocol for Agentic Web Access
terms.txt specifies a robots.txt-style protocol for per-path, per-purpose AI crawler consent and compensation, with enforcement adding 0.20-0.65 ms per request.