terms.txt: A Consent and Compensation Protocol for Agentic Web Access
terms.txt specifies a robots.txt-style protocol for per-path, per-purpose AI crawler consent and compensation, with enforcement adding 0.20-0.65 ms per request.
The paper documents that automated clients now make up most web requests, that training dominates Cloudflare-classified crawling, and that the largest AI platforms fetch thousands of pages per returned visitor while robots.txt cannot express identity, purpose, terms, or price. It specifies terms.txt plus an origin-enforced exchange using Web Bot Auth signatures, signed intent, delegation tokens, HTTP 402 negotiation, and signed receipts. A dependency-free implementation adds 0.20 to 0.65 ms per request on one vCPU.
- robots.txt cannot express identity, purpose, terms, or price
- AI platforms fetch thousands of pages per returned visitor
- Exchange uses Web Bot Auth signatures and HTTP 402 negotiation
- Reference implementation overhead: 0.20-0.65 ms per request on one vCPU
- Vendors
- Cloudflare
- Products
- terms.txtWeb Bot Auth
From the original article
1 min read at arxiv.orgThe open web ran on an unwritten bargain: sites admitted crawlers, and search engines sent visitors back. Public measurements show that bargain breaking under AI crawlers and agents. Automated clients now make up most requests, training dominates Cloudflare-classified crawling, and the largest AI platforms fetch thousands of pages for each visitor they return.
By Rajarshi Chowdhury · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.