terms.txt: A Consent and Compensation Protocol for Agentic Web Access
terms.txt specifies a robots.txt-style protocol for per-path, per-purpose AI crawler consent and compensation, with enforcement adding 0.20-0.65 ms per request.
The paper documents that automated clients now make up most web requests, that training dominates Cloudflare-classified crawling, and that the largest AI platforms fetch thousands of pages per returned visitor while robots.txt cannot express identity, purpose, terms, or price. It specifies terms.txt plus an origin-enforced exchange using Web Bot Auth signatures, signed intent, delegation tokens, HTTP 402 negotiation, and signed receipts. A dependency-free implementation adds 0.20 to 0.65 ms per request on one vCPU.
- robots.txt cannot express identity, purpose, terms, or price
- AI platforms fetch thousands of pages per returned visitor
- Exchange uses Web Bot Auth signatures and HTTP 402 negotiation
- Reference implementation overhead: 0.20-0.65 ms per request on one vCPU
Full article131 words · extracted from arxiv.org · click to collapse
The open web ran on an unwritten bargain: sites admitted crawlers, and search engines sent visitors back. Public measurements show that bargain breaking under AI crawlers and agents. Automated clients now make up most requests, training dominates Cloudflare-classified crawling, and the largest AI platforms fetch thousands of pages for each visitor they return. The web's common control, robots.txt, cannot express identity, purpose, terms, or price, can be circumvented, and newer alternatives are largely proprietary CDN features. We specify terms.txt, a robots.txt-style file for per-path, per-purpose machine-access terms, plus an origin-enforced exchange using Web Bot Auth signatures, signed intent, delegation tokens, HTTP 402 negotiation, and signed receipts. We define what the exchange can enforce, audit, and leave to contract. A dependency-free implementation adds 0.20 to 0.65 ms per request on one vCPU.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.11152