Agentic Crawlers Record More Trackers Than Standard Methods
LLM-driven agents record 1.5-1.7 times more third-party trackers than non-agentic crawlers across 2,000 websites.
Stories per day · 30 days
2 RSS
Relatedopenai2ai-crawl-control1ai-discovery1ai-training1chatgpt1cloudflare1content-preferences1gpt-61ietf1llm-agents1privacy1robots-txt1tracking1
4 stories, newest first · one card per story
Stories tagged #web-crawling
OpenAI will honor a shared robots.txt AI training preference signal
Site operators can state one robots.txt preference for multiple providers; OpenAI says implementation is still months away.
1
Cloudflare and OpenAI: making AI discovery better for everyone
A Cloudflare-OpenAI pilot helped ChatGPT index hundreds of millions of new pages from about one percent of Cloudflare's network.
7
terms.txt: A Consent and Compensation Protocol for Agentic Web Access
terms.txt specifies a robots.txt-style protocol for per-path, per-purpose AI crawler consent and compensation, with enforcement adding 0.20-0.65 ms per request.