Cloudflare partners with Internet Archive to make sites available when their origin servers are down
Creepy crawlies
git.kernel.org spends more CPU rendering commit pages for abusive AI-era scrapers than on all legitimate access including git clones.
Konstantin Ryabitsev reports that abusive crawler 'background radiation' at git.kernel.org consumes more CPU cycles rendering commit HTML than all legitimate access, with 14 cores across five geo-distributed nodes dedicated to scraper traffic. Simon Willison highlights the implications for crawlable web services like Datasette.
Trends in Web Threats: Old Web Skimmer Still Active Today
Unit 42 detected 577,000 landing URL incidents in Q1 2022, with an old web skimmer family still actively stealing payment card data.
Palo Alto Unit 42 detected 577,275 landing URL incidents (116,643 unique) and 2,043,862 malicious host URL incidents (180,370 unique) between January and March 2022. Web threat volumes declined after the November 2021 holiday peak, but an old web skimmer family remained active. Business and economy sites overtook personal sites as the most common apparently benign entry points. Most malicious domains geolocated to the United States, Germany, and Russia, though proxy servers and VPNs obscure true locations.
terms.txt: A Consent and Compensation Protocol for Agentic Web Access
terms.txt specifies a robots.txt-style protocol for per-path, per-purpose AI crawler consent and compensation, with enforcement adding 0.20-0.65 ms per request.
The paper documents that automated clients now make up most web requests, that training dominates Cloudflare-classified crawling, and that the largest AI platforms fetch thousands of pages per returned visitor while robots.txt cannot express identity, purpose, terms, or price. It specifies terms.txt plus an origin-enforced exchange using Web Bot Auth signatures, signed intent, delegation tokens, HTTP 402 negotiation, and signed receipts. A dependency-free implementation adds 0.20 to 0.65 ms per request on one vCPU.