Creepy crawlies
git.kernel.org spends more CPU rendering commit pages for abusive AI-era scrapers than on all legitimate access including git clones.
Konstantin Ryabitsev reports that abusive crawler 'background radiation' at git.kernel.org consumes more CPU cycles rendering commit HTML than all legitimate access, with 14 cores across five geo-distributed nodes dedicated to scraper traffic. Simon Willison highlights the implications for crawlable web services like Datasette.
- 14 CPU cores across five nodes handle scraper-only commit rendering
- Scraper load exceeds all legitimate access including git clones
- Underlines AI crawler strain on public infrastructure
- Products
- git.kernel.orgDatasette
- Organizations
- Linux kernel
From the original article
Creepy crawlies Konstantin Ryabitsev discusses how bad the "background radiation" of abusive crawlers has become from the perspective of git. kernel. org , the official Git repository for the Linux kernel: TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones.
simonwillison.net
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.