Creepy crawlies
git.kernel.org spends more CPU rendering commit pages for abusive AI-era scrapers than on all legitimate access including git clones.
Konstantin Ryabitsev reports that abusive crawler 'background radiation' at git.kernel.org consumes more CPU cycles rendering commit HTML than all legitimate access, with 14 cores across five geo-distributed nodes dedicated to scraper traffic. Simon Willison highlights the implications for crawlable web services like Datasette.
- 14 CPU cores across five nodes handle scraper-only commit rendering
- Scraper load exceeds all legitimate access including git clones
- Underlines AI crawler strain on public infrastructure
Creepy crawlies Konstantin Ryabitsev discusses how bad the "background radiation" of abusive crawlers has become from the perspective of git.kernel.org , the official Git repository for the Linux kernel: TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html. I worry about this a lot from the perspective of Datasette, which serves a huge number of crawlable web pages. Via Hacker News Tags: crawling , git , linux , datasette , ai-ethics
This source does not provide full text. Read it at simonwillison.net.