Wellcome Collection search failed under a distributed crawler flood
Original title: Crawlers, floods and fleets: a year of keeping our digital collections open
About 10,000 one-off requests in seven minutes on July 6 bypassed address-based rate limits and broke search.
Wellcome Collection described crawler and bot traffic against its online collections between June and September 2026. Named AI crawlers drove egress costs up, including one crawler making more than 2.5 million requests in four days, before compression, a rewritten robots.txt, and CDN rate limits reduced load. On July 6, about 10,000 addresses each sent one uncacheable search request in seven minutes, and the search service returned errors to nearly everyone within five minutes. A JavaScript challenge restored service for people while also blocking AI agents and link-preview services.
Unconfirmed: The delay before crawler load fell is Wellcome's inference about how long the heaviest crawler took to fetch robots.txt.
- Egress cost had risen nearly fourfold since January while requests grew by about half.
- Compressing IIIF traffic cut that internal data movement by more than nine tenths.
- A July 5 burst peaked at 40,000 requests a minute and was mostly rate-limited.
- By the second night, some traffic used headless browsers that solved the challenge.
- Catalogue API health checks had not been wired to alert a person.
Impact: search outage and sharply higher cloud egress costs
- 1.Compress IIIF responses and rate-limit named crawlers at the CDN.
- 2.Use a tested JavaScript challenge in front of search pages that bots can overwhelm.
- 3.Connect catalogue API health checks to alerts that reach a person.
- Products
- IIIF
- Victims
- Wellcome Collection
From the original article
13 min read at stacks.wellcomecollection.org
A printer’s workshop: on the left a printing press, on the right and centre workmen engaged in various tasks . Engraving after L. J.
By stacks.wellcomecollection.org via objectif_lune · stacks.wellcomecollection.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.