Rewinding the exponential: How we keep hardware usage down
Cloudflare details pipeline changes that deferred about 2.5 years of hardware growth.
Cloudflare's data team described containing hardware growth while the pipeline behind Logs, the SQL API, and analytics reached about two billion events per second by May 2026. Changes over five years are equivalent to deferring about two and a half years of hardware expansion. Replacing a monolithic schema in 2022 cut backbone bandwidth 35-40%, and switching compression from LZ4 to zstd in 2024 reduced compressed throughput nearly 40% with little CPU impact. In 2026 the team reworked replication, decentralized sampling, and added aggregations so analytics no longer double-writes every event to two regions.
- The pipeline handled about two billion events per second by May 2026.
- Splitting the 2022 monolith schema cut backbone bandwidth 35-40%.
- Moving from LZ4 to zstd in 2024 reduced compressed traffic nearly 40%.
- 2026 changes cover replication, decentralized sampling, and pipeline aggregations.
- Vendors
- Cloudflare
- Products
- Cloudflare LogsClickHouseLogfwdr
- Organizations
- Cloudflare
From the original article
11 min read at blog.cloudflare.com
Our strategies to optimize our data pipeline throughput without compromising data quality.
By Cathy Chi · blog.cloudflare.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.