GitHub rebuilds Git storage for thousands of concurrent agent writes
Original title: GitHub's new Git storage architecture
Internal tests reached 5,000 concurrent agents and 7,200 pushes per second on one repository.
GitHub described Project Rubicon, a Git storage design so many agents can write to one repository at once. Pack data lives in Azure Blob Storage, and only reference updates need agreement, so storage, request serving, and repository maintenance scale separately. Spokes still routes and load-balances Git client, API, and web requests. GitHub announced the work at GitHub Universe after moving nearly all repositories into Azure.
Unconfirmed: The concurrency and push-rate figures are GitHub's own internal test results.
- Tests showed 5,000 agents and 7,200 pushes per second on one repo.
- Busiest GitHub.com repositories today see dozens of writes per second.
- Only reference updates need ordering; Git objects are content-addressed.
- One repository's daily requests grew about 4x from March to May.
- Virtual replicas catch up by fetching only data a request needs.
From the original article
14 min read at github.blog
GitHub’s new Git storage architecture is built so thousands of agents can push to a single repository at once. Here’s how it works. Last week at GitHub Universe, we announced our new Git storage architecture, capable of handling thousands of writes per second with reads that scale on demand.
By DAlperin · github.blog
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.