Inside GitHub's new Git storage architecture

In this article
GitHub has unveiled a new Git storage architecture built around one observation: only reference updates (pointers such as branches that point to commits) need agreement among parties. As a result, data storage, request serving and repository maintenance can scale independently of one another. In internal testing, this architecture withstood 5,000 concurrent agents working on a single repository, at 7,200 pushes per second, while still serving read traffic with consistently low latency. For developers, Git is still Git: the way you push, fetch and branch does not change.
Why a redesign is needed for agent scale
The platform's scale is viewed along two axes. The horizontal axis is the number of developers and agents working across many repositories; older approaches such as tenant partitioning, ensuring isolation or fairness between tenants, and load-based routing are retained. The vertical axis is the number of people and agents working on a single repository together, and most new requests come from this axis.
For a single repository, the new architecture must meet the following requirements:
Accept thousands of writes per second and support multiple write nodes, because pack indexing and object connectivity checks are the most CPU-intensive work, and a single node would quickly become a compute bottleneck.
Reduce commit latency, because agents must wait for each push to finish before moving on to the next step.
Record changes in order for each reference, while being able to quickly sequence them across references.
Keep read costs low when nothing has changed, which is the common case given the very large read-to-write ratio.
Allow each component to scale independently.
Overview of the Rubicon architecture
The design is named Rubicon and began in May 2026, after the initial migration to Azure. In every experimental variant, pack data lives in Azure Blob Storage. The final variant was chosen because it maximizes throughput while preserving guarantees on correctness, durability, availability and latency.
Requests from Git clients, the API and the web interface still enter through Spokes, the component that continues to handle routing, load balancing and maintaining the roles of the placement and balancing service. The data plane beneath Spokes is the entirely new part. Spokes forwards each request to one of a small group of virtual replicas assigned to the repository. A virtual replica is a fleet of disposable machines serving reads, writes or both, and its local disk is just a cache that can be rebuilt at any time.
When possible, Spokes sends a repository's traffic to the same virtual replicas to keep caches warm and latency low, but correctness does not depend on which replica a request lands on. If a replica disappears, the remaining replicas keep serving and the replacement replica fills its cache as traffic arrives. When a repository gets busier, the system adds virtual replicas.
Durable state lives in two places:
Git objects, in the form of packs along with auxiliary data that speeds up reads, live in Azure Blob Storage.
The references and the list of packs that make up the repository (the pack manifest) live in a separate replicated log service called Ref-CAS, short for Reference Compare-and-Swap.
The write path goes like this: a virtual replica indexes the incoming pack and uploads it to Blob Storage, then performs a transaction against Ref-CAS to add the pack to the manifest and update the references. Later, a compaction node merges that pack with others to keep reads fast. On the read path, the virtual replica checks the reference version with Ref-CAS, pulls the needed packs from Blob Storage into its cache, and then serves from there.
Reducing coordination: reference updates at high throughput
Git objects are immutable and named by their content, so two writers can store them at the same time without needing to agree with each other. The part that needs consensus in a push is the reference update, and this is very small compared with the whole body of data. Most of what sits in Blob Storage is written once to its own key and never changes, which suits object storage well. References are the opposite: they are small state that many writes compete to update.
Two approaches based on Blob Storage were tried, and both hit limits:
The first approach is a write-ahead log for all of a repository's references, indexed by a single blob, with each update being a conditional write to that blob. Reads are cheap, but only one conditional write can proceed at any moment, so the write ceiling is one per round trip to the blob. In practice, write throughput reached only a few hundred per second.
The second approach splits references across multiple sharded blobs, with each reference hashed into one shard. Writes to different shards no longer wait on each other, but a read must check every shard and is only as fast as the slowest one. The team traded write latency for read latency, even though reads outnumber writes by orders of magnitude.
The result is the Ref-CAS service, a small consensus group dedicated to references. Each update is a compare-and-swap confirmed by a quorum: a transaction locks only the references it touches, checks that each reference still has the value the writer expects, and becomes visible once a majority of the log's replicas have acknowledged it. Because agents mostly push to their own branches, most updates do not contend and are replicated in parallel. When two pushes update the same reference, only one is appended to the log.
Each commit to Ref-CAS receives a log sequence number (LSN), and the repository's version is the LSN of its latest commit. On a read, the virtual replica sends the version of the reference state it currently holds. If that version is still the latest, the response carries no references, so an unchanged repository costs just one comparison. If the replica has fallen behind, it receives the current references and version, along with pointers to the pack data.
Ref-CAS never holds or validates object data, only pointers to it. A virtual replica uploads a push's packs to Blob Storage before committing the reference transaction, so a reference only ever points to objects that are already durable. As a result, reference state alone can be handled at tens of thousands of operations per second.
Separating storage from compute: compaction at scale
Each push adds a pack to Blob Storage. Git can serve from multiple packs, but each additional pack makes lookups a little more expensive, so the number of packs cannot grow without bound. In the old architecture, compaction (merging packs) ran right on the replicas serving traffic, and had to run on every replica. In the new architecture, it runs entirely in the background on dedicated nodes, directly on durable storage, and uses Ref-CAS to tell virtual replicas about the changed state.
Each compaction node takes ownership of a portion of the repository's data through a lease, so compaction can scale horizontally. The process on a single shard goes through the following steps:
The node acquires a lease on the shard in Blob Storage, so that only one node merges a given shard at any time.
The node reads the manifest to find new packs and collects them into an open segment.
When the segment reaches its limit by pack count, size or age, the node seals the segment and merges its packs into fewer, larger packs.
While merging, the shard's next segment begins taking in input packs.
Before publishing, the node verifies that the new pack contains exactly the same objects as the input packs, uploads the new pack and adds it to the manifest.
A second transaction removes the input packs from the manifest, and a cleanup step deletes them afterward.
To scale, this process runs independently for many shards at once. The hash of each pack maps to a shard, so adding shards and nodes adds compaction throughput. A merged pack that is still under the size limit returns to a shard according to its own hash and is merged again, so packs grow tier by tier until they hit the limit. The result is a pack count that stays nearly flat even as push volume rises.
Compaction only merges packs but keeps every object. Garbage collection is the step that removes objects that are no longer referenced, for example because a branch was deleted or force-pushed. This step works similarly to compaction, using separate workers for the compute-heavy part. With these approaches, latency stays flat as write volume grows.
What comes next
The goal is for every repository, from a developer's side project to the busiest monorepo, to run on the same architecture with no special handling for busy repositories. A single architecture also means every improvement reaches every repository at the same time. The architecture is currently in limited private preview, with a broader rollout early next year.
Planned architectural improvements include:
Reading only what is needed: fetching one object from Blob Storage instead of the whole pack that contains it.
Organizing packs by type: grouping commits, trees and blobs into separate packs.
Partial replicas: serving requests without needing every pack on local disk.
Materialized views: keeping repository data in ready-to-use form, such as prebuilt archives or a near-real-time shallow copy of the default branch.
Global replicas: placing replicas in any region, close to where they are used.
Scale to zero: scaling the fleet to the repositories that are actually active.
For agent operations, the next items include working without a clone, an atomic commit API to land a multi-file change in a single operation, and an event stream so agents can subscribe to changes in real time. On the Git side, GitHub continues contributing to the project by improving partial clone, shallow clone and sparse checkout, removing hidden scaling costs, speeding up history traversal with metadata such as commit-graph, and warning in advance about common pitfalls.


