OpenAI has replaced the backend that serves its giant source-code repository with a new system called Pando, and the company says the switch has made code pushes far faster and more reliable. Pando now handles all development in OpenAI's monorepo, and it has cut median push latency by nearly five times.

The problem started with Codex. The company says its coding tools have changed the pace of work across the company, letting engineers run many tasks at once. That means more branches, more pushes, more merges and a steady stream of fetches and clones from development environments, research jobs and continuous integration systems. Each push can also trigger several jobs that fetch the same commit at once.

Why did OpenAI replace its Git backend?

Around the end of last year, the systems behind the monorepo began to strain. Engineers saw slow fetches and pushes and failed merges. At times, reliability dropped below 90 percent. The team had to figure out whether it had hit a limit of Git itself or a limit in the way OpenAI served its repository.

Why the existing designs fell short

OpenAI's engineers evaluated two common ways of scaling Git. The first, a three-phase commit approach, keeps a full copy of the repository on several replicas. A push must be accepted by a majority of those replicas before it is published, which requires several rounds of communication. The engineers found that this approach puts disk performance and network latency directly on the push path, so write delays grow as volume rises.

The second approach orders all updates through a shared write-ahead log stored in blob storage. Pushes to unrelated branches still have to update the same manifest, which creates a bottleneck. Adding storage capacity alone did not fix it. Replicas also had to read and index packs before they could serve them, and when writes outpaced that work, clients waited longer to fetch new commits.

How Pando handles pushes and fetches

Why did OpenAI choose causal consistency for Pando?

Pando takes a different path. It relaxes the consistency model, so the system no longer provides a single, point-in-time view of every branch. Instead, it uses causal consistency. Each branch is updated independently, and pushes to unrelated branches can proceed in parallel. A reader looking at several branches might see them at different points in time, but the company says users and agents care mainly about the order of updates to a single branch.

How does Pando handle pushes and fetches?

When a client pushes, Pando streams the incoming packfile to blob storage and validates it in a single pass. It does not unpack the files into a local repository. The system then records a new version of the branch in a metadata store and updates the branch pointer in one transaction. That allows a client to read its own write right away.

Pando also reuses packfiles, the compressed bundles Git uses to transfer objects. Instead of building a custom pack for every client, the server can send the same stored bytes to many readers. A background process combines small packs into larger ones over time, similar to compaction in log-structured merge trees. Pando serves fetches through packfile URIs, and a caching layer delivers the files from regional caches. OpenAI reports cache hit rates above 99 percent. In tests using standard cloud virtual machines, clone and fetch operations saturated 10-gigabit network links.

To keep pushes small, Pando advertises only a designated trunk branch by default. The company says this trade-off works for its monorepo. The median packfile produced this way was under 20 KiB, and the 99th percentile was under 3 MiB.

Built quickly, with heavy use of AI

OpenAI says it built Pando in 10 weeks, from kickoff to production. The core design came from a single engineer. Teams of agents implemented much of the system, and a handful of engineers deployed it. Pando began serving production traffic in April.

The team wrote the system in Rust and built a conformance test suite so agents could check their changes against standard Git behavior. It also used deterministic simulation testing to find race conditions. Agents wrote formal specifications in Lean and TLA+ to check properties of the design, the company said.

Rollout happened in stages. Pando first ran as a read- and write-through proxy in front of the old system, which kept the two in sync. Teams then moved over one at a time without downtime. OpenAI also shipped its public Git distribution to developer machines and the ChatGPT desktop app, and it used an internal wrapper around the Git command-line tool to migrate clients in phases. Hundreds of integrations that relied on older APIs, webhooks and repository downloads were moved alongside the Git traffic.

The company says the results include better merge reliability and a median push latency reduction of nearly 80 percent. Pando now delivers nearly a petabyte of Git data per day using only a handful of CPU cores. In one hour, it served more than 57 terabytes of Git packs, with more than 99 percent of those bytes coming from local disk cache.

OpenAI also said it has fed fixes back to the open-source Git project. Work on downloading and indexing packs surfaced problems with concurrent downloads and interrupted transfers, and the company said it contributed those fixes upstream, along with an option called explicit haves that supports more efficient force pushes. Some of the changes are already in upstream Git, and others are in OpenAI's public repository. The company is also exploring whether Pando could support workloads where many agents collaborate on one Git repository.

Dieser Artikel wurde mit Unterstützung von KI verfasst.
News Factory APP - agentische News für besseres SEO & AEO.