OpenAI found one overloaded storage worker kept attracting more traffic. That is metastability

The client stopped and the failure continued

OpenAI has published an unusually useful engineering detail from Habitat, its application-storage service. During a burst of traffic, part of the Python fleet became overloaded. Engineers stopped the client responsible for the burst and expected recovery. Instead, a subset of processes remained degraded and received increasing amounts of work until they were restarted.

That is the defining clue. Ordinary overload is pressure while demand exceeds capacity. Remove the demand and the service recovers. A metastable failure reaches a bad state that sustains itself after the initiating pressure disappears.

OpenAI traced the behaviour to connection pooling and load distribution. Before changes, some tail processes served five to ten times the average number of concurrent requests. Python's aiohttp TCPConnector reused connections in last-in, first-out order. The connection returned most recently was selected for the next request. Under healthy conditions that can improve locality. Under uneven load it can keep a busy path hot.

The incident is valuable because no exotic database fault was required. A reasonable default, interacting with persistent connections and overloaded workers, created positive feedback.

How LIFO can feed the hot worker again

A connection pool does not choose a fresh server for every request. It keeps established connections and reuses them. If only a handful of persistent connections were created during a busy period, much of one client's traffic can remain attached to the workers behind those connections.

With LIFO reuse, the connection that has just completed work sits at the front of the next selection. A fast healthy connection is reused often, which is efficient. The problem appears when timing, concurrency and queueing cause the pool to cycle repeatedly through a small, unlucky set. Those workers receive more concurrent work, their event loops fall behind, responses queue, and the system's own retry or scheduling behaviour can preserve the imbalance.

Think of ten loading bays and a dispatcher who always sends the next lorry to the bay that most recently returned a radio. During a surge, three bays become the active set. One bay starts slowing down, but its radio remains inside the dispatcher's active pile. The empty bays do not help if no new connection sends work to them. Removing the original convoy does not empty the queue already built at the hot bay, and normal arriving traffic can keep it from catching up.

OpenAI tested the hypothesis by limiting how long connections could be reused. That bounded the degradation and pointed investigators toward the pool. The general method is as useful as the specific fix: change one piece that limits feedback, then observe whether the failure can still sustain itself.

The self-reinforcing path described in OpenAI's Habitat account.

Stage	What happens	What an operator sees
Burst	A client opens or heats a small set of pooled connections	Traffic rises but fleet capacity still appears available
Imbalance	Selected workers receive far more concurrency than the average	Tail workers reach 5 to 10 times average load
Delay	CPU work and queued callbacks increase asyncio scheduling delay	Long tail latency with moderate average utilisation
Reuse	The active connection set continues receiving requests	Hot workers stay hot while neighbours remain cooler
Metastability	Queued and newly routed work prevent recovery after the burst	Stopping the initiating client does not heal the service
Reset	Connection lifetime is bounded or workers restart	Load redistributes and the bad state clears

The event loop made tail latency the real capacity limit

Habitat's Python architecture adds context. Asyncio lets one process keep many I/O operations in flight, but it does not make CPU work execute in parallel on that process's thread. When response processing, serialization or configuration work consumes the thread, ready network operations wait for the event loop to return.

OpenAI measured the difference between when a small background task should run and when it actually ran. Under high utilisation, scheduling delay reached hundreds of milliseconds and in edge cases several seconds. The service therefore kept concurrency per process low and scaled through many processes.

The team also found synchronized feature-flag refreshes. Every process polled a large configuration each minute with no jitter and parsed rules for every production service. A single pod could run up to eight Python processes, so the same CPU-heavy work arrived together. Reducing the configuration, extending the interval and adding jitter spread the work and reduced tail latency.

This is an operational reminder that average CPU is often the wrong health signal for an asynchronous service. A process can show acceptable average utilisation while callbacks wait long enough to violate user latency. Event-loop delay is the queue the standard dashboard forgot to display.

Why one slow database call becomes a product problem

OpenAI says an average user request can trigger hundreds of database calls. At that fan-out, the slowest call becomes visible to the user. If each call is usually fast but has a small chance of landing in the tail, multiplying calls makes encountering at least one slow result increasingly likely.

As a simple illustration, suppose each database call has a 0.1% chance of being unusually slow and a request makes 300 independent calls. The chance of avoiding every slow call is roughly 74%. That leaves about a one-in-four chance of encountering at least one. Real calls are not independent and the number is hypothetical, but it demonstrates why tiny tail probabilities become product behaviour at large fan-out.

The team later rewrote the service in Rust. OpenAI reports the Rust version now handles 95% of production traffic and is six times more CPU efficient and fifteen times more memory efficient than the Python version. That is a substantial outcome, but it should not erase the earlier lesson. A faster language creates headroom. It does not automatically correct a load-balancing feedback loop, synchronized work or unbounded retries. Architecture and runtime efficiency solve different parts of the system.

What another engineering team should copy

Reusable checks for asynchronous services with persistent connection pools.

Control	Why it matters	Useful signal
Graph load per worker	Fleet averages hide the unlucky tail	Maximum and percentile concurrency by process
Measure event-loop delay	CPU utilisation does not show callbacks waiting	Scheduled versus actual callback time
Jitter periodic work	Synchronized refreshes create artificial bursts	Latency aligned to minute or refresh boundaries
Bound connection reuse	A small hot pool can preserve bad routing	Connection age, requests per connection and backend spread
Test recovery	Removing the trigger may not clear accumulated state	Time to recover after load stops
Give retries a budget	Retries can become the demand sustaining overload	Attempt count and retry traffic as a share of total

The P.K. view

This incident is more useful than a polished scale number because it shows the difference between capacity and distribution. The fleet had workers. The requests did not reach them evenly.

Metastability should change incident procedure. Teams often ask what caused the spike and stop once that client, release or job is disabled. The second question is whether the system can return to a healthy state using only its normal control loops. If not, the incident remains active even after the trigger is gone.

The signal to carry into a review is the five-to-ten-times worker imbalance. Any platform dashboard that reports only fleet averages can declare that system healthy while a minority of processes collapse. Measure the tails by instance, measure the event loop, and run a recovery test in which demand is removed without restarting anything.

OpenAI's fix story is not "rewrite Python in Rust". The durable lesson is that a connection-selection rule helped traffic remember the wrong part of the fleet. Faster workers are valuable. A system that can forget a bad state is safer.