Cloudflare says 155 Workers AI GLM 5.3 responses held other customers' data after a KV cache mismatch

155 responses, a 681,280 token pool and a 1,048,576 token window

Cloudflare says that between 27 August and 9 September 2026 a small number of customers using the GLM 5.3 model on its Workers AI platform received responses that contained data from other customers' requests, and that 155 responses were affected, all to prompts longer than 681,280 tokens. The cause, as Cloudflare tells it, was two things at once: the model was deployed with a context window of 1,048,576 tokens while the cache pool on the prefill GPUs held only 681,280, and a bug in the SGLang inference engine let an over length request carry on instead of failing. The pool was short of the window by 367,296 tokens, which is 35.03 percent of it (derived). Cloudflare's post is dated 26 October 2026; this briefing read it, the matching SGLang post and the GitHub records on 30 October 2026, between about 15:37 and 15:50 UTC.

What that does not establish is most of what a customer would want to know. The post gives no number of affected customers beyond "a small number", does not say whether any of them were told, and does not say what kinds of data the 155 responses held. It does not establish that anyone read another customer's data on purpose: Cloudflare says it found no evidence of deliberate exploitation. And it does not say whether any other engine or model outside Cloudflare's service has the same flaw. Cloudflare operates Workers AI and is the only source for the incident's scope, and it is also the party that chose the configuration. That is a reason to read its account closely, not to doubt it.

The mechanism, step by step

Workers AI runs GLM 5.3 on SGLang in a disaggregated layout. A pool of prefill GPUs reads the prompt and builds the model's key value (KV) cache for it, then hands that cache over RDMA, through a transfer library called Mooncake, to a pool of decode GPUs that generate the reply. Cloudflare says the KV cache in GPU memory is a shared pool of fixed size pages, and that the pages are not zeroed between requests: the next request that gets a page overwrites it, so the engine itself has to track who owns which page.

Two numbers sit at the centre. The pool size is not a setting: SGLang works it out at startup from the GPU memory left after the weights load. The context length is a separate setting. Cloudflare's wording is that nothing forces the two to agree and that checking is the operator's job. On its GLM 5.3 prefill nodes the pool came to 681,280 tokens against a context length of 1,048,576. Cloudflare says it knew of the mismatch before launch, saw successful HTTP status codes and a clean streaming body in testing, and assumed the engine would handle the gap gracefully.

The path of an over length request, as Cloudflare lays it out for the engine bug:

1. The router tells the decode node the prompt length and decode reserves that many KV pages (eight, in Cloudflare's example).
2. The prompt is longer than the prefill pool, so the scheduler rejects it. The rejection function records the error and swaps the prompt for a one token placeholder.
3. In disaggregated mode, the admission queues on both prefill and decode check whether a request is finished, not whether it was rejected. The rejection is stored in a different field, so a rejected request reports itself as live and is admitted.
4. Prefill runs a trivial pass on the placeholder, sends one page of KV to decode and signals that the transfer is complete.
5. Decode takes that signal to mean prefill has overwritten the earlier state with the full prompt. It has not. Decode attends over all its reserved pages; only the first was written by this request, and the rest still hold what earlier requests left there.
6. The client gets HTTP 200 instead of HTTP 400, with a prompt token count that matches the long prompt and a coherent reply, which Cloudflare says arrived faster than a full prefill should take and contained content from other users' KV pages.

Cloudflare says every request over the threshold behaved this way, and that on a single node deployment the same rejection produced an error as intended. It says GLM 5.3 was the only model affected, because every other disaggregated deployment on Workers AI has pools larger than its context length on both prefill and decode, so no request could reach the rejection path. The figure below is Cloudflare's own drawing of the two paths, followed by this site's drawing of the two numbers and the path.

Zoom

Zoom

The figures, and which are derived

One consequence of the arithmetic is worth stating plainly. Cloudflare says every request over the threshold behaved this way, so no special technique was needed to reach the rejected path; that reading is this site's inference from the statement. Cloudflare's analysis is that after it set aside its own testing and the researcher's, the requests it identified were consistent in timing and volume with ordinary long context use. The post does not describe how it identified them or whether it could tell, for each of the 155, whose data appeared in whose response.

A public bug report 52 days before launch, and what the record shows

Cloudflare's timeline, in UTC, runs as follows. GLM 5.3 launched on Workers AI on 27 August with the pool smaller than the window, and Cloudflare says the exposure was present from launch. A report arrived through its HackerOne programme at 23:11 on 8 September. Verification began at 08:00 on 9 September, 8 hours 49 minutes later (derived). The incident was declared at 13:22, the model was set to private at 14:00 so that requests returned HTTP 400, the root cause was identified at 14:35, all prefill nodes were on a corrected configuration at 14:52, and at 19:40 the model went back to public after a rebuilt deployment with an engine patch. Cloudflare says it met the SGLang maintainers on 10 September, that further hardening reached all its production SGLang deployments between 11 and 19 September, and that SGLang v0.5.21 came out on 2 October and v0.5.22 on 19 October. The SGLang release page shows v0.5.22 released on 19 October.

Zoom

The earliest dated item is a public one. SGLang issue 30233, opened by a user on 6 July 2026, is titled as aborted prefill requests (input too long) causing decode to generate garbage from incomplete KV cache. That is 52 days before the Workers AI launch (derived), and Cloudflare says it noticed the issue while investigating. The opening post of that issue, as read, describes garbage output from uninitialised cache and does not mention other users' data. Cloudflare found the same path returning other customers' data, because the pages were not blank but stale. That the reporter did not describe a data exposure is the record; that the same defect would expose whatever the pages held is this site's inference from Cloudflare's account, not a statement in the issue. Cloudflare also says it had been flagging this class of issue to SGLang since v0.5.10.

The fix history has a wrinkle that matters to operators. Cloudflare says the fix for this incident was already on SGLang main in pull request 36988, merged on 6 September for a related failure path but not yet released; that page shows it merged on 6 September and describes it as retiring an aborted request's prefill result, in the context of vision language model encoder and embedding validation failures. Cloudflare's table lists 36988 as the change that stops the exposure, in v0.5.20, and pull request 38935, in v0.5.21, as rejecting these requests earlier. The author of 38935 describes it as a waste and latency fix rather than a correctness fix, and says the stale cache symptoms they first wrote it against do not reproduce once 36988 is in. So the change that closes the leak shipped in a release before the one Cloudflare names in its advice, and Cloudflare's advice to run v0.5.22 or later is a recommendation that covers both and the related fixes.

A successful response is not a correct one

Three comfortable labels did the work of controls here. The first is the context window: an advertised 1,048,576 tokens read as a capacity, when the memory behind it could hold 681,280. The second is the status code. Cloudflare says its pre launch tests saw successful responses and clean streams, and that its gap was that it "only validated HTTP response codes and engine performance at long contexts, without validating model output and coherence". A leak that returns HTTP 200 and a fluent answer passes every check that looks at the envelope. The third is an assumption. Cloudflare says it "expected the engine to return an error for any prompt it could not process", and the engine, on the path it took, did not.

The lesson Cloudflare draws is concrete: it now compares pool size with context length on the running engine before any disaggregated deployment change ships, instead of inferring it from response codes. It also says its own canary marker tests, which this site reads as planted marker text that should never appear in another request's output, were used to validate the fix before the model came back.

Stated and not stated

The last row is not about Cloudflare's incident. The SGLang post describes a second and wider problem in the same family: when a request is aborted while prefill is still writing its KV cache to decode, writes already sent cannot be recalled, and if the slots are reused the late writes land in another request's cache. SGLang says default configurations in disaggregated mode were affected from the 0.4 line in early 2025 until v0.5.21, or v0.5.22 for the MORI transfer backend, and that a client can raise the odds by sending long requests and disconnecting mid transfer, though it cannot choose the victim. Cloudflare says the bug in its incident was different, because a rejected request took the handoff every time rather than by a timing accident. Both posts treat them as one class of memory safety bug. This site reports SGLang's description as SGLang's, and has not tested either.

What SGLang operators should check, in order

Cloudflare and SGLang give operators the steps below. The order and the last two items are this site's ranking and reading, not theirs. No product is recommended.

A longer term point from Cloudflare is that configuration checks and lifecycle fixes reduce the ways a page can be misused but do not make misuse impossible. It describes a prototype that tags every physical KV page and checks the mapping before decode reads, which it says cost under 1 percent in throughput, and which it disabled after reliability problems. It lists zeroing KV pages on release as a draft proposal. Cloudflare says pages are not zeroed today for performance reasons.

Technical details

Neither post prints indicators of compromise, hashes, domains or a log line from the incident, and the researcher's reproduction script is not published in the posts read. What they do print is the name of the affected model, the names of the engine functions and fields involved, the flags and settings an operator can use, and the symptom. Those are below, each copied from the post that printed it. Nothing here is exploit code or a proof of concept.

The affected model identifier on Workers AI, as Cloudflare prints it. Customers can use it to match their own request logs to the model and the window.

@cf/zai-org/glm-5.3

The names Cloudflare gives for the rejection path and the check that missed it. A reader who runs SGLang can search their version of the code for these names to see whether a rejected request is retired before the handoff.

Req.set_finish_with_abort
req.finished()
finished_reason
to_finish

The symptom Cloudflare describes for the leaking path. This is a description of what a client saw, not a signature: a normal long prompt can also return a coherent answer.

The client got HTTP 200 instead of HTTP 400. prompt_tokens matched the long prompt and the reply was coherent.

The corruption checks Cloudflare tells operators to turn on where the model supports them. Both are flags on the SGLang engines, and the checksum flag is for both the prefill and the decode engine, per the SGLang post.

--disaggregation-enable-kv-checksum
--kv-canary

Settings the SGLang post gives for the deferred release hold on older builds, and the metric to watch afterwards. The metric has an outcome label that reports whether holds ended on drain confirmation, timeout or error. The post says a healthy deployment releases almost all holds through drain confirmation.

SGLANG_DISAGGREGATION_DEFERRED_DECODE_KV_RELEASE=1
SGLANG_MOONCAKE_MAX_TRANSFER_BATCH_INDICES
SGLANG_DISAGGREGATION_DEFERRED_DECODE_KV_RELEASE_TIMEOUT
sglang:decode_deferred_kv_release_total

The upgrade commands from the SGLang v0.5.22 release page, which Cloudflare names as the version to run or exceed.

uv pip install --prerelease=allow sglang==0.5.22
docker pull lmsysorg/sglang:v0.5.22

The question to ask your provider and your own team

This was not an attack on the record: Cloudflare says it found no evidence of one, and the report came through its HackerOne programme. It is a case of a pool size that needed to be at least the context length, with an engine underneath that did not stop the request when it was not. Cloudflare chose to publish the mismatch, the missed test and the timeline, and that is useful. It also leaves open how many people's data moved, and to whom.

Any service that advertises a long context window is making a claim about memory it has to hold. If a request quietly fell off the safe path and came back with HTTP 200, what is the test in your deployment, or your provider's, that would notice that the answer contained someone else's words?

This site's earlier briefing on GLM 5.3 covered a different matter, an outside cyber evaluation, and is here.

Key facts