Skip to main content
LeoNex News

KV cache leaking between requests in LLM serving: lessons from Workers AI

Illustration: KV cache leaking between requests in LLM serving: lessons from Workers AI
In this article

Since August 27, 2026, the GLM-5.3 model on Workers AI (@cf/zai-org/glm-5.3) has had a cross-customer isolation bug: some responses contained data from another customer's request sent to the same model just before. The bug resulted from two overlapping problems: a mismatched SGLang deployment configuration and a bug in how SGLang handles overly long requests in disaggregated prefill/decode mode. The incident has been fixed and the model was restored on September 9, with the disclosure coordinated with a post from the SGLang team.

Scope of impact and timeline

Researcher Gadi Evron of Knostic reported it through the HackerOne program on September 8. According to the report, very long prompts sent to the model sometimes received back text belonging to another user's session. Cloudflare reproduced the bug the next morning and took the model offline within 40 minutes of declaring the incident. In total, 155 responses were affected, all of them responses to prompts longer than 681,280 tokens. After excluding Cloudflare's own tests and the researcher's attempts, the remaining requests matched ordinary long-context usage in timing and volume, and there was no evidence of deliberate exploitation. No other model on Workers AI was affected.

Key milestones of the incident:

Technical cause

The pool size is not a configuration value: SGLang computes it itself at startup from the amount of GPU memory left after loading the weights. The model's context length is a separate configuration value, and nothing forces the two to match.

The second problem is an SGLang bug in disaggregated mode. When a prompt is longer than the pool, the scheduler marks the request as rejected and replaces the prompt with a one-token placeholder so that the request runs through the normal batching mechanism without allocating real memory. In a single-node deployment, the client receives an error. In disaggregated mode, the admission queues of both prefill and decode check whether the request has finished, not whether it was rejected, so the rejected request is still treated as alive.

  1. The router tells the decode node the prompt length, and decode reserves 8 KV pages for the request.

  2. The prefill admission queue accepts the rejected request because finished() still returns False.

  3. Prefill runs a trivial forward pass on a single placeholder token, sends one KV page to decode, and reports the transfer as complete.

  4. Decode accepts the handoff, starts generating a response, and attention reads all 8 reserved pages.

  5. Only page P0 was written by this request. Pages P1 through P7 still contain data from earlier requests, including other users' prompts and partially generated output.

  6. The client receives HTTP 200 instead of HTTP 400, with a coherent response containing content from other people's KV pages.

Remediation

After taking the model down, the team made two groups of changes before reopening it.

Hardening the system and coordinating with SGLang

This bug belongs to the class of KV cache memory-safety bugs, where a page is freed, reused, or read while another party still holds access to it. There are two common causes: page lifecycle within a node, when multiple tasks hold the same page, and race conditions between nodes, when a request is cancelled while prefill is still writing KV to decode over RDMA and writes already sent cannot be recalled. After the incident, Cloudflare did the following on every disaggregated deployment in production.

One day after the incident, Cloudflare met with the SGLang maintainers at RadixArk. The two sides agreed that RadixArk will investigate the reported race conditions, bring the idea of integrity checks upstream, and build reproduction tests for this class of bugs. The patch for the incident shipped in SGLang v0.5.20, with follow-up releases in v0.5.21 and v0.5.22. Some of the changes that have gone upstream include:

Recommendations for SGLang operators

For anyone running SGLang in disaggregated prefill/decode mode, Cloudflare recommends updating the SGLang dependency to v0.5.22 or later and taking the following steps.

In the long term, the approach being pursued is to detect, at read time, that a request is about to attend over a page that does not belong to it, and to fail only that request instead of returning someone else's data. The prototype attaches to each physical KV page a tag that changes every time the page is reallocated, each request records the tags it expects, and the mapping is checked before decode reads the cache; in testing, the cost was under 1% in throughput and tail latency. The prototype was turned off because of reliability issues and because it is tied to specific attention backends, so Cloudflare is working with the SGLang team to build a production version that works with every model. In parallel, Cloudflare is applying its formal verification experience to KV cache transfer engines, starting by modeling and proving the safety of the RDMA-based KV cache transfer protocol.

Key takeaways

  • From August 27 to September 9, 155 GLM-5.3 responses to prompts longer than 681,280 tokens may have contained other customers' data; there is no evidence of deliberate exploitation.

  • The cause was two overlapping problems: the prefill KV pool was smaller than the context window, and in disaggregated mode SGLang did not cancel rejected requests but still transferred stale cache to decode.

  • The fix consists of enlarging the pool to roughly 2.04 million tokens and patching the engine to drop rejected requests before handoff.

  • Only GLM 5.3 was affected; the upstream fixes are in SGLang v0.5.22, and operators are advised to update.

  • The long-term direction is to check page ownership at read time and to formally prove the KV cache transfer protocol.

More in Technology Insights

Illustration: Next.js information leak via "use cache" (CVE-2026-119790)
TECH Technology Insights

Next.js information leak via "use cache" (CVE-2026-119790)

A Moderate-severity vulnerability in Next.js allows cached data for one root param value to be returned to requests belonging to a different root param value. The bug appears when the same 'use cache' function is called both directly during render and through unstable_cache(), and that function reads a root param. The cause…

3 min read

Illustration: Q3 2026 DDoS report: many small "swarms", fewer big blows
TECH Technology Insights

Q3 2026 DDoS report: many small "swarms", fewer big blows

Short, medium-intensity, high-volume attacks, a hallmark of rented DDoS services (booters), accelerated this quarter. The main targets were computer software, gambling and casinos, and the information technology sector. Compared with the same period in 2025, network-layer attacks rose 208%, of which…

3 min read

Email newsletter

Receive a selection of new LeoNex News articles by email.

We only use this address to send the newsletter.

Receive a selection of new LeoNex News articles by email. You will receive a confirmation email; the subscription only takes effect after you confirm. Every email includes an unsubscribe link. Read the privacy policy