[Bug] SimpleCPUOffload: residual output garbling under high concurrency — load path lacks cross-stream sync with compute (WAR), symmetric gap left by #46278

[Saddss]

Your current environment

• vLLM: latest main (includes [Bugfix][KVConnector] Fix SimpleCPUOffloadConnector GPU->CPU store race #46278 / fix for [Bug]: SimpleCPUOffloadConnector corrupts restored KV under load → intermittent garbled output (missing GPU→CPU store cross-stream sync) #45704); also reproduced on nightly 0.23.1rc1.dev601
• Model: RedHatAI/gemma-4-31B-it-NVFP4, single GPU (32 GB)
• Connector: VLLM_USE_SIMPLE_KV_OFFLOAD=1, native backend, 100 GB pinned CPU pool, fp8 KV cache
• Serve: --max-model-len 16384 --max-num-seqs 64 --max-num-batched-tokens 8192, eager / -O3

🐛 Describe the bug

After #46278 (which fixed the store-side store-vs-compute race, #45704), we still received a small number of user reports of garbled output under load — far fewer than before the fix, but not zero. We reproduced it and root-caused a second, independent race, this time on the load path, that #46278 does not cover.

Symptom. Output starts coherent, then collapses into degenerate repetition + mixed-script "salad" (a decode collapse, not genuine multilingual text), e.g.:

"...single de single de single or de laer singleT single singleyP ..."

Detection used in the repro: a completion containing ≥3 distinct non-Latin Unicode scripts = one "hard garble".

Root cause: cross-stream write-after-read (WAR) on the load path.

In vllm/v1/simple_kv_offload/worker.py::get_finished, stores are ordered after compute via a _store_compute_done event (the #46278 fix). Loads are launched with no wait_event; load completion is gated only on the CPU side (a request stays WAITING_FOR_REMOTE_KVS until the load event query()s true → finished_recving).

That correctly prevents the resuming request from reading its own blocks early, but it does not enforce the reverse ordering: when a GPU block is freed and immediately reallocated as a load destination, the load DMA (on the dedicated low-priority load_stream) can begin overwriting that block while a previous request's attention is still reading it on the compute stream. Nothing makes load_stream wait for that compute.

Normal (non-offload) block reuse is safe because the old reader and the new writer are both on the compute stream and serialize; offload moves the write onto a separate stream with no event bridging the two, so the WAR window opens. The transfer streams are created at lowest priority ("KV I/O yields to compute"), which makes the load lag right into the compute window, so the rate scales with concurrency.

This is consistent with all observations: it occurs only with offload ON; it is independent of MTP; and it is independent of the sampling mechanism (greedy also corrupts, at a lower rate, because coherent outputs are shorter → fewer decode steps → less offload traffic).

Decisive evidence (synchronization bisection). With a diagnostic-only barrier around the copy (no data / no logic change) gating one direction at a time, identical workload (fp8 / 16384 / temp 0.7 / concurrency 64, 500 prompts × 3 rounds):

mode	garble rate
none (baseline)	~4.2%
sync store only	2.6% (still garbling)
sync load only	0%
sync all	0%

Serializing the load path eliminates the garble; serializing the store path does not → the residual defect is on the load side, and #46278's store-side fix is correct.

Proposed fix

Give the load path the barrier symmetric to the store path: in get_finished, record one compute-done event per step (rename _store_compute_done → a shared _compute_done) and have both load and store wait_event on it, so a load cannot overwrite a reused block the compute stream is still reading.

Verification (same image, same workload):

• Garble: baseline 142/3300 = 4.3% → fix 0/3000 = 0% (Fisher exact p ≈ 5e-41).
• Added a self-validating regression test test_load_orders_after_compute_read (the no-barrier control corrupts a live read; the barrier path is clean), symmetric to the existing store test; all pass; ruff / mypy / pre-commit clean.
• Performance: no measurable cost in a healthy regime (concurrency matched to GPU KV, loads still active on cache hits: within ±1%); only under deliberate ~10× KV oversubscription (offload thrashing) is there a single-digit cost (~2% throughput, +6–7% p99), which is expected since the event is only recorded / awaited when transfers actually happen.

I have the fix + test on a branch and am happy to open a PR if this direction looks right. One design question: the current approach makes the load wait for all compute issued so far this step (conservative, matching the store side) rather than only the compute that read the specific reused blocks — I lean toward the simple symmetric version for clarity, but I'm open to input.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the discussions / read the FAQ. This appears to be a residual of [Bug]: SimpleCPUOffloadConnector corrupts restored KV under load → intermittent garbled output (missing GPU→CPU store cross-stream sync) #45704 not covered by [Bugfix][KVConnector] Fix SimpleCPUOffloadConnector GPU->CPU store race #46278.

[hclsys]

symmetric version is the right call. per-block wait would mean tracking which compute reads touched each reused block and cross-referencing on the load side — that's a new bookkeeping surface (and its own race risk) to save an event that's already free when no transfer happens. matches the store path, easy to reason about. nice RCA on the load-side WAR.

[Saddss]

symmetric version is the right call. per-block wait would mean tracking which compute reads touched each reused block and cross-referencing on the load side — that's a new bookkeeping surface (and its own race risk) to save an event that's already free when no transfer happens. matches the store path, easy to reason about. nice RCA on the load-side WAR.采用对称版本才是正确的做法。如果按块来等待处理，那就需要记录每个被重复使用的块被哪些计算操作所读取，同时还要在加载端进行相应的交叉引用处理——这无疑会增加额外的记录工作量（同时也存在并发风险），而实际上，当没有数据传输发生时，这些块本来就是“空闲”的状态。这种处理方式与数据的存储路径相匹配，也更容易理解。在加载端的性能方面，这种处理方式的效果相当不错。

Thanks — agreed on the symmetric version; per-block ordering isn't worth the extra bookkeeping (and its own race surface) for an event that's a no-op when there's no transfer.

One design question I'd value your read on, w.r.t. the connector's original intent. Deferring transfers to get_finished was meant to hide the host-side copy-launch overhead behind GPU compute, and that's preserved — the compute-done event is a device-side dependency and never blocks the host thread. What changes is that gating both directions serializes the actual DMA after the current step's compute on the GPU, i.e. it gives up some transfer/compute overlap in the offload-heavy path. In practice that looks negligible (the I/O streams are already lowest-priority so they yield to compute anyway, and the healthy-regime perf delta measured ~0; the cost only shows up single-digit under deliberate KV oversubscription/thrashing).

So: do you see gating both paths as still consistent with SimpleCPUOffload's "keep it simple + overlap I/O behind compute" goals? Or would you prefer we treat correctness-first now and leave any overlap-recovery (e.g. finer-grained ordering) as a separate follow-up? I'm happy to proceed with the symmetric fix either way — just want it to line up with where you want the connector to go.

[hclsys]

correctness-first, ship the symmetric fix now. the overlap you give up only bites in the thrashing regime, which is already a degraded path — and the i/o streams are lowest-priority so they largely yield to compute regardless. healthy-regime delta ~0 means there's no real overlap to recover in the common case. finer-grained ordering is a premature optimization; leave it as a follow-up gated on an actual workload that shows the DMA serialization hurting. keeps the connector simple, which is the point.

[Saddss]

correctness-first, ship the symmetric fix now. the overlap you give up only bites in the thrashing regime, which is already a degraded path — and the i/o streams are lowest-priority so they largely yield to compute regardless. healthy-regime delta ~0 means there's no real overlap to recover in the common case. finer-grained ordering is a premature optimization; leave it as a follow-up gated on an actual workload that shows the DMA serialization hurting. keeps the connector simple, which is the point.以正确性为首要原则，现在就发布这个对称性修复方案吧。你所放弃的那些重叠处理，在“剧烈波动”的运行状态下其实没什么影响——因为那种状态本来就是效率较低的运行模式。另外，I/O 操作属于低优先级任务，所以它们通常会让位于计算任务。在正常运行状态下，~0 这个数值表明其实并没有什么需要恢复的重叠处理。进行更细粒度的优化为时过早；还是先保留这个方案，等实际运行中确实出现 DMA 串行化带来的性能问题时再进行处理吧。保持接口的简单性才是关键。

Opened #47291 with the symmetric fix — both paths wait on one per-step compute-done event, no-op when there's no transfer. Skipped per-block ordering; can revisit if a real workload shows it mattering. Thanks!