Repository navigation
[Bugfix][KV Offload] Do not store sliding-window chunks with null blocks - #63652
Conversation
With blocks_per_chunk > 1, a sliding-window group can have a null block inside a chunk whose last block is real: a GPU prefix-cache hit leaves the blocks before the window null, and a sliding-window free zeroes pending store positions. _build_store_jobs checked only each chunk's last block, stored the chunk without the null block's slot, and the whole chunk became a CPU hit. A later request whose window covers that slot loaded KV that was never written. Skip a chunk when any block that the store mask counts as reachable is null. With no mask, every block counts. Add a scheduler-level test, and an LLM-level test (Gemma-3, 8 blocks per chunk, store_threshold=2) that checks a request hitting the CPU tier at such a chunk produces the same output as a cold run. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Stan <273962421+pepmach@users.noreply.github.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment Once the PR is approved or has the If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. Agent GuidelinesIMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban. 🚀 |
|
/ci run |
|
✅ @pepmach, CI is now available for this PR.
|
|
✅ Triggered Buildkite CI #97248 for commit ℹ️ This PR is 37 commits behind upstream |
Conflict in test_offloading_connector.py: keep main's new test_cpu_offloading_sliding_window_partial_chunk (vllm-project#63652) CUDA-only, as main has it (not yet run on ROCm), and this branch's CUDA-or-ROCm gate on test_cpu_offloading_metrics. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
…cks (vllm-project#63652) Signed-off-by: Stan <273962421+pepmach@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: vllm-agent <claw@inferact.ai> Signed-off-by: Christopher Mackenzie <christopher.mackenzie.software@gmail.com>
…cks (vllm-project#63652) Signed-off-by: Stan <273962421+pepmach@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: vllm-agent <claw@inferact.ai> Signed-off-by: Christopher Mackenzie <christopher.mackenzie.software@gmail.com>
Overview
Fixes #63653.
With offloaded chunks of several GPU blocks,
OffloadingConnectorcould store a sliding-window chunk that contained a null block. A later CPU hit then loaded the never-written slot as KV, and the request's output changed with no error.Claims
Validation
Two new tests:
test_sliding_window_chunk_with_null_block_is_not_storedintests/v1/kv_connector/unit/offloading_connector/test_scheduler.py: scheduler level, sync and async scheduling.test_cpu_offloading_sliding_window_partial_chunkintests/v1/kv_connector/unit/test_offloading_connector.py: LLM level, CUDA,google/gemma-3-1b-itwithblocks_per_chunk=8andstore_threshold=2. Request B's GPU hit ends mid-chunk, which leaves null sliding-window blocks inside chunk 0, and B stores that chunk. The test asserts that this hit length is reached, so the case is exercised, and that a request hitting the chunk on the CPU tier produces the same output as a cold run.Scheduler-level test without the fix (main
24324ef0), failing at the store check for request B:Line 2299 is
runner.run(decoded_tokens=[EOS_TOKEN_ID], expected_stored=((0, 0), (0, 1))): on main, B also stores sliding-window block 1 without block 0. With the fix,2 passed in 1.29s.LLM-level test: not run on a GPU, because I don't have one. CI runs it on CUDA. I ran its helper
_sliding_window_partial_chunk_teston CPU against a tiny random Gemma-3 (512-token window, GPU block 32, 8 blocks per chunk). The offload copy was done by an out-of-tree spec that runsgpu_worker.py's transfer code withmemmovein place of the CUDA kernel (described in the linked issue).assert out_c.outputs[0].token_ids == cold_c.outputs[0].token_ids: C's greedy tokens change, and C loaded 114,688 bytes that no store wrote.End to end on CPU with the same spec, comparing C's output against a run without offloading:
Model evaluation: the fix changes only which chunks get stored, so an affected request recomputes a chunk instead of loading it. An accuracy benchmark wouldn't isolate the bug, because the bug needs a specific hit-then-evict pattern. The runs above construct that pattern and compare against a run without offloading. The LLM-level test does the same in CI against a cold run on
google/gemma-3-1b-it.Suites, run with
pytest -q <suite>on CPU (macOS arm64,VLLM_TARGET_DEVICE=cpu) at24324ef0:tests/v1/kv_connector/unit/offloading_connector/tests/v1/kv_connector/unit/test_offloading_connector.pytests/v1/coretests/v1/kv_offload(withoutcpu/test_gpu_worker.py)The failures on both sides are environment limits on this Mac:
tests/v1/core, the hybrid test model needscausal_conv1d_weight_pack, which the CPU build lacks, and one test's block size is unsupported;kv_offloadfailures arecpu/test_shared_offload_region.py, which needs/dev/shm.pre-commit run --fileson the changed files passes, including ruff 0.14.0, typos, mypy and the Buildkite test-wiring check.The branch merges cleanly into main
237d2ffd. On that merge, the scheduler-level test, the LLM-level helper on CPU and theoffloading_connectorsuite (632 passed, 2 skipped) all pass.Details
Root cause, with lines at
24324ef0:vllm/v1/core/single_type_kv_cache_manager.py:1004-1037). A sliding-window free also zeroes pending store positions (offloading/scheduler.py:1503-1522).prefix_cache_retention_interval=0, a GPU hit lands on an earlier prompt's replay boundary. That boundary is block-aligned but not necessarily chunk-aligned, so the null run can end mid-chunk._build_store_jobsselected chunks by their last block only (offloading/scheduler.py:1880-1891,1917), then dropped null blocks from the store spec (1995).vllm/v1/kv_offload/cpu/gpu_worker.py:635-654), whilecomplete_storemarked all of it ready (vllm/v1/kv_offload/cpu/manager.py:370-387).The fix adds one check to key selection. After the existing reachability test, a chunk is skipped if any block the store mask marks reachable is null. With no mask, any null block skips it. No GPU block exists to copy for that position, so the chunk can't be completed. A later request that would have hit this chunk computes it instead: it loses a partial CPU hit and avoids loading stale KV. Null blocks that the mask marks unreachable are still allowed. A request whose window excludes them holds null blocks there too, and its load skips those slots.
Test changes:
RequestRunnerrejectedblocks_per_chunk > 1whenever custom KV cache groups were passed. It now accepts them when every group uses the runner's block size._wait_for_prefix_cache_resettakesreset_connector, so the LLM-level test can clear the CPU tier before its cold run.Not a duplicate: as of 2026-10-26, no open or closed issue or PR reports this bug. I ran 20 keyword searches over issues and PRs and checked the 171 open PRs that touch the offloading connector or
kv_offload/. The closest are these two:store_triggers/on_compute.pyand keeps the last-block check. If it lands first, this check goes there.AI assistance: Claude (Anthropic) found this bug with a randomized checker over the offloading connector. It also wrote the reproductions, the CPU end-to-end harness, both tests and the fix, and ran the suites above. I reviewed every changed line and reran the tests and the CPU end-to-end check. The commit carries a
Co-authored-bytrailer.Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
Need help on this PR? Tag
@codesmith-botwith what you need.Need help on this PR? Tag
@codesmith-botwith what you need.