[Bugfix][KV Offload] Reset a resumed streaming session's block table in OffloadingConnector and SimpleCPUOffloadConnector by albertoperdomo2 · Pull Request #60810 · vllm-project/vllm

albertoperdomo2 · 2026-10-09T08:43:54Z

Overview

OffloadingConnector and SimpleCPUOffloadConnector corrupts the CPU KV cache for streaming-input sessions. A session resumed for its next input chunk arrives as NewRequestData with its full block table, but yield_req_data flagged it as a delta, so the connector appended it to the table it already held.

Claims

A resumed streaming session's new chunks are offloaded from their own GPU blocks, so requests reusing that prefix from the CPU tier get correct KV.
yield_req_data reports is_full_table (True for new requests) instead of preempted; the connector resets on it. NIXL and SimpleCPUOffload ignore the flag, so their behavior is unchanged.

Validation

test_streaming_session_stores_new_chunks (sync and async): the session's second chunk stores blocks (0, 1, 2) on main (earlier chunks' blocks) and (2, 3, 4) with this PR.
pytest tests/v1/kv_connector/unit/offloading_connector/ tests/v1/simple_kv_offload/: all pass.
E2E ran, and after a three-chunk streaming session, a greedy request on its 192-token prefix is served from the CPU tier and compared with a full recompute.
- main: the outputs differ (the CPU-tier output loops on 5, 276, 183, 9).
- this PR: all 32 tokens match.

Details

_build_store_jobs pairs chunk j's key with block_ids[j], so the duplicated table ([1, 2, 3, 1, 2, 3, 4, 5, 6] vs [1, 2, 3, 4, 5, 6]) stored earlier chunks' KV under later chunks' keys. For every other request the table is already empty when the reset runs, so it is a no-op for them.

SimpleCPUOffloadConnector duplicates the table the same way. It doesn't corrupt KV (it keys by block hash), but it skips offloading the session's new blocks. That will be a follow-up.

AI assistance (codex) was used; I reviewed every changed line and ran the validation above.

Pull Request Checklist

I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).
AI assistance was used during the creation of this PR (tests).
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>

albertoperdomo2 · 2026-10-15T10:17:38Z

cc: @orozery @ApostaC

albertoperdomo2 · 2026-10-17T11:33:01Z

/ci run

github-actions · 2026-10-17T11:33:17Z

❌ @albertoperdomo2, A reviewer with write access must run /ci run, approve the PR, or add the ready label first.

orozery

LGTM!
Thanks for the fix @albertoperdomo2 !
@ivanium PTAL for the simple cpu changes.

github-actions · 2026-10-21T15:33:21Z

✅ @albertoperdomo2, CI is now available for this PR.

/ci run starts upstream CI; /amd-ci run starts AMD CI only.
Run commands test your branch with every commit currently on its upstream target branch. If it is behind, the bot merges the target branch into it first (pull before pushing again). Append --allow-stale to a run command to test an outdated branch as is, at your own risk.
/ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
/amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
/ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

ivanium

Overall good; left a small question.

ivanium · 2026-10-21T20:04:22Z

-                    if new_block_id_groups[g] is not None:
+            if is_full_table:
+                state.block_ids = tuple([] for _ in range(num_groups))
+                state.num_stored_blocks = [0] * num_groups


Small Q: Should we preserve num_stored_blocks here and only replace block_ids? Or maybe I missed sth

nice catch, yes, you are right: streaming continuation preserves the prefix, so num_stored_blocks remains valid (new requests initialize it to zero and preemption resets it above). I’ll remove this reset and only replace block_ids 👍🏽
cc: @orozery

Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>

orozery · 2026-10-25T06:23:07Z

/ci run

github-actions · 2026-10-25T06:23:33Z

✅ Triggered Buildkite CI #97104 for commit fe973a776985.

ℹ️ This PR is 128 commits behind upstream main. Merged upstream main into this branch as fe973a776985 before starting CI. Pull the branch before pushing more changes.

…in OffloadingConnector and SimpleCPUOffloadConnector (vllm-project#60810) Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: vllm-agent <claw@inferact.ai> Signed-off-by: Christopher Mackenzie <christopher.mackenzie.software@gmail.com>

…in OffloadingConnector and SimpleCPUOffloadConnector (vllm-project#60810) Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: vllm-agent <claw@inferact.ai>

…e flag Chunked local attention blocks hold token positions, like full and sliding-window attention, so select them by position too: the block that straddles a chunk boundary was not copied again. A UniformTypeKVCacheSpecs group already takes this path, because generate_scheduler_kv_cache_config gives the scheduler one of its layer specs. Since vllm-project#60810, yield_req_data reports a new request, and so a preempted request that comes back as new, with is_full_table=True. The extra reset on preempted_req_ids is redundant: drop it and name the flag after what it means. Skip the last-chunk SSM copy for "align" SSM groups, whose block table is not append-only; they keep main's per-step copy. Reword two stale comments. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Sohom Chakraborty <16609933+sohom-cs@users.noreply.github.com>

fix: Reset resumed streaming session's block table

b7052cb

Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>

claude Bot reviewed Oct 9, 2026

View reviewed changes

albertoperdomo2 requested review from ApostaC, NickLucche, ivanium, orozery, wzhao18 and xuechendi as code owners October 9, 2026 08:43

Merge branch 'main' into fix/offload-streaming-session-block-ids

c3b0b51

mergify Bot added bug Something isn't working kv-connector labels Oct 9, 2026

albertoperdomo2 added 2 commits October 9, 2026 20:34

Merge branch 'main' into fix/offload-streaming-session-block-ids

34d9136

Merge branch 'main' into fix/offload-streaming-session-block-ids

e016a28

snadampal reviewed Oct 13, 2026

View reviewed changes

Comment thread vllm/distributed/kv_transfer/kv_connector/v1/offloading/scheduler.py Resolved

Comment thread vllm/distributed/kv_transfer/kv_connector/v1/offloading/scheduler.py Resolved

albertoperdomo2 added 3 commits October 14, 2026 07:59

chore: Rename loop variable to be more accurate

9d6e1b8

Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>

fix: Reset SimpleCPUOffload block table for resumed streaming sessions

3aae52b

Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>

Merge branch 'main' into fix/offload-streaming-session-block-ids

1d584b9

albertoperdomo2 changed the title ~~[Bugfix][KV Offload] Reset a resumed streaming session's block table in OffloadingConnector~~ [Bugfix][KV Offload] Reset a resumed streaming session's block table in OffloadingConnector and SimpleCPUOffloadConnector Oct 14, 2026

Merge branch 'main' into fix/offload-streaming-session-block-ids

3b799b8

albertoperdomo2 added 3 commits October 16, 2026 08:51

Merge branch 'main' into fix/offload-streaming-session-block-ids

38f13b4

Merge branch 'main' into fix/offload-streaming-session-block-ids

6ada3af

Merge branch 'main' into fix/offload-streaming-session-block-ids

da25bb5

Merge branch 'main' into fix/offload-streaming-session-block-ids

e26bc84

orozery approved these changes Oct 21, 2026

View reviewed changes

orozery added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 21, 2026

Merge branch 'main' into fix/offload-streaming-session-block-ids

dcde27b

ivanium reviewed Oct 21, 2026

View reviewed changes

fix: Preserve num_store_blocks on streaming continuation

7122c97

Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>

AdvayMonga mentioned this pull request Oct 22, 2026

[Bugfix][KV Offload] Reset the store cursor when a request restarts from token 0 #63010

Merged

6 tasks done

Merge branch 'main' into fix/offload-streaming-session-block-ids

7d21aa4

Merge branch 'main' into fix/offload-streaming-session-block-ids

fe973a7

orozery merged commit 921ed7f into vllm-project:main Oct 25, 2026
105 checks passed

albertoperdomo2 deleted the fix/offload-streaming-session-block-ids branch October 25, 2026 08:03

sohom-cs mentioned this pull request Oct 27, 2026

[Bugfix][KV Offload] Do not offload the partially computed block of a request that finished with an error #63937

Merged

1 of 6 tasks

sohom-cs mentioned this pull request Oct 29, 2026

[Bugfix][NIXL] Re-save the block that straddles a chunked-prefill boundary in host-buffer mode #59102

Open

1 of 6 tasks

Navigation Menu

Repository navigation

[Bugfix][KV Offload] Reset a resumed streaming session's block table in OffloadingConnector and SimpleCPUOffloadConnector - #60810

[Bugfix][KV Offload] Reset a resumed streaming session's block table in OffloadingConnector and SimpleCPUOffloadConnector#60810
orozery merged 16 commits into
vllm-project:mainfrom
albertoperdomo2:fix/offload-streaming-session-block-ids

albertoperdomo2 commented Oct 9, 2026 •

edited

Loading

Uh oh!

claude Bot left a comment

Uh oh!

Uh oh!

Uh oh!

albertoperdomo2 commented Oct 15, 2026

Uh oh!

albertoperdomo2 commented Oct 17, 2026

Uh oh!

github-actions Bot commented Oct 17, 2026

Uh oh!

orozery left a comment

Uh oh!

github-actions Bot commented Oct 21, 2026

Uh oh!

ivanium left a comment

Uh oh!

ivanium Oct 21, 2026

Uh oh!

albertoperdomo2 Oct 21, 2026

Uh oh!

orozery commented Oct 25, 2026

Uh oh!

github-actions Bot commented Oct 25, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

albertoperdomo2 commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Claims

Validation

Details

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

Uh oh!

Uh oh!

albertoperdomo2 commented Oct 15, 2026

Uh oh!

albertoperdomo2 commented Oct 17, 2026

Uh oh!

github-actions Bot commented Oct 17, 2026

Uh oh!

orozery left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 21, 2026

Uh oh!

ivanium left a comment

Choose a reason for hiding this comment

Uh oh!

ivanium Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

albertoperdomo2 Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

orozery commented Oct 25, 2026

Uh oh!

github-actions Bot commented Oct 25, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

albertoperdomo2 commented Oct 9, 2026 •

edited

Loading