Repository navigation
[Bugfix][Core] Exempt exactly the blocks an async KV load writes from zeroing - #59504
Merged
Merged
Conversation
… zeroing The scheduler skipped zeroing for the blocks covering the loaded token range, which only matches full attention. A non-prefix-cacheable sliding window gets fresh blocks below the local hit, and a circular buffer keeps its block at index 0; both were zeroed while the connector wrote them. Let the managers leave the blocks they allocate for an async load out of the zeroing list and drop the token-range skip set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Move the shared loaded-group API from vllm-project#59197 so replay and zeroing use the same connector declaration. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
ivanium
marked this pull request as ready for review
October 1, 2026 22:25
ivanium
requested review from
ApostaC,
NickLucche,
WoosukKwon,
alexm-redhat,
heheda12345,
njhill,
orozery,
robertgshaw2-redhat,
wzhao18,
xuechendi and
ywang96
as code owners
October 1, 2026 22:25
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Member
|
@claude review |
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
…oing # Conflicts: # vllm/v1/core/sched/scheduler.py Signed-off-by: Nick Hill <nickhill123@gmail.com>
Member
|
/ci run |
|
❌ This PR is 1 commit behind upstream |
Member
|
/ci run |
|
✅ Triggered Buildkite CI #92719 for commit |
Contributor
CI selector (shadow): 131 test steps (170 jobs) instead of 75 (92 jobs)Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today's rules. How it works. Feedback welcome: reply here if it would skip a step this change needs, or runs something unrelated.
Selector would run (131)
Would skip (today's rules run them) (11)
Would add (today's rules do not run them) (67)
AMD mirrors: would skip (11)
AMD mirrors: would add (57)
17 changed files · base |
Member
|
/ci run |
|
✅ Triggered Buildkite CI #92723 for commit |
Member
|
/ci retry |
|
✅ Triggered Buildkite CI #92743 for commit |
njhill
enabled auto-merge (squash)
October 3, 2026 05:48
whx-sjtu
added a commit
to whx-sjtu/vllm
that referenced
this pull request
Oct 3, 2026
Remove the connector zeroing hook and reuse the loaded-group allocation contract introduced by vllm-project#59504. Keep synchronous exemptions limited to hybrid models, preserve local tails and unloaded groups, and retain the full external hit when prefix replay rewinds the computed-token count. Adapt the scheduler and MoRI READ regressions to the allocation contract; keep the upstream removal of token-range slicing and per-step exclusion state. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: whx-sjtu <xiaowang990929@gmail.com>
6 tasks done
whx-sjtu
added a commit
to whx-sjtu/vllm
that referenced
this pull request
Oct 4, 2026
Remove the connector zeroing hook and reuse the loaded-group allocation contract introduced by vllm-project#59504. Keep synchronous exemptions limited to hybrid models, preserve local tails and unloaded groups, and retain the full external hit when prefix replay rewinds the computed-token count. Adapt the scheduler and MoRI READ regressions to the allocation contract; keep the upstream removal of token-range slicing and per-step exclusion state. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: whx-sjtu <xiaowang990929@gmail.com>
ivanium
added a commit
to ivanium/vllm
that referenced
this pull request
Oct 6, 2026
…window Use the loaded-group contract introduced by vllm-project#59504 to retain transferred windows. Keep this commit limited to replay decisions and their regression test. Co-authored-by: Claude Code <noreply@anthropic.com> Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
nick-oconnor
added a commit
to nick-oconnor/vllm
that referenced
this pull request
Oct 6, 2026
Add SM120-GLM53-FLASH.md: the kept and dropped ocnr changes with their upstream status, the deployed serve args, the auto-fit memory model, and the boot-verify checklist that must pass before deploying a build. Carries the 0.31 cut, first onto the v0.31.0 tag and then onto main. Moving to main drops the vllm-project#55222 and vllm-project#56531 carries (both merged upstream after the release fork) and picks up vllm-project#59504, which the tag base leaves open. Records why vllm-project#57635 had to be adapted rather than replayed -- upstream rescoped flashinfer_autotune for PP, but the leader-broadcast pattern the carry removes is still there -- and that the CUDAGraph estimate is not the cause of the 1M-context regression and must not be disabled. Signed-off-by: Nick O'Connor <nick@ocnr.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Prevent allocator zeroing from racing async KV loads, including sliding-window blocks below a local prefix hit and circular-buffer blocks at index 0. Keep zeroing enabled for groups the selected connector does not load.
Claims
get_loaded_kv_cache_group_ids(request), moved from [Bugfix][DSv4.1] Skip SWA bounded replay for KV loads that carry the window #59197: prefix-cacheable groups by default, all transfer groups for NIXL/Mooncake P/D, and request-selected delegation for MultiConnector. [Bugfix][DSv4.1] Skip SWA bounded replay for KV loads that carry the window #59197 consumes the same API for replay decisions.Validation
Reproduction commands
Earlier end-to-end results, collected before this API move: DeepSeek-V4.1-Flash P/D with NIXL, 1P1D TP4 + TP4, two-pass GSM8K 5-shot, concurrency 512. Pass 2 includes decode-side prefix hits. “Other wrong” excludes prompts with N % 32 == 31 (#59330); “t < 16” counts those with (N - 1) % 128 < 16. These model evaluations were not rerun for the API refactor.
Details
Token-range slicing cannot identify all transferred blocks for non-prefix-cacheable windows or circular buffers. Allocation already knows the new physical blocks; the connector now identifies which groups it restores. The scheduler passes those group IDs to allocation, replacing
_skip_zero_block_idsandget_zeroing_block_ids_in_range. Store/offloading connectors retain zeroing for their excluded scratch/window groups.Failed-load recovery is unchanged: multi-group loads free and retry; single-group partial failures re-record the unwritten suffix. Tests cover async/sync loads with and without the window, circular-buffer zeroing, and MultiConnector selection.
Not a duplicate: #59197 uses this shared contract to skip replay; #59164 concerns synchronous MoRIIO READ destinations; #55085 handles unwritten Mooncake Store tail pages; #52187 concerns worker-side per-group zeroing.
Pull Request Checklist
/pr-checklistskill.