[BugFix] Resolve multiple async kv load deadlock - #44560
Merged
Merged
Conversation
Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
requested review from
ApostaC,
WoosukKwon,
alexm-redhat,
heheda12345,
orozery,
robertgshaw2-redhat and
ywang96
as code owners
June 4, 2026 17:19
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| @@ -386,7 +392,8 @@ def allocate_slots( | |||
| num_tokens_main_model=num_tokens_main_model, | |||
| ) | |||
|
|
|||
| if num_blocks_to_allocate > self.block_pool.get_num_free_blocks(): | |||
| available_blocks = self.block_pool.get_num_free_blocks() - reserved_blocks | |||
Contributor
There was a problem hiding this comment.
This estimation is pessimistic, so it might reduce utilization? I'm not sure of the trade-off vs preemption (which only happens at the peak usage).
Contributor
There was a problem hiding this comment.
Discussed offline. This is correct for PD case and might affect throughput for non-PD on some edge cases. But those cases are rare and should run in PD anyway.
WoosukKwon
approved these changes
Jun 5, 2026
njhill
enabled auto-merge (squash)
June 6, 2026 22:37
njhill
added a commit
to njhill/vllm
that referenced
this pull request
Jun 7, 2026
Resolve conflicts with the async KV-load deadlock fix (vllm-project#44560) in the KV cache admission path: - allocate_slots() now takes both num_running_reqs (watermark sizing) and reserved_blocks (in-flight prefill reservation). - The free-block admission check stacks both: a request is admitted only if num_blocks_to_allocate + watermark_blocks <= free_blocks - reserved_blocks. - scheduler passes len(self.running) and the in-flight-prefill reservation. Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
knight0528
pushed a commit
to knight0528/vllm
that referenced
this pull request
Jun 8, 2026
Signed-off-by: Nick Hill <nickhill123@gmail.com>
ekagra-ranjan
pushed a commit
to ekagra-ranjan/vllm
that referenced
this pull request
Jun 9, 2026
Signed-off-by: Nick Hill <nickhill123@gmail.com> Signed-off-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com>
waqahmed-amd-fi
pushed a commit
to waqahmed-amd-fi/vllm
that referenced
this pull request
Jun 10, 2026
Signed-off-by: Nick Hill <nickhill123@gmail.com> Signed-off-by: Waqar Ahmed <waqar.ahmed@amd.com>
Saddss
pushed a commit
to Saddss/vllm
that referenced
this pull request
Jun 14, 2026
Signed-off-by: Nick Hill <nickhill123@gmail.com>
divineearthly
pushed a commit
to divineearthly/vllm
that referenced
this pull request
Jun 19, 2026
Signed-off-by: Nick Hill <nickhill123@gmail.com> Signed-off-by: divineearthly <divineearthly@gmail.com>
nkzhenhua
pushed a commit
to nkzhenhua/vllm
that referenced
this pull request
Jun 24, 2026
Signed-off-by: Nick Hill <nickhill123@gmail.com>
MingqiWang-coder
added a commit
to vLLM-HUST/vllm-hust
that referenced
this pull request
Jun 30, 2026
Cherry-pick 62 bugfix/security PRs from upstream vllm-project/vllm main (2026-05-03 to 2026-06-17), covering scheduler, engine core, model runner, worker, attention, KV cache, compilation, and structured output fixes. Security (4): vllm-project#43286 vllm-project#44744 vllm-project#45118 vllm-project#45252 Bugfix (56): vllm-project#35536 vllm-project#36616 vllm-project#38895 vllm-project#39155 vllm-project#39324 vllm-project#39562 vllm-project#39805 vllm-project#40398 vllm-project#40726 vllm-project#40727 vllm-project#40737 vllm-project#40749 vllm-project#40961 vllm-project#41119 vllm-project#41133 vllm-project#41233 vllm-project#41237 vllm-project#41411 vllm-project#41496 vllm-project#41549 vllm-project#41674 vllm-project#41873 vllm-project#41895 vllm-project#42040 vllm-project#42112 vllm-project#42289 vllm-project#42479 vllm-project#42585 vllm-project#42692 vllm-project#42706 vllm-project#42709 vllm-project#42739 vllm-project#42967 vllm-project#43001 vllm-project#43079 vllm-project#43125 vllm-project#43160 vllm-project#43616 vllm-project#43669 vllm-project#43719 vllm-project#43768 vllm-project#43808 vllm-project#43961 vllm-project#43982 vllm-project#43988 vllm-project#43998 vllm-project#44057 vllm-project#44560 vllm-project#44574 vllm-project#44568 vllm-project#44603 vllm-project#44744 vllm-project#45195 vllm-project#45345 vllm-project#45383 vllm-project#45487 vllm-project#45564 vllm-project#45673 Runner fix (2): vllm-project#44568 vllm-project#44603 Skipped: vllm-project#43781 (ROCm-specific, not applicable to Ascend NPU) Conflict resolutions: - Manual merge: vllm-project#43286 vllm-project#45118 vllm-project#42112 vllm-project#43160 vllm-project#43719 vllm-project#44560 - Upstream-preferred (-X theirs): vllm-project#43808 vllm-project#43988 vllm-project#42967 vllm-project#35536 vllm-project#45195 - Test files (--theirs): vllm-project#44744 vllm-project#41895 vllm-project#42040 vllm-project#41233 vllm-project#45345 vllm-project#43982 Co-authored-by: GitHub Copilot Signed-off-by: MingqiWang-coder <mingqiwang@hust.edu.cn>
MingqiWang-coder
added a commit
to vLLM-HUST/vllm-hust
that referenced
this pull request
Jun 30, 2026
Cherry-pick 62 bugfix/security PRs from upstream vllm-project/vllm main (2026-05-03 to 2026-06-17), covering scheduler, engine core, model runner, worker, attention, KV cache, compilation, and structured output fixes. Security (4): vllm-project#43286 vllm-project#44744 vllm-project#45118 vllm-project#45252 Bugfix (56): vllm-project#35536 vllm-project#36616 vllm-project#38895 vllm-project#39155 vllm-project#39324 vllm-project#39562 vllm-project#39805 vllm-project#40398 vllm-project#40726 vllm-project#40727 vllm-project#40737 vllm-project#40749 vllm-project#40961 vllm-project#41119 vllm-project#41133 vllm-project#41233 vllm-project#41237 vllm-project#41411 vllm-project#41496 vllm-project#41549 vllm-project#41674 vllm-project#41873 vllm-project#41895 vllm-project#42040 vllm-project#42112 vllm-project#42289 vllm-project#42479 vllm-project#42585 vllm-project#42692 vllm-project#42706 vllm-project#42709 vllm-project#42739 vllm-project#42967 vllm-project#43001 vllm-project#43079 vllm-project#43125 vllm-project#43160 vllm-project#43616 vllm-project#43669 vllm-project#43719 vllm-project#43768 vllm-project#43808 vllm-project#43961 vllm-project#43982 vllm-project#43988 vllm-project#43998 vllm-project#44057 vllm-project#44560 vllm-project#44574 vllm-project#44568 vllm-project#44603 vllm-project#44744 vllm-project#45195 vllm-project#45345 vllm-project#45383 vllm-project#45487 vllm-project#45564 vllm-project#45673 Runner fix (2): vllm-project#44568 vllm-project#44603 Skipped: vllm-project#43781 (ROCm-specific, not applicable to Ascend NPU) Conflict resolutions: - Manual merge: vllm-project#43286 vllm-project#45118 vllm-project#42112 vllm-project#43160 vllm-project#43719 vllm-project#44560 - Upstream-preferred (-X theirs): vllm-project#43808 vllm-project#43988 vllm-project#42967 vllm-project#35536 vllm-project#45195 - Test files (--theirs): vllm-project#44744 vllm-project#41895 vllm-project#42040 vllm-project#41233 vllm-project#45345 vllm-project#43982 Co-authored-by: GitHub Copilot Signed-off-by: MingqiWang-coder <mingqiwang@hust.edu.cn>
MingqiWang-coder
added a commit
to vLLM-HUST/vllm-hust
that referenced
this pull request
Jul 2, 2026
Cherry-pick 62 bugfix/security PRs from upstream vllm-project/vllm main (2026-05-03 to 2026-06-17), covering scheduler, engine core, model runner, worker, attention, KV cache, compilation, and structured output fixes. Security (4): vllm-project#43286 vllm-project#44744 vllm-project#45118 vllm-project#45252 Bugfix (56): vllm-project#35536 vllm-project#36616 vllm-project#38895 vllm-project#39155 vllm-project#39324 vllm-project#39562 vllm-project#39805 vllm-project#40398 vllm-project#40726 Runner fix (2): vllm-project#44568 vllm-project#44603 Skipped: vllm-project#43781 (ROCm-specific, not applicable to Ascend NPU) Conflict resolutions: - Manual merge: vllm-project#43286 vllm-project#45118 vllm-project#42112 vllm-project#43160 vllm-project#43719 vllm-project#44560 - Upstream-preferred (-X theirs): vllm-project#43808 vllm-project#43988 vllm-project#42967 vllm-project#35536 vllm-project#45195 - Test files (--theirs): vllm-project#44744 vllm-project#41895 vllm-project#42040 vllm-project#41233 vllm-project#45345 vllm-project#43982 Co-authored-by: GitHub Copilot Signed-off-by: MingqiWang-coder <mingqiwang@hust.edu.cn>
ohsono
pushed a commit
to ohsono/vllm
that referenced
this pull request
Jul 3, 2026
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Dao007forever
pushed a commit
to Dao007forever/vllm
that referenced
this pull request
Jul 18, 2026
Signed-off-by: Nick Hill <nickhill123@gmail.com>
philippesic
pushed a commit
to philippesic/vllm-semantic-cache
that referenced
this pull request
Jul 19, 2026
Signed-off-by: Nick Hill <nickhill123@gmail.com>
plasticchris
pushed a commit
to plasticchris/vllm
that referenced
this pull request
Jul 20, 2026
Signed-off-by: Nick Hill <nickhill123@gmail.com>
xmdhb
pushed a commit
to xmdhb/vllm-hust
that referenced
this pull request
Jul 24, 2026
…ulable queue head Requests parked in WAITING_FOR_REMOTE_KVS hold their allocated blocks and are only promoted back to WAITING when the waiting-queue traversal reaches them and consumes their finished_recving signal. When the queue head fails block allocation, the traversal stops with a bare break; if nothing is running at that point, no future event can free blocks and the scheduler freezes permanently (Running: 0, Waiting: N, forever). The exact FCFS scenario of vllm-project#45388 was already fixed by vllm-project#44560's admission gate, but the gate only protects an FCFS admission-time invariant: under priority scheduling a later, higher-priority, unschedulable request can still strand a parked async load that holds the very blocks it needs (deterministic regression test included), and post-admission events can invalidate the reserved-fit invariant the same way. Fix: keep the bare break only while something is running (completion will free blocks; queue-order admission is preserved). With nothing running, skip the unschedulable request via the existing step_skipped_waiting mechanism and keep scanning so parked requests can be promoted, scheduled, and eventually release the blocks the head needs. Fixes the permanent-deadlock family of vllm-project#45388, vllm-project#42371. Co-authored-by: Claude Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Wayne Chiu <waynehacking8@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a different fix for the issue targeted by #40968.
The changes are simpler and the resulting behavior should also be better - we avoid async-loading new kv if it would occupy blocks required for other in-flight chunked prefills or async-loading requests.
Note: we apply this throttle to async kv loads only, rather than chunked-prefill requests in general, since the latter are naturally throttled by batch capacity and are also already eligible for preemption.
cc @Dao007forever