Conversation
chfeng-cs
requested review from
ApostaC,
WoosukKwon,
alexm-redhat,
heheda12345,
njhill,
orozery,
robertgshaw2-redhat and
ywang96
as code owners
June 13, 2026 04:04
Signed-off-by: Ethan Feng <ethan.fengch@gmail.com>
chfeng-cs
force-pushed
the
fix/hybrid-kv-load-failure-recovery
branch
from
June 13, 2026 04:09
e988e29 to
ee4b118
Compare
seungrokj
added a commit
to SemiAnalysisAI/InferenceX
that referenced
this pull request
Jul 7, 2026
Apply vllm-project/vllm#45497 scheduler fix at runtime via gist to resolve ValueError in _update_requests_with_invalid_blocks with hybrid KV cache manager. Expand lmcache conc-list to [16, 32, 48, 64]. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
cquil11
added a commit
to SemiAnalysisAI/InferenceX
that referenced
this pull request
Jul 9, 2026
* fix: pin LMCache to commit 1720917e for DSv4 MI355X agentic vLLM Pin LMCache to a specific commit to work around vLLM scheduler ValueError with hybrid KV cache manager. Bump vLLM image to nightly, tune LMCache config (blocking timeout, transfer mode), and narrow sweep to lmcache conc=72 for focused debugging. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: add perf-changelog entry for DSv4 MI355X agentic vLLM LMCache pin Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: switch LMCache to dev branch for PR #3853 fix Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: patch vLLM scheduler.py for hybrid KV cache and expand conc sweep Apply vllm-project/vllm#45497 scheduler fix at runtime via gist to resolve ValueError in _update_requests_with_invalid_blocks with hybrid KV cache manager. Expand lmcache conc-list to [16, 32, 48, 64]. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: increase LMCache mq_timeout from 600s to 6000s Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: add DEP sweep and reduce dram-utilization for DSv4 MI355X agentic Add DP-attention conc=[40,56,72] sweep, narrow lmcache to conc=[32,40], and reduce dram-utilization from 0.80 to 0.60. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: expand DEP sweep to high conc [100,150,200,250,300] for MI355X Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: reset DSv4 MI355X agentic sweep to baseline conc range Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Cameron Quilici <cjquilici@gmail.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Fix #45474.
Fix a scheduler crash in KV load failure recovery for hybrid KV cache models. The existing path assumed
get_block_ids()returned a single KV cache group, but hybrid models return one block-id list per group.This PR keeps the existing precise recovery behavior for single-group requests and adds a conservative fallback for multi-group requests: if any invalid block belongs to the request, reset its computed KV prefix and reschedule it for recomputation.
This is a tradeoff for the failure path. Precise recovery for hybrid KV would need group-aware invalid-block metadata and a way to map failed blocks back to the earliest safe token prefix across all KV cache groups, which is a larger change. This PR instead prioritizes correctness and keeping the engine alive over minimizing recomputation. Since KV load failures are expected to be uncommon, the extra recomputation only applies on the error path.
Happy to work on precise recovery for hybrid KV if maintainers think it is necessary.
Test Plan
Run the focused invalid-block recovery tests and ruff on the modified files.
Test Result
All passed.
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.