Skip to content

[Core][KV Connector] Avoid hybrid KV load failure crash - #45497

Open
chfeng-cs wants to merge 1 commit into
vllm-project:mainfrom
chfeng-cs:fix/hybrid-kv-load-failure-recovery
Open

chfeng-cs wants to merge 1 commit into
vllm-project:mainfrom
chfeng-cs:fix/hybrid-kv-load-failure-recovery

Conversation

@chfeng-cs

@chfeng-cs chfeng-cs commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

Purpose

Fix #45474.
Fix a scheduler crash in KV load failure recovery for hybrid KV cache models. The existing path assumed get_block_ids() returned a single KV cache group, but hybrid models return one block-id list per group.

This PR keeps the existing precise recovery behavior for single-group requests and adds a conservative fallback for multi-group requests: if any invalid block belongs to the request, reset its computed KV prefix and reschedule it for recomputation.

This is a tradeoff for the failure path. Precise recovery for hybrid KV would need group-aware invalid-block metadata and a way to map failed blocks back to the earliest safe token prefix across all KV cache groups, which is a larger change. This PR instead prioritizes correctness and keeping the engine alive over minimizing recomputation. Since KV load failures are expected to be uncommon, the extra recomputation only applies on the error path.

Happy to work on precise recovery for hybrid KV if maintainers think it is necessary.

Test Plan

Run the focused invalid-block recovery tests and ruff on the modified files.

Test Result

pytest tests/v1/kv_connector/unit/test_invalid_blocks_correctness.py::test_hybrid_invalid_blocks_fallback_recomputes_from_beginning -q
pytest tests/v1/kv_connector/unit/test_invalid_blocks_correctness.py::test_sync_recompute_blocks_not_freed_for_running_requests tests/v1/kv_connector/unit/test_invalid_blocks_correctness.py::test_sync_fail_invalid_blocks_evicted tests/v1/kv_connector/unit/test_invalid_blocks_correctness.py::test_hybrid_invalid_blocks_fallback_recomputes_from_beginning -q
ruff check vllm/v1/core/sched/scheduler.py tests/v1/kv_connector/unit/test_invalid_blocks_correctness.py

All passed.


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Signed-off-by: Ethan Feng <ethan.fengch@gmail.com>
@chfeng-cs
chfeng-cs force-pushed the fix/hybrid-kv-load-failure-recovery branch from e988e29 to ee4b118 Compare June 13, 2026 04:09
seungrokj added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Jul 7, 2026
Apply vllm-project/vllm#45497 scheduler fix at runtime via gist to
resolve ValueError in _update_requests_with_invalid_blocks with hybrid
KV cache manager. Expand lmcache conc-list to [16, 32, 48, 64].

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
cquil11 added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Jul 9, 2026
* fix: pin LMCache to commit 1720917e for DSv4 MI355X agentic vLLM

Pin LMCache to a specific commit to work around vLLM scheduler
ValueError with hybrid KV cache manager. Bump vLLM image to nightly,
tune LMCache config (blocking timeout, transfer mode), and narrow
sweep to lmcache conc=72 for focused debugging.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: add perf-changelog entry for DSv4 MI355X agentic vLLM LMCache pin

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: switch LMCache to dev branch for PR #3853 fix

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: patch vLLM scheduler.py for hybrid KV cache and expand conc sweep

Apply vllm-project/vllm#45497 scheduler fix at runtime via gist to
resolve ValueError in _update_requests_with_invalid_blocks with hybrid
KV cache manager. Expand lmcache conc-list to [16, 32, 48, 64].

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: increase LMCache mq_timeout from 600s to 6000s

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: add DEP sweep and reduce dram-utilization for DSv4 MI355X agentic

Add DP-attention conc=[40,56,72] sweep, narrow lmcache to conc=[32,40],
and reduce dram-utilization from 0.80 to 0.60.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: expand DEP sweep to high conc [100,150,200,250,300] for MI355X

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: reset DSv4 MI355X agentic sweep to baseline conc range

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Cameron Quilici <cjquilici@gmail.com>
@mergify mergify Bot added the scheduler label Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Engine core dies on KV-connector load failure for hybrid-KV models — _update_requests_with_invalid_blocks assumes a single KV cache group

1 participant