[v0.23.0][BugFix] Report async KV load failures to scheduler by Pz1116 · Pull Request #13099 · vllm-project/vllm-ascend

Pz1116 · 2026-07-29T11:50:51Z

What this PR does / why we need it?

When AscendStore runs with load_async=true, KVCacheStoreRecvingThread was created without the worker's shared invalid-block set and lock. The receive thread therefore recorded backend load failures in a private set, while KVPoolWorker.get_block_ids_with_load_errors() returned an empty set. As a result, kv_load_failure_policy=recompute did not reach the scheduler recovery path.

This PR passes the worker-owned invalid-block set and lock into the async receive thread and adds a regression test covering the factory wiring and worker-visible failure state.

Does this PR introduce any user-facing change?

Yes. For non-layerwise AscendStore with load_async=true, failed KV loads are now reported to the scheduler so requests can be rescheduled when kv_load_failure_policy=recompute is configured.

How was this patch tested?

Added test_async_recv_thread_shares_invalid_block_state.
Ran 8 focused AscendStore async-load and failure-reporting unit tests successfully.
Ran all changed-file pre-commit hooks successfully.

vllm-project/vllm@ee0da84

vLLM version: v0.23.0
vLLM main: vllm-project/vllm@ee0da84

Share invalid block state between KVPoolWorker and the async receive thread so load failures reach the scheduler's recompute policy. Add a regression test for the thread factory wiring. Signed-off-by: Pz1116 <zpbzpb123123@gmail.com>

gemini-code-assist · 2026-07-29T11:50:59Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a bug in the AscendStore async loading mechanism where KV load failures were being isolated within the receive thread. By passing the worker's shared invalid-block set and lock to the thread, the system can now properly propagate load errors to the scheduler, enabling correct recovery behavior when the failure policy is set to recompute.

Highlights

Async KV Load Failure Reporting: Fixed an issue where asynchronous KV load failures were not correctly reported to the scheduler by ensuring the receive thread shares the worker's invalid-block state.
Regression Testing: Added a new unit test to verify that the KVCacheStoreRecvingThread correctly receives and updates the shared invalid-block set and lock.

New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request ensures that invalid_block_ids and invalid_block_ids_lock are passed to KVCacheStoreRecvingThread when starting KV transfer threads, and adds a unit test to verify this behavior. There are no review comments to address, so I have no additional feedback to provide.

Suggested PR Title:

[Ops][BugFix] Pass invalid block state to KVCacheStoreRecvingThread

Suggested PR Summary:

### What this PR does / why we need it?
This PR updates `_start_kv_transfer_threads` in `pool_worker.py` to pass `invalid_block_ids` and `invalid_block_ids_lock` to the `KVCacheStoreRecvingThread` initialization. This ensures that the receiving thread can share and update the invalid block state correctly.

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
A new unit test `test_async_recv_thread_shares_invalid_block_state` has been added in `test_pool_worker.py` to verify that the state is correctly shared and updated.

github-actions · 2026-07-29T11:52:36Z

👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:‌‌

A PR should do only one thing, smaller PRs enable faster reviews.
Every PR should include unit tests and end-to-end tests ‌to ensure it works and is not broken by other future PRs.
Write the commit message by fulfilling the PR description to help reviewer and future developers understand.

If CI fails, you can run linting and testing checks locally according Contributing and Testing.

Tip

💡 Consider Linking a Related Issue or RFC

Your PR title contains the [BugFix] tag, indicating a bug fix or new feature.

Linking a related issue or RFC in the PR description is strongly encouraged — it gives reviewers helpful context and speeds up the review. You can use any of these keywords:

Fixes #<issue_number>
Closes #<issue_number>
Resolves #<issue_number>
Refs #<rfc_or_issue_number> (for RFCs)

🙏 Thanks for helping us keep the project well-organized!

### What this PR does / why we need it? Forward-port #13099 to `releases/v0.24.0rc`. When AscendStore runs with `load_async=true`, `KVCacheStoreRecvingThread` was created without the worker's shared invalid-block set and lock. The receive thread therefore recorded backend load failures in a private set, while `KVPoolWorker.get_block_ids_with_load_errors()` returned an empty set. As a result, `kv_load_failure_policy=recompute` did not reach the scheduler recovery path. This PR passes the worker-owned invalid-block set and lock into the async receive thread and adds a regression test covering the factory wiring and worker-visible failure state. ### Does this PR introduce _any_ user-facing change? Yes. For non-layerwise AscendStore with `load_async=true`, failed KV loads are now reported to the scheduler so requests can be rescheduled when `kv_load_failure_policy=recompute` is configured. ### How was this patch tested? - All 82 tests in `tests.ut.distributed.ascend_store.test_pool_worker` passed. - All changed-file pre-commit hooks passed. - `git diff --check` passed. vllm-project/vllm@85c09e9 - vLLM version: v0.24.0 - vLLM main: vllm-project/vllm@85c09e9 Signed-off-by: Pz1116 <zpbzpb123123@gmail.com>

### What this PR does / why we need it? Forward-port #13099 to `main`. When AscendStore runs with `load_async=true`, `KVCacheStoreRecvingThread` was created without the worker's shared invalid-block set and lock. The receive thread therefore recorded backend load failures in a private set, while `KVPoolWorker.get_block_ids_with_load_errors()` returned an empty set. As a result, `kv_load_failure_policy=recompute` did not reach the scheduler recovery path. This PR passes the worker-owned invalid-block set and lock into the async receive thread and adds a regression test covering the factory wiring and worker-visible failure state. ### Does this PR introduce _any_ user-facing change? Yes. For non-layerwise AscendStore with `load_async=true`, failed KV loads are now reported to the scheduler so requests can be rescheduled when `kv_load_failure_policy=recompute` is configured. ### How was this patch tested? - All 82 tests in `tests.ut.distributed.ascend_store.test_pool_worker` passed. - Changed-file code quality hooks passed; the gitleaks hook could not run locally because `wget` is unavailable. - `git diff --check` passed. vllm-project/vllm@d02df74 - vLLM version: v0.25.1 - vLLM main: vllm-project/vllm@d02df74 Signed-off-by: Pz1116 <zpbzpb123123@gmail.com>

### What this PR does / why we need it? Forward-port #13099 to `releases/v0.25.1rc`. When AscendStore runs with `load_async=true`, `KVCacheStoreRecvingThread` was created without the worker's shared invalid-block set and lock. The receive thread therefore recorded backend load failures in a private set, while `KVPoolWorker.get_block_ids_with_load_errors()` returned an empty set. As a result, `kv_load_failure_policy=recompute` did not reach the scheduler recovery path. This PR passes the worker-owned invalid-block set and lock into the async receive thread and adds a regression test covering the factory wiring and worker-visible failure state. ### Does this PR introduce _any_ user-facing change? Yes. For non-layerwise AscendStore with `load_async=true`, failed KV loads are now reported to the scheduler so requests can be rescheduled when `kv_load_failure_policy=recompute` is configured. ### How was this patch tested? - All 82 tests in `tests.ut.distributed.ascend_store.test_pool_worker` passed. - Changed-file code quality hooks passed; the gitleaks hook could not run locally because the target branch's wrapper is not executable. - `git diff --check` passed. vllm-project/vllm@fe784ff - vLLM version: v0.25.1 - vLLM main: vllm-project/vllm@fe784ff Signed-off-by: Pz1116 <zpbzpb123123@gmail.com>

Forward-port vllm-project#13099 to `main`. When AscendStore runs with `load_async=true`, `KVCacheStoreRecvingThread` was created without the worker's shared invalid-block set and lock. The receive thread therefore recorded backend load failures in a private set, while `KVPoolWorker.get_block_ids_with_load_errors()` returned an empty set. As a result, `kv_load_failure_policy=recompute` did not reach the scheduler recovery path. This PR passes the worker-owned invalid-block set and lock into the async receive thread and adds a regression test covering the factory wiring and worker-visible failure state. Yes. For non-layerwise AscendStore with `load_async=true`, failed KV loads are now reported to the scheduler so requests can be rescheduled when `kv_load_failure_policy=recompute` is configured. - All 82 tests in `tests.ut.distributed.ascend_store.test_pool_worker` passed. - Changed-file code quality hooks passed; the gitleaks hook could not run locally because `wget` is unavailable. - `git diff --check` passed. vllm-project/vllm@d02df74 - vLLM version: v0.25.1 - vLLM main: vllm-project/vllm@d02df74 Signed-off-by: Pz1116 <zpbzpb123123@gmail.com>

Forward-port vllm-project#13099 to `main`. When AscendStore runs with `load_async=true`, `KVCacheStoreRecvingThread` was created without the worker's shared invalid-block set and lock. The receive thread therefore recorded backend load failures in a private set, while `KVPoolWorker.get_block_ids_with_load_errors()` returned an empty set. As a result, `kv_load_failure_policy=recompute` did not reach the scheduler recovery path. This PR passes the worker-owned invalid-block set and lock into the async receive thread and adds a regression test covering the factory wiring and worker-visible failure state. Yes. For non-layerwise AscendStore with `load_async=true`, failed KV loads are now reported to the scheduler so requests can be rescheduled when `kv_load_failure_policy=recompute` is configured. - All 82 tests in `tests.ut.distributed.ascend_store.test_pool_worker` passed. - Changed-file code quality hooks passed; the gitleaks hook could not run locally because `wget` is unavailable. - `git diff --check` passed. vllm-project/vllm@d02df74 - vLLM version: v0.25.1 - vLLM main: vllm-project/vllm@d02df74 Signed-off-by: Pz1116 <zpbzpb123123@gmail.com> Signed-off-by: lijiaqi139 <lijiaqi139@huawei.com>

Forward-port vllm-project#13099 to `main`. When AscendStore runs with `load_async=true`, `KVCacheStoreRecvingThread` was created without the worker's shared invalid-block set and lock. The receive thread therefore recorded backend load failures in a private set, while `KVPoolWorker.get_block_ids_with_load_errors()` returned an empty set. As a result, `kv_load_failure_policy=recompute` did not reach the scheduler recovery path. This PR passes the worker-owned invalid-block set and lock into the async receive thread and adds a regression test covering the factory wiring and worker-visible failure state. Yes. For non-layerwise AscendStore with `load_async=true`, failed KV loads are now reported to the scheduler so requests can be rescheduled when `kv_load_failure_policy=recompute` is configured. - All 82 tests in `tests.ut.distributed.ascend_store.test_pool_worker` passed. - Changed-file code quality hooks passed; the gitleaks hook could not run locally because `wget` is unavailable. - `git diff --check` passed. vllm-project/vllm@d02df74 - vLLM version: v0.25.1 - vLLM main: vllm-project/vllm@d02df74 Signed-off-by: Pz1116 <zpbzpb123123@gmail.com> Signed-off-by: lijiaqi139 <lijiaqi139@huawei.com> Signed-off-by: jiaqi-lee <15316070896@163.com>

### What this PR does / why we need it? Forward-port vllm-project#13099 to `main`. When AscendStore runs with `load_async=true`, `KVCacheStoreRecvingThread` was created without the worker's shared invalid-block set and lock. The receive thread therefore recorded backend load failures in a private set, while `KVPoolWorker.get_block_ids_with_load_errors()` returned an empty set. As a result, `kv_load_failure_policy=recompute` did not reach the scheduler recovery path. This PR passes the worker-owned invalid-block set and lock into the async receive thread and adds a regression test covering the factory wiring and worker-visible failure state. ### Does this PR introduce _any_ user-facing change? Yes. For non-layerwise AscendStore with `load_async=true`, failed KV loads are now reported to the scheduler so requests can be rescheduled when `kv_load_failure_policy=recompute` is configured. ### How was this patch tested? - All 82 tests in `tests.ut.distributed.ascend_store.test_pool_worker` passed. - Changed-file code quality hooks passed; the gitleaks hook could not run locally because `wget` is unavailable. - `git diff --check` passed. vllm-project/vllm@d02df74 - vLLM version: v0.25.1 - vLLM main: vllm-project/vllm@d02df74 Signed-off-by: Pz1116 <zpbzpb123123@gmail.com>

Forward-port vllm-project#13099 to `main`. When AscendStore runs with `load_async=true`, `KVCacheStoreRecvingThread` was created without the worker's shared invalid-block set and lock. The receive thread therefore recorded backend load failures in a private set, while `KVPoolWorker.get_block_ids_with_load_errors()` returned an empty set. As a result, `kv_load_failure_policy=recompute` did not reach the scheduler recovery path. This PR passes the worker-owned invalid-block set and lock into the async receive thread and adds a regression test covering the factory wiring and worker-visible failure state. Yes. For non-layerwise AscendStore with `load_async=true`, failed KV loads are now reported to the scheduler so requests can be rescheduled when `kv_load_failure_policy=recompute` is configured. - All 82 tests in `tests.ut.distributed.ascend_store.test_pool_worker` passed. - Changed-file code quality hooks passed; the gitleaks hook could not run locally because `wget` is unavailable. - `git diff --check` passed. vllm-project/vllm@d02df74 - vLLM version: v0.25.1 - vLLM main: vllm-project/vllm@d02df74 Signed-off-by: Pz1116 <zpbzpb123123@gmail.com> Signed-off-by: lijiaqi139 <lijiaqi139@huawei.com> Signed-off-by: jiaqi-lee <15316070896@163.com>

### What this PR does / why we need it? Forward-port vllm-project#13099 to `main`. When AscendStore runs with `load_async=true`, `KVCacheStoreRecvingThread` was created without the worker's shared invalid-block set and lock. The receive thread therefore recorded backend load failures in a private set, while `KVPoolWorker.get_block_ids_with_load_errors()` returned an empty set. As a result, `kv_load_failure_policy=recompute` did not reach the scheduler recovery path. This PR passes the worker-owned invalid-block set and lock into the async receive thread and adds a regression test covering the factory wiring and worker-visible failure state. ### Does this PR introduce _any_ user-facing change? Yes. For non-layerwise AscendStore with `load_async=true`, failed KV loads are now reported to the scheduler so requests can be rescheduled when `kv_load_failure_policy=recompute` is configured. ### How was this patch tested? - All 82 tests in `tests.ut.distributed.ascend_store.test_pool_worker` passed. - Changed-file code quality hooks passed; the gitleaks hook could not run locally because `wget` is unavailable. - `git diff --check` passed. vllm-project/vllm@d02df74 - vLLM version: v0.25.1 - vLLM main: vllm-project/vllm@d02df74 Signed-off-by: Pz1116 <zpbzpb123123@gmail.com>

Forward-port vllm-project#13099 to `main`. When AscendStore runs with `load_async=true`, `KVCacheStoreRecvingThread` was created without the worker's shared invalid-block set and lock. The receive thread therefore recorded backend load failures in a private set, while `KVPoolWorker.get_block_ids_with_load_errors()` returned an empty set. As a result, `kv_load_failure_policy=recompute` did not reach the scheduler recovery path. This PR passes the worker-owned invalid-block set and lock into the async receive thread and adds a regression test covering the factory wiring and worker-visible failure state. Yes. For non-layerwise AscendStore with `load_async=true`, failed KV loads are now reported to the scheduler so requests can be rescheduled when `kv_load_failure_policy=recompute` is configured. - All 82 tests in `tests.ut.distributed.ascend_store.test_pool_worker` passed. - Changed-file code quality hooks passed; the gitleaks hook could not run locally because `wget` is unavailable. - `git diff --check` passed. vllm-project/vllm@d02df74 - vLLM version: v0.25.1 - vLLM main: vllm-project/vllm@d02df74 Signed-off-by: Pz1116 <zpbzpb123123@gmail.com> Signed-off-by: lijiaqi139 <lijiaqi139@huawei.com> Signed-off-by: jiaqi-lee <15316070896@163.com> (cherry picked from commit f08ee66)

Forward-port vllm-project#13099 to `main`. When AscendStore runs with `load_async=true`, `KVCacheStoreRecvingThread` was created without the worker's shared invalid-block set and lock. The receive thread therefore recorded backend load failures in a private set, while `KVPoolWorker.get_block_ids_with_load_errors()` returned an empty set. As a result, `kv_load_failure_policy=recompute` did not reach the scheduler recovery path. This PR passes the worker-owned invalid-block set and lock into the async receive thread and adds a regression test covering the factory wiring and worker-visible failure state. Yes. For non-layerwise AscendStore with `load_async=true`, failed KV loads are now reported to the scheduler so requests can be rescheduled when `kv_load_failure_policy=recompute` is configured. - All 82 tests in `tests.ut.distributed.ascend_store.test_pool_worker` passed. - Changed-file code quality hooks passed; the gitleaks hook could not run locally because `wget` is unavailable. - `git diff --check` passed. vllm-project/vllm@d02df74 - vLLM version: v0.25.1 - vLLM main: vllm-project/vllm@d02df74 Signed-off-by: Pz1116 <zpbzpb123123@gmail.com> Signed-off-by: lijiaqi139 <lijiaqi139@huawei.com> Signed-off-by: jiaqi-lee <15316070896@163.com> (cherry picked from commit f08ee66) Signed-off-by: jiaqi-lee <15316070896@163.com>

…#14687) ### What this PR does / why we need it? This is the v0.26.0 release-blocker backport rollup. It contains only the 19 audited high-severity correctness and stability fixes that were still missing from releases/v0.26.0rc at 1f95052. The branch has 23 physical commits because #14619 is preserved as its complete five-commit atomic series. Cross-branch equivalents are deduplicated, and every logical fix remains independently reviewable and revertible. #### Included fixes | # | Source / target PR | Severity | Problem fixed | |---:|---|---|---| | 1 | #13538 | P0 correctness | Qwen3-VL MoE + FlashComm1 + deepstack used the wrong residual tensor and could silently corrupt output. | | 2 | #13600 | P0 deadlock | MRV1/MRV2 main and draft update streams could mutually wait during repeated full-graph execution. | | 3 | #13498 | P0 data corruption | Float32 Mamba state could overwrite the shared bf16 hidden-state cache buffer. | | 4 | #13902 | P0 correctness | RL weight reload left ACL graphs referencing stale W8A8-MXFP8 weight addresses. | | 5 | #12359 / #12371 | P0 correctness | Mooncake reformatted KV before all TP/CP pulls for a request completed, causing TP inequality or reordered KV. | | 6 | #13111 / #13113 / #13110 | P1 KV correctness | Multi-KV-group save/load incorrectly reused group-0 block size for every group. | | 7 | #13116 / #13117 / #13099 | P1 state consistency | Async KV load failures were not shared with the scheduler, preventing recompute recovery. | | 8 | #13308 / #13310 / #13307 | P1 crash | Memcache batch query/allocation before lazy initialization could assert in scheduler or worker. | | 9 | #13012 | P1 hang/corruption | Level-2 sleep/wake could lose the MoE loader and leave EPLB tensors pointing at released storage. | | 10 | #13414 | P1 crash/correctness | Dynamic EPLB initialized W8A8 scales for only the first expert weight. | | 11 | #14001 | P1 crash | MiniMax-M3 index_q was reshaped using total size instead of the per-head dimension. | | 12 | #14394 | P1 crash/hang | MRV2 FULL_DECODE_ONLY dropped graph padding when runtime mode was FULL. | | 13 | #13136 | P1 crash | P/D + DP zero-token ranks compared None with MC2 capacity and raised TypeError. | | 14 | #13183 | P1 unavailable | ec_both was treated as producer-only and skipped KV specification/data needed by its consumer role. | | 15 | #13123 | P1 OOB/device error | MRV2 dummy-token remainder was concentrated on one request and could exceed max_model_len. | | 16 | #13159 | P1 crash | MRV2 num_nans used the wrong Triton libdevice and the penalty kernel could exceed the CANN grid limit. | | 17 | #12940 | P1 crash | DFlash profiling used total query count instead of actual input tokens for RoPE/graph capture. | | 18 | #13394 via #13405 | P0 correctness | RL sampling tensor lifetime errors could produce Inf/OOV tokens and contaminate later output. | | 19 | #14142 via #14619 | P1 long-run/state correctness | P/D rejection left stale KV/accounting and unsafe retry/replay behavior could leak, duplicate, or return wrong responses. | #### Backport policy - Selected the audited v0.26 release-adapted commits where available; the closed rollup #14337 was not revived wholesale. - Kept only one canonical copy of fixes duplicated across 0.23, 0.25, and main. - Manually adapted #13136, the core #13123 input-batch hunk, and #13159 to preserve current v0.26/MegaMoe/model-runner behavior. - Used the current v0.26 target change from #13405 and the complete five-commit #14619 series. - Intentionally excluded performance-only, UX-only, conditional-support, low-confidence, and owner-unsettled fixes from this release window. ### Does this PR introduce _any_ user-facing change? Yes, behavior is corrected for the affected configurations: crashes, deadlocks, hangs, incorrect output, stale KV state, and data corruption are prevented. There is no new public API, CLI option, or configuration requirement. ### How was this patch tested? Local validation completed: - Audited manifest: 19/19 logical fixes, 23/23 expected source commits; missing 0, duplicate 0, unexpected 0. - All 23 commits retain source provenance and Signed-off-by trailers. - Ruff lint and format checks passed for all 38 changed Python files. - Python syntax compilation passed for all 38 changed Python files. - git diff --check, codespell, forbidden-import, package-init, context-manager, and filename checks passed. - The final worktree is clean at 86b2ca8. The backports retain or add focused tests for AscendStore, Mooncake rejection cleanup, fused MoE/EPLB, W8A8-MXFP8 reload, worker sleep/wake, MRV2 graph padding, penalty-grid limits, hidden-state extraction, and two-card speculative DP. NPU UT/E2E was not run locally because the available Windows environment has no vLLM, PyTorch/torch_npu, pytest, or Ascend device. CI and targeted NPU regression are therefore required before merge, especially: - MRV1/MRV2 full-graph repeated-iteration deadlock/teardown. - Mooncake TP2/TP4 out-of-order pull KV equality and P/D rejection cleanup. - Qwen3-VL FlashComm1 + deepstack fixed-seed correctness. - Level-2 sleep/wake, dynamic EPLB, and multi-round RL weight reload. - P/D + DP zero-token ranks, ec_both, DFlash profile/graph, and RL Inf/OOV sampling. - Proxy retry and streaming replay behavior from #14619. #### Review checklist - [x] Only the 19 approved release-critical logical fixes are included. - [x] One logical fix per commit; #14619 remains an atomic five-commit series. - [x] No performance-only backports are included. - [x] Source provenance and sign-offs are retained. - [ ] Repository CI passes. - [ ] Targeted Ascend NPU correctness and long-run tests pass. - vLLM version: v0.26.0 - vLLM main: vllm-project/vllm@d02df74 --------- Signed-off-by: kyle-zhangchi <chiiiiiizhang@gmail.com> Signed-off-by: lijiaqi139 <lijiaqi139@huawei.com> Signed-off-by: jiaqi-lee <15316070896@163.com> Signed-off-by: tyy0829 <1455207791@qq.com> Signed-off-by: yejj710 <abyss1999@163.com> Signed-off-by: jiajinzhu2 <jiajinzhu@huawei.com> Signed-off-by: chenyue1122 <oyoy7102@163.com> Signed-off-by: XuRongSheng <1843167357@qq.com> Signed-off-by: muziyuhui666 <lijianfu9@huawei.com> Signed-off-by: Pz1116 <zpbzpb123123@gmail.com> Signed-off-by: zouyida2052 <zouyida2002@gmail.com> Signed-off-by: likailong <likailong5@huawei.com> Signed-off-by: hanxi-java <634498162@qq.com> Signed-off-by: Liam <ml646@duke.edu> Signed-off-by: AuroraEmiya <Sakura.iostream@gmail.com> Signed-off-by: HF-001 <1670186653@qq.com> Signed-off-by: wangxiaoteng <wangxiaoteng@huawei.com> Signed-off-by: Hcm03 <chengminhua1@huawei.com> Signed-off-by: zhuyixiang <zhuyixiang2014@163.com> Signed-off-by: moonseeker <2290166829@qq.com> Co-authored-by: kyle-zhangchi <chiiiiiizhang@gmail.com> Co-authored-by: tyy0829 <87685049+tyy0829@users.noreply.github.com> Co-authored-by: yejj <abyss1999@163.com> Co-authored-by: jiajinzhu2 <jiajinzhu@huawei.com> Co-authored-by: CHENYUE <56943221+PHOEBEMOON0802@users.noreply.github.com> Co-authored-by: Xu Rongsheng <73730571+MmMmaru@users.noreply.github.com> Co-authored-by: yjyang62 <yangjinyang5@huawei.com> Co-authored-by: muziyuhui666 <lijianfu9@huawei.com> Co-authored-by: CXY-Katrina <katrina.cxy@gmail.com> Co-authored-by: cywang250805 <wangchaoyu7@huawei.com> Co-authored-by: Bill845514379 <huangjianbao2@huawei.com> Co-authored-by: yejj710 <yejj710@gmail.com> Co-authored-by: AuroraEmiya <Sakura.iostream@gmail.com> Co-authored-by: HaoxinZong <116423146+HaoxinZong@users.noreply.github.com> Co-authored-by: pz1116 <47019764+Pz1116@users.noreply.github.com> Co-authored-by: zouyida2052 <zouyida2002@gmail.com> Co-authored-by: iKeybot <92210799+iKeybot-code@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: 韩熙 <63780107+hanxi-java@users.noreply.github.com> Co-authored-by: zouzy <38661932+zouzy5137@users.noreply.github.com> Co-authored-by: AuroraEmiya <92282919+AuroraEmiya@users.noreply.github.com> Co-authored-by: Liam <ml646@duke.edu> Co-authored-by: kx <1670186653@qq.com> Co-authored-by: wangxiaoteng888 <56506195+wangxiaoteng888@users.noreply.github.com> Co-authored-by: Hcm03 <chengminhua1@huawei.com> Co-authored-by: zhuyixiang <zhuyixiang2014@163.com> Co-authored-by: moonseeker <2290166829@qq.com>

…vllm-project#14687) ### What this PR does / why we need it? This is the v0.26.0 release-blocker backport rollup. It contains only the 19 audited high-severity correctness and stability fixes that were still missing from releases/v0.26.0rc at 1f95052. The branch has 23 physical commits because vllm-project#14619 is preserved as its complete five-commit atomic series. Cross-branch equivalents are deduplicated, and every logical fix remains independently reviewable and revertible. #### Included fixes | # | Source / target PR | Severity | Problem fixed | |---:|---|---|---| | 1 | vllm-project#13538 | P0 correctness | Qwen3-VL MoE + FlashComm1 + deepstack used the wrong residual tensor and could silently corrupt output. | | 2 | vllm-project#13600 | P0 deadlock | MRV1/MRV2 main and draft update streams could mutually wait during repeated full-graph execution. | | 3 | vllm-project#13498 | P0 data corruption | Float32 Mamba state could overwrite the shared bf16 hidden-state cache buffer. | | 4 | vllm-project#13902 | P0 correctness | RL weight reload left ACL graphs referencing stale W8A8-MXFP8 weight addresses. | | 5 | vllm-project#12359 / vllm-project#12371 | P0 correctness | Mooncake reformatted KV before all TP/CP pulls for a request completed, causing TP inequality or reordered KV. | | 6 | vllm-project#13111 / vllm-project#13113 / vllm-project#13110 | P1 KV correctness | Multi-KV-group save/load incorrectly reused group-0 block size for every group. | | 7 | vllm-project#13116 / vllm-project#13117 / vllm-project#13099 | P1 state consistency | Async KV load failures were not shared with the scheduler, preventing recompute recovery. | | 8 | vllm-project#13308 / vllm-project#13310 / vllm-project#13307 | P1 crash | Memcache batch query/allocation before lazy initialization could assert in scheduler or worker. | | 9 | vllm-project#13012 | P1 hang/corruption | Level-2 sleep/wake could lose the MoE loader and leave EPLB tensors pointing at released storage. | | 10 | vllm-project#13414 | P1 crash/correctness | Dynamic EPLB initialized W8A8 scales for only the first expert weight. | | 11 | vllm-project#14001 | P1 crash | MiniMax-M3 index_q was reshaped using total size instead of the per-head dimension. | | 12 | vllm-project#14394 | P1 crash/hang | MRV2 FULL_DECODE_ONLY dropped graph padding when runtime mode was FULL. | | 13 | vllm-project#13136 | P1 crash | P/D + DP zero-token ranks compared None with MC2 capacity and raised TypeError. | | 14 | vllm-project#13183 | P1 unavailable | ec_both was treated as producer-only and skipped KV specification/data needed by its consumer role. | | 15 | vllm-project#13123 | P1 OOB/device error | MRV2 dummy-token remainder was concentrated on one request and could exceed max_model_len. | | 16 | vllm-project#13159 | P1 crash | MRV2 num_nans used the wrong Triton libdevice and the penalty kernel could exceed the CANN grid limit. | | 17 | vllm-project#12940 | P1 crash | DFlash profiling used total query count instead of actual input tokens for RoPE/graph capture. | | 18 | vllm-project#13394 via vllm-project#13405 | P0 correctness | RL sampling tensor lifetime errors could produce Inf/OOV tokens and contaminate later output. | | 19 | vllm-project#14142 via vllm-project#14619 | P1 long-run/state correctness | P/D rejection left stale KV/accounting and unsafe retry/replay behavior could leak, duplicate, or return wrong responses. | #### Backport policy - Selected the audited v0.26 release-adapted commits where available; the closed rollup vllm-project#14337 was not revived wholesale. - Kept only one canonical copy of fixes duplicated across 0.23, 0.25, and main. - Manually adapted vllm-project#13136, the core vllm-project#13123 input-batch hunk, and vllm-project#13159 to preserve current v0.26/MegaMoe/model-runner behavior. - Used the current v0.26 target change from vllm-project#13405 and the complete five-commit vllm-project#14619 series. - Intentionally excluded performance-only, UX-only, conditional-support, low-confidence, and owner-unsettled fixes from this release window. ### Does this PR introduce _any_ user-facing change? Yes, behavior is corrected for the affected configurations: crashes, deadlocks, hangs, incorrect output, stale KV state, and data corruption are prevented. There is no new public API, CLI option, or configuration requirement. ### How was this patch tested? Local validation completed: - Audited manifest: 19/19 logical fixes, 23/23 expected source commits; missing 0, duplicate 0, unexpected 0. - All 23 commits retain source provenance and Signed-off-by trailers. - Ruff lint and format checks passed for all 38 changed Python files. - Python syntax compilation passed for all 38 changed Python files. - git diff --check, codespell, forbidden-import, package-init, context-manager, and filename checks passed. - The final worktree is clean at 86b2ca8. The backports retain or add focused tests for AscendStore, Mooncake rejection cleanup, fused MoE/EPLB, W8A8-MXFP8 reload, worker sleep/wake, MRV2 graph padding, penalty-grid limits, hidden-state extraction, and two-card speculative DP. NPU UT/E2E was not run locally because the available Windows environment has no vLLM, PyTorch/torch_npu, pytest, or Ascend device. CI and targeted NPU regression are therefore required before merge, especially: - MRV1/MRV2 full-graph repeated-iteration deadlock/teardown. - Mooncake TP2/TP4 out-of-order pull KV equality and P/D rejection cleanup. - Qwen3-VL FlashComm1 + deepstack fixed-seed correctness. - Level-2 sleep/wake, dynamic EPLB, and multi-round RL weight reload. - P/D + DP zero-token ranks, ec_both, DFlash profile/graph, and RL Inf/OOV sampling. - Proxy retry and streaming replay behavior from vllm-project#14619. #### Review checklist - [x] Only the 19 approved release-critical logical fixes are included. - [x] One logical fix per commit; vllm-project#14619 remains an atomic five-commit series. - [x] No performance-only backports are included. - [x] Source provenance and sign-offs are retained. - [ ] Repository CI passes. - [ ] Targeted Ascend NPU correctness and long-run tests pass. - vLLM version: v0.26.0 - vLLM main: vllm-project/vllm@d02df74 --------- Signed-off-by: kyle-zhangchi <chiiiiiizhang@gmail.com> Signed-off-by: lijiaqi139 <lijiaqi139@huawei.com> Signed-off-by: jiaqi-lee <15316070896@163.com> Signed-off-by: tyy0829 <1455207791@qq.com> Signed-off-by: yejj710 <abyss1999@163.com> Signed-off-by: jiajinzhu2 <jiajinzhu@huawei.com> Signed-off-by: chenyue1122 <oyoy7102@163.com> Signed-off-by: XuRongSheng <1843167357@qq.com> Signed-off-by: muziyuhui666 <lijianfu9@huawei.com> Signed-off-by: Pz1116 <zpbzpb123123@gmail.com> Signed-off-by: zouyida2052 <zouyida2002@gmail.com> Signed-off-by: likailong <likailong5@huawei.com> Signed-off-by: hanxi-java <634498162@qq.com> Signed-off-by: Liam <ml646@duke.edu> Signed-off-by: AuroraEmiya <Sakura.iostream@gmail.com> Signed-off-by: HF-001 <1670186653@qq.com> Signed-off-by: wangxiaoteng <wangxiaoteng@huawei.com> Signed-off-by: Hcm03 <chengminhua1@huawei.com> Signed-off-by: zhuyixiang <zhuyixiang2014@163.com> Signed-off-by: moonseeker <2290166829@qq.com> Co-authored-by: kyle-zhangchi <chiiiiiizhang@gmail.com> Co-authored-by: tyy0829 <87685049+tyy0829@users.noreply.github.com> Co-authored-by: yejj <abyss1999@163.com> Co-authored-by: jiajinzhu2 <jiajinzhu@huawei.com> Co-authored-by: CHENYUE <56943221+PHOEBEMOON0802@users.noreply.github.com> Co-authored-by: Xu Rongsheng <73730571+MmMmaru@users.noreply.github.com> Co-authored-by: yjyang62 <yangjinyang5@huawei.com> Co-authored-by: muziyuhui666 <lijianfu9@huawei.com> Co-authored-by: CXY-Katrina <katrina.cxy@gmail.com> Co-authored-by: cywang250805 <wangchaoyu7@huawei.com> Co-authored-by: Bill845514379 <huangjianbao2@huawei.com> Co-authored-by: yejj710 <yejj710@gmail.com> Co-authored-by: AuroraEmiya <Sakura.iostream@gmail.com> Co-authored-by: HaoxinZong <116423146+HaoxinZong@users.noreply.github.com> Co-authored-by: pz1116 <47019764+Pz1116@users.noreply.github.com> Co-authored-by: zouyida2052 <zouyida2002@gmail.com> Co-authored-by: iKeybot <92210799+iKeybot-code@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: 韩熙 <63780107+hanxi-java@users.noreply.github.com> Co-authored-by: zouzy <38661932+zouzy5137@users.noreply.github.com> Co-authored-by: AuroraEmiya <92282919+AuroraEmiya@users.noreply.github.com> Co-authored-by: Liam <ml646@duke.edu> Co-authored-by: kx <1670186653@qq.com> Co-authored-by: wangxiaoteng888 <56506195+wangxiaoteng888@users.noreply.github.com> Co-authored-by: Hcm03 <chengminhua1@huawei.com> Co-authored-by: zhuyixiang <zhuyixiang2014@163.com> Co-authored-by: moonseeker <2290166829@qq.com>

fix(kv_pool): report async load failures to scheduler

74dd825

Share invalid block state between KVPoolWorker and the async receive thread so load failures reach the scheduler's recompute policy. Add a regression test for the thread factory wiring. Signed-off-by: Pz1116 <zpbzpb123123@gmail.com>

Pz1116 requested review from LCAIZJ, MengqingCao and wangxiyuan as code owners July 29, 2026 11:50

gemini-code-assist Bot reviewed Jul 29, 2026

View reviewed changes

github-actions Bot added the module:tests label Jul 29, 2026

Pz1116 added the ready label Jul 29, 2026

This was referenced Jul 29, 2026

[v0.24.0rc][BugFix] Report async KV load failures to scheduler #13115

Merged

[BugFix] Report async KV load failures to scheduler #13116

Merged

[v0.25.1rc][BugFix] Report async KV load failures to scheduler #13117

Merged

zzzzwwjj approved these changes Jul 29, 2026

View reviewed changes

zzzzwwjj merged commit 3c14186 into vllm-project:releases/v0.23.0 Jul 29, 2026
21 checks passed

jiaqi-lee mentioned this pull request Aug 20, 2026

[v0.26.0rc][BugFix] Backport critical correctness and stability fixes #14687

Merged

6 tasks

AuroraEmiya mentioned this pull request Aug 31, 2026

[Feature] Add Triton support for GLM5 lightning indexer beyond 2048 pool length AuroraEmiya/vllm-ascend#7

Closed

Navigation Menu

[v0.23.0][BugFix] Report async KV load failures to scheduler - #13099

[v0.23.0][BugFix] Report async KV load failures to scheduler#13099
zzzzwwjj merged 1 commit into
vllm-project:releases/v0.23.0from
Pz1116:bugfix/v0230-async-load-failure-reporting

Pz1116 commented Jul 29, 2026 •

edited by github-actions Bot

Loading

Uh oh!

gemini-code-assist Bot commented Jul 29, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

github-actions Bot commented Jul 29, 2026

💡 Consider Linking a Related Issue or RFC

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Pz1116 commented Jul 29, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

gemini-code-assist Bot commented Jul 29, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

github-actions Bot commented Jul 29, 2026

💡 Consider Linking a Related Issue or RFC

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Pz1116 commented Jul 29, 2026 •

edited by github-actions Bot

Loading