[BugFix][Sampler] Fix inf in Qwen30b RL sampling by recording stream for q - #13394
Merged
Merged
Conversation
Record stream for tensor q allocated on global_stream to prevent the caching allocator from reusing its memory while probs.div_(q) runs on the current stream. Signed-off-by: Hcm03 <chengminhua1@huawei.com>
Contributor
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses a critical race condition in the sampling logic on Ascend NPUs. By explicitly recording the stream for the tensor q, the fix ensures that the memory buffer is not recycled by the caching allocator while a cross-stream division operation is still pending, thereby preventing the generation of invalid tokens and -inf logprobs. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
Suggested PR Title:\n\nmarkdown\n[Ops][BugFix] Record stream on tensor q in sampler\n\n\nSuggested PR Summary:\n\nmarkdown\n### What this PR does / why we need it?\nThis pull request adds `q.record_stream` to prevent the tensor `q` from being deallocated prematurely during asynchronous NPU execution.\n\nFeedback: Calling `torch.npu.current_stream()` directly is expensive as it constructs a new stream object on each call. It is recommended to use the cached `current_stream()` helper from `vllm_ascend.utils` instead.\n\n### Does this PR introduce _any_ user-facing change?\nNo.\n\n### How was this patch tested?\nNo test details provided.\n
| for i, generator in generators.items(): | ||
| q[i].exponential_(generator=generator) | ||
| torch.npu.current_stream().wait_stream(global_stream()) | ||
| q.record_stream(torch.npu.current_stream()) |
Contributor
There was a problem hiding this comment.
In vllm_ascend, calling torch.npu.current_stream() is expensive because it constructs a new stream object on every call (as documented in vllm_ascend/utils.py). Since random_sample is executed in the hot path of every generation step, we should use the cached current_stream() helper from vllm_ascend.utils to avoid this overhead. Please also update line 42 to use current_stream() and import it from vllm_ascend.utils at the top of the file.
Suggested change
| q.record_stream(torch.npu.current_stream()) | |
| q.record_stream(current_stream()) |
References
- Avoid calling torch.npu.current_stream() directly as it is expensive and constructs a new stream object on each call. Use the cached current_stream() helper instead.
Contributor
|
👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:
If CI fails, you can run linting and testing checks locally according Contributing and Testing. Tip 💡 Consider Linking a Related Issue or RFCYour PR title contains the [BugFix] tag, indicating a bug fix or new feature. Linking a related issue or RFC in the PR description is strongly encouraged — it gives reviewers helpful context and speeds up the review. You can use any of these keywords:
🙏 Thanks for helping us keep the project well-organized! |
weijinqian0
approved these changes
Aug 6, 2026
weijinqian0
enabled auto-merge (squash)
August 6, 2026 08:44
HMCCMH
pushed a commit
to hotTea123/vllm-ascend
that referenced
this pull request
Aug 12, 2026
…for q (vllm-project#13394) ### What this PR does / why we need it? This PR fixes an `inf` sampling issue observed in Qwen3 30B RL workloads on Ascend. In `random_sample()`, tensor `q` is allocated on `global_stream()` but consumed on `torch.npu.current_stream()` via `probs.div_(q)`. The code only establishes an execution dependency (`wait_stream`), but does not declare the storage lifetime of `q` on the consumer stream. Without `q.record_stream(torch.npu.current_stream())`, the caching allocator may reclaim and reuse `q`'s buffer on the sampling stream before the queued division on the current stream has read it. On Ascend, `exponential_()` is a multi-kernel in-place sequence (`uniform_ -> neg_ -> add_(1) -> masked_fill_ -> log_ -> mul_(-1)`), so the next decode step's negative intermediate state can overwrite the old `q` address while the previous consumer is still reading it. For Qwen3 30B RL (vocab size 151936 vs. tokenizer length 151669), the masked tail of 267 ids has exactly zero probability. With a corrupted `q`, `0 / negative_q = -0` wins `argmax` and selects the first masked OOV id (`151669`), whose raw logprob is `-inf`. The OOV token then enters the autoregressive context and corrupts all subsequent outputs. Adding `q.record_stream(torch.npu.current_stream())` declares that `q` must remain alive on the consumer stream, preventing the allocator from recycling the buffer too early. Reproduction with the real `exponential_()` intermediate-state race: - Before fix: 64/64 rows selected the masked OOV id `151669` with raw logprob `-inf` - After fix: 0/64 rows invalid, sampled ids stay legal Note: the same fix is being submitted against all supported release branches (`releases/v0.13.0` -> `main`). ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? - Reproduced the allocator-reuse race with real `exponential_()` intermediate states (`uniform_ -> neg_`): 64/64 sampled rows selected the masked OOV id `151669` with raw logprob `-inf` before the fix; after the fix 0/64 rows are invalid and sampled ids stay legal. - Verified natural address reuse of `q` across consecutive decode steps is eliminated with `record_stream`. - Existing sampler unit tests pass. - NPU hardware verification pending on the original RL environment (`bsz=256, decode>=10k, TP=4, n=8`). - vLLM version: v0.26.0 - vLLM main: vllm-project/vllm@d02df74 Signed-off-by: Hcm03 <chengminhua1@huawei.com>
MmMmaru
pushed a commit
to jiaqi-lee/vllm-ascend
that referenced
this pull request
Aug 19, 2026
…for q (vllm-project#13394) ### What this PR does / why we need it? This PR fixes an `inf` sampling issue observed in Qwen3 30B RL workloads on Ascend. In `random_sample()`, tensor `q` is allocated on `global_stream()` but consumed on `torch.npu.current_stream()` via `probs.div_(q)`. The code only establishes an execution dependency (`wait_stream`), but does not declare the storage lifetime of `q` on the consumer stream. Without `q.record_stream(torch.npu.current_stream())`, the caching allocator may reclaim and reuse `q`'s buffer on the sampling stream before the queued division on the current stream has read it. On Ascend, `exponential_()` is a multi-kernel in-place sequence (`uniform_ -> neg_ -> add_(1) -> masked_fill_ -> log_ -> mul_(-1)`), so the next decode step's negative intermediate state can overwrite the old `q` address while the previous consumer is still reading it. For Qwen3 30B RL (vocab size 151936 vs. tokenizer length 151669), the masked tail of 267 ids has exactly zero probability. With a corrupted `q`, `0 / negative_q = -0` wins `argmax` and selects the first masked OOV id (`151669`), whose raw logprob is `-inf`. The OOV token then enters the autoregressive context and corrupts all subsequent outputs. Adding `q.record_stream(torch.npu.current_stream())` declares that `q` must remain alive on the consumer stream, preventing the allocator from recycling the buffer too early. Reproduction with the real `exponential_()` intermediate-state race: - Before fix: 64/64 rows selected the masked OOV id `151669` with raw logprob `-inf` - After fix: 0/64 rows invalid, sampled ids stay legal Note: the same fix is being submitted against all supported release branches (`releases/v0.13.0` -> `main`). ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? - Reproduced the allocator-reuse race with real `exponential_()` intermediate states (`uniform_ -> neg_`): 64/64 sampled rows selected the masked OOV id `151669` with raw logprob `-inf` before the fix; after the fix 0/64 rows are invalid and sampled ids stay legal. - Verified natural address reuse of `q` across consecutive decode steps is eliminated with `record_stream`. - Existing sampler unit tests pass. - NPU hardware verification pending on the original RL environment (`bsz=256, decode>=10k, TP=4, n=8`). - vLLM version: v0.26.0 - vLLM main: vllm-project/vllm@d02df74 Signed-off-by: Hcm03 <chengminhua1@huawei.com>
shiqiangA
pushed a commit
to shiqiangA/vllm-ascend
that referenced
this pull request
Aug 20, 2026
…for q (vllm-project#13394) ### What this PR does / why we need it? This PR fixes an `inf` sampling issue observed in Qwen3 30B RL workloads on Ascend. In `random_sample()`, tensor `q` is allocated on `global_stream()` but consumed on `torch.npu.current_stream()` via `probs.div_(q)`. The code only establishes an execution dependency (`wait_stream`), but does not declare the storage lifetime of `q` on the consumer stream. Without `q.record_stream(torch.npu.current_stream())`, the caching allocator may reclaim and reuse `q`'s buffer on the sampling stream before the queued division on the current stream has read it. On Ascend, `exponential_()` is a multi-kernel in-place sequence (`uniform_ -> neg_ -> add_(1) -> masked_fill_ -> log_ -> mul_(-1)`), so the next decode step's negative intermediate state can overwrite the old `q` address while the previous consumer is still reading it. For Qwen3 30B RL (vocab size 151936 vs. tokenizer length 151669), the masked tail of 267 ids has exactly zero probability. With a corrupted `q`, `0 / negative_q = -0` wins `argmax` and selects the first masked OOV id (`151669`), whose raw logprob is `-inf`. The OOV token then enters the autoregressive context and corrupts all subsequent outputs. Adding `q.record_stream(torch.npu.current_stream())` declares that `q` must remain alive on the consumer stream, preventing the allocator from recycling the buffer too early. Reproduction with the real `exponential_()` intermediate-state race: - Before fix: 64/64 rows selected the masked OOV id `151669` with raw logprob `-inf` - After fix: 0/64 rows invalid, sampled ids stay legal Note: the same fix is being submitted against all supported release branches (`releases/v0.13.0` -> `main`). ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? - Reproduced the allocator-reuse race with real `exponential_()` intermediate states (`uniform_ -> neg_`): 64/64 sampled rows selected the masked OOV id `151669` with raw logprob `-inf` before the fix; after the fix 0/64 rows are invalid and sampled ids stay legal. - Verified natural address reuse of `q` across consecutive decode steps is eliminated with `record_stream`. - Existing sampler unit tests pass. - NPU hardware verification pending on the original RL environment (`bsz=256, decode>=10k, TP=4, n=8`). - vLLM version: v0.26.0 - vLLM main: vllm-project/vllm@d02df74 Signed-off-by: Hcm03 <chengminhua1@huawei.com>
6 tasks
kunpengW-code
pushed a commit
that referenced
this pull request
Aug 24, 2026
…#14687) ### What this PR does / why we need it? This is the v0.26.0 release-blocker backport rollup. It contains only the 19 audited high-severity correctness and stability fixes that were still missing from releases/v0.26.0rc at 1f95052. The branch has 23 physical commits because #14619 is preserved as its complete five-commit atomic series. Cross-branch equivalents are deduplicated, and every logical fix remains independently reviewable and revertible. #### Included fixes | # | Source / target PR | Severity | Problem fixed | |---:|---|---|---| | 1 | #13538 | P0 correctness | Qwen3-VL MoE + FlashComm1 + deepstack used the wrong residual tensor and could silently corrupt output. | | 2 | #13600 | P0 deadlock | MRV1/MRV2 main and draft update streams could mutually wait during repeated full-graph execution. | | 3 | #13498 | P0 data corruption | Float32 Mamba state could overwrite the shared bf16 hidden-state cache buffer. | | 4 | #13902 | P0 correctness | RL weight reload left ACL graphs referencing stale W8A8-MXFP8 weight addresses. | | 5 | #12359 / #12371 | P0 correctness | Mooncake reformatted KV before all TP/CP pulls for a request completed, causing TP inequality or reordered KV. | | 6 | #13111 / #13113 / #13110 | P1 KV correctness | Multi-KV-group save/load incorrectly reused group-0 block size for every group. | | 7 | #13116 / #13117 / #13099 | P1 state consistency | Async KV load failures were not shared with the scheduler, preventing recompute recovery. | | 8 | #13308 / #13310 / #13307 | P1 crash | Memcache batch query/allocation before lazy initialization could assert in scheduler or worker. | | 9 | #13012 | P1 hang/corruption | Level-2 sleep/wake could lose the MoE loader and leave EPLB tensors pointing at released storage. | | 10 | #13414 | P1 crash/correctness | Dynamic EPLB initialized W8A8 scales for only the first expert weight. | | 11 | #14001 | P1 crash | MiniMax-M3 index_q was reshaped using total size instead of the per-head dimension. | | 12 | #14394 | P1 crash/hang | MRV2 FULL_DECODE_ONLY dropped graph padding when runtime mode was FULL. | | 13 | #13136 | P1 crash | P/D + DP zero-token ranks compared None with MC2 capacity and raised TypeError. | | 14 | #13183 | P1 unavailable | ec_both was treated as producer-only and skipped KV specification/data needed by its consumer role. | | 15 | #13123 | P1 OOB/device error | MRV2 dummy-token remainder was concentrated on one request and could exceed max_model_len. | | 16 | #13159 | P1 crash | MRV2 num_nans used the wrong Triton libdevice and the penalty kernel could exceed the CANN grid limit. | | 17 | #12940 | P1 crash | DFlash profiling used total query count instead of actual input tokens for RoPE/graph capture. | | 18 | #13394 via #13405 | P0 correctness | RL sampling tensor lifetime errors could produce Inf/OOV tokens and contaminate later output. | | 19 | #14142 via #14619 | P1 long-run/state correctness | P/D rejection left stale KV/accounting and unsafe retry/replay behavior could leak, duplicate, or return wrong responses. | #### Backport policy - Selected the audited v0.26 release-adapted commits where available; the closed rollup #14337 was not revived wholesale. - Kept only one canonical copy of fixes duplicated across 0.23, 0.25, and main. - Manually adapted #13136, the core #13123 input-batch hunk, and #13159 to preserve current v0.26/MegaMoe/model-runner behavior. - Used the current v0.26 target change from #13405 and the complete five-commit #14619 series. - Intentionally excluded performance-only, UX-only, conditional-support, low-confidence, and owner-unsettled fixes from this release window. ### Does this PR introduce _any_ user-facing change? Yes, behavior is corrected for the affected configurations: crashes, deadlocks, hangs, incorrect output, stale KV state, and data corruption are prevented. There is no new public API, CLI option, or configuration requirement. ### How was this patch tested? Local validation completed: - Audited manifest: 19/19 logical fixes, 23/23 expected source commits; missing 0, duplicate 0, unexpected 0. - All 23 commits retain source provenance and Signed-off-by trailers. - Ruff lint and format checks passed for all 38 changed Python files. - Python syntax compilation passed for all 38 changed Python files. - git diff --check, codespell, forbidden-import, package-init, context-manager, and filename checks passed. - The final worktree is clean at 86b2ca8. The backports retain or add focused tests for AscendStore, Mooncake rejection cleanup, fused MoE/EPLB, W8A8-MXFP8 reload, worker sleep/wake, MRV2 graph padding, penalty-grid limits, hidden-state extraction, and two-card speculative DP. NPU UT/E2E was not run locally because the available Windows environment has no vLLM, PyTorch/torch_npu, pytest, or Ascend device. CI and targeted NPU regression are therefore required before merge, especially: - MRV1/MRV2 full-graph repeated-iteration deadlock/teardown. - Mooncake TP2/TP4 out-of-order pull KV equality and P/D rejection cleanup. - Qwen3-VL FlashComm1 + deepstack fixed-seed correctness. - Level-2 sleep/wake, dynamic EPLB, and multi-round RL weight reload. - P/D + DP zero-token ranks, ec_both, DFlash profile/graph, and RL Inf/OOV sampling. - Proxy retry and streaming replay behavior from #14619. #### Review checklist - [x] Only the 19 approved release-critical logical fixes are included. - [x] One logical fix per commit; #14619 remains an atomic five-commit series. - [x] No performance-only backports are included. - [x] Source provenance and sign-offs are retained. - [ ] Repository CI passes. - [ ] Targeted Ascend NPU correctness and long-run tests pass. - vLLM version: v0.26.0 - vLLM main: vllm-project/vllm@d02df74 --------- Signed-off-by: kyle-zhangchi <chiiiiiizhang@gmail.com> Signed-off-by: lijiaqi139 <lijiaqi139@huawei.com> Signed-off-by: jiaqi-lee <15316070896@163.com> Signed-off-by: tyy0829 <1455207791@qq.com> Signed-off-by: yejj710 <abyss1999@163.com> Signed-off-by: jiajinzhu2 <jiajinzhu@huawei.com> Signed-off-by: chenyue1122 <oyoy7102@163.com> Signed-off-by: XuRongSheng <1843167357@qq.com> Signed-off-by: muziyuhui666 <lijianfu9@huawei.com> Signed-off-by: Pz1116 <zpbzpb123123@gmail.com> Signed-off-by: zouyida2052 <zouyida2002@gmail.com> Signed-off-by: likailong <likailong5@huawei.com> Signed-off-by: hanxi-java <634498162@qq.com> Signed-off-by: Liam <ml646@duke.edu> Signed-off-by: AuroraEmiya <Sakura.iostream@gmail.com> Signed-off-by: HF-001 <1670186653@qq.com> Signed-off-by: wangxiaoteng <wangxiaoteng@huawei.com> Signed-off-by: Hcm03 <chengminhua1@huawei.com> Signed-off-by: zhuyixiang <zhuyixiang2014@163.com> Signed-off-by: moonseeker <2290166829@qq.com> Co-authored-by: kyle-zhangchi <chiiiiiizhang@gmail.com> Co-authored-by: tyy0829 <87685049+tyy0829@users.noreply.github.com> Co-authored-by: yejj <abyss1999@163.com> Co-authored-by: jiajinzhu2 <jiajinzhu@huawei.com> Co-authored-by: CHENYUE <56943221+PHOEBEMOON0802@users.noreply.github.com> Co-authored-by: Xu Rongsheng <73730571+MmMmaru@users.noreply.github.com> Co-authored-by: yjyang62 <yangjinyang5@huawei.com> Co-authored-by: muziyuhui666 <lijianfu9@huawei.com> Co-authored-by: CXY-Katrina <katrina.cxy@gmail.com> Co-authored-by: cywang250805 <wangchaoyu7@huawei.com> Co-authored-by: Bill845514379 <huangjianbao2@huawei.com> Co-authored-by: yejj710 <yejj710@gmail.com> Co-authored-by: AuroraEmiya <Sakura.iostream@gmail.com> Co-authored-by: HaoxinZong <116423146+HaoxinZong@users.noreply.github.com> Co-authored-by: pz1116 <47019764+Pz1116@users.noreply.github.com> Co-authored-by: zouyida2052 <zouyida2002@gmail.com> Co-authored-by: iKeybot <92210799+iKeybot-code@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: 韩熙 <63780107+hanxi-java@users.noreply.github.com> Co-authored-by: zouzy <38661932+zouzy5137@users.noreply.github.com> Co-authored-by: AuroraEmiya <92282919+AuroraEmiya@users.noreply.github.com> Co-authored-by: Liam <ml646@duke.edu> Co-authored-by: kx <1670186653@qq.com> Co-authored-by: wangxiaoteng888 <56506195+wangxiaoteng888@users.noreply.github.com> Co-authored-by: Hcm03 <chengminhua1@huawei.com> Co-authored-by: zhuyixiang <zhuyixiang2014@163.com> Co-authored-by: moonseeker <2290166829@qq.com>
Leetrytry
pushed a commit
to Leetrytry/vllm-ascend
that referenced
this pull request
Sep 11, 2026
…vllm-project#14687) ### What this PR does / why we need it? This is the v0.26.0 release-blocker backport rollup. It contains only the 19 audited high-severity correctness and stability fixes that were still missing from releases/v0.26.0rc at 1f95052. The branch has 23 physical commits because vllm-project#14619 is preserved as its complete five-commit atomic series. Cross-branch equivalents are deduplicated, and every logical fix remains independently reviewable and revertible. #### Included fixes | # | Source / target PR | Severity | Problem fixed | |---:|---|---|---| | 1 | vllm-project#13538 | P0 correctness | Qwen3-VL MoE + FlashComm1 + deepstack used the wrong residual tensor and could silently corrupt output. | | 2 | vllm-project#13600 | P0 deadlock | MRV1/MRV2 main and draft update streams could mutually wait during repeated full-graph execution. | | 3 | vllm-project#13498 | P0 data corruption | Float32 Mamba state could overwrite the shared bf16 hidden-state cache buffer. | | 4 | vllm-project#13902 | P0 correctness | RL weight reload left ACL graphs referencing stale W8A8-MXFP8 weight addresses. | | 5 | vllm-project#12359 / vllm-project#12371 | P0 correctness | Mooncake reformatted KV before all TP/CP pulls for a request completed, causing TP inequality or reordered KV. | | 6 | vllm-project#13111 / vllm-project#13113 / vllm-project#13110 | P1 KV correctness | Multi-KV-group save/load incorrectly reused group-0 block size for every group. | | 7 | vllm-project#13116 / vllm-project#13117 / vllm-project#13099 | P1 state consistency | Async KV load failures were not shared with the scheduler, preventing recompute recovery. | | 8 | vllm-project#13308 / vllm-project#13310 / vllm-project#13307 | P1 crash | Memcache batch query/allocation before lazy initialization could assert in scheduler or worker. | | 9 | vllm-project#13012 | P1 hang/corruption | Level-2 sleep/wake could lose the MoE loader and leave EPLB tensors pointing at released storage. | | 10 | vllm-project#13414 | P1 crash/correctness | Dynamic EPLB initialized W8A8 scales for only the first expert weight. | | 11 | vllm-project#14001 | P1 crash | MiniMax-M3 index_q was reshaped using total size instead of the per-head dimension. | | 12 | vllm-project#14394 | P1 crash/hang | MRV2 FULL_DECODE_ONLY dropped graph padding when runtime mode was FULL. | | 13 | vllm-project#13136 | P1 crash | P/D + DP zero-token ranks compared None with MC2 capacity and raised TypeError. | | 14 | vllm-project#13183 | P1 unavailable | ec_both was treated as producer-only and skipped KV specification/data needed by its consumer role. | | 15 | vllm-project#13123 | P1 OOB/device error | MRV2 dummy-token remainder was concentrated on one request and could exceed max_model_len. | | 16 | vllm-project#13159 | P1 crash | MRV2 num_nans used the wrong Triton libdevice and the penalty kernel could exceed the CANN grid limit. | | 17 | vllm-project#12940 | P1 crash | DFlash profiling used total query count instead of actual input tokens for RoPE/graph capture. | | 18 | vllm-project#13394 via vllm-project#13405 | P0 correctness | RL sampling tensor lifetime errors could produce Inf/OOV tokens and contaminate later output. | | 19 | vllm-project#14142 via vllm-project#14619 | P1 long-run/state correctness | P/D rejection left stale KV/accounting and unsafe retry/replay behavior could leak, duplicate, or return wrong responses. | #### Backport policy - Selected the audited v0.26 release-adapted commits where available; the closed rollup vllm-project#14337 was not revived wholesale. - Kept only one canonical copy of fixes duplicated across 0.23, 0.25, and main. - Manually adapted vllm-project#13136, the core vllm-project#13123 input-batch hunk, and vllm-project#13159 to preserve current v0.26/MegaMoe/model-runner behavior. - Used the current v0.26 target change from vllm-project#13405 and the complete five-commit vllm-project#14619 series. - Intentionally excluded performance-only, UX-only, conditional-support, low-confidence, and owner-unsettled fixes from this release window. ### Does this PR introduce _any_ user-facing change? Yes, behavior is corrected for the affected configurations: crashes, deadlocks, hangs, incorrect output, stale KV state, and data corruption are prevented. There is no new public API, CLI option, or configuration requirement. ### How was this patch tested? Local validation completed: - Audited manifest: 19/19 logical fixes, 23/23 expected source commits; missing 0, duplicate 0, unexpected 0. - All 23 commits retain source provenance and Signed-off-by trailers. - Ruff lint and format checks passed for all 38 changed Python files. - Python syntax compilation passed for all 38 changed Python files. - git diff --check, codespell, forbidden-import, package-init, context-manager, and filename checks passed. - The final worktree is clean at 86b2ca8. The backports retain or add focused tests for AscendStore, Mooncake rejection cleanup, fused MoE/EPLB, W8A8-MXFP8 reload, worker sleep/wake, MRV2 graph padding, penalty-grid limits, hidden-state extraction, and two-card speculative DP. NPU UT/E2E was not run locally because the available Windows environment has no vLLM, PyTorch/torch_npu, pytest, or Ascend device. CI and targeted NPU regression are therefore required before merge, especially: - MRV1/MRV2 full-graph repeated-iteration deadlock/teardown. - Mooncake TP2/TP4 out-of-order pull KV equality and P/D rejection cleanup. - Qwen3-VL FlashComm1 + deepstack fixed-seed correctness. - Level-2 sleep/wake, dynamic EPLB, and multi-round RL weight reload. - P/D + DP zero-token ranks, ec_both, DFlash profile/graph, and RL Inf/OOV sampling. - Proxy retry and streaming replay behavior from vllm-project#14619. #### Review checklist - [x] Only the 19 approved release-critical logical fixes are included. - [x] One logical fix per commit; vllm-project#14619 remains an atomic five-commit series. - [x] No performance-only backports are included. - [x] Source provenance and sign-offs are retained. - [ ] Repository CI passes. - [ ] Targeted Ascend NPU correctness and long-run tests pass. - vLLM version: v0.26.0 - vLLM main: vllm-project/vllm@d02df74 --------- Signed-off-by: kyle-zhangchi <chiiiiiizhang@gmail.com> Signed-off-by: lijiaqi139 <lijiaqi139@huawei.com> Signed-off-by: jiaqi-lee <15316070896@163.com> Signed-off-by: tyy0829 <1455207791@qq.com> Signed-off-by: yejj710 <abyss1999@163.com> Signed-off-by: jiajinzhu2 <jiajinzhu@huawei.com> Signed-off-by: chenyue1122 <oyoy7102@163.com> Signed-off-by: XuRongSheng <1843167357@qq.com> Signed-off-by: muziyuhui666 <lijianfu9@huawei.com> Signed-off-by: Pz1116 <zpbzpb123123@gmail.com> Signed-off-by: zouyida2052 <zouyida2002@gmail.com> Signed-off-by: likailong <likailong5@huawei.com> Signed-off-by: hanxi-java <634498162@qq.com> Signed-off-by: Liam <ml646@duke.edu> Signed-off-by: AuroraEmiya <Sakura.iostream@gmail.com> Signed-off-by: HF-001 <1670186653@qq.com> Signed-off-by: wangxiaoteng <wangxiaoteng@huawei.com> Signed-off-by: Hcm03 <chengminhua1@huawei.com> Signed-off-by: zhuyixiang <zhuyixiang2014@163.com> Signed-off-by: moonseeker <2290166829@qq.com> Co-authored-by: kyle-zhangchi <chiiiiiizhang@gmail.com> Co-authored-by: tyy0829 <87685049+tyy0829@users.noreply.github.com> Co-authored-by: yejj <abyss1999@163.com> Co-authored-by: jiajinzhu2 <jiajinzhu@huawei.com> Co-authored-by: CHENYUE <56943221+PHOEBEMOON0802@users.noreply.github.com> Co-authored-by: Xu Rongsheng <73730571+MmMmaru@users.noreply.github.com> Co-authored-by: yjyang62 <yangjinyang5@huawei.com> Co-authored-by: muziyuhui666 <lijianfu9@huawei.com> Co-authored-by: CXY-Katrina <katrina.cxy@gmail.com> Co-authored-by: cywang250805 <wangchaoyu7@huawei.com> Co-authored-by: Bill845514379 <huangjianbao2@huawei.com> Co-authored-by: yejj710 <yejj710@gmail.com> Co-authored-by: AuroraEmiya <Sakura.iostream@gmail.com> Co-authored-by: HaoxinZong <116423146+HaoxinZong@users.noreply.github.com> Co-authored-by: pz1116 <47019764+Pz1116@users.noreply.github.com> Co-authored-by: zouyida2052 <zouyida2002@gmail.com> Co-authored-by: iKeybot <92210799+iKeybot-code@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: 韩熙 <63780107+hanxi-java@users.noreply.github.com> Co-authored-by: zouzy <38661932+zouzy5137@users.noreply.github.com> Co-authored-by: AuroraEmiya <92282919+AuroraEmiya@users.noreply.github.com> Co-authored-by: Liam <ml646@duke.edu> Co-authored-by: kx <1670186653@qq.com> Co-authored-by: wangxiaoteng888 <56506195+wangxiaoteng888@users.noreply.github.com> Co-authored-by: Hcm03 <chengminhua1@huawei.com> Co-authored-by: zhuyixiang <zhuyixiang2014@163.com> Co-authored-by: moonseeker <2290166829@qq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it?
This PR fixes an
infsampling issue observed in Qwen3 30B RL workloads on Ascend.In
random_sample(), tensorqis allocated onglobal_stream()but consumed ontorch.npu.current_stream()viaprobs.div_(q). The code only establishes an execution dependency (wait_stream), but does not declare the storage lifetime ofqon the consumer stream.Without
q.record_stream(torch.npu.current_stream()), the caching allocator may reclaim and reuseq's buffer on the sampling stream before the queued division on the current stream has read it. On Ascend,exponential_()is a multi-kernel in-place sequence (uniform_ -> neg_ -> add_(1) -> masked_fill_ -> log_ -> mul_(-1)), so the next decode step's negative intermediate state can overwrite the oldqaddress while the previous consumer is still reading it.For Qwen3 30B RL (vocab size 151936 vs. tokenizer length 151669), the masked tail of 267 ids has exactly zero probability. With a corrupted
q,0 / negative_q = -0winsargmaxand selects the first masked OOV id (151669), whose raw logprob is-inf. The OOV token then enters the autoregressive context and corrupts all subsequent outputs.Adding
q.record_stream(torch.npu.current_stream())declares thatqmust remain alive on the consumer stream, preventing the allocator from recycling the buffer too early.Reproduction with the real
exponential_()intermediate-state race:151669with raw logprob-infNote: the same fix is being submitted against all supported release branches (
releases/v0.13.0->main).Does this PR introduce any user-facing change?
No.
How was this patch tested?
exponential_()intermediate states (uniform_ -> neg_): 64/64 sampled rows selected the masked OOV id151669with raw logprob-infbefore the fix; after the fix 0/64 rows are invalid and sampled ids stay legal.qacross consecutive decode steps is eliminated withrecord_stream.bsz=256, decode>=10k, TP=4, n=8).