Skip to content

[Bugfix] Guard DeepSeek V4 MRV1 piecewise CUDA graphs - #51768

Merged
WoosukKwon merged 1 commit into
mainfrom
agent/fix-dsv4-indexer-cudagraph-metadata
Aug 11, 2026
Merged

WoosukKwon merged 1 commit into
mainfrom
agent/fix-dsv4-indexer-cudagraph-metadata

Conversation

@WoosukKwon

@WoosukKwon WoosukKwon commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

Default DeepseekV4ForCausalLM to Model Runner V2 and reject only the known-broken configuration:

  • DeepSeek V4
  • Model Runner V1
  • PIECEWISE or FULL_AND_PIECEWISE CUDA graphs

The validation runs after CUDA-graph mode resolution. MRV1 therefore remains available with eager/NONE, FULL, or FULL_DECODE_ONLY; other model architectures are unaffected.

Why

#51430 exposed a correctness problem in the legacy V1 model runner's breakable PIECEWISE CUDA-graph path. The same attention implementation retains normal GSM8K accuracy and MTP acceptance with Model Runner V2.

Failing this specific combination avoids silent output corruption while preserving MRV1 for configurations that do not exercise the affected path.

The error directs users to either select MRV2 or disable PIECEWISE CUDA graphs.

Relationship to #51750

This is an alternative to #51750, not a duplicate. #51750 exactly reverts #51430 and restores the wider MRV1 eager region. This PR leaves the attention implementation unchanged, defaults DeepSeek V4 to the already-correct MRV2 path, and fails closed if a user explicitly reaches the broken MRV1 PIECEWISE path.

The required duplicate searches found no other open PR adding this configuration guard.

Model validation

GPU validation ran through Slurm on a GB200 node with TP=2, FP4 indexer cache, MTP with two speculative tokens, and CUDA graphs enabled.

code runner questions GSM8K MTP draft acceptance
exact post-#51430 commit 635dd6aae MRV2 256 0.9570 80.7–81.4%
unmodified current main MRV2 1,319 0.9500 81.1–82.1%
exact post-#51430 commit 635dd6aae MRV1 + PIECEWISE 1,319 0.0311 collapsed to 3.1–5.5%

Tests

env -u VLLM_USE_V2_MODEL_RUNNER \
  .venv/bin/python -m pytest tests/test_config.py \
  -k 'v2_model_runner or mrv1_piecewise' -q
# 29 passed

.venv/bin/pre-commit run ruff-check --files \
  vllm/config/vllm.py tests/test_config.py
.venv/bin/pre-commit run ruff-format --files \
  vllm/config/vllm.py tests/test_config.py
.venv/bin/pre-commit run mypy-3.12 --files \
  vllm/config/vllm.py tests/test_config.py --hook-stage manual

The tests cover both rejected PIECEWISE modes and the allowed MRV2, eager/NONE, FULL, FULL_DECODE_ONLY, and non-DeepSeek-V4 cases.

AI assistance

AI assistance was used to investigate the runner-specific regression, implement this change, run validation, and prepare this PR.

Before marking ready for review, the human submitter must:

  • review and understand every changed line
  • independently confirm the relevant test results

@mergify mergify Bot added deepseek Related to DeepSeek models nvidia bug Something isn't working labels Aug 11, 2026
@WoosukKwon
WoosukKwon force-pushed the agent/fix-dsv4-indexer-cudagraph-metadata branch from 1410285 to 7349476 Compare August 11, 2026 06:29
@WoosukKwon WoosukKwon changed the title [Bugfix] Fix DeepSeek V4 indexer CUDA graph replay [Bugfix] Default DeepSeek V4 to Model Runner V2 Aug 11, 2026
@mergify mergify Bot added the mrv2 Model Runner V2 specific label Aug 11, 2026
@WoosukKwon
WoosukKwon marked this pull request as ready for review August 11, 2026 06:32

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

@WoosukKwon WoosukKwon added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 11, 2026
@WoosukKwon
WoosukKwon force-pushed the agent/fix-dsv4-indexer-cudagraph-metadata branch from 7349476 to 9a35f17 Compare August 11, 2026 06:37
@WoosukKwon WoosukKwon changed the title [Bugfix] Default DeepSeek V4 to Model Runner V2 [Bugfix] Require Model Runner V2 for DeepSeek V4 Aug 11, 2026
@WoosukKwon
WoosukKwon marked this pull request as draft August 11, 2026 06:38
Co-authored-by: OpenAI Codex <codex@openai.com>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
@WoosukKwon
WoosukKwon force-pushed the agent/fix-dsv4-indexer-cudagraph-metadata branch from 9a35f17 to 28cfd2e Compare August 11, 2026 06:44
@WoosukKwon WoosukKwon changed the title [Bugfix] Require Model Runner V2 for DeepSeek V4 [Bugfix] Guard DeepSeek V4 MRV1 piecewise CUDA graphs Aug 11, 2026
@WoosukKwon
WoosukKwon marked this pull request as ready for review August 11, 2026 07:19
@WoosukKwon

Copy link
Copy Markdown
Collaborator Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #83314 for commit 28cfd2e196d8.

@WoosukKwon
WoosukKwon merged commit 87668ab into main Aug 11, 2026
94 of 96 checks passed
@WoosukKwon
WoosukKwon deleted the agent/fix-dsv4-indexer-cudagraph-metadata branch August 11, 2026 14:34
@github-project-automation github-project-automation Bot moved this to Done in NVIDIA Aug 11, 2026
njhill added a commit to njhill/vllm that referenced this pull request Aug 14, 2026
vllm-project#51430 narrowed the DeepSeek V4 eager cudagraph region, which corrupts
MRV1 output, and vllm-project#51768 responded by defaulting the model to MRV2 and
rejecting MRV1 + PIECEWISE. That default costs ROCm, where MRV1 is still
the faster runner for this model.

Choose the region from the runner instead: MRV1 wraps the whole
attention body in `_prepare_and_attn_eager`, restoring the pre-vllm-project#51430
region it needs, while MRV2 keeps the narrow region and its shorter
TTFT. The nested `_sparse_indexer_and_attn` break runs inline, since
`add_eager` clears `_capturing` first.

MRV1 + PIECEWISE is then correct on every platform, so drop the
rejection. CUDA still defaults to MRV2; ROCm defaults to MRV1 until MRV2
is competitive there.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Signed-off-by: Nick Hill <nickhill123@gmail.com>
Fangzhou-Ai added a commit to Fangzhou-Ai/vllm that referenced this pull request Aug 16, 2026
vllm-project#51430/vllm-project#51768 moved DSV4 to MRV2 and a narrow eager region. That is a
large decode TPOT regression on ROCm. Default ROCm back to MRV1, wrap
the full attention body in the eager break for MRV1 only, and drop the
MRV1+PIECEWISE rejection. CUDA keeps MRV2 and the narrow region.

Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Cursor Grok 4.6 <cursoragent@cursor.com>
Signed-off-by: fai <fangzhouai@gmail.com>
shen-shanshan pushed a commit to shen-shanshan/vllm that referenced this pull request Aug 17, 2026
…roject#51768)"

Restore Model Runner V1 after the prerequisite eager-region revert makes piecewise graph execution correct again and recovers ROCm decode throughput.

Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Fangzhou Ai <fangzhou.ai@amd.com>
zyp2014 pushed a commit to zyp2014/vllm that referenced this pull request Aug 21, 2026
…1768)

Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working deepseek Related to DeepSeek models mrv2 Model Runner V2 specific nvidia ready ONLY add when PR is ready to merge/full CI is needed

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant