Skip to content

[AMD] Prevent DeepSeek-V4 MTP GPU faults during graph replay - #38663

Draft
bingxche wants to merge 1 commit into
sgl-project:mainfrom
bingxche:bingxche/fix-dsv4-target-verify-gpu-plan
Draft

bingxche wants to merge 1 commit into
sgl-project:mainfrom
bingxche:bingxche/fix-dsv4-target-verify-gpu-plan

Conversation

@bingxche

@bingxche bingxche commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

What broke

PR #26383 (deaba74745d7) introduced this regression. It changed HIP-Radix target verification to build full metadata early from CPU sequence lengths. Before that change, CUDA Graph target verification built its compression plan from GPU data. After it, fixed-size MTP graph replay accidentally started using the CPU planner.

The CPU planner fills a temporary pinned-memory buffer and starts an asynchronous copy to the GPU. CUDA Graph captures that copy for later replay, but the temporary CPU buffer is not kept as graph state. A replay can therefore use stale plan offsets. The C128 write kernel then writes outside its allocation, causing the multi-rank GPU memory fault and scheduler exit -6. The later connection failures and KeyError: answer happen only because the server has already crashed.

PR #30964 later added the GPU planner for ragged target verification, where CPU sequence lengths are unavailable. Its check covered only ragged layouts, so fixed-size MTP graph replay remained on the unsafe path. In short: #26383 introduced the bad graph behavior; #30964 left that fixed-size case uncovered.

Failing job: https://github.com/sgl-project/sglang/actions/runs/33538157363/job/99957472899

Fix

Use the existing GPU planner in the two cases that require it: CUDA Graph replay, and ragged verification without CPU sequence lengths. Fixed-size eager verification keeps the existing CPU planner because it does not replay the temporary host buffer and avoids an extra GPU planning launch. Normal prefill, decode, and draft extension are unchanged.

Validation

ROCm 7.2.4, 8x MI355X, DeepSeek-V4-Pro FP4 MTP:

  • 1319/1319 answers completed
  • Accuracy 0.944655; average accepted tokens 2.936088
  • 2 registered tests passed
  • No GPU memory fault, C128 out-of-bounds, fatal HSA, connection, or missing-answer signatures

The full MTP run used SGLang 9a05b470 plus this planner change. The planner hunk and the MTP path it exercises are unchanged on the current PR base.

Checklist

  • Pre-commit passed
  • No user-facing API change

CI States

Latest PR Test (Base): ❌ Run #34335075601
Latest PR Test (Extra): ❌ Run #34335075206
Latest PR Test (AMD ROCm 10): ❌ Run #34335075672

Signed-off-by: bingxche <bingxche@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant