[thanhhao98]

Motivation

MiniMax-M3 with an EAGLE3 draft does not start on CUDA: TARGET_VERIFY reaches the sparse layers' forward_extend, which needs the extend_seq_lens that verify batches lack, so the warm-up fails with AttributeError: 'NoneType' object has no attribute 'device'. #41497 fixed ROCm by verifying a linear draft chain with the sparse decode kernels, one causal row per draft token, but every gate it added is is_hip. This PR covers TP, not DP attention.

Modifications

• Run the [AMD] Fix MiniMax-M3 EAGLE3 verification on ROCm #41497 linear-verify path on every non-NPU GPU: forward_extend hands verify to forward_decode on a shallow copy, one row per draft token, so the dense layers keep their per-request metadata and the draft KV is stored once. _init_rocm_linear_verify_metadata becomes _init_linear_verify_metadata; the ROCm test only follows the rename.
• On CUDA, accept only EAGLE chains (topk == 1) via the existing draft_is_chain_layout; tree drafts and other algorithms raise NotImplementedError at startup. ROCm keeps its rule: any draft with topk == 1.
• Refuse HiSparse with speculative decoding at startup: its swap-in keeps one device buffer and one LRU slot set per request, while verify scores bs × ndt rows.
• Bound verify capture by the context length on every platform (CUDA capture took the bound from the 1-token dummy lengths, a one-column score buffer that _decode_score_kernel overruns at replay); the eager bound counts the draft rows.
• Clamp rows past the context length to it, which keeps the verify bound at the context length instead of context + ndt: those rows predict positions that are never committed (the scheduler caps max_new_tokens below the context length and trims over-accepted drafts), and at 1M context the decode top-k stays in its 8192 register bucket.
• Keep verify rows on Triton when MSA decode is on, since its plan covers decode batches only; MSA with the decode graph and speculative decoding is still refused at startup.

Accuracy Tests

MiniMax-M3-NVFP4, 4x GB300, TP4, 1M context, thinking enabled, SGLANG_DISABLE_MSA=1, lmsysorg/sglang:nightly-dev-cu13-20261012-e6072935 + this patch; EAGLE3 Inferact/MiniMax-M3-EAGLE3-GQA (3 steps, topk 1, 4 draft tokens, unquantized draft), real verification.

same build, one tray each	no spec	EAGLE3
GSM8K 1319 q (5-shot)	0.861	0.864
GSM8K 40 q at --parallel 1 (bs=1 decode graph)	0.800	0.875
passkey at 131K / 524K / 990K tokens	pass	pass
mean acceptance length (GSM8K / long context)	–	2.81 / 2.90

• New test_minimax_linear_verify.py (CUDA, H100 and B200 CI; MSA decode on, so verify must stay on Triton) compares verify rows with the same rows decoded alone: eager at prefixes 1000 and 5118 (crosses a block boundary) is bit-exact; a graph captured on 1-token dummies, replayed with one request two tokens short of a 64K context, matches inside the context and clamps rows past it.
• On GB300, dropping the eager draft-token term fails the eager cases and dropping the clamp fails the replay cases.

Speed Tests and Profiling

Mean TPOT at bs=1 (prompt cached, 256 output tokens, real acceptance), same build, one GB300 tray per arm:

prompt tokens	no spec	EAGLE3	speedup
8K	4.34 ms	2.23 ms	1.95x
131K	6.00 ms	3.40 ms	1.76x
524K	9.78 ms	5.43 ms	1.80x
990K	13.88 ms	8.30 ms	1.67x

Reproduce

Server, on lmsysorg/sglang:nightly-dev-cu13-20261012-e6072935 with this PR, 4× GB300 (the no-spec arm drops the --speculative-* flags):

SGLANG_DISABLE_MSA=1 python3 -m sglang.launch_server --model-path nvidia/MiniMax-M3-NVFP4 --trust-remote-code --tp 4 --context-length 1048576 --reasoning-parser minimax-m3 --tool-call-parser minimax-m3 --default-chat-template-kwargs '{"thinking_mode":"enabled"}' --enable-metrics --random-seed 42 --speculative-algorithm EAGLE3 --speculative-draft-model-path Inferact/MiniMax-M3-EAGLE3-GQA --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --speculative-draft-model-quantization unquant

GSM8K (5-shot): python3 -m sglang.test.few_shot_gsm8k --num-shots 5 --num-questions 1319, and --num-questions 40 --parallel 1 for the bs=1 row.

Passkey: one chat request per size whose prompt is filler text with a random 5-digit key at two thirds depth, sized to 131K / 524K / 990K tokens with the server's chat template; pass when the key is in the answer. bs=1 TPOT: on the same server, a follow-up "count upward" turn on that cached conversation, 256 tokens with ignore_eos; the 8K row warms an 8K prefix first.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests.
• Update documentation.
• Provide accuracy and speed benchmark results.
• Follow the SGLang code style guidance.

---

CI States

Latest PR Test (Base): ❌ Run #40291392525
Latest PR Test (Extra): ❌ Run #40291392550
Latest PR Test (AMD ROCm 10): ❌ Run #40291392514

[thanhhao98]

@nvpohanh could you pls help to review this, thanks!

[nvpohanh]

[by Claude Code] This review is done by agents. Please feel free to disagree if they do not make sense.

[by Claude Code] Severity: style | Confidence: High

Issue

The Accuracy Tests and Speed Tests sections give the model, GPUs, TP, image, and EAGLE3 settings, but no commands. Others cannot rerun the GSM8K, passkey, or TPOT numbers.

Fix

Add the exact server launch commands for the no-spec and EAGLE3 arms, the GSM8K and passkey eval commands, and the benchmark command behind the bs=1 TPOT table, including input/output lengths and how the prompt was cached.

[thanhhao98]

These PRs should be reviewed first in the M3 PR series:

• [Perf] minimax-m3: Split the score-only prefill indexer over the KV range #45452
• [Perf] [NVIDIA] minimax-m3: Score EAGLE verify rows in one pass over the index K #45451

They will help to improve the performance.
Could you please help review them @ispobock