[mgoin]

Overview

Add a Hopper (SM90) backend to DeepSeek-V4.1's specialized decode executor (#61868), for DeepSeek-V4.1-Flash at TP4 on H200 with DSpark k=5. It is opt-in, as on GB300, through --additional-config '{"decode_backend":"latency_specialized"}'. Decode steps of up to 192 rows run in the executor; larger steps, prefill and anything it does not serve fall back to the stock path over the same weights.

Stacked on #61869 (DSpark low-latency decode), #57770 (split-vocabulary greedy argmax) and #64892 (SM90 mHC overlap behind the fused all-reduce). Pure TP4 with Humming MoE also needs humming#177 (humming-kernels 0.1.19, #64802).

Reviewing: the Hopper executor commit is the new work, plus the merge of main on top of it, which adapts the stack to main's DraftLogitsCache and the moved CuTe DSL primitives. Everything before them comes from #61869 and #64892. #57770 is merged in main. #61869 is still open, so its changes are in this diff until it lands.

Duplicate check: there is no other open Hopper backend for the specialized decode. #62958 and #64442 only touch DSpark sampling.

Claims

• Median ITL on H200 TP4, DSpark k=5, on this branch merged with main: −38% at c1 (8.65 → 5.34 ms) and about −5% at c8 on real prompts; −37% at c1 and −14% at c8 at 64K context. At c16 and above the gain was about 5–7% before the merge and has not been re-measured since.
• Quality is unchanged within noise on GPQA-Diamond, AIME25 and 128K needle retrieval; greedy logprob drift is within the stock run-to-run floor.
• No host syncs in steady-state decode (VLLM_GPU_SYNC_CHECK=error, re-run on the merged build: 0 at c1, c8 and c32).
• The Hopper commit changes no core components (model runner, scheduler, KV cache manager); the model_runner.py changes in this diff come from [Perf][DSv4.1] DSpark draft on the specialized decode #61869. Everything in the Hopper commit is under vllm/models/deepseek_v41/nvidia/specialized_decode/, and SM100 behavior is unchanged.

Validation

Setup: 4× H200, --tensor-parallel-size 4 --max-model-len 200000 --max-num-seqs 64, DSpark k=5, Humming MoE with humming#177. "Off" is the same build with the executor disabled, so it includes every stacked PR. Runs are in ABBA order (on, off, off, on).

Speed, real prompts, output length 1024, temperature 0.7. Median ITL in ms:

conc	1	2	4	8	16	24	32
executor off	8.76	10.31	12.81	16.49	21.26	26.61	29.83
executor on	5.40	6.88	10.57	14.27	19.84	25.02	28.31
Δ	−38.3%	−33.3%	−17.5%	−13.5%	−6.6%	−6.0%	−5.1%

Acceptance lengths match between arms (2.79–3.15).

Long context, 64K prompts, two on/off pairs. Median ITL in ms:

conc	on	off	Δ
1	5.64, 5.65	8.96, 8.96	−37%
8	15.20, 13.58	15.59, 16.15	about −9%

Quality.

Eval	on	off	on − off (paired)
GPQA-Diamond, 8 × 198	79.9	80.6	−0.6 ± 0.6
AIME25, 8 × 30	86.2	85.8	+0.4 ± 1.8
128K needle, single and multikey, c1 and c8	12/12 each	12/12 each

• On the final build, GPQA-Diamond at 3 repeats scores 80.0.
• Greedy logprobs at identical prefixes, 24 prompts:
  • on vs off: mean |Δ logprob| on the chosen token 2.2e-2;
  • off vs off (noise floor): 2.6e-2;
  • every divergence happens at a top-1 margin of at most 0.25 nats.

The concurrency sweep and the 8-repeat evals ran before a fused-indexer fix at long context. On the final build c1 and c8 measure 5.39 and 14.40 ms, unchanged.

After merging main (f1df4a3ae9). Median ITL in ms, on a new 160-prompt set from RedHatAI/speculator_benchmarks:

Point	on	off	Δ
real prompts, c1	5.34	8.64, 8.66	−38%
real prompts, c8	14.36	14.64, 15.65	about −5%
64K context, c1	5.63	8.89	−37%
64K context, c8	13.63	15.93	−14%

Check	on	off
128K needle, single and multikey, c1 and c8	12/12 each	12/12 each
GPQA-Diamond, 3 repeats	79.3 (80.3 / 79.3 / 78.3)	not re-run

Only one on run per point is clean. The tables above this one are from before the merge and used an earlier prompt set, so compare numbers only within a table. The c8 gain is smaller here than the −13.5% in the sweep above. The cause isn't established: main, the prompt set and acceptance length (3.25 on, 3.18–3.50 off) all changed, and the runs are few. A clean c1–c32 sweep on the merged build is still to do; the attempt so far overlapped other work on the GPUs and was discarded.

Tests. pytest tests/models/deepseek_v41/nvidia/specialized_decode/test_hopper.py on H200: 19 passed.

• The tests compare each Hopper kernel and step against the stock path or an FP32 reference.
• The fused indexer has a long-context case: one indexer shared across ratio-1 and ratio-2 layers, rows longer than the candidate coverage, and a NaN-poisoned output.
• test_hopper is wired into Buildkite (models_basic.yaml).
• CI has no 4× H200 runner for the end-to-end numbers above.

Reproduce:

vllm serve deepseek-ai/DeepSeek-V4.1-Flash --tensor-parallel-size 4 \
  --max-model-len 200000 --max-num-seqs 64 --tokenizer-mode deepseek_v41 \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5}'
vllm bench serve --backend vllm --endpoint /v1/completions --dataset-name custom \
  --dataset-path real_prompts.jsonl --skip-chat-template --custom-output-len 1024 \
  --temperature 0.7 --max-concurrency 1   # likewise 2, 4, 8, 16, 24, 32

The executor-on arm adds --additional-config '{"decode_backend":"latency_specialized"}' to vllm serve. The executor-off arm is the default.

Details

Where the time goes. At c1 the stock step is mostly fixed per-kernel cost, not bandwidth:

• an isolated ~10 MB GEMV reaches only about 50% of HBM;
• each kernel boundary costs ~0.7–1.8 µs.

So the executor:

• chains kernels with PDL;
• replaces the small-batch MoE, dense FP8 linears, sparse MLA and indexer with SM90 CuTe DSL kernels written for these shapes;
• fuses the small ops around them.

The MoE runs near its streaming limit (about 4 TB/s).

Compatibility. The kernels read the weights in stock's layouts:

• Humming MXFP4 × block-FP8 for experts;
• Humming W8A16 for dense layers.

So steps the executor doesn't serve fall back to stock with one copy of the weights. The exception is an FP8 copy of wo_a, because stock SM90 keeps it only as BF16. It costs about 0.36 GB per GPU (1.44 GB in FP8 across 43 layers, sharded TP4).

Size. About 10.8K lines under hopper/, mostly kernels:

Area	Lines
MoE	4.4K
sparse MLA	2.3K
dense GEMM	1.6K
indexer	0.9K
attention-chain fusions	0.7K

Plus +342/−85 in the shared specialized-decode package, which lets the executor select its backend by architecture and records metadata builds for the Hopper builders. Plus 1.0K lines of tests.

It mirrors the GB300 backend's structure, so the two share the executor runtime, step schedule and fallbacks.

Limitations.

• TP4 on H200 only.
• Steps above 192 rows use stock.
• The gain shrinks with batch size, as the per-kernel overhead is amortized.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @mgoin.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @mgoin.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork