[maeehart]

Overview

This PR ports the DeepSeek-V4.1-Flash mono decode kernels of #60397 to gfx942 (MI300X and MI325X). A standard decoder layer then runs as 2 kernels instead of 32. The path is opt-in with --monokernel, as on gfx950.

The PR was rebased onto main after #61082 as one commit. The commits after it merge main and answer the second review (see Details). gfx942 now runs the same layers whole as gfx950, with the switch of #61082 (see Details). The latest commits take the PR out of shared code, as RFC #60904 asks, and merge main with #62131 (see Details). The AgentX and GSM8K comparisons below are from 67206b7 and the commits before it, when the switch was VLLM_ROCM_MONO_DECODE=1. The checks of the latest commits are at the end of Validation.

Claims

MI325X, TP4, DSpark with 5 drafts. Each AgentX run is 3,600 s and is scored with InferenceX's own result code.

• AgentX at concurrency 1, against main: interactivity P50 goes from 216.85 to 503.53 tokens/s/user (2.32×), P90 from 182.25 to 411.26 (2.26×), and total throughput per GPU from 4,024 to 6,546 tokens/s (1.63×).
• AgentX at concurrency 2, 4 and 8, against InferenceX's published MI325X points: P90 goes from 153.22 to 368.64 (2.41×), from 142.75 to 316.28 (2.22×) and from 81.52 to 162.61 (1.99×).
• A standard decoder layer takes 115 µs instead of 320 µs. A decode period of one request takes 7.2 ms instead of 17.9 ms.
• GSM8K strict match at concurrency 7, 3 runs each: main 0.9247 and this PR 0.9270 on average (pooled p 0.49). The difference is not significant.
• On a2ce22d at concurrency 7, GPQA-Diamond is 85.0%, AIME25 99.0% and RULER NIAH 40 of 40 in every cell, against 85.3%, 99.4% and 40 of 40 for the reference arm of [ROCm][Perf][DSv4.1] Mono decode layer for DeepSeek-V4.1-Flash on MI355X #60397 on MI355X.
• The gfx942 code runs only with --monokernel, on gfx942. Outside vllm/models/deepseek_v41/ and the tests, the PR adds an 8-line hook to the DSpark speculator and two CI jobs.
• On gfx950 all 76 builds of the mono kernels, the 12 of the DSpark draft's runner included, have the same register and LDS use as on main and the same count of each instruction, apart from 1 or 2 more s_waitcnt in K2 and the FFN launch. The draft runner's scratch is now laid out for its MoE, see Details.
• On MI355X with --monokernel at TP2, two runs of each arm on 6caadd0, before [ROCm][Perf][DSv4.1] Mono decode for the DSpark draft layers #62131: GSM8K strict match 0.9257 and 0.9242 against 0.9249 and 0.9272 for main (see Validation).

Limits on gfx942

• TP4 only. At TP2 the path logs one line and every layer runs vLLM's path.
• Decode steps of up to 42 rows, which is 7 requests with 5 DSpark drafts. A gfx942 workgroup has 64 KB of LDS, against 160 KB on gfx950, so a GEMV pass takes 6 tokens instead of 16. A larger step, prefill, a mixed step and a PIECEWISE graph run vLLM's path on every layer and every rank.
• gfx942 has no scaled FP4 or FP8 MFMA and uses FNUZ FP8, so the kernels read their own copies of the weights. At TP4 the weights take 170.11 GiB a rank instead of 98.88 GiB. On MI325X with --gpu-memory-utilization 0.9 the KV cache still holds 11.18 million tokens. MI300X (192 GiB) does not fit this well, so the numbers here are for MI325X.

Validation

deepseek-ai/DeepSeek-V4.1-Flash on MI325X, TP4. The main arm is vllm/vllm-openai-rocm:nightly-81198e97ba7eee2a22540caaa756b7fdddcb4d93. The PR arm is the same image with the vllm/ changes of #60397 and this PR copied over its vllm package. Both arms use the same command and environment, apart from the switch, which was VLLM_ROCM_MONO_DECODE=1 on those commits.

AgentX, concurrency 1, against main. One 3,600 s run per arm, each on a fresh server.

	main	PR	PR / main
Measured requests, no errors	239	306
Interactivity P50 (tokens/s/user)	216.85	503.53	2.32×
Interactivity P90 (tokens/s/user)	182.25	411.26	2.26×
TTFT P50 (s)	0.510	0.544
Total throughput per GPU (tokens/s)	4,024	6,546	1.63×

AgentX, concurrency 2, 4 and 8. There is no main run at these concurrencies, so the reference is InferenceX's published MI325X TP4 points (Actions run 35637516947). They come from an older nightly. At concurrency 1 the published P90 is 189.90, close to our main run.

Concurrency	P90, published	P90, PR	Throughput per GPU, published	Throughput per GPU, PR
2	153.22	368.64 (2.41×)	4,255	6,729 (1.58×)
4	142.75	316.28 (2.22×)	6,315	8,882 (1.41×)
8	81.52	162.61 (1.99×)	10,998	14,839 (1.35×)

At concurrency 8, steps of more than 42 rows ran vLLM's path during 0.14% of the decode time.

Accuracy. lm_eval gsm8k, 5-shot, all 1,319 questions, greedy, DSpark with block rejection. At concurrency 7 every decode step of this PR takes the mono path. Strict match of each run:

	main	This PR
Concurrency 1	0.9340, 0.9318	0.9272
Concurrency 7	0.9242, 0.9257, 0.9242	0.9295, 0.9242, 0.9265, 0.9303

Two runs of the same code differ by up to 0.6 points, so the runs are pooled with a Cochran-Mantel-Haenszel test with one stratum a question. At concurrency 7 this PR scores 0.23 points more than main (p 0.49). Over all runs the mono path scores 0.14 points less (p 0.54). The mono kernels quantize the activations to MXFP8, where vLLM's gfx942 path keeps bf16.

GPQA-Diamond, AIME25 and RULER NIAH. A build of a2ce22d with --monokernel at TP4 and concurrency 7, with the sample counts of #60397: GPQA-Diamond with 2 option orders and 4 seeds at up to 32K tokens, AIME25 with 16 seeds at up to 64K tokens, and NIAH with niah_single_1, niah_multikey_2 and niah_multikey_3 at 32K, 128K and 256K, 40 samples a cell. The reference is the MI355X result of #60397. The runs are in the comments for NIAH, for AIME25 and for GPQA-Diamond.

	This PR, MI325X TP4	#60397 mono, MI355X TP2	#60397 reference, MI355X TP2
GPQA-Diamond	85.0% (1347 of 1584)	85.1 ± 2.2%	85.3 ± 2.2%
AIME25	99.0% (475 of 480)	98.3 ± 0.8%	99.4 ± 0.3%
RULER NIAH, 9 cells	40 of 40 in each	40 of 40 in each	40 of 40 in each

A debugging patch, not part of this PR, ran vLLM's own layer after each mono layer on the same inputs, in steps of 6 to 42 rows. No output had a non-finite value, and the index layers' top-512 lists matched vLLM's in all 481,944 entries.

gfx950. main and this PR on the same MI355X node, two runs each, in the order PR, main, main, PR. All four runs use --monokernel at TP2, DSpark with 5 drafts, probabilistic drafts and block rejection, and the same server arguments (see Reproduce). main is vllm/vllm-openai-rocm:nightly-39983a5e7a5a575a7829f5d5ada50a33d9337ec1, and the PR runs use the same image with this PR's files at 6caadd0 copied in. At 6caadd0 the PR still held a change of _prepare_dflash_inputs_kernel that it no longer has (see Changes outside the model directory), and main did not have #62131 yet. The speed rows use random prompts of 1024 tokens and 1024 output tokens, 16 requests at concurrency 1 and 64 at concurrency 8.

	main, run 1	main, run 2	This PR, run 1	This PR, run 2
GSM8K 5-shot, 1319 questions, concurrency 8, strict match	0.9249	0.9272	0.9257	0.9242
Concurrency 1, output throughput (tokens/s)	520.34	579.54	584.43	577.95
Concurrency 1, median TPOT (ms)	1.59	1.56	1.54	1.48
Concurrency 1, median ITL (ms)	7.60	7.59	7.36	7.38
Concurrency 1, median TTFT (ms)	66.30	66.55	67.09	67.02
Concurrency 8, output throughput (tokens/s)	2038.63	1986.51	2049.16	2099.40
Concurrency 8, median TPOT (ms)	3.37	3.35	3.47	3.17
Concurrency 8, median ITL (ms)	17.06	16.99	16.84	16.77
Concurrency 8, median TTFT (ms)	84.37	83.77	84.51	83.91

GSM8K is the same within the spread of the runs, and the median TTFT differs by less than 1 ms. With DSpark a decode step streams several tokens, so the median ITL is the time of one decode step. It was about 0.2 ms shorter with the PR, from that _prepare_dflash_inputs_kernel change. The TPOT also depends on how many drafts are accepted. In the first main run a few first tokens took more than a second, which lowered its throughput at concurrency 1.

Tests. The kernel tests are in tests/kernels/models/amd/ and run in a new one-GPU MI300 job in .buildkite/test_areas/kernels.yaml. A new four-GPU MI300 job in .buildkite/test_areas/models_basic.yaml runs the real-model and graph replay tests on gfx942 at TP4. On MI325X all of them pass on 6cb1400, and the two TP4 files take about 10 minutes. On MI355X the two TP4 files pass their CDNA4 cases on 0597d94 too, and their gfx942 cases skip there.

File	Compared against
test_dsv41_gfx942_topk.py	vLLM's top_k_per_row_decode and its candidate mask, for contexts from 100 to 300,000 tokens
test_dsv41_gfx942_cand_logits.py	AITER's dense logits at the candidate positions, bit for bit
test_dsv41_gfx942_sparse_indexer.py	vLLM's ROCm indexer op on a decode step, for a layer without candidate blocks, for layer 20, which writes them, and for a layer that reads them. The gfx942 indexer layer runs its own copy of that op's decode path, and this test fails when the two stop agreeing
test_dsv41_gfx942_decode_meta.py	the old SWA metadata ops and top-k packing, on 800 random batches
test_dsv41_gfx942_fused_markov.py	vLLM's Markov sampling loop, and each draft token against vLLM's _sample_logits on the same step's logits, exactly
test_dsv41_gfx942_draft_lm_head.py	an fp32 GEMM. It also checks that wvSplitK gets no 5-row GEMM with --monokernel and keeps it without
test_dsv41_mono_gfx942.py	torch references for K2's key lists, the first layer's seam, the GEMV cores and the gfx942 FP4 and FP8 conversions
tests/models/test_dsv41_mono_layers.py	vLLM's own decoder layers in a five-layer model with random weights, at TP4 with the FP8 indexer, at steps of 8 rows (two GEMV passes, the second one padded) and 42 rows (the limit), every layer at every decode step
tests/models/test_dsv41_mono_numerics.py	the composed runner at TP4 on the copies made at load: the padded step against its rows run alone, and 300 replays of steps of 6, 8 and 42 rows in one CUDA graph against the first run, bit for bit

On f0a549d a TP4 server with the mono path also answered the sanity prompts correctly, one at a time and 7 at once, and three GSM8K runs at concurrency 7 gave strict match 0.9303, 0.9272 and 0.9257. On the rebased commit pre-commit passes, and the 60 tests of tests/models/test_dsv41_mono_decode.py pass on CPU. On a2ce22d, which merges main into the rebased commit, the new MI300 job passes in CI, and a TP4 server with the mono path ran GPQA-Diamond, AIME25 and NIAH (see Accuracy). The commits after it merge main again and follow four changes on main: #58689, which renamed the V4.1 ROCm attention classes, #55090, which split the skinny GEMM check, #58800, which moved the gfx1250 check out of rocm_unquantized_gemm_impl, and #61161, which mixes the draft step into the seed of the draft sampler. The fused Markov launch now mixes in its draft step too, so its draft tokens stay the same as those of vLLM's loop.

The latest commits move the gfx942 top-k launches into an indexer layer in the model folder, add a test that compares that layer with vLLM's indexer op, drop the _prepare_dflash_inputs_kernel change, and merge main with #62131. On 6cb1400 pre-commit passes, the 76 tests of tests/models/test_dsv41_mono_decode.py pass on CPU, and the kernel tests and the two TP4 files pass on MI325X with the ROCm 7.2 nightly and with the ROCm 10 nightly. The new indexer test fails on a copy of the layer whose logits take the head weights of the wrong rows. With the ROCm 7.2 nightly a TP4 server with the mono path answered the sanity prompts correctly, one at a time and 7 at once, and three GSM8K runs at concurrency 7 gave strict match 0.9310, 0.9303 and 0.9295. With the ROCm 10 nightly such a server also answered the sanity prompts correctly, and GSM8K gave 0.9295, 0.9272 and 0.9242. In one of three starts with the mono path on ROCm 10 the server hung before it was ready: three ranks stopped in AITER's custom all-reduce while it registered the buffers of the decoder replay CUDA graphs, which main also captures. On gfx950 the 76 builds compile, see Claims, and on MI355X a TP2 server with the mono path ran the DSpark draft layers whole with the fix of #62131's scratch and answered the sanity prompts correctly.

Reproduce

Server, the same for both arms apart from --monokernel, which the main arm leaves out:

export VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1 AITER_TRITON_LOG_LEVEL=ERROR
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 VLLM_ROCM_FLYDSL_QR_INT4=1
export VLLM_DSV41_FUSED_COMBINE_TOPK_SWA=1 VLLM_USE_BREAKABLE_CUDAGRAPH=1
export VLLM_USE_RUST_FRONTEND=1 VLLM_USE_V2_MODEL_RUNNER=1 OMP_NUM_THREADS=1
export VLLM_ENGINE_READY_TIMEOUT_S=3600
vllm serve deepseek-ai/DeepSeek-V4.1-Flash --tensor-parallel-size 4 --monokernel \
  --language-model-only --tokenizer-mode deepseek_v41 --tool-call-parser deepseek_v41 \
  --enable-auto-tool-choice --reasoning-parser deepseek_v41 \
  --moe-backend aiter_triton_mxfp4_bf16 --gpu-memory-utilization 0.9 \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"synthetic","synthetic_acceptance_length":3.51,"enable_adaptive_verification":false}' \
  --max-model-len 1048576 --max-num-seqs 128 --max-num-batched-tokens 16384 \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' --max-cudagraph-capture-size 1024 \
  --engram-config '{"cpu_offload":true}'

GSM8K uses "rejection_sample_method":"block" instead of the two synthetic keys, with lm_eval 0.4.9.2 and num_concurrent=1 for the concurrency 1 runs:

lm_eval --model local-completions --tasks gsm8k --num_fewshot 5 \
  --model_args model=deepseek-ai/DeepSeek-V4.1-Flash,base_url=http://127.0.0.1:8000/v1/completions,num_concurrent=7,max_retries=3,tokenized_requests=False \
  --gen_kwargs max_gen_toks=512,temperature=0.0,top_p=1.0

AgentX is the InferenceX AgentX client (agentx-standalone.md) with the arguments of the InferenceX AgentX sweep and --concurrency 1, 2, 4 or 8. InferenceX's python -m infx.results.agentic.process_agentic_result scored each run with TP=4 GPU_COUNT=4.

The gfx950 runs, with the same server for both arms:

export VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1 VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4
export VLLM_USE_BREAKABLE_CUDAGRAPH=1 VLLM_USE_V2_MODEL_RUNNER=1 VLLM_ENGINE_READY_TIMEOUT_S=3600
vllm serve deepseek-ai/DeepSeek-V4.1-Flash --tensor-parallel-size 2 --monokernel \
  --language-model-only --tokenizer-mode deepseek_v41 --moe-backend aiter --gpu-memory-utilization 0.9 \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":false}' \
  --max-model-len 32768 --max-num-seqs 64 --max-num-batched-tokens 16384 \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' --engram-config '{"cpu_offload":true}'

The speed rows are python3 -m sglang.bench_serving --backend vllm --dataset-name random --random-input-len 1024 --random-output-len 1024 --random-range-ratio 1.0 --request-rate inf, with --max-concurrency 1 --num-prompts 16 and with --max-concurrency 8 --num-prompts 64. GSM8K is the lm_eval command above with num_concurrent=8 and --gen_kwargs max_gen_toks=2048.

Kernel tests on one GPU, and the TP4 tests on four:

pytest -v -s tests/kernels/models/amd/
pytest -v -s tests/models/test_dsv41_mono_numerics.py tests/models/test_dsv41_mono_layers.py

Details

• Layers. gfx942 runs every backbone layer whole, as gfx950 does since [ROCm][Perf][DSv4.1] Mono decode on every backbone layer #61082. The index layers use vLLM's FP8 indexer, because the MXFP4 indexer cache needs CDNA4. vLLM's compressor and indexer run between K1 and K2, as on gfx950. On gfx942 mono/keylist.py then writes the layer's key lists, so K2 is the build without the key table. On gfx942 the 3 DSpark draft layers take the FFN launch after vLLM's attention, built for the draft's 128 experts and top 3, on the target's runner. On gfx950 they run whole on a runner of their own, as [ROCm][Perf][DSv4.1] Mono decode for the DSpark draft layers #62131 added.
• gfx942 helper ops in vllm/models/deepseek_v41/amd/ops/: the sparse indexer's decode top-512 (topk512_gfx942.cu) and its candidate logits, the SWA decode metadata and the top-k packing as one launch each, and DSpark's Markov sampling as one launch a draft step. With --monokernel on gfx942 the V4.1 ROCm attention takes the indexer layer of sparse_indexer.py. It runs vLLM's ROCm indexer op on every step except a step of only decode rows, which runs the decode path of that op with the top-k launches in front of vLLM's ops. The DSpark speculator asks the draft model's sample_sequential first, and the DSv4.1 draft model runs the fused Markov launches there. draft_lm_head.py pads the DSpark draft's 5-row logits GEMM to 6 rows, so that vLLM's dispatch takes the torch GEMM instead of wvSplitK. For the 32320 × 5120 lm_head shard that takes 96 µs instead of 295 µs on MI325X. Each helper leaves a call it cannot take to vLLM's own op. topk512_gfx942.cu is built with hipcc when the model is created. If the build fails, a warning is logged and the indexer runs vLLM's top-k ops.
• Weight copies. The routed experts are copied in the byte order of the gfx942 GEMVs, without vLLM's padding columns, and the dense linears as e4m3 bytes, because vLLM dequantizes those to bf16 on gfx942. The copies are made when vLLM processes the weights at load, so vLLM counts them as model memory when it sizes the KV cache. A reload writes the new copies into the first ones in place, because the kernels and the captured CUDA graphs hold their addresses.
• Changes outside the model directory. One hook in the DSpark speculator: _sample_sequential asks the draft model's sample_sequential first, if the model has one, and runs its own loop when the call returns False. Only the DSv4.1 draft model has one, and it takes a step only with --monokernel on gfx942. The rest are the two CI jobs. The earlier hooks in rocm_aiter_mla_sparse.py are now the indexer layer of sparse_indexer.py. The change of the padding loops of _prepare_dflash_inputs_kernel is dropped, because [Perf][Spec Decode] Give the DFlash padding loops their own block width, 2048 -> 16 iterations #58633 parallelises the same padding.
• Merge of [ROCm][Perf][DSv4.1] Mono decode for the DSpark draft layers #62131. [ROCm][Perf][DSv4.1] Mono decode for the DSpark draft layers #62131 runs the DSpark draft layers whole on gfx950, on a runner of their own whose MoE takes one FP4 group an ug task (DRAFT_UG_GROUPS). Its scratch was laid out for the default two groups, and with one group the ug flags take twice the room. So the flags of the upper half of the tasks were written over the start of the normed rows that the MoE reads as its input, at TP 2 and 4 and for every step width. Only the draft layers could be affected, not the target's output, and in a GSM8K check on MI355X main accepted as many drafts as this PR, so the overlap did not show there. scratch_layout and scratch_bytes now take the runner's groups. run_ffn checks that each MoE region of a build fits the layout, and that check found the overlap when the draft runner's K2 was built. The FFN launch reads the target's layout, because it runs on the target's runner, also for a gfx942 draft layer.
• Since the first review. The branch is rebased onto main as one commit. The gfx942 ops moved from vllm/model_executor/layers/dsv41_gfx942/ into vllm/models/deepseek_v41/amd/ops/, as RFC [RFC]: A standard for Specialized (MonoKernels) Decodes in vllm/models #60904 asks for model kernels. VLLM_ROCM_MONO_SHADOW and its layer-by-layer check are removed. The rebase onto [ROCm][Perf][DSv4.1] Mono decode on every backbone layer #61082 takes its switch and its layer plan on gfx942 too. That removes the gfx942 paths of their own for the first layer, the window layers and the index layers, apart from mono/keylist.py.
• Since the second review. The kernel tests moved to tests/kernels/models/amd/. The fused Markov test checks each draft token against vLLM's _sample_logits on the same step's cached logits. The change to vLLM's skinny GEMM dispatch is reverted, and only the DSpark draft's lm_head avoids wvSplitK, with --monokernel on gfx942. test_dsv41_mono_layers.py and a gfx942 graph replay test in test_dsv41_mono_numerics.py run at TP4 in a new four-GPU MI300 job. On gfx942 the first layer's output bound in test_dsv41_mono_layers.py is 6% instead of 3%, because vLLM's gfx942 MoE keeps bf16 activations and the kernels quantize them to MXFP8, which alone moves the FFN output of the test's random weights by 4.65%. The residual stream then differs by up to 2.5% instead of 0.4 to 0.7%, so past the first layer a routing near-tie there is a gap of up to 0.02 instead of 0.01, and up to a third of the tokens may flip an expert instead of a quarter (22% seen).
• Relation to other PRs. Draft [ROCm][Perf][DSv4] Standalone mono decode layer for DeepSeek-V4-Pro on MI355X #61482 moves the FlyDSL stages to vllm/models/common/amd/mono/, and this PR edits the files it moves. Whichever of the two merges second needs a rebase. If this PR merges first, its gfx942 tests can check [ROCm][Perf][DSv4] Standalone mono decode layer for DeepSeek-V4-Pro on MI355X #61482's move on MI325X. [ROCm][Perf][DSv4.1] Mono decode for the DSpark draft layers #62131 is merged into this branch, see Merge of [ROCm][Perf][DSv4.1] Mono decode for the DSpark draft layers #62131. [Perf][Spec Decode] Give the DFlash padding loops their own block width, 2048 -> 16 iterations #58633 parallelises the padding of _prepare_dflash_inputs_kernel, which this PR changed until the latest commits, and neither PR needs the other. [Core] Shared abstractions for specialized decode kernels #61311 moves amd/mono/ to amd/specialized_decode/mono/ and replaces mono_decode.py, as RFC [RFC]: A standard for Specialized (MonoKernels) Decodes in vllm/models #60904 describes. A trial merge of the two conflicts in 15 files, among them mono_decode.py and tests/models/test_dsv41_mono_decode.py, which [Core] Shared abstractions for specialized decode kernels #61311 deletes and this PR changes. So whichever of the two merges second needs a port. [Perf][Spec Decode][DSv4.1] DSpark draft on the specialized decode, with a fused Markov loop and a post-forward CUDA graph #61869 adds a fused Markov sampler at the start of _sample_sequential, next to this PR's hook, and the two merge there without a conflict.
• Follow-ups. Steps of more than 42 rows on gfx942, so that 8 or more concurrent requests with 5 drafts also take the mono path. One copy of the routed experts instead of two. Other TP sizes on gfx942. The whole DSpark draft layers on gfx942, as on gfx950. topk512_gfx942.cu as a compiled op in csrc/.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

This PR was written with AI assistance (Cursor). The test commands and their results are under Validation.

[mawong-amd]

Preliminary AI-assisted review. The analysis and testing were done with Claude Code (Opus 5.5 and GLM 5.3); I reviewed the results.

As on #60397, this is an actively developing area, so these are suggestions ranked by how strongly I'd recommend them, not a checklist. The #60397 review comments carry over here and aren't repeated; its author has since pushed the deployment checks and test hardening there (dea9fa3, b9423a8), which this branch doesn't have yet.

Thanks for this, and for the very thorough description. Tested on MI300X (192 GiB, 4 GPUs): this branch installed into vllm/vllm-openai-rocm:nightly-rocm100-8cbd5d0300 (covers the merge-base). All 237 new gfx942 kernel tests pass on MI300X. E2E with the description's server command, adapted (block rejection, --max-model-len 65536, --max-cudagraph-capture-size 96):

	mono off	mono on
Weights per rank	98.88 GiB	169.07 GiB
KV cache	63.79 GiB, 4.15M tokens (util 0.9)	fails at util 0.9 (−8.25 GiB); 2.31 GiB, 149K tokens at 0.955
Single request, 512 tokens, mean ITL	6.10 ms	2.70 ms (2.26×)
Kernels per decode period (torch profiler, rank 0)	1,885	342; K1 ~24 µs, K2 93–97 µs, draft FFN launch 52 µs
GSM8K 5-shot, first 400, greedy, concurrency 1	0.930 ± 0.013	0.950 ± 0.011

So the speedup, kernel counts and accuracy match the description on MI300X too.

Would like to see before merge

1. topk512_gfx942.cu is JIT-built with torch.utils.cpp_extension.load at the first decode step. vLLM builds its kernels through csrc/ and CMake; a runtime hipcc build needs hipcc and a writable extension cache at serve time, fails mid-serving instead of at startup, and has no fallback. Please move it into csrc/ (or at least build at startup with a fallback to top_k_per_row_decode).

2. tests/models/test_dsv41_mono_decode.py fails 6 of 26 with this PR, none of them hardware-dependent: the CDNA message is now "needs CDNA3/4", the mocks lack layer.engram, runner.ffn() takes topk, and a path needs the TP group. It runs in the MI355 Basic Models step, so CI goes red once it runs.

3. MI300X doesn't really fit. With the copies, the description's own --gpu-memory-utilization 0.9 fails at startup on MI300X; the ceiling is ~0.955 (only ~184 of 192 GiB free at worker startup), leaving 2.31 GiB of KV — 149K tokens, about two 64K requests, and not one at the recipe's 1M context. Most of the +70 GiB is the routed-expert copies (~69 GiB). I'd scope the claims to MI325X, or document the MI300X limits.

4. The rebase onto #60397's new head needs care. The parent's new MonoLayerWeights.check asserts the gfx950 layout (fp4x2 w13, E8M0 block scales, padded intermediate); on gfx942 the tensors are the weights942 copies (FNUZ bytes, tile-major, padding cut), so the check needs a gfx942 variant or the copies go unchecked — the exact gap the parent's fix closed. The unit test file also changed there (new MoE-rejection and padded-step cases), so it's a reconcile, not just fixing the 6 failures.

Recommended, fine as follow-ups

5. Reloaded weights don't reach the copies. _capture_plain_experts skips the rebuild when mono942 exists, and the dense copies are cached on first use, so after reload_weights or an RL weight update, prefill uses the new weights while ≤6-row decode silently uses the old ones. Captured graphs hold the copies' pointers, so a fix would copy_ into them in place.

6. The two changes outside the switch (the wvSplitK ≤4 rows on gfx942 for every model, and the DFlash padding on every platform) each have a single measurement behind them. They'd be easier to land and to bisect as their own small PRs.

Nits

• The dense copies (~1.8 GiB) are built at the first eager decode step, which by default is the cudagraph memory-profiling pass, so they're counted as graph memory (estimate 5.74 GiB with mono vs 3.40 without). That works, but only by accident: with graph-memory profiling off or enforce_eager they'd escape accounting. Building them at load would be clearer.
• common/arch.py decides GFX942 from the ARCH env var, while _on_gfx942() asks the platform. A stray ARCH (a generic name) can make the weight copies and the kernel builds disagree.
• A gfx942 twin of the parent's new 300-replay graph stress (both widths, one graph, bit for bit) would guard the FNUZ copies' path; the shadow mode (VLLM_ROCM_MONO_SHADOW) already provides the harness.

For what it's worth, the FNUZ re-encoding traced as exact: the bf16 → e4m3 round trip, the halved bit patterns with scale code +1, subnormals and −0. The checkpoint's scale codes (dense 109–121, experts 115–127) are well clear of the E8M0 edge cases.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @maeehart.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[maeehart]

/ci run

[github-actions]

✅ Triggered Buildkite CI #95642 for commit 0f80888cb2d8.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @maeehart.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[maeehart]

@tjtanaa @Fangzhou-Ai @dllehr-amd I pushed 6cb1400. This PR now leaves common code alone, as RFC #60904 asks:

• The gfx942 top-k launches moved out of rocm_aiter_mla_sparse.py into an indexer layer in the model folder (ops/sparse_indexer.py). It runs vLLM's ROCm indexer op on every step except a step of only decode rows. The new test_dsv41_gfx942_sparse_indexer.py runs the layer and vLLM's op on the same decode step and fails when the two stop agreeing.
• The DSpark speculator no longer imports DSv4.1 code. It asks the draft model's sample_sequential first, and only the DSv4.1 draft model has one.
• The _prepare_dflash_inputs_kernel change is dropped. [Perf][Spec Decode] Give the DFlash padding loops their own block width, 2048 -> 16 iterations #58633 parallelises the same padding loops, and this PR does not need it.

Outside vllm/models/deepseek_v41/ and the tests, the PR changes only that 8-line hook in the DSpark speculator and adds the two MI300 CI jobs.

The branch also merges main with #62131. On gfx950 the draft layers run as #62131 runs them. On gfx942 they keep the FFN launch after vLLM's attention.

@Fangzhou-Ai, the merge found a bug in #62131. Its draft runner builds the MoE with DRAFT_UG_GROUPS = 1, but scratch_layout sizes the MoE regions for the default two groups, and with one group the ugf flags take twice the room. So the flags of the upper half of the ug tasks are written over the start of the normed rows, which the draft MoE reads as its input, at TP 2 and 4 and for every step width. The target's output is not affected. c688ebe passes the runner's ug_groups to scratch_layout and scratch_bytes. This PR checks that each MoE region of a build fits the layout, and that check stopped the build of the draft runner's K2, so the fix has to come with this PR. On MI355X at TP2 with --monokernel, servers of main and of c688ebe both ran the draft layers whole and answered the sanity prompts correctly, and on GSM8K they accepted as many drafts, so the overlap did not show in the drafts there.

The checks of 6cb1400 are at the end of Validation in the description.

[maeehart]

/ci run

[github-actions]

✅ Triggered Buildkite CI #95911 for commit 4d0efb49acbc.

ℹ️ This PR is 42 commits behind upstream main. Merged upstream main into this branch as 4d0efb49acbc before starting CI. Pull the branch before pushing more changes.

[maeehart]

/ci run

[github-actions]

❌ CI already ran on this commit: https://buildkite.com/vllm/ci/builds/95911. No new CI build was started. Push new changes to run CI again, use /ci retry to rerun failed jobs, or ask a reviewer with write access to rerun it.

[maeehart]

/ci run

[github-actions]

✅ Triggered Buildkite CI #95974 for commit 2ee39949d9b7.

ℹ️ This PR is 1 commit behind upstream main. Merged upstream main into this branch as 2ee39949d9b7 before starting CI. Pull the branch before pushing more changes.