[DeepSeek] V4.1-Flash: point the AMD image at a ROCm nightly and select the CK a8w4 MoE experts

[Fangzhou-Ai]

Summary

Two AMD-only changes for DeepSeek-V4.1-Flash. NVIDIA is unchanged.

   docker_image:
     nvidia: "vllm/vllm-openai:deepseekv41-flash-0909"
-    amd: "vllm/vllm-openai-rocm:deepseekv41-flash-0909"
+    amd: "vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657"

   hardware_overrides:
     amd:
       extra_args:
         - "--moe-backend"
-        - "aiter_triton_mxfp4_bf16"
+        - "aiter"

Why the nightly image

The deepseekv41-flash-0909 tag predates vllm-project/vllm#56503, which moves the mHC
delayed pre block off the eager Torch reference and onto AITER. Without it, MI355X runs
the hyper-connection seams as unfused eager Torch ops.

tag	vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657
digest	sha256:960228cfcb5de9f4cd22d28998d1125be62b546c3d220a884f570343e99ffcee
published	2026-09-12T05:27:48Z
vLLM commit	eed1f3d0c6043bd494424a22443ee198dd56f657

eed1f3d is 11 commits ahead of, and 0 behind, #56503's merge commit, so the change is
included.

Why plain aiter rather than a named kernel

aiter_triton_mxfp4_bf16 maps to exactly one entry, the Triton W4A16 kernel, so decode
ran _moe_gemm_a16w4. Plain aiter opens vLLM's full priority list, whose head is the
Composable Kernel backend, and the experts become the a8w4 family:

mfma_moe1_silu_mul_afp8_wfp4_bf16_t32x128x256_pm1_async_gui_v33.kd
mfma_moe2_afp8_wfp4_bf16_cshuffle_t32x128x128_vscale_fix3_fp4opt_v1_pm1.kd

Two things make this non-obvious, so the flag carries a comment: the backend is named
..._BF16, and this checkpoint declares activation_scheme: dynamic with no
input_scale tensors. CK quantizes activations to FP8 internally regardless.

Decode traces on MI355X TP4 at 131k context:

concurrency	MoE GEMM (ms/step)	whole step
1	2.72 → 2.30	−4.2%
4	4.91 → 4.27	−5.3%
16	9.06 → 7.18	−8.7%

gsm8k strict-match is unchanged at 0.9719.

Log level

AITER_TRITON_LOG_LEVEL=ERROR is added to the AMD env. Every warning aiter.ops.triton
emits is about Gluon availability, and Gluon supports only gfx1250 — on gfx950 that is a
fixed property rather than a condition worth reporting. It accounted for 98% of the lines
in a server log (411k of 417k lines, 30 MiB of 32 MiB) with no throughput effect.

Validation

scripts/build-recipes-api.mjs could not be run — the host has Node but no npm/pnpm, so
js-yaml is unavailable. The file was validated by parsing it with PyYAML instead,
confirming that it loads, that both docker_image entries resolve, that the AMD override
resolves to --gpu-memory-utilization 0.9 --moe-backend aiter, and that the edits inside
the guide: | literal block preserve the block's indentation. No schema keys were added
or removed.

Runtime on MI355X TP4 with DSpark drafting 5 tokens: the CK path serves correctly on this
nightly, and single-user decode measures 265 tok/s p50 interactivity against 259 tok/s
for the Triton kernel on the same box and image. Broader concurrency validation is tracked
downstream.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
vllm-recipes	Ready	Preview	Sep 12, 2026 9:57pm UTC

[gemini-code-assist]

Code Review

This pull request updates the AMD Docker image for the DeepSeek-V4.1-Flash model to a specific ROCm nightly tag to support the AITER mHC path, and updates the prerequisites guide accordingly. The reviewer suggests using the full image tag instead of a truncated version in the guide text to ensure it is fully copy-pasteable.

[gemini-code-assist]

The image tag in the prerequisites guide is truncated with an ellipsis (...). To ensure the guide is fully copy-pasteable and accurate for users who want to manually pull or verify the image, please use the full image tag vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657.

    this architecture, so the Install block only offers Docker. On AMD, use the
    `vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657` image instead — it carries the AITER
    mHC path from vllm-project/vllm#56503, which the 0909 tag predates.

[maeehart]

I served this same digest (sha256:960228cfcb5de9f4cd22d28998d1125be62b546c3d220a884f570343e99ffcee, vLLM eed1f3d0c) on MI355X TP4 with the recipe DSpark JSON (enable_adaptive_verification:true). Weights load. Then maybe_create_adaptive_verification_manager dies during initialize_kv_cache with:

ValueError: Adaptive verification trims verification requests on device, which the DeepseekV41IndexerBackend attention backend does not support. Pass enable_adaptive_verification=false in the speculative config, or use a backend that does.

DeepseekV41IndexerBackend.supports_device_cpu_query_lens_mismatch() is False on ROCm. The helpers it inherits (_supports_varlen_paged_mqa_logits and _supports_flattened_device_query_lens in vllm/v1/attention/backends/mla/indexer.py) are CUDA plus DeepGEMM only (SM100 / SM90). HIP rocm_aiter_sparse_attn_indexer already consumes flattened decode_metadata. The engine still refuses to start because that capability helper is False.

So this nightly plus the published Speculative decoding tick cannot start on AMD. NVIDIA is unchanged.

Until that helper is True on ROCm (vllm-project/vllm#56620), the AMD command needs enable_adaptive_verification:false. DSpark itself still works. Do not put the replacement JSON in hardware_overrides.amd.extra_args. Features emit last in resolveCommand, so dedupeArgs would keep the feature's true and drop the extra_args value. The supported override is a feature-level hardware replacement:

  spec_decoding:
    hardware_overrides:
      amd:
        args:
          - "--speculative-config"
          - '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":false}'

The image pin for vllm-project/vllm#56503 is still the right move. The generated AMD command with Speculative decoding ticked is not, on this digest. Follow-up with that override: #963.

[maeehart]

Follow-up: #963 puts the AMD enable_adaptive_verification:false replacement on features.spec_decoding.hardware_overrides, which is the path resolveCommand actually honors.

[Fangzhou-Ai]

Thanks @maeehart — confirming your finding independently, and agreeing with both the diagnosis and the fix location.

We hit the same wall on MI355X TP4 with this digest while validating the pin. Our InferenceX benchmark script for this model already pins enable_adaptive_verification:false in the DSpark config for exactly this reason; the comment above it references the run where the engine refused to start. So this is not specific to your setup — it reproduces on any AMD launch of this digest with the feature left at its default.

Your reading of the config layering matches ours too: since features emit last in resolveCommand, dedupeArgs would keep the feature's true and silently drop anything placed in hardware_overrides.amd.extra_args. The feature-level features.spec_decoding.hardware_overrides.amd.args replacement in #963 is the path that actually takes effect.

One sequencing note for maintainers. #962 and #963 both edit models/deepseek-ai/DeepSeek-V4.1-Flash.yaml, so whichever lands second needs a rebase. More importantly, if #962 merges on its own, recipes.vllm.ai will generate an AMD command that cannot start whenever Speculative decoding is ticked, until #963 follows. Two ways to avoid that window, and I am happy with either:

• Merge [DeepSeek] V4.1-Flash: AMD DSpark without adaptive verification #963 immediately after [DeepSeek] V4.1-Flash: point the AMD image at a ROCm nightly and select the CK a8w4 MoE experts #962, treating them as one unit.
• Or I fold the enable_adaptive_verification:false hardware override into [DeepSeek] V4.1-Flash: point the AMD image at a ROCm nightly and select the CK a8w4 MoE experts #962 (with credit to @maeehart) and [DeepSeek] V4.1-Flash: AMD DSpark without adaptive verification #963 closes as absorbed.

@maeehart let me know which you prefer for your patch; I'll follow your lead rather than take it over.

On the upstream side, once vllm-project/vllm#56620 makes supports_device_cpu_query_lens_mismatch() True for DeepseekV41IndexerBackend on ROCm, the override should be removed again. Worth a short comment in the YAML so it doesn't outlive the reason for it.

[maeehart]

Keep #963 as its own PR. Merge it immediately after #962.

Do not fold the override into #962 expecting vllm-project/vllm#56620 to remove it soon. I served with that indexer helper forced True on ROCm. The first maybe_create check passed. The second check then raised because DeepseekV41ROCMAiterSparseSWABackend reports AttentionCGSupport.UNIFORM_BATCH rather than ALWAYS. I am closing #56620.

Hub nightly eed1f3d0c (sha256:960228cfcb5de9f4cd22d28998d1125be62b546c3d220a884f570343e99ffcee) on MI355X TP4 with enable_adaptive_verification:false did start. The recipe 17*19 curl returned 323. The YAML comment now says to remove the AMD override only after a serve with true actually starts.

[esmeetu]

@Fangzhou-Ai Can you help resolve the conflicts? Also, could we reduce the number of AI comments a bit? :)

[chunfangamd]

Superseded by #965.

#965 is branched from current main (dde6825, the #963 merge) rather than from this branch, so it applies without the conflict flagged here and preserves the features.spec_decoding.hardware_overrides.amd block from #963 byte-for-byte. Same four AMD changes, plus MI355X runtime evidence and full (non-truncated) image tags in the Prerequisites text.

@Fangzhou-Ai I don't have push access on this repo, so could you close this one? If anything here should carry over that I missed, say so and I'll fold it into #965.

[Fangzhou-Ai]

superseded by #965 close this PR now