[DeepSeek][ROCm] Align V4.1-Flash MI355X image and MoE backend

[chunfangamd]

Summary

• Pin the AMD image to vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657. The deepseekv41-flash-0909 tag predates [ROCm][DSV4.1][Perf] Use AITER mHC for the delayed pre block vllm#56503, which moves the mHC delayed pre block off the eager Torch reference and onto AITER.
• Pass plain --moe-backend aiter so vLLM's priority list selects the CK a8w4 experts rather than the Triton W4A16 _moe_gemm_a16w4 kernel that the specific backend name pins.
• Set AITER_TRITON_LOG_LEVEL=ERROR for the gfx950 Gluon-fallback warning spam. Log hygiene, not a throughput claim.
• Record mi355x: verified, and pin --max-num-seqs 128 for single_node_tp on MI355X only so the published command reproduces the verified run (reasoning below).
• Make the Docker install note and the guide's Prerequisites vendor-aware: the MI355X page no longer tells AMD users to pull the NVIDIA image, and both tags render in full.
• Preserve the AMD DSpark enable_adaptive_verification:false override merged in [DeepSeek] V4.1-Flash: AMD DSpark without adaptive verification #963, byte-for-byte.

Corresponding InferenceX change: SemiAnalysisAI/InferenceX#3058 (head 559ef7560a74d0fb8da64fdfc0c357ad2fc33c4f).

This is a clean replacement for #962, branched from vllm-project/recipes:main at dde68254124431b82d66c816eca36e4e6d24e47d — the #963 merge commit. It reuses none of #962's commits or its merge base, so it applies without the conflict #962 carries.

Scheduler-default correspondence

#3058 pins MAX_NUM_SEQS=128. This recipe emits the flag, scoped to MI355X + single_node_tp, because omitting it is not equivalent here.

At the pinned vLLM commit eed1f3d0c6, SchedulerConfig.DEFAULT_MAX_NUM_SEQS is indeed 128 — but that constant is only the fallback EngineArgs._set_default_max_num_seqs_and_batched_tokens_args uses when the usage context is missing from get_batch_defaults(). For vllm serve (UsageContext.OPENAI_API_SERVER) on a GPU reporting at least 160 GiB — MI355X is 288 GB — that map returns max_num_seqs = 1024 and max_num_batched_tokens = 16384. performance_mode defaults to balanced, so neither is doubled, and the closing min(1024, 16384) clamp leaves 1024. An unpinned MI355X command would therefore serve 8x the scheduler capacity the verified run used.

Scope is strategy_overrides.single_node_tp.hardware_overrides.mi355x.extra_args, which buildArgs applies as an additive exact-GPU layer, so it does not replace the recipe-level amd block. The rendered matrix below confirms nothing else moves. If you would rather the recipe track vLLM's default and leave reproducibility to the guide, I am happy to drop this hunk — it is self-contained.

Validation

• node scripts/build-recipes-api.mjs: ✓ JSON API: 190 models (165 with recommended_command, 797 default-hw alternatives, 1638 per-hw renderings), 193 promoted variants, 10 strategies, 2 kv-store deployments, 2 platforms. No errors, no synthesis warning for this recipe.
• pnpm lint: not run. next lint finds no committed ESLint config and drops into an interactive setup prompt, exiting 1. Pre-existing on main and unrelated to this change.
• git diff --check: clean. One file changed, +33/-7. No public/, node_modules/, or site/ output staged.

Generated MI355X / single_node_tp / speculative decoding + text-only on:

VLLM_ENGINE_READY_TIMEOUT_S=3600 VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1
AITER_TRITON_LOG_LEVEL=ERROR VLLM_USE_BREAKABLE_CUDAGRAPH=1 VLLM_USE_RUST_FRONTEND=1

vllm serve deepseek-ai/DeepSeek-V4.1-Flash \
  --tokenizer-mode deepseek_v41 \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.9 \
  --moe-backend aiter \
  --max-num-seqs 128 \
  --tool-call-parser deepseek_v41 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v41 \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":false}' \
  --language-model-only

Exactly one --moe-backend, exactly one --speculative-config, and no synthetic-acceptance fields. Install → Docker on MI355X renders docker pull vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657 — full tag, no ellipsis.

Regression checks through the same builder:

Profile	Result
MI350X single_node_tp	Identical to MI355X except no --max-num-seqs; existing verification unaffected
MI355X TEP / multi-node TP / PD	No --max-num-seqs 128 leak; PD keeps its own 32 per pool
GB200 and H200 single_node_tp, GB200 PD	Image still vllm/vllm-openai:deepseekv41-flash-0909, no AMD env vars, enable_adaptive_verification:true

MI355X runtime evidence

InferenceX run 34710937012 (attempt 2), job agentic eval / … cluster:mi355x-amds vllm TP4 … | eval-only. Its launch line matches the serving subset above — same nightly image, MI355X, TP4, --moe-backend aiter, --gpu-memory-utilization 0.9, --max-num-seqs 128, and DSpark with real block rejection and enable_adaptive_verification:false. Server reached ready and served the eval:

gsm8k 5-shot — strict-match 0.9689 ± 0.0048, flexible-extract 0.9682 ± 0.0048, both PASS.

The agentic arms at concurrency 1/2/4/8/16/32 also completed on the same image and hardware, but they set rejection_sample_method: synthetic with a fixed acceptance length for throughput measurement. That is deliberately not represented here: the public AMD path keeps real block rejection, which is the arm cited above.

The CK-path decode and latency deltas are recorded in #3058's perf-changelog.yaml — MoE GEMM 2.72 → 2.30 ms/step at concurrency 1, 4.91 → 4.27 at 4, 9.06 → 7.18 at 16, for whole-step gains of 4.2% / 5.3% / 8.7%, and AgentX concurrency 1 mean ITL 4.01 → 3.91 ms with mean TTFT 681 → 639 ms. Those traces are InferenceX's measurement, not something I re-derived; the gsm8k figures above are read directly from the linked job log.

The benchmark script blob is identical at the run's head (ab76c131) and at #3058's current head (559ef756), so this evidence maps to the configuration #3058 ships today.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
vllm-recipes	Ready	Preview	Sep 13, 2026 4:35pm UTC

[gemini-code-assist]

Code Review

This pull request updates the DeepSeek-V4.1-Flash model configuration to support AMD ROCm nightly docker images, adds support for the MI355X hardware, switches the MoE backend to aiter to leverage Composable Kernel performance, and introduces hardware-specific overrides like pinning --max-num-seqs to 128 for MI355X. Feedback points out that the AMD hardware overrides are missing critical ROCm execution and optimization arguments (such as --distributed-executor-backend mp, --compilation-config, and --max-num-batched-tokens 8192) which are necessary for stable and optimized multi-GPU execution.

[gemini-code-assist]

The hardware_overrides.amd.extra_args block is missing critical ROCm execution and optimization arguments that are used in the verified MI355X validation command in the guide (lines 956-971) and are standard across other ROCm recipes (like DeepSeek-V4-Flash.yaml). Specifically, it should include --distributed-executor-backend mp (required for stable multi-GPU execution on ROCm) and --compilation-config '{"mode": 3, "cudagraph_mode": "FULL_DECODE_ONLY"}' (critical for compilation and decode performance), as well as --max-num-batched-tokens 8192.

Without these, the synthesized command generated by the UI/API for AMD users will be unoptimized and may fail or hang during multi-GPU execution.

    extra_args:
      - "--distributed-executor-backend"
      - "mp"
      - "--gpu-memory-utilization"
      - "0.9"
      - "--max-num-batched-tokens"
      - "8192"
      # Plain "aiter" opens vLLM's full MoE priority list, whose head is the
      # Composable Kernel backend, giving the a8w4 experts. Naming
      # aiter_triton_mxfp4_bf16 instead pins the Triton W4A16 _moe_gemm_a16w4.
      - "--moe-backend"
      - "aiter"
      - "--compilation-config"
      - '{"mode": 3, "cudagraph_mode": "FULL_DECODE_ONLY"}'

[chunfangamd]

Checked all three against the run this recipe is verified against and against vLLM at the pinned commit (eed1f3d0c6). Declining them.

--compilation-config '{"mode": 3, "cudagraph_mode": "FULL_DECODE_ONLY"}' — mode 3 is CompilationMode.VLLM_COMPILE, the Inductor-based piecewise-compilation path. DeepseekV41ForCausalLM is not torch-compiled upstream, which is precisely why this block sets VLLM_USE_BREAKABLE_CUDAGRAPH=1 a few lines below, and why the ROCm sparse SWA backend reporting UNIFORM_BATCH forced the feature-level DSpark override merged in #963. Adding this would abort at engine init, not improve decode.

--distributed-executor-backend mp and --max-num-batched-tokens 8192 — neither appears in the verified MI355X launch line (benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh in SemiAnalysisAI/InferenceX#3058). That script passes --max-num-batched-tokens 16384, which is also vLLM's own default for UsageContext.OPENAI_API_SERVER on a GPU reporting ≥160 GiB (EngineArgs.get_batch_defaults), so 8192 would halve the verified value rather than match it.

On the cited precedent: DeepSeek-V4-Flash.yaml is a different architecture, and its AMD block carries those flags legitimately. V4.1-Flash is the one without torch.compile support. None of the three are in V4.1-Flash's AMD block on main today either, so adding them would be outside this PR's scope — aligning the existing recipe with #3058.

[ivanium]

LGTM with a small nit. Approved to unblock merging

[ivanium]

nit: maybe we can also add --max-cudagraph-capture-size 1024 for dspark 6 for better perf. Not a blocker for merging this PR though.

[chunfangamd]

Thanks, @ivanium, for the suggestion! The current recipe is to align with the sweep run SemiAnalysisAI/InferenceX#3058. We will test --max-cudagraph-capture-size 1024
cc: @Fangzhou-Ai

[Fangzhou-Ai]

I dig into it a little bit, I think this setting is for benchmarking purposes only. In general we don't need the cap on max-num-seqs and max-cudagraph-capture-size. Mayabe we should revise this PR into a benchmark-only recipe? @ivanium @AndreasKaratzas

[Fangzhou-Ai]

But we do want to pin the image to the latest nightly now, DSV4.1 has been enabled since this Saturday.

[ivanium]

how about we add a perf tuning section in the overview on how to set these parameters and potentially link to the InferenceX submission?

[Fangzhou-Ai]

yep that's what I'm (my codex) planning to do now.

[Fangzhou-Ai]

please check #968