[ROCm] Document DeepSeek-V4-Pro MI355X 8K/1K InferenceX reproduction

[chunfangamd]

Summary

Adds the exact single-node 8× MI355X fixed-sequence (8K/1K) reproduction for SemiAnalysisAI/InferenceX#2792 to the DeepSeek-V4-Pro guide. Both arms of that lane are documented: STP (no speculative decoding) and real MTP with two speculative tokens.

This updates the existing Guide only. It does not change general recipe defaults — no new model file or variant, and no edits to hardware_overrides.amd, variant defaults, or the strategy builders.

The recipe already carried a general AMD configuration and an Agentic MI355X reproduction, but neither is an exact replacement for this lane:

Setting	Fixed-sequence #2792	General AMD	Agentic MI355X
Workload	ISL 8192 / OSL 1024, conc 4–512	general	agentic-coding traces
--gpu-memory-utilization	0.8	0.9	0.86
Prefix caching	--no-enable-prefix-caching	not set	--enable-prefix-caching
CUDA graph mode	FULL_AND_PIECEWISE	FULL_DECODE_ONLY	FULL_AND_PIECEWISE
--moe-backend	explicit aiter	not set	aiter
Speculation	none, or real MTP K=2	none	synthetic MTP K=3
--max-num-seqs / --max-num-batched-tokens	not passed	512 / 8192	8 / 8192
Image	immutable Sep-1 nightly + digest	floating :nightly	not pinned

Recipe-site structural and render checks were run locally (below). The GPU performance and eval results come from the linked InferenceX run — no MI355X benchmark was rerun in this repository.

Source and validation provenance

• InferenceX PR: #2792, reviewed at head 7393456f925f4962fc9bb6438567e73f7ca12bf8
• STP script: dsv4_fp4_mi355x_vllm.sh
• MTP script: dsv4_fp4_mi355x_vllm_mtp.sh
• Config matrix: configs/amd-master.yaml — keys dsv4-fp4-mi355x-vllm and dsv4-fp4-mi355x-vllm-mtp
• Successful sweep: run 33538769698 (success)
• Image: vllm/vllm-openai-rocm:nightly-7c5dc571cbd1064ecc8a9b1045637ff647aa22cb, digest sha256:f0bdaf5217a09949842b45c1ea1f12260d3205ec81f143b320dfc2eb3ec95e55, vLLM commit 7c5dc571
• ROCm DeepSeek-V4 MTP support: vllm-project/vllm#43385 (merged 2026-05-24)

Run 33538769698 tested commit 7a209b121932696b7efd437ac50d19c53efc7a94, not the current PR head. All eight concurrency points passed on both arms, plus lm-eval GSM8K eval-only jobs at concurrency 128 and 512 (InferenceX reports em_strict 0.948–0.960, n_eff 1319, on this image).

The PR head is 28 commits ahead of the tested commit, so I compared the two directly: neither fixed-sequence script changed, and the configs/amd-master.yaml changes in that range touch only the Kimi-K3, MiniMax-M3, and DSV4-SGLang entries — not the two DSV4 vLLM keys. The configuration documented here is therefore identical at the tested commit and at the current head.

Configuration mapping

Recipe item	Value	Source
Checkpoint	deepseek-ai/DeepSeek-V4-Pro — the FP8 (Preview) variant, not the 0813 default	amd-master.yaml model:
Hardware	one node, 8× MI355X	runner: mi355x, multinode: false
Image	pinned nightly tag + digest (above)	amd-master.yaml, perf-changelog.yaml
Environment	VLLM_ROCM_USE_AITER=1, VLLM_ROCM_USE_AITER_MOE=1, VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1, VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4	both scripts
Parallelism	--tensor-parallel-size 8 --data-parallel-size 1; EP 1, no --enable-expert-parallel	PARALLEL_ARGS; no dp-attn/ep key in the config
Cache	--kv-cache-dtype fp8, --no-enable-prefix-caching	both scripts
Scheduling / executor	--async-scheduling, --distributed-executor-backend mp	both scripts
Memory	--gpu-memory-utilization 0.8	both scripts
MoE	--moe-backend aiter	both scripts
Parsing	--tokenizer-mode deepseek_v4, --reasoning-parser deepseek_v4	both scripts
Compilation	'{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"}'	both scripts
STP arm	no --speculative-config	dsv4_fp4_mi355x_vllm.sh
MTP arm	'{"method":"mtp","num_speculative_tokens":2}' (real verification)	NUM_SPEC_TOKENS=2 in the MTP script
Workload	ISL 8192, OSL 1024; concurrency 4, 8, 16, 32, 64, 128, 256, 512; 10 × concurrency requests	config isl/osl/conc-start/conc-end; run job names
Chat formatting	required on MTP benchmark requests — a client-side concern	MTP script's benchmark client

Two accuracy notes carried into the guide:

• Chat formatting. InferenceX passes --dsv4 to its own vendored benchmark client, which selects a private DeepSeek-V4 encoder. That is not a vllm serve option and is not copied here. The guide instead shows the upstream-supported mechanism, vllm bench serve --backend openai-chat --endpoint /v1/chat/completions, and states plainly that the two are not byte-identical.
• Shared-expert fusion. VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 is exported for recipe parity, but vLLM's own eligibility check does not currently pass for this mixed FP4+FP8 checkpoint, so the fusion self-disables at startup — as InferenceX's own changelog entry for #2792 states. The guide says this rather than claiming the fused path executed.

I also verified the recipe's builder already emits exactly {"method":"mtp","num_speculative_tokens":2} for the MTP mode on the FP8 (Preview) variant, so the interactive command builder and this guide section agree.

Scope and non-goals

• Guide-only benchmark reproduction; the general recommended command is unchanged.
• No new model file, no new or duplicate fp4 variant — the mixed FP4+FP8 preview checkpoint stays under the established FP8 (Preview) variant.
• No change to general AMD/default command-builder behavior: hardware_overrides.amd, variant defaults, strategy_overrides, and docker_image are untouched.
• The existing Agentic MI355X section is preserved, with an explicit table distinguishing its synthetic MTP K=3 / high-concurrency setup from this fixed-sequence real-MTP2 setup.
• No synthetic-acceptance or golden-acceptance settings added.
• No InferenceX scheduler, SLURM, monitoring, or perf-changelog.yaml plumbing.
• No generated public/ files committed.
• No --max-model-len, --max-num-seqs, --max-num-batched-tokens, or --block-size values invented — the lane does not pass them, and the guide says so explicitly.

Validation

Commands actually run, with real outcomes:

• node scripts/build-recipes-api.mjs — pass (✓ JSON API: 187 models … 9 strategies)
• pnpm build — pass; /deepseek-ai/DeepSeek-V4-Pro prerendered
• Generated-output diff — snapshotted public/deepseek-ai/ before and after the edit. Only 3 files differ (the recipe and its two promoted variant aliases, which embed the same guide), and the semantic delta is limited to guide + meta.date_updated. recommended_command and by_hardware hash identically, and the whole per-hardware/per-strategy rendering tree under public/deepseek-ai/DeepSeek-V4-Pro/ is byte-identical. No generated command changed.
• Flag-by-flag comparison against the live InferenceX scripts — all four environment variables and all twelve serve arguments match; confirmed that --max-model-len, --max-num-seqs, --max-num-batched-tokens, --block-size, --enable-expert-parallel, --enable-prefix-caching, tool-calling flags, and --dsv4 are absent.
• bash -n on all three new fenced bash blocks — pass; also executed the env block to confirm the four variables export as documented, and that --num-prompts expands to 10 × concurrency.
• Rendered-HTML inspection of the built page — 9 headings render as real heading tags, both tables render as <table>, all 3 code blocks render as <pre>, and the long image tag/digest, JSON quoting, and FP8 (Preview) reference survive intact.
• Link check — the cited InferenceX blobs resolve at the pinned SHA, run 33538769698 is success, and vllm#43385 is merged.
• git diff --check / git diff --cached --check — clean; YAML top-level key order unchanged; exactly one file staged.

Not run: pnpm validate fails, but this is a pre-existing baseline failure unrelated to this PR — its script targets hooks/validate.mjs, which is not tracked in main. I reproduced the identical MODULE_NOT_FOUND in a clean worktree on unmodified upstream/main (e88ec7c). pnpm lint was also skipped: this PR touches no JavaScript.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
vllm-recipes	Ready	Preview	Sep 5, 2026 5:02am UTC

[gemini-code-assist]

Code Review

This pull request updates the metadata and adds a comprehensive guide for reproducing the fixed-sequence benchmark (InferenceX MI355X 8K/1K) for the DeepSeek-V4-Pro model. The review feedback correctly identifies that the --no-enable-prefix-caching flag used in the STP launch command is invalid in vLLM and will cause startup failures, recommending its removal and a corresponding update to the comparison table.

[gemini-code-assist]

The flag --no-enable-prefix-caching is not a valid vLLM argument. In standard vLLM, prefix caching is disabled by default, and there is no --no-enable-prefix-caching option in the argument parser. Passing this unrecognized argument will cause the vllm serve command to fail at startup with an unrecognized arguments error.

To disable prefix caching, you should simply omit the --enable-prefix-caching flag.

    --async-scheduling \
    --distributed-executor-backend mp \

[gemini-code-assist]

Since --no-enable-prefix-caching is not a valid vLLM flag and prefix caching is disabled by default, this table entry should be updated to reflect that prefix caching is disabled by default.

  | Prefix caching | disabled (default) | --enable-prefix-caching |