Repository navigation
[ROCm][Recipe] MiniMax-M3.1-Flash: MI355X PD, DBO / SP prefill, Quark MXFP4, mono decode - #63257
Merged
esmeetu merged 14 commits intoOct 23, 2026
Merged
Conversation
… install Add an `fp8` variant for H100/H200 (TP8, FP8 KV and indexer caches, Marlin MoE, DSpark off) serving the same checkpoint, keep the NVFP4 default on Blackwell and MI355X, and hide the pip install tab. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: esmeetu <jasonailu87@gmail.com>
…args Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: esmeetu <jasonailu87@gmail.com>
…ption - pd_cluster on MI355X renders the guide's 1P1D: one TP4 engine per role, BLHNC KV layout, prefill without indexer_decode_cp plus the HugeTLB SimpleCPUOffloadConnector, decode on the default graphs. - single_node_tp: TP4 on MI355X (Hopper keeps the fp8 variant's TP8). - New opt-in feature "DBO / SP prefill" (MI355X single node): SP + DBO, DBO or SP; guide section with the flags and the 32K prefill times. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Taoyu Zhu <zhutaoyu97@gmail.com>
- New variant `mxfp4` = amd/MiniMax-M3.1-Flash-Quark-MXFP4: routed experts in OCP MXFP4, run as W4A4 on the AITER MoE (--moe-backend aiter, no shared-expert fusion). MI355X only. - `--quantization-config.moe mxfp8` moves from the recipe-level `amd` override to the default variant: only that checkpoint stores NVFP4 experts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Taoyu Zhu <zhutaoyu97@gmail.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
- The merge of main added a second `fp8:` variant block (main and the hopper branch both carried it); the duplicated key failed the recipe validation. Keep one. - On MI355X and Blackwell the brand overrides replace the variant's FP8 KV and indexer caches with NVFP4, so the picker offered a command that is not the Hopper setup (on MI355X also TP8 and NVFP4 experts without the MXFP8 requant). Limit it to sm90 (H100, H200). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Taoyu Zhu <zhutaoyu97@gmail.com>
…MI355X) New variant `mxfp4_mono`: the Quark export on the ROCm mono decode kernels (VLLM_ROCM_MINIMAX_M3_MONO=1), which run a decode step of up to 16 tokens as one kernel per layer. They need FP8 KV and FP8 indexer K caches on the AITER sparse PA path (VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1) and FULL_DECODE_ONLY graphs; other steps run on the normal path. A variant rather than a feature: mono is offered for the Quark checkpoint only, and a modes feature needs a mode for every variant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Taoyu Zhu <zhutaoyu97@gmail.com>
Move mono decode from a separate `mxfp4_mono` variant to an opt-in feature `mono_decode` (single node TP, MI355X). Its `mono` mode applies to the Quark MXFP4 variant only; the other variants get an empty mode, so enabling the feature there changes nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Taoyu Zhu <zhutaoyu97@gmail.com>
The mono mode is marked unsupported on MI300X, MI325X and MI350X too: it was measured on MI355X only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Taoyu Zhu <zhutaoyu97@gmail.com>
The NVFP4 variant showed 533 GB, the MI355X footprint after the MXFP8 re-quantization. Show the stored NVFP4 size (285 GB) and keep 533 GB as an MI355X override (measured 105.0 GiB per TP4 rank). The Quark variant goes from 285 to 293 GB (measured 56.9 GiB per TP4 rank). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Taoyu Zhu <zhutaoyu97@gmail.com>
esmeetu
approved these changes
Oct 23, 2026
benenzhu
marked this pull request as ready for review
October 23, 2026 05:06
Member
|
/ci run |
|
✅ Triggered Buildkite CI #96813 for commit |
|
✅ @benenzhu, CI is now available for this PR.
|
benenzhu
changed the title
esmeetu
enabled auto-merge (squash)
October 23, 2026 05:45
auto-merge was automatically disabled
October 23, 2026 05:50
Head branch was modified
Contributor
Author
|
/ci run |
|
✅ Triggered Buildkite CI #96829 for commit ℹ️ This PR is 1 commit behind upstream |
github-actions Bot
added a commit
to shantanusharma/recipes
that referenced
this pull request
Oct 23, 2026
…mni@17bd19f3d775 vllm-project/vllm@main:recipes 19c0e170d257 → 8fae0ff58bae [latest.vllm] vllm-project/vllm@19c0e17...8fae0ff Commits: - vllm-project/vllm@da180efbcddc [ROCm][Recipe] MiniMax-M3.1-Flash: MI355X PD, DBO / SP prefill, Quark MXFP4, mono decode (vllm-project/vllm#63257) Recipes: - MiniMaxAI/MiniMax-M3.1-Flash (modified): https://recipes.vllm.ai/MiniMaxAI/MiniMax-M3.1-Flash vllm-project/vllm-omni@main:recipes 27fd77b48ccd → 17bd19f3d775 [latest.vllm-omni] vllm-project/vllm-omni@27fd77b...17bd19f Commits: - vllm-project/vllm-omni@439a3bb4b008 [Hardware][Ascend] Add a Drama 3 NPU runner path (vllm-project/vllm-omni#9106) Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
github-actions Bot
added a commit
to vllm-project/recipes
that referenced
this pull request
Oct 23, 2026
…mni@c829af446829 vllm-project/vllm@main:recipes 0510bd667956 → c3117c7441dd [latest.vllm] vllm-project/vllm@0510bd6...c3117c7 Commits: - vllm-project/vllm@da180efbcddc [ROCm][Recipe] MiniMax-M3.1-Flash: MI355X PD, DBO / SP prefill, Quark MXFP4, mono decode (vllm-project/vllm#63257) Recipes: - MiniMaxAI/MiniMax-M3.1-Flash (modified): https://recipes.vllm.ai/MiniMaxAI/MiniMax-M3.1-Flash vllm-project/vllm-omni@main:recipes ea08ab6182e6 → c829af446829 [latest.vllm-omni] vllm-project/vllm-omni@ea08ab6...c829af4 Commits: - vllm-project/vllm-omni@439a3bb4b008 [Hardware][Ascend] Add a Drama 3 NPU runner path (vllm-project/vllm-omni#9106) Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
bantianyue
pushed a commit
to bantianyue/vllm
that referenced
this pull request
Oct 28, 2026
… MXFP4, mono decode (vllm-project#63257) Signed-off-by: esmeetu <jasonailu87@gmail.com> Signed-off-by: Taoyu Zhu <zhutaoyu97@gmail.com> Co-authored-by: esmeetu <jasonailu87@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: vllm-agent <claw@inferact.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
AMD MI355X additions to the MiniMax-M3.1-Flash recipe (
recipes/models/MiniMaxAI/MiniMax-M3.1-Flash.yamlonly). No other open PR changes this recipe.Claims
pd_clusterrenders the guide's 1P1D. One TP4 engine per role; BLHNC KV layout and NIXL with 4 threads; the prefill node dropsindexer_decode_cpand offloads KV to host memory in 1 GiB HugeTLB pages (SimpleCPUOffloadConnector).single_node_tp(Hopper keeps the FP8 variant's TP8).amd/MiniMax-M3.1-Flash-Quark-MXFP4, MXFP4 W4A4 experts on the AITER MoE (--moe-backend aiter, no shared-expert fusion).--quantization-config.moe mxfp8moves into the default variant, since only that checkpoint stores NVFP4 experts.FULL_DECODE_ONLYgraphs. On the released image (aiter 0.1.26) it also needs [ROCm][MiniMax-M3] Backport #61889: fix sparse PA KV block count #63270, the backport of [ROCm] Fix MiniMax-M3 sparse PA KV block count #61889, to start.fp8variant, limited to Hopper (the merge ofmainhad duplicated the key; on MI355X and Blackwell it rendered NVFP4 caches). VRAM minimums per GPU: the NVFP4 variant shows 285 GB, and MI355X gets 533 GB after the MXFP8 re-quantization.Validation
node scripts/validate-source.mjs --source vllm --dir <vllm>/recipespasses on vllm-project/recipesmain. Rendered commands were checked for the default, Quark, and FP8 variants on MI355X, B300, and H200, with and without the new features.Rendered commands served on MI355X TP4 with
vllm/vllm-openai-rocm:minimax-m31-flash, DSpark 3, AIME26 (16 epochs, max_tokens 98304, thinking on) and MMMU-Pro (all, 3460):¹ Same image, KV/indexer/MoE settings and DSpark 3, but served by our eval script (block verification), not the rendered command.
Details
mono_decodecarries an empty mode for the other variants: enabling it there changes nothing.--limit-mm-per-prompt '{"video": 0}'. With video inputs on, one Quark server ran out of memory in the vision encoder at MMMU concurrency 256 (--gpu-memory-utilization0.92).AI assistance (Claude Code) was used to write the recipe changes and run the validation above; I reviewed the changes.
Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
🤖 Generated with Claude Code