[AMD] MiniMax-M3 MXFP8 MI355X vLLM: gate AITER sparse PA to 8k1k high-conc + native linear default / MiniMax-M3 MXFP8 MI355X vLLM：AITER 稀疏分页注意力门控至 8k1k 高并发 + 默认原生 linear - #2187
Conversation
…lation linear Bump the minimaxm3-fp8-mi355x-vllm image to nightly-9e57de7197f234f9d9187715d96e07e007048c0f, which carries the merged AITER page-16 sparse paged-attention path (vllm-project/vllm#47287). Enable it via VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1 (with VLLM_ROCM_USE_AITER=1 and fp8 KV cache on TP4, num_kv_heads == 1 per rank) plus the recipe's quick all-reduce knobs VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0 and VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256. Drop --linear-backend emulation now that the Triton MXFP8 dense-linear GEMM is optimized. Deliberately do NOT add the #47269 --hf-overrides use_index_cache/index_topk_freq indexer skip: it reduces model-architecture FLOPs, disallowed by docs/PR_REVIEW_CHECKLIST.md. Sparse PA is a kernel-level speedup only. Verified locally on MI355X (gfx950) TP4 8k1k: conc1 10/10 and conc512 5120/5120 completed with zero failures (conc512 ~28.6k tok/s total, ~7.1k tok/s/GPU). 中文：将 minimaxm3-fp8-mi355x-vllm 镜像升级到 nightly-9e57de7197f234f9d9187715d96e07e007048c0f，该镜像已包含合并进主分支的 AITER page-16 稀疏分页注意力(sparse paged attention, vllm-project/vllm#47287)。 通过 VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1（配合 VLLM_ROCM_USE_AITER=1、fp8 KV cache 及 TP4，每个 rank num_kv_heads == 1）启用，并补上 recipe 的 quick all-reduce 调优开关 VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0 与 VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256。由于 Triton MXFP8 稠密 linear GEMM 已优化，移除 --linear-backend emulation。刻意不添加 #47269 的 --hf-overrides use_index_cache/index_topk_freq indexer 跳层开关：它会减少模型 架构 FLOPs，违反 docs/PR_REVIEW_CHECKLIST.md。sparse PA 仅为 kernel 级加速。 已在 MI355X(gfx950) TP4 8k1k 本地验证：conc1 10/10、conc512 5120/5120 全部成功 （conc512 约 28.6k tok/s 总吞吐，约 7.1k tok/s/GPU）。 Co-authored-by: Cursor <cursoragent@cursor.com>
… for 8k1k high-conc On the sparse-PA nightly the dense-linear backend crossover reversed vs #2003: the native Triton MXFP8 GEMM now wins in the memory-bound low-concurrency regime, while --linear-backend emulation (bf16 hipBLASLT) still wins in the compute-bound high-concurrency regime. Measured on gfx950 MXFP8 + sparse PA: emulation ~+3-5% at 8k1k conc 64-512 but ~-3% at conc 1-8. Gate emulation to isl>=8192 && conc>=64; native everywhere else (all 1k1k, 8k1k conc<64). Overridable via LINEAR_BACKEND. 中文：在 sparse-PA nightly 上，稠密 linear 后端的性能拐点相比 #2003 发生反转： 原生 Triton MXFP8 GEMM 现在在访存受限的低并发区间更快，而 --linear-backend emulation（bf16 hipBLASLT）仍在计算受限的高并发区间更快。gfx950 MXFP8 + sparse PA 实测：emulation 在 8k1k conc 64-512 约 +3-5%，在 conc 1-8 约 -3%。因此将 emulation 限定在 isl>=8192 且 conc>=64，其余（所有 1k1k、8k1k conc<64）使用原生 路径。可通过 LINEAR_BACKEND 覆盖。 Co-authored-by: Cursor <cursoragent@cursor.com>
…high-conc AITER page-16 sparse PA is a long-context/high-batch optimization: it wins at 8k1k conc>=64 but adds overhead at short context (1k1k) and low concurrency. Gate both VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT (sparse PA) and --linear-backend emulation behind a single MM3_HIGH_CONC_FASTPATH condition (isl>=8192 && conc>=64); everything else falls back to the #2003 path (non-shuffled Triton attention + native linear) so there is no regression outside the target regime. Net vs #2003: 8k1k tput/gpu conc128 +3.3%, conc256 +2.5%, conc512 +4.7% (peak), neutral elsewhere. Overridable via MM3_HIGH_CONC_FASTPATH / LINEAR_BACKEND. 中文：AITER page-16 稀疏分页注意力(sparse PA)是长上下文/大批量优化：在 8k1k conc>=64 有收益，但在短上下文(1k1k)和低并发下带来额外开销。将 VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT（sparse PA）与 --linear-backend emulation 统一 用 MM3_HIGH_CONC_FASTPATH 条件（isl>=8192 且 conc>=64）门控；其余情况回退到 #2003 路径（非 shuffle 的 Triton 注意力 + 原生 linear），从而在目标区间之外不产生回退。 相对 #2003：8k1k tput/gpu conc128 +3.3%、conc256 +2.5%、conc512 +4.7%（峰值）， 其余基本持平。可通过 MM3_HIGH_CONC_FASTPATH / LINEAR_BACKEND 覆盖。 Co-authored-by: Cursor <cursoragent@cursor.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
1 similar comment
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
中文：将 perf-changelog 的 pr-link 设置为 #2187。 Co-authored-by: Cursor <cursoragent@cursor.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Beyond the inline finding, I also checked two other candidates and ruled them out: the perf-changelog pr-link field is correctly filled in (points to this PR, not a placeholder), and the MM3_HIGH_CONC_FASTPATH conc>=64 threshold's apparent regression at conc64 8k1k (-0.9%) is within the same run-to-run noise band as the PR's own flagged conc8 outlier, not a real gating-boundary bug.
Extended reasoning...
Checked the gating threshold (isl>=8192 && conc>=64) against the PR's own benchmark table for a boundary regression, and checked the perf-changelog pr-link field for an unfilled placeholder. Both were consistent with intended behavior/correctly filled, so no further findings beyond the inline nit.
… concurrencies Switch VLLM_ROCM_QUICK_REDUCE_QUANTIZATION from INT6 to INT4 unconditionally (#2003 used INT6); accuracy is guarded by the 8k1k evals. The AITER quick all-reduce tuning knobs (CAST_BF16_TO_FP16=0, QUANTIZATION_MIN_SIZE_KB=256) remain in the 8k1k high-conc fast path. Aimed at lifting the low/mid-conc points (e.g. the 8k1k conc8 dip) via cheaper all-reduces. 中文：将 VLLM_ROCM_QUICK_REDUCE_QUANTIZATION 从 INT6 无条件改为 INT4（#2003 用的是 INT6），准确性由 8k1k evals 保障。AITER quick all-reduce 调优开关 (CAST_BF16_TO_FP16=0、QUANTIZATION_MIN_SIZE_KB=256) 仍保留在 8k1k 高并发快速路径中。 目的是通过更廉价的 all-reduce 提升低/中并发点（例如 8k1k conc8 的回退）。 Co-authored-by: Cursor <cursoragent@cursor.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29479669373 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29479669373 |
|
/reuse-sweep-run |
|
vllm recipe PR: vllm-project/recipes#654 |
❌❌❌ REJECTED ❌❌❌@seungrokj — blocking: the linked upstream recipe PR vllm-project/recipes#654 is still OPEN (not merged), and the sign-off leaves two checklist items unchecked without explanation. Merge #654, then re-sign. ✅ Check 0 (CODEOWNER): PASS — |
|
vllm-project/recipes#654 is merged. |
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
Additional detail section:
- insert any additional info here
recipe at https://github.com/vllm-project/recipes/blob/main/models/MiniMaxAI/MiniMax-M3.yaml
Signed: seungrokj
✅✅✅ Verdict: PASS ✅✅✅✅ Check 0 (CODEOWNER): PASS — |
|
@functionstackx @cquil11 can you plz approve this ? |
✅✅✅ Verdict: PASS ✅✅✅✅ Check 0 (CODEOWNER): PASS — |
✅✅✅ Verdict: PASS ✅✅✅✅ Check 0 (CODEOWNER): PASS — |
4 participants
Summary
Bumps
minimaxm3-fp8-mi355x-vllmtovllm/vllm-openai-rocm:nightly-9e57de7197f234f9d9187715d96e07e007048c0f(carries the merged AITER page-16 sparse paged-attention path, vllm-project/vllm#47287) and re-tunes the serve command for that nightly:VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1, withVLLM_ROCM_USE_AITER=1+ fp8 KV cache on TP4 wherenum_kv_heads == 1per rank). Kernel-level speedup of the same sparse-attention computation — no FLOP reduction (we deliberately do not use the #47269--hf-overrides use_index_cache/index_topk_freqindexer-skip, perdocs/PR_REVIEW_CHECKLIST.md).MM3_HIGH_CONC_FASTPATH=isl>=8192 && conc>=64). Sparse PA is a long-context/high-batch optimization and regresses short context (1k1k) and low concurrency, so bothVLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUTand--linear-backend emulationare enabled only in that regime. Everything else falls back to the prior [AMD] MiniMax-M3 MXFP8 MI355X vLLM: nightly + AITER-on TP4 + emulatin linear / MiniMax-M3 MXFP8 MI355X vLLM：升级 nightly + 启用 AITER TP4 + emulation linear #2003 path (non-shuffled Triton attention + native linear).--linear-backend emulationfrom [AMD] MiniMax-M3 MXFP8 MI355X vLLM: nightly + AITER-on TP4 + emulatin linear / MiniMax-M3 MXFP8 MI355X vLLM：升级 nightly + 启用 AITER TP4 + emulation linear #2003). On this nightly the native Triton MXFP8 GEMM is faster in the memory-bound low/mid-concurrency regime; emulation (bf16 hipBLASLT) only wins at 8k1k high concurrency and stays gated there.VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4; [AMD] MiniMax-M3 MXFP8 MI355X vLLM: nightly + AITER-on TP4 + emulatin linear / MiniMax-M3 MXFP8 MI355X vLLM：升级 nightly + 启用 AITER TP4 + emulation linear #2003 used INT6) — cheaper collectives lift the whole curve and eliminate the low/mid-conc dips; accuracy is guarded by the 8k1k evals. The AITER tuning knobs (CAST_BF16_TO_FP16=0,QUANTIZATION_MIN_SIZE_KB=256) stay in the 8k1k high-conc fast path.Search space (TP4, conc 1-512 at 1k1k and 8k1k) is unchanged.
Performance vs #2003 (tput/GPU) — every concurrency is a win
e2e run (test-config, mi355x-amds): https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29342987628
No regression at any concurrency (gains +0.2% to +11.6%); peak throughput +8.5% (1k1k conc512) / +8.6% (8k1k conc512). Switching to INT4 quick all-reduce resolved the earlier 8k1k conc8 (-4.7% -> +0.2%) and conc64 (-0.9% -> +2.4%) dips seen with INT6.
Test plan
test-configgreen onmi355x-amds(run 29342987628), all 20 throughput + 2 eval jobsvllm-project/recipes) reflecting sparse PA + gated linear-backend + INT4 for MXFP8 MI355X (for CODEOWNER sign-off)中文说明
将
minimaxm3-fp8-mi355x-vllm镜像升级到nightly-9e57de7197f234f9d9187715d96e07e007048c0f（已包含合并进主分支的 AITER page-16 稀疏分页注意力 vllm-project/vllm#47287），并针对该 nightly 重新调优 serve 命令：VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1，配合VLLM_ROCM_USE_AITER=1与 fp8 KV cache，TP4 下每 ranknum_kv_heads == 1）。这是对同一稀疏注意力计算的 kernel 级加速，不减少 FLOPs（按docs/PR_REVIEW_CHECKLIST.md，刻意不使用 #47269 的--hf-overridesindexer 跳层开关）。MM3_HIGH_CONC_FASTPATH=isl>=8192 且 conc>=64）。稀疏分页注意力是长上下文/大批量优化，在短上下文(1k1k)与低并发下会回退性能，因此VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT与--linear-backend emulation仅在该区间启用，其余回退到 [AMD] MiniMax-M3 MXFP8 MI355X vLLM: nightly + AITER-on TP4 + emulatin linear / MiniMax-M3 MXFP8 MI355X vLLM：升级 nightly + 启用 AITER TP4 + emulation linear #2003 路径（非 shuffle Triton 注意力 + 原生 linear）。--linear-backend emulation）。在该 nightly 上原生 Triton MXFP8 GEMM 在低/中并发更快；emulation 仅在 8k1k 高并发占优并被门控于此。VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4；[AMD] MiniMax-M3 MXFP8 MI355X vLLM: nightly + AITER-on TP4 + emulatin linear / MiniMax-M3 MXFP8 MI355X vLLM：升级 nightly + 启用 AITER TP4 + emulation linear #2003 为 INT6）——更廉价的集合通信抬升整条曲线并消除低/中并发的回退；准确性由 8k1k evals 保障。AITER 调优开关（CAST_BF16_TO_FP16=0、QUANTIZATION_MIN_SIZE_KB=256）保留在 8k1k 高并发快速路径。搜索空间（TP4，1k1k 与 8k1k 并发 1-512）保持不变。相对 #2003 每个并发点均为正收益（+0.2% ~ +11.6%），峰值吞吐 +8.5%(1k1k)/+8.6%(8k1k)；改用 INT4 quick all-reduce 后消除了此前 INT6 下 8k1k conc8(-4.7%->+0.2%)、conc64(-0.9%->+2.4%) 的回退。