perf(amd): MiniMax-M3 ATOM AgentX on 1007 nightly with mono decode at every TP4 point / MiniMax-M3 ATOM AgentX 更新至 1007 nightly 并在所有 TP4 点位启用 mono decode

[yhl-amd]

Summary

Move minimaxm3-fp4-mi355x-atom-agentic-mtp to the 2026-10-07 ATOM nightly and turn MiniMax-M3 mono decode on at every TP4 point. Follows the layout of #3388.

• Image rocm/atom-dev:nightly_202609281543 (ATOM 0.1.7.dev43+gaa0c5c3a1) → rocm/atom-dev:nightly_202610070333 (digest sha256:2f605a3e8b459a0385f784edb0335c11892277fee650fe42044e4af39e36dd01, ATOM 0.1.7.dev60+g7fc0926c6, ROCm 7.2.4).
  • feat(v41): mono decode, FP4 index plane, shared mono framework ROCm/ATOM#2479 moves MiniMax-M3 mono decode onto the shared mono framework, publishes every mono mailbox flag only after the stores it announces have drained, and rewrites the sparse selection as exact top-k by counting.
  • fix(mono): zero the mailboxes and fence the ranks in one FlyDSL launch ROCm/ATOM#2483 rebuilds the per-step mailbox clear and cross-rank fence as one FlyDSL launch, replacing a Triton fence that Triton 3.7 on gfx950 compiled into a loop that could spin for ever.
  • fix(minimax-m3): decide a mono request's long context in one place ROCm/ATOM#2422 fixes the mono long-context decision.
  • The upstream recipes/MiniMax-M3-Agentic-InferenceX.md is unchanged.
• Mono decode on at every TP4 point: remove ATOM_MONO_ENABLE=0 from the indexer-CP points C15, C20, C24 and C28 (perf(amd): enable MiniMax-M3 ATOM FlyDSL decode / 启用 MiniMax-M3 ATOM FlyDSL 解码 #3388 had set it after they hung in run 36493321464). Mono serves decode steps of up to 16 tokens (four requests with EAGLE3 K3), covering all of TP4 C1, C2 and C4. ATOM builds mono for TP4 only, so TP2 runs the unfused path.
• TP2: add concurrency 4 (max-num-seqs 8); the GPU-resident TP2 points are now C1, C2 and C4.
• Record LMCache 0.5.6.dev139+gf22dec28, the wheel in the new image, for both DRAM-offload arms; the offload settings are unchanged.
• The recipe cites feat(services): lmcache-server kind and lmcache-mp connector NVIDIA/srt-slurm#507 (in srt-slurm since v2.36.0) for extra-kv-connectors instead of the dropped 507-lmcache-server-atom-sglang.patch.
• No data-type or precision change to the EAGLE3 draft model (Inferact/MiniMax-M3-EAGLE3-GQA): online_quant_config (ptpc_fp8), kv_cache_dtype fp8 and index-cache-dtype fp8 are unchanged; mono decode changes the target model's decode kernels, not draft precision.

Validation

• validate_perf_changelog, infx.matrix.plan (19 throughput points, full MiniMax vendor eval at each point), git diff --check.
• pytest infx/tests/{launch,srt_slurm,matrix} with srtctl v2.39.1: 730 passed.

中文

• 更新镜像至 rocm/atom-dev:nightly_202610070333（ATOM 0.1.7.dev60+g7fc0926c6，含 fix(mono): zero the mailboxes and fence the ranks in one FlyDSL launch ROCm/ATOM#2483 的 mono step fence 修复）。
• 去掉 TP4 C15/C20/C24/C28 的 ATOM_MONO_ENABLE=0，所有 TP4 点位启用 MiniMax-M3 mono decode（每步最多 16 个 token，即 EAGLE3 K3 下 4 个请求，覆盖 C1/C2/C4）；ATOM 只为 TP4 构建 mono，TP2 走非融合路径。
• 新增 TP2 C4，LMCache 版本记录为 0.5.6.dev139+gf22dec28，offload 配置不变。
• 风险：fix(minimax-m3): order mono stores before their done flags; keep pad rows out of routing ROCm/ATOM#2452 中的 pad row 修复尚未合入；本配方捕获 1–18 的全部 batch，4 个请求以内不会出现 pad row。

🤖 Generated with Claude Code

[github-actions]

Thanks for the contribution!

• Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
• PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
• After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.

中文

感谢你的贡献！

• **审阅：**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER，请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，再通过 Slack 联系核心维护者。必须严格遵循模板，并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have，才能触发签核验证。
• **PR 验证：**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目，并且恰好带有一个主标签时，才会运行扫描：full-sweep-fail-fast（强烈推荐；canary 加逐矩阵 fail-fast）、full-sweep-enabled（有 canary；矩阵任务在失败后继续运行）或 non-canary-full-sweep-enabled（无 canary，也无 fail-fast）。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
• **合并后：**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复；参见重新运行失败任务的说明。

[claude]

I reviewed this PR and didn't find any bugs. Because it re-enables MiniMax-M3 mono decode at TP4 C15/C20/C24/C28 — concurrency points that previously had it explicitly disabled after a hang in production (run 36493321464) — and the PR's own risk note says GPU sweep validation is still pending and an open upstream fix (ROCm/ATOM#2452) isn't in this image yet, a human look would still be worthwhile before merging.

What was reviewed:

• Mono-decode re-enable at C15/C20/C24/C28 (max-num-seqs 30-56) vs. the PR's claim that mono only needs to cover 4-request/16-token steps — the batch-size gap is real but ATOM's internal mono step-batching isn't visible from this repo to confirm either way.
• Docs claiming mono also covers LMCache points C40/C48 — ruled out as a doc bug: those blocks never set ATOM_MONO_ENABLE at all (mono was already on by default there), so the wording is consistent.
• Image/LMCache version bumps, new TP2 C4 point, and the dropped patch reference are applied consistently across agentic.yaml, amd-master.yaml, perf-changelog.yaml, and both docs.

Extended reasoning...

The diff is a benchmark-config-only change (srt-slurm recipe yaml, matrix config, perf changelog, and English/Chinese docs) for an AMD MI355X MiniMax-M3 ATOM recipe: no application or security-sensitive code is touched. The decisive factor for deferring rather than approving is that the change reverts a prior safety workaround (ATOM_MONO_ENABLE=0) that was added specifically because mono decode hung in production at these same TP4 concurrency points, and the PR author's own "Risk" section says a related upstream fix is not yet in this image and GPU sweep validation is still pending, so correctness at the wider batch sizes is not yet confirmed.

This review covers commit 26d7e83, which is no longer the latest commit on this pull request; later commits are not covered by it.

[github-actions]

View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=37598205362

View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=37598205362

[seungrokj]

/reuse-sweep-run 37598205362

[seungrokj]

As a PR reviewer and CODEOWNER, I have reviewed this and have:

• Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
• Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
• Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
• Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
• Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
• Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison.
• For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
• Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
• Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
• If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
• If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
• Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
  • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
• Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
• If this PR uses append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it.
• If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
• Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.

Additional detail section:

• insert any additional info here
• recipe at https://github.com/ROCm/ATOM/blob/main/recipes/MiniMax-M3-Agentic-InferenceX.md
• this is the draft model precision check: 1. The PR doesn't touch any draft setting. It changes three files: agentic.yaml, amd-master.yaml and perf-changelog.yaml. The changes are:
• the image bump nightly_202609281543 → nightly_202610070333
• removing ATOM_MONO_ENABLE=0 at TP4 C15/C20/C24/C28
• adding TP2 C4
• the LMCache version string

draft-model, method: eagle3, num-speculative-tokens, kv_cache_dtype, index-cache-dtype and online_quant_config are all unchanged.

2. The image bump doesn't change draft precision either. I compared the ATOM commits inside the two images, aa0c5c3a1 and 7fc0926c6, which are 17 commits apart:

• Unchanged: eagle3_llama.py, linear.py, quant_spec.py, the quant parsers, the loader, layernorm, and the eagle3 architecture mapping in config.py.
• eagle_proposer.py and drafter.py did change (#2367, #2479). The changes only restore kv_indptr/context_lens during draft-graph warmup and change how anchors and draft token IDs are passed. None of them touch dtype or quantization.
• Mono decode only replaces the target's decode kernels (MiniMaxM3SparseForCausalLM); the draft is a separate model class.

3. What each draft layer actually runs in, the same before and after. The checkpoint Inferact/MiniMax-M3-EAGLE3-GQA is LlamaForCausalLMEagle3, stored as bfloat16 with no quantization_config, and ATOM maps it to Eagle3LlamaModel (atom/config.py:1469).

┌────────────────────────────────────────────────────┬───────────────────────────────────────────────────────────────┐
│ Draft layer │ Precision │
├────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────┤
│ embed_tokens, fc, qkv_proj, o_proj, gate_up_proj, │ BF16 │
│ down_proj, lm_head │ │
├────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────┤
│ input_layernorm, hidden_norm, │ BF16 │
│ post_attention_layernorm, norm │
├────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────┤
│ Draft attention KV cache dtype: fp8 via atom_config.kv_cache_dtype │
│ │ (eagle3_llama.py:318) │
├───────────────────────────────────────────────────────────────────────────────────────┤
│ Indexer cache │ Not applicable: the draft has no indexer, so │
│ nly affects the target │
└────────────────────────────────────────────────────┴───────────────────────────────────────────────────────────────┘

The linears are BF16 because Eagle3Llama builds them without passing quant_config. LinearBase then falls back to LayerQuantConfig(), which is QuantType.No witant_spec.py:81).

Signed: seungrokj

[github-actions]

⚠️ Verdict: WARN ⚠️

⚠️ Check 15 (PR description): WARN — The body says the image is rocm/atom-dev:nightly_202610061452 (ATOM 0.1.7.dev58+g08735170a, manifest sha256:b0ebacc…) and that upstream recipes/MiniMax-M3-Agentic-InferenceX.md is unchanged "between aa0c5c3a and 08735170". At the pinned SHA, amd-master.yaml and agentic.yaml pin rocm/atom-dev:nightly_202610070333, which is ATOM 0.1.7.dev60+g7fc0926c6 per perf-changelog. The body also says "GPU sweep pending", but run 37598205362 has already passed. The PR title still says "1006 nightly". Please update the body and title to describe the 1007 image and the completed sweep.

Passed and not applicable checks

✅ Check 0 (CODEOWNER): PASS — @seungrokj is listed as an owner of inferencex-e2e/configs/amd-master.yaml. The recipe file and perf-changelog.yaml fall only under the catch-all owner.

✅ Check 1 (Passing sweep on in-PR commit): PASS — In-PR commit 4b94f096 has 19/19 agentic / and 19/19 agentic eval / check-runs with conclusion success (run 37598205362, attempt 1). Between that commit and the pinned head, only main merges changed the tree. They did not touch the MiniMax-M3 ATOM recipe or its amd-master.yaml entry.

✅ Check 2 (Evals pass): PASS — agg_eval_all.json covers all 19 points with minimax_m3_full scores between 0.97 and 0.99. An extra gsm8k run on TP4 c48 scored 0.969, against a bar of 0.90. Every point reports infrastructure_success. All points used rocm/atom-dev:nightly_202610070333, the image this PR pins.

➖ Check 3 (Upstream recipe): N/A — The only affected entry is framework: atom. No vLLM or SGLang production serving configuration is affected. The ROCm/ATOM recipe link is informational only.

✅ Check 4 (Reuse command): PASS — @seungrokj, who has COLLABORATOR association, posted /reuse-sweep-run 37598205362 on its own line.

✅ Check 5 (Latest checklist template): PASS — Every item in the current PR_REVIEW_CHECKLIST.md template is present in the sign-off and checked.

✅ Check 6 (Upstream images / engine-first): PASS — (a) This is an ATOM entry, so the vLLM/SGLang upstream-image rule does not apply. (b) The PR modifies an existing ATOM entry rather than adding one. vLLM MiniMax-M3 MI355X submissions landed first: #1935 and #1939, then AgentX in #2118.

✅ Check 7 (Deprecated models/scenarios): PASS — In MODELS.md, minimaxm3 agentic coding is active. The EAGLE3-GQA draft is the agreed plan-of-record draft model. Review date is 2026-10-08.

✅ Check 8 (No architecture hacks / event publication): PASS — No --hf-overrides or model-config edits. ptpc_fp8 online quantization of the target is allowed precision lowering, with evals passing. publish_events_and_metrics is a TRT-LLM setting that ROCm/ATOM does not implement (zero code-search hits), so it does not apply here.

✅ Check 9 (Spec-decode chat template): PASS — The recipe sets AIPERF_APPLY_CHAT_TEMPLATE: 'true' with default-chat-template-kwargs set to thinking_mode: enabled.

✅ Check 10 (No engine patches): PASS — runners/srt-slurm/patches/ contains only a README. minimaxm3-atom-eagle3-deps.sh downloads the draft and installs the tokenizer deps sentencepiece and tiktoken. Neither is an engine patch.

✅ Check 11 (Golden simulated AL): PASS — The harness injected --set roles.agg.args.spec-decode-acceptance-length=2.78, confirmed in the TP4 c1 srt-single-node-submission.json. This matches the golden value in minimaxm3_eagle3_gqa.yaml for thinking_on with K=3. Eval-only runs keep real acceptance.

➖ Check 12 (Append-only): N/A — The new perf-changelog entry has no append-only: true.

✅ Check 13 (Draft runs as shipped): PASS — The Inferact/MiniMax-M3-EAGLE3-GQA checkpoint (LlamaForCausalLMEagle3) is stored in bfloat16 with no quantization_config. In ATOM 7fc0926c6, eagle3_llama.py builds every linear without quant_config. Online quantization in linear.py:1164 is gated on quant_config is not None, so ptpc_fp8 never reaches the draft and it runs in BF16. The FP8 KV cache comes from the target-wide kv_cache_dtype, which is allowed. eagle3_llama.py, linear.py and quant_spec.py are unchanged from aa0c5c3a1 to 7fc0926c6. SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not used; this is an ATOM recipe.

✅ Check 14 (Pareto coverage): PASS — The curve minimaxm3 / agentic-coding / mi355x_atom / fp4 / p90_e2el has 9 frontier points out of 19 measured, with 0 invalid. Points span TP4 c5, c8, c12, c15, c20, c24 and c28, the TP4 c40 DRAM point, and the TP2 c25 DRAM point. Counted with infx.workflows.pareto_coverage from the run 37598205362 bmk_agentic_* artifacts. Throughput is tput_per_gpu; x is P90 E2EL.

Assessed commit: 6acaeab387acd9635a89cc3fd51ea54de7ce0601.

[seungrokj]

@edwingao28 plz review this