feat(amd): restore DSv4.1-Flash ATOM AgentX on srt-slurm with 1007 nightly / 以 srt-slurm 恢复 DSv4.1-Flash ATOM AgentX 并更新至 1007 nightly

[yhl-amd]

Summary

Restore dsv41flash-fp4-mi355x-atom-agentic-dspark (#3387) on the native srt-slurm framework and move it to the 2026-10-07 ATOM nightly. #3387's benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_atom_mtp.sh was deleted by #3461, and #3463 then dropped the master-config entry and docs, so this arm currently does not run. Following the pattern in #3388 and the other ATOM AgentX recipes, it is now a srt-recipe row.

• New recipe inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/dsv41flash/atom/mi355x-fp4-mtp/agentic.yaml: engine: atom, atomesh frontend (same digest as the other ATOM recipes), one override_tp{2,4}_c* variant per point.
• Image rocm/atom-dev:nightly_202609250902 → rocm/atom-dev:nightly_202610070333.
• Points change from [AMD][AgentX] Add DeepSeek-V4.1-Flash ATOM TP2/TP4 / 新增 DeepSeek-V4.1-Flash ATOM TP2/TP4 #3387: TP2 [1, 2, 4, 5, 6, 8, 16, 32, 64, 128] (adds 4, 5, 6 and 128), TP4 [1, 4, 5, 6, 8, 32] (adds 1, 4, 5 and 6; drops 2, 16 and 64). No expert parallelism or KV offload. Concurrency 1–8 covers the batch sizes that V4.1 mono decode serves (up to 8 requests a step).
• Server settings follow the upstream recipe as before: BF16 KV, FP4 index cache (FP8 before; follows upstream at 7fc0926c), max-num-seqs 128, 16K batching/prefill chunks, prefix caching with block size 16, 8K state checkpoints, level 3, FULL graphs, five-token DSpark, dsml_v41 tool parsing. The one difference: every point now captures the dense list [1..32, 48, 64, 128] (set once in base) instead of only concurrency 32.
• Golden AL is no longer written in the recipe. Per AGENTS.md, infx/srt_slurm/synthetic_acceptance.py adds --spec-decode-acceptance-length 3.51 (the dsv41flash_dspark thinking_on K5 value) to AgentX throughput runs and leaves eval-only runs on real acceptance.
• Replay semianalysis_cc_traces_weka_062126 for 3600 s per point with five warmup requests per lane; AIPerf TCP/drain timeouts 900000/300 as in [AMD][AgentX] Add DeepSeek-V4.1-Flash ATOM TP2/TP4 / 新增 DeepSeek-V4.1-Flash ATOM TP2/TP4 #3387.
• Two points set ATOM_PREFILL_DECODE_INTERVAL (see below): TP4 c32 = 16, TP2 c64 = 4. All other points are unchanged.
• Server env OMP_NUM_THREADS=4, ATOM_NUMA_BIND=0, ATOM_DISABLE_MMAP=true, AITER_LOG_LEVEL=WARNING is unchanged; srtctl masks GPUs 0..TP-1 via ROCR_VISIBLE_DEVICES. The existing mi355x-amds shared-HF-cache rule already covers deepseek-ai/DeepSeek-V4.1-Flash, so no launcher or policy change is needed.

Image

Pin: rocm/atom-dev:nightly_202610070333, digest sha256:2f605a3e8b459a0385f784edb0335c11892277fee650fe42044e4af39e36dd01, ATOM 0.1.7.dev60+g7fc0926c6, ROCm 7.2.4.

Relevant changes since the 0925 pin (ATOM 4685e3cf):

• feat: MiniMax-M3 mono decode with 1M context and indexer CP; V4.1 group32 FP8 and BF16 RoPE ROCm/ATOM#2419: group32 FP8 weight preshuffle and BF16 RoPE tables for DeepSeek-V4.1.
• feat(v41): mono decode, FP4 index plane, shared mono framework ROCm/ATOM#2479: V4.1 mono decode. It is on by default and serves decode steps of up to 8 requests for this TP2/TP4, BF16-KV, five-token DSpark deployment. It also adds the FP4 index plane that the upstream recipe now uses.
• [Config] feat(configs): DeepSeek-V4.1-Flash TP4 bf16 GEMMs in the dsv41 bf16 table ROCm/aiter#6174 (in the image's AITER): gfx950 tuned rows for the six DeepSeek-V4.1-Flash TP4 bf16 GEMM (N, K) pairs that previously fell back to torch, including the DSpark verify batches and the TP4 lm_head.
• fix(mono): zero the mailboxes and fence the ranks in one FlyDSL launch ROCm/ATOM#2483: replaces the Triton mono step fence, which Triton 3.7 on gfx950 compiled into a loop that could spin forever. On nightly_202610061452 and nightly_202610061600 the TP2 c1 canary hung in warmup on the DSpark draft's first mono step and never served (runs 37486079757 and 37493871230).

Draft precision

The DSpark draft (layers 37–39, loaded inline from the checkpoint) keeps its checkpoint weights: no --online_quant_config is passed, and its attention linears stay per_1x32 FP8 (e4m3, FP8 activations). Two compute precisions change with the image bump, checked against ATOM 7fc0926c6:

• Routed-expert activations MXFP4 → MXFP8. The draft's DraftBlock reuses the V4.1 MoE, which sets moe_fp8_activations = True (feat(v41): mono decode, FP4 index plane, shared mono framework ROCm/ATOM#2479); Mxfp4MoEMethod then quantizes expert inputs to FP8 instead of MXFP4. Expert weights stay MXFP4.
• RoPE cos/sin tables FP32 → BF16 (feat: MiniMax-M3 mono decode with 1M context and indexer CP; V4.1 group32 FP8 and BF16 RoPE ROCm/ATOM#2419); the draft builds its own rotary embedding from the same class.

Numerics-only changes (same dtypes, different kernels or summation order): group32 preshuffled FP8 wkv GEMM with inverse RoPE fused into the output projection, fixed-order router GEMV, and the mono decode kernels. --kv_cache_dtype bf16 and --index-cache-dtype fp4 touch cache storage only; the draft has no indexer, so the FP4 index cache does not apply to it.

The perf-changelog.yaml entry still says "no data-type or precision change to the DSpark draft"; that line predates this check and should be updated to match.

TP4 c32: prefill-decode interval 16

TP4 c32 sets ATOM_PREFILL_DECODE_INTERVAL=16. ATOM runs prefill and decode as separate forwards. Each new turn adds about 2.3 prefill-only forwards, and every running decode waits for them. In CI 36681286104, about 40% of TP4 c32 decode time went to these waits; for vLLM the figure was about 20%. With the interval, ATOM's PrefillDelayer (off by default) runs at least 16 decode passes after each prefill and batches the prefills that arrive in the meantime.

Local 3600 s replay (MI355X, same trace):

TP4 c32	P90 interactivity	ITL p50 / p90 (ms)	TTFT p50 / p90 (ms)	tok/s/GPU
interval 0 (before)	108.2	4.93 / 9.24	511 / 1201	53.3k
interval 4	137.9	4.64 / 7.25	886 / 1759	53.7k
interval 16	150.9	4.41 / 6.63	1140 / 2357	53.8k

For reference, vLLM TP4 c32 in CI is 131.0. Throughput does not change because AgentX is paced by the trace: users are idle about 74% of the time. Decode time saved per turn (−1.04 s) is roughly offset by added TTFT (+0.96 s).

TP2 c64: prefill-decode interval 4

TP2 c64 sets ATOM_PREFILL_DECODE_INTERVAL=4. In CI this point matched vLLM on interactivity (31.6 vs 31.5) but delivered 7.4% less throughput (118.7k vs 128.2k tok/s/GPU). About a third of ATOM's time went to prefill-only forwards. At this load, one batched prefill forward serves several requests, so throughput rises as well as interactivity.

Local 3600 s replay (MI355X, same trace):

TP2 c64	P90 interactivity	tok/s/GPU	requests	TTFT p50 / p90 (s)
interval 0 (before)	30.3	105.4k	6487	0.57 / 2.3
interval 4	38.8	115.2k (+9.3%)	6941	2.2 / 6.4
interval 8	41.7	113.6k	6856	2.8 / 10.1
interval 16	52.1	113.8k	6807	4.3 / 15.3

Interval 4 gives the most throughput for the smallest increase in TTFT.

Validation

Done at an earlier revision of this PR (image nightly_202609291501/nightly_202610011450, ATOM 74fd942b); not re-run for the 1007 image:

• git diff --check, YAML parsing, and a byte-prefix check on perf-changelog.yaml (append-only) passed.
• Ran the repository's build_overrides (synthetic-acceptance connector) on every variant, deep-merged over base: every matrix point mapped to exactly one variant with matching image, model, GPU count and CONC. Throughput gets spec-decode-acceptance-length=3.51; eval-only gets no override.
• Every ATOM flag in the recipe and the dsml_v41 parser existed at ATOM 74fd942b; the srtctl ATOM backend passes arg keys verbatim.
• The pydantic matrix generator, validate_perf_changelog, and srtctl override expansion are left to CI.

Background: why c8/c16 on the 0925 pin are not a valid baseline

Measured on nightly_202609291501, before the move to 1007; kept because it explains why the #3387 numbers should not be compared against.

CI run 36681286104 on nightly_202609291501 shows TP2 c8 X = 207.9, versus 239.9 on the #3387 nightly_202609250902 run. A local MI355X 3600 s replay reproduces it (207.6). The 0925 number is not a valid baseline: before ATOM#2419 (ATOM_GROUP32_WEIGHT_PRESHUFFLE, on by default from 0929), DSV4.1-Flash TP2 small-batch decode produces garbage text.

Checked locally with real acceptance (no --spec-decode-acceptance-length), greedy chat requests:

Image / setting	1–4 concurrent requests	64 concurrent
nightly_202609250902 / _251613 (old default path)	garbage	coherent
0925 with DSpark disabled, eager mode, or dual-stream MoE off	garbage	–
nightly_202609291501 + ATOM_GROUP32_WEIGHT_PRESHUFFLE=0	garbage	–
nightly_202609291501 default	coherent	coherent

• The throughput sweep forces acceptance (AL 3.51), so output text is meaningless by design and the corruption is invisible. Corrupted hidden states collapse MoE routing onto a few experts: the routed MoE GEMM reads about 7x less HBM (L2 hit 63% vs 11%, DRAM read requests 0.31M vs 2.16M per call), so decode looks faster.
• The 0925 eval passed (GSM8K 0.953) because it ran at concurrency 64, where the old path is correct.
• Root cause: on the old path the FP8 group32 linears run AITER's Triton packed GEMM (gemm_afp8wfp8 / the earlier group32 kernel). It loads masked scale lanes with other=127, and on Triton 3.7/3.8 those lanes read stale LDS. At small M (5–6 rows: shapes 2304x5120, 1792x5120, 16384x1280, 4096x1280) each call then returns about 1000 NaN/Inf values. Isolated kernel tests start with clean LDS and pass. Applying ROCm/aiter#6007 (other=0) to nightly_202609291501 with ATOM_GROUP32_WEIGHT_PRESHUFFLE=0 gives coherent output and 0 non-finite values across 2000 checked calls. The default path uses the FlyDSL preshuffled GEMM for these linears, so it is unaffected.
• With correct numerics, the 0929 default, preshuffle off + #6007, and preshuffle off + #6007 + [Triton/Gluon] Restore direct-to-LDS scale copies for small-M packed afp8wfp8 ROCm/aiter#6048 have the same decode step time within run-to-run noise (TP2, batch 1–32).

Conclusion: do not set ATOM_GROUP32_WEIGHT_PRESHUFFLE=0 for this arm, and do not use the c8/c16 points measured on 0925 as a reference.

中文

概要

将 dsv41flash-fp4-mi355x-atom-agentic-dspark（#3387）迁移到原生 srt-slurm 框架，并更新为 2026-10-07 ATOM nightly。#3387 的 dsv41flash_fp4_mi355x_atom_mtp.sh 已在 #3461 中删除，#3463 随后移除了 master config 条目和文档，该配置目前不会运行。本 PR 参考 #3388 及其他 ATOM AgentX 配方，改为 srt-recipe 行。

• 新增配方 inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/dsv41flash/atom/mi355x-fp4-mtp/agentic.yaml：engine: atom，atomesh frontend（与其他 ATOM 配方相同 digest），每个测试点对应一个 override_tp{2,4}_c* 变体。
• 镜像由 rocm/atom-dev:nightly_202609250902 更新为 rocm/atom-dev:nightly_202610070333。
• 测试点相对 [AMD][AgentX] Add DeepSeek-V4.1-Flash ATOM TP2/TP4 / 新增 DeepSeek-V4.1-Flash ATOM TP2/TP4 #3387 的变化：TP2 [1, 2, 4, 5, 6, 8, 16, 32, 64, 128]（新增 4、5、6、128），TP4 [1, 4, 5, 6, 8, 32]（新增 1、4、5、6，去掉 2、16、64），不启用 EP 或 KV 卸载。并发 1–8 覆盖 V4.1 mono decode 支持的 batch 大小（每步最多 8 个请求）。
• 服务参数沿用上游配方：BF16 KV，FP4 index cache（此前为 FP8，跟随上游 7fc0926c），max-num-seqs 128，16K batching/prefill chunk，block size 16 的前缀缓存，8K 状态检查点，level 3，FULL graph，五 token DSpark，dsml_v41 工具解析。唯一区别：所有点统一捕获 [1..32, 48, 64, 128]（在 base 中设置），而不再只针对并发 32。
• 配方不再写入 golden AL：按 AGENTS.md，由 infx/srt_slurm/synthetic_acceptance.py 为 AgentX 吞吐测试添加 --spec-decode-acceptance-length 3.51（thinking_on K5），eval-only 保持真实 acceptance。
• 每点回放 semianalysis_cc_traces_weka_062126 3600 秒，每条 lane 预热 5 个请求。
• 两个点设置 ATOM_PREFILL_DECODE_INTERVAL（见下文）：TP4 c32 = 16，TP2 c64 = 4，其余点不变。

镜像

rocm/atom-dev:nightly_202610070333，digest sha256:2f605a3e…，ATOM 0.1.7.dev60+g7fc0926c6，ROCm 7.2.4。相对 0925（ATOM 4685e3cf）的相关改动：

• feat: MiniMax-M3 mono decode with 1M context and indexer CP; V4.1 group32 FP8 and BF16 RoPE ROCm/ATOM#2419：为 DeepSeek-V4.1 新增 group32 FP8 权重预重排和 BF16 RoPE 表。
• feat(v41): mono decode, FP4 index plane, shared mono framework ROCm/ATOM#2479：V4.1 mono decode，默认开启，在本 TP2/TP4、BF16 KV、五 token DSpark 部署下服务最多 8 个请求的 decode 步；同时新增上游配方使用的 FP4 index plane。
• [Config] feat(configs): DeepSeek-V4.1-Flash TP4 bf16 GEMMs in the dsv41 bf16 table ROCm/aiter#6174（镜像内 AITER）：为此前回退到 torch 的 6 组 DeepSeek-V4.1-Flash TP4 bf16 GEMM (N, K) 新增 gfx950 调优条目，包括 DSpark verify batch 和 TP4 lm_head。
• fix(mono): zero the mailboxes and fence the ranks in one FlyDSL launch ROCm/ATOM#2483：替换 Triton mono step fence。Triton 3.7 在 gfx950 上会把它编译成可能无限自旋的循环；在 nightly_202610061452 和 nightly_202610061600 上，TP2 c1 canary 在 warmup 中卡在 DSpark draft 的第一个 mono 步，始终未能服务（run 37486079757、37493871230）。

Draft 精度

DSpark draft（第 37–39 层，从检查点内联加载）的权重保持检查点精度：未传入 --online_quant_config，attention 线性层仍为 per_1x32 FP8（e4m3，FP8 激活）。对照 ATOM 7fc0926c6，镜像升级带来两处计算精度变化：

• 路由专家激活 MXFP4 → MXFP8：draft 的 DraftBlock 复用 V4.1 MoE，其设置 moe_fp8_activations = True（feat(v41): mono decode, FP4 index plane, shared mono framework ROCm/ATOM#2479），Mxfp4MoEMethod 因此把专家输入量化为 FP8 而非 MXFP4；专家权重仍为 MXFP4。
• RoPE cos/sin 表 FP32 → BF16（feat: MiniMax-M3 mono decode with 1M context and indexer CP; V4.1 group32 FP8 and BF16 RoPE ROCm/ATOM#2419）。

仅数值层面的变化（dtype 不变）：group32 预重排 FP8 wkv GEMM 并将 inverse RoPE 融合到输出投影、固定顺序的 router GEMV、mono decode kernel。--kv_cache_dtype bf16 和 --index-cache-dtype fp4 只影响缓存存储；draft 没有 indexer，FP4 index cache 不作用于 draft。

perf-changelog.yaml 中仍写着 draft “无数据类型或精度变化”，该描述早于本次核对，需要同步更新。

TP4 c32：prefill-decode interval 16

TP4 c32 设置 ATOM_PREFILL_DECODE_INTERVAL=16。ATOM 的 prefill 和 decode 分开跑，每个新请求带来约 2.3 次纯 prefill forward，期间所有在跑的 decode 都要等待。在 CI 36681286104 里，TP4 c32 约 40% 的 decode 时间花在这种等待上，vLLM 约 20%。开启后，ATOM 的 PrefillDelayer（默认关闭）每次 prefill 后至少先跑 16 步 decode，并把这期间到达的 prefill 攒到一起跑。

本地 3600 秒回放：P90 交互性 108.2 → 150.9（interval 4 为 137.9），ITL p90 9.24 → 6.63 ms，TTFT p50 0.51 → 1.14 s，吞吐不变（53.3k → 53.8k tok/s/GPU）。CI 上 vLLM TP4 c32 为 131.0。吞吐不变是因为 AgentX 由 trace 节奏决定，用户约 74% 的时间空闲；每轮省下的 decode 时间（−1.04 s）基本被增加的 TTFT（+0.96 s）抵消。

TP2 c64：prefill-decode interval 4

TP2 c64 设置 ATOM_PREFILL_DECODE_INTERVAL=4。CI 上这个点的交互性与 vLLM 持平（31.6 对 31.5），吞吐却低 7.4%（118.7k 对 128.2k tok/s/GPU），ATOM 约 1/3 的时间花在纯 prefill forward 上。在这个负载下，攒批后一次 prefill forward 能服务多个请求，所以吞吐和交互性都会提升。本地 3600 秒回放：交互性 30.3 → 38.8，吞吐 105.4k → 115.2k tok/s/GPU（+9.3%），完成请求 6487 → 6941，TTFT p90 2.3 → 6.4 s。interval 8 和 16 的交互性更高（41.7、52.1），但吞吐不如 4，TTFT p90 分别达到 10.1 s 和 15.3 s，因此选 4。

验证

在本 PR 较早的版本（镜像 nightly_202609291501/nightly_202610011450，ATOM 74fd942b）上完成，未针对 1007 镜像重跑：git diff --check、YAML 解析、changelog 追加检查通过；build_overrides 检查每个点唯一匹配一个变体，吞吐注入 AL 3.51，eval 不注入；配方中的 ATOM 参数和 dsml_v41 解析器在 74fd942b 上均存在。pydantic matrix generator、validate_perf_changelog 和 srtctl override 展开由 CI 覆盖。

背景：为什么 0925 上的 c8/c16 不能作为基线

以下在 nightly_202609291501 上测得（早于升级到 1007），保留用于说明为何不应与 #3387 的数字比较。

CI 36681286104（nightly_202609291501）的 TP2 c8 X 为 207.9，#3387 的 nightly_202609250902 为 239.9；本地回放复现为 207.6。但在 ATOM#2419（ATOM_GROUP32_WEIGHT_PRESHUFFLE，0929 起默认开启）之前，DSV4.1-Flash TP2 小 batch decode 会输出乱码：0925 镜像在 1–4 个并发请求时输出乱码，64 个并发时正常；关闭 DSpark、eager 模式或关闭双 stream MoE 仍是乱码；0929 设置 ATOM_GROUP32_WEIGHT_PRESHUFFLE=0 同样乱码；只有 0929 默认配置输出正常。

• 吞吐 sweep 强制接受长度（AL 3.51），看不出输出错误。错误的 hidden state 让 MoE 路由塌缩到少数专家，routed MoE GEMM 读取的显存约少 7 倍，所以 decode 显得更快。
• 0925 的 eval（GSM8K 0.953）在 c64 下运行，大 batch 时旧路径正确，因此没有发现问题。
• 根因：旧路径下 FP8 group32 线性层使用 AITER 的 Triton packed GEMM，对被 mask 的 scale 用 other=127 加载，在 Triton 3.7/3.8 上会读到 LDS 残留数据；小 M（5–6 行）时每次调用约 1000 个 NaN/Inf。应用 ROCm/aiter#6007（other=0）后恢复正常。默认路径使用 FlyDSL 预重排 GEMM，不受影响。
• 计算正确时，各方案的 decode step 时间在波动范围内一致。

结论：不要为该配置设置 ATOM_GROUP32_WEIGHT_PRESHUFFLE=0；0925 上测得的 c8/c16 不应作为参照。

🤖 Generated with Claude Code

[github-actions]

Thanks for the contribution!

• Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
• PR verification: Sweeps only run on labeled PRs. Add full-sweep-fail-fast (strongly recommended); use full-sweep-enabled only when matrix jobs should continue after a failure.
• After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.

中文

感谢你的贡献！

• **审阅：**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER，请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，再通过 Slack 联系核心维护者。必须严格遵循模板，并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have，才能触发签核验证。
• **PR 验证：**扫描仅在带有标签的 PR 上运行。强烈建议添加 full-sweep-fail-fast；仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。
• **合并后：**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复；参见重新运行失败任务的说明。

[claude]

Claude finished @yhl-amd's task in 40s —— View job

---

LGTM - no blocking issues found

[github-actions]

View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=37721285904

View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=37721285904

[claude]

I reviewed this PR and didn't find any bugs. Because it touches the CODEOWNER-covered amd-master.yaml and docs/MODELS.md/MODELS_zh.md, a human (codeowner) look would still be worthwhile.

What was reviewed: the new ATOM srt-slurm recipe (dsv41flash/atom/mi355x-fp4-mtp/agentic.yaml) against sibling ATOM recipes (dsv4, kimik3, minimaxm3, dsv41flash/vllm) to confirm the gpus:-only TP-override pattern, ep/dp-attn/spec-decoding: draft_model master-config key usage, and CONC-list/master-config alignment all match established conventions; the golden AL 3.51 claim was cross-checked against infx/golden_al_distribution/dsv41flash_dspark.yaml (thinking_on, K5) and the atom engine path in infx/srt_slurm/synthetic_acceptance.py; the perf-changelog.yaml change is a pure append after the last existing entry.

Extended reasoning...

Config/docs-only PR (new ATOM benchmark recipe, master-config entry, docs, append-only changelog) with no security-sensitive surface touched. It closely mirrors several existing sibling recipes and I verified the non-obvious claims (golden AL value, TP-via-gpus override convention, synthetic-acceptance ATOM support) directly in the repo rather than taking the PR description at face value, finding no discrepancies. Decision is defer rather than approve solely because two of the changed files (amd-master.yaml, docs/MODELS.md/MODELS_zh.md) are CODEOWNER-covered paths per .github/CODEOWNERS.

[seungrokj]

/reuse-sweep-run 37597985033

[seungrokj]

As a PR reviewer and CODEOWNER, I have reviewed this and have:

• Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
• Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
• Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
• Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
• Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
• Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison.
• For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
• Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
• Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
• If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
• If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
• Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
  • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
• Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
• If this PR uses append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it.
• If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
• Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.

Additional detail section:

• insert any additional info here
• recipe at https://github.com/ROCm/ATOM/blob/main/recipes/DeepSeek-V4.1-Flash-Agentic.md
• regarding the datatype of draft model
  only one weight changes datatype. Of the draft's weights, only attn.wo_a runs at a different dtype on the new image. It is the same tensor in all three draft stages. Every other draft weight keeps its dtype. Some of them are now stored in a different memory layout, but the data type is the same.

I diffed ATOM 4685e3cf (old image) against 7fc0926c6 (new image). DraftAttention and DraftBlock subclass V4.1's Attention and Block, so any change in those applies to every draft stage.

The one change: wo_a goes from BF16 to FP8

• Old pin: Attention.process_weights_after_loading turned the checkpoint's FP8 wo_a (with E8M0 32x32 scales) into BF16 after loading. It deleted the scale and set quant_type = QuantType.No, and _project_out ran a BF16 einsum.
• New pin: that hook is gone. wo_a stays FP8 e4m3 with E8M0 32x32 scales, laid out in 16x16 tiles (needs_preshuffled_weight = True). _project_out now quantizes the attention output to MXFP8 and calls batched_gemm_a8w8_mxscale_bpreshuffle. So wo_a goes from a BF16 GEMM to an FP8×MXFP8 GEMM.
• The PR's "Draft precision" section doesn't list this. Its phrase "inverse RoPE fused into the output projection" is the same change, but the PR doesn't say the weight's dtype changed. This change comes from #2479 or #2419; I didn't check which one.

Every draft weight, old vs. new

┌────────────────────────────────────────────────────────┬────────────────────────────────────────┬───────────────────────────────────────────────────────────────────────────────────────┬───────────────┐
│ Weight (per draft stage) │ Old pin │ New pin │ Changed? │
├────────────────────────────────────────────────────────┼────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┼───────────────┤
│ attn.wqkv_a, attn.wq_b, attn.wo_b │ FP8 + E8M0 1x32/32x32, original layout │ Same dtype, preshuffled 16x16 on gfx950 (ATOM_GROUP32_WEIGHT_PRESHUFFLE=1 by default) │ Layout only │
├────────────────────────────────────────────────────────┼────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┼───────────────┤
│ attn.wo_a │ BF16, converted at load │ FP8 + E8M0, preshuffled │ Dtype changes │
├────────────────────────────────────────────────────────┼────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┼───────────────┤
│ ffn.shared_experts.gate_up_proj, .w2 │ FP8 + E8M0 group32 │ Same dtype, preshuffled │ Layout only │
├────────────────────────────────────────────────────────┼────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┼───────────────┤
│ ffn.experts.w13_weight, .w2_weight (routed experts) │ MXFP4 (fp4x2) + E8M0 │ MXFP4. With FP8 activations, gate and up are now interleaved (is_guinterleave) │ Layout only │
├────────────────────────────────────────────────────────┼────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┼───────────────┤
│ ffn.gate.weight │ BF16 │ BF16, now run by a fixed-order GEMV │ No │
├────────────────────────────────────────────────────────┼────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┼───────────────┤
│ gate.e_score_correction_bias, bias_vl, attn_sink, hc_* │ FP32 │ FP32 │ No │
├────────────────────────────────────────────────────────┼────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┼───────────────┤
│ attn_norm, ffn_norm, q_norm, kv_norm │ BF16 │ BF16 │ No │
├────────────────────────────────────────────────────────┼────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┼───────────────┤
│ main_proj (stage 0 only) │ FP8 group32 │ Same dtype, preshuffled │ Layout only │
├────────────────────────────────────────────────────────┼────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┼───────────────┤
│ main_norm, norm, Markov and confidence heads │ Unchanged │ Unchanged │ No │
├────────────────────────────────────────────────────────┼────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────┼───────────────┤
│ Embedding and LM head │ Shared with the target model │ Shared with the target model │ No │
└────────────────────────────────────────────────────────┴────────────────────────────────────────┴───────────────────────────────────────────────────────────────────────────────────────┴───────────────┘

Signed: seungrokj

[github-actions]

⚠️ Verdict: WARN ⚠️

⚠️ Check 15 (PR description): WARN — The body's "Draft precision" section says "Two compute precisions change with the image bump" and lists only routed-expert activations (MXFP4 to MXFP8) and the RoPE tables (FP32 to BF16). It leaves out a third change: the draft's attn.wo_a moved from a BF16 GEMM (FP8 weight dequantized at load on the 0925 pin) to an FP8 x MXFP8 batched_gemm_a8w8_mxscale_bpreshuffle at ATOM 7fc0926c6 attention.py L246-322, as the sign-off itself records. The committed perf-changelog.yaml entry at the assessed SHA still says "No data-type or precision change to the DSpark draft". The body admits this line is stale, but it was not fixed. Please update the body and the changelog entry to list all three draft compute-precision changes.

Passed and not applicable checks

✅ Check 0 (CODEOWNER): PASS — @seungrokj is a named owner of the changed inferencex-e2e/configs/amd-master.yaml. The recipe, changelog and MODELS{,_zh}.md row edits are covered as a recognized CODEOWNER on this AMD-only PR.

✅ Check 1 (Sweep on in-PR commit): PASS — Run 37597985033 ran on in-PR commit d3ba89f3. All 15 agentic / jobs, the TP2 c1 canary and agentic eval / succeeded. The recipe file at d3ba89f3 is byte-identical to the one at 368c95ad, and later commits only merge main and update the commit message.

✅ Check 2 (Evals): PASS — eval_results_all: GSM8K em_strict 0.973 (±0.004, n=1319), TP2 c128, real acceptance, image rocm/atom-dev:nightly_202610070333, the same image as the config.

➖ Check 3 (Upstream recipe): N/A — The PR affects only framework: atom (ATOM vendor engine). No vLLM/SGLang production serving configuration is added or changed. The sign-off links the ATOM upstream recipe for reference.

✅ Check 4 (Reuse command): PASS — /reuse-sweep-run 37597985033 was posted by seungrokj (COLLABORATOR).

✅ Check 5 (Latest checklist): PASS — All 18 items of the current template are present and checked.

✅ Check 6 (Upstream images / ordering): PASS — (a) Not applicable to an ATOM entry. (b) Vendor-specific atom entry; open-source entries dsv41flash-fp4-mi355x-vllm-agentic-dspark and dsv41flash-fp4-mi355x-sglang-agentic-dspark already exist for dsv41flash on cluster:mi355x-amds.

✅ Check 7 (Deprecated models): PASS — In MODELS.md, dsv41flash lists agentic coding as active and no deprecated scenarios.

✅ Check 8 (No arch hacks): PASS — No --hf-overrides or model edits. ATOM_PREFILL_DECODE_INTERVAL is a scheduling knob. publish_events_and_metrics is a TRT-LLM srtctl key that ATOM does not support, and ATOM server metrics were collected (70 metrics, atom adapter).

✅ Check 9 (Spec-decode chat template): PASS — The AgentX replay goes through AIPerf's agentx chat scenario. Throughput uses simulated AL, and the eval uses lm-eval local-chat-completions with --apply_chat_template.

✅ Check 10 (No engine patches): PASS — The recipe adds no patches, inline rewrites, wheel installs or file overrides. The launch command is plain atom.entrypoints.openai_server.

✅ Check 11 (Agentic golden AL): PASS — The run log shows --spec-decode-acceptance-length 3.51 injected by infx/srt_slurm/synthetic_acceptance.py. This equals dsv41flash_dspark.yaml thinking_on K=5 (3.51), matching num-speculative-tokens: 5. The eval keeps real acceptance.

➖ Check 12 (Append-only): N/A — The new changelog entry has no append-only: true.

✅ Check 13 (Draft runs as shipped): PASS — The draft is the inline DSpark head of deepseek-ai/DeepSeek-V4.1-Flash. The effective command has no --online_quant_config, dtype override or draft quantization flag, and SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE does not apply (ATOM, unset). At pinned ATOM 7fc0926c6, wo_a is served at its stored FP8 + E8M0 32x32 precision by default, and the other draft weights keep their stored dtypes. The activation and RoPE changes are image defaults that are not set by the submission. The sign-off records the per-weight evidence.

✅ Check 14 (Pareto coverage): PASS — One curve, dsv41flash / agentic-coding / mi355x ATOM / fp4 / p90 E2EL, from run 37597985033 artifact results_bmk (source d3ba89f3, image nightly_202610070333): 16 measured points, 9 on the frontier (9/5). The helper could not be executed in this sandbox, so the count applies its exact upper_right algorithm to the artifact data by hand.

Assessed commit: 368c95ad9912698e272ff3787b897fd424720b98.

[seungrokj]

@Oseltamivir @edwingao28 plz review this PR ?