[AMD][AgentX] Add DeepSeek-V4-Pro FP4 ATOM PD-disagg AgentX hicache MTP on MI355X / 在 MI355X 上添加 DeepSeek-V4-Pro FP4 ATOM PD 分离式 AgentX hicache MTP recipe

[seungrokj]

Summary

Add the ATOM PD-disaggregated agentic arm for DeepSeek-V4-Pro-0813 MXFP4 on MI355X — dsv4-fp4-mi355x-atom-disagg-agentic-lmcache-dspark — the ATOM counterpart to dsv4-fp4-mi355x-sglang-disagg-agentic-hicache-mtp, so the two engines are directly comparable on the AgentX trace workload. 1P1D TP8, mooncake RDMA KV transfer, atomesh PD router, cluster:mi355x-amds. Modeled on ATOM recipes/DeepSeek-V4-Agentic-PD-Max.md.

Docker image

rocm/atom-dev:nightly_202609221542

Recipe / search space

• TP8 (EP1): conc 1, 16 — MTP, GPU-resident KV, dram-utilization 0.80.
• DP-attention: conc 64, 128 — MTP, GPU-resident KV, dram-utilization 0.80.
• DP-attention + CPU offload: conc 256 — DSpark draft model + lmcache host KV tier (multi connector, max_pending_saves=8, slot_sidecar_staging_slots=4, LRU, 256-token chunks), dram-utilization 0.50, dp-sticky idx2idx routing.
• conc 1/16/64 keep the shape run 35643358891 used and change only the image.

Per ATOM DeepSeek-V4-Agentic-PD-Max recipe: prefix caching on, FP8 KV and index cache, block-size 256, TBO on prefill only, max-num-seqs = 2× concurrency.

online_quant

This config does not use online_quant. server_atom.sh only emits --online_quant_config (ONLINE_QUANT_ARG) when the model entry sets online_quant_config / online_quant_dpa_config in benchmarks/multi_node/amd_utils/models_atom.yaml (read at server_atom.sh:65-66, assembled at :156-172, injected into the prefill/decode server commands at :317/561/652). The DeepSeek-V4-Pro-AgentX entry (and its -0813-AgentX alias) defines neither field, so ONLINE_QUANT_ARG stays empty and --online_quant_config is never passed. Online (runtime) quantization is unnecessary here because the served DeepSeek-V4-Pro-0813 checkpoint is already offline MXFP4-quantized; online_quant (e.g. the ptpc_fp8 configs in deprecated/models_atom.yaml) is for BF16 / unquantized checkpoints.

DSpark accuracy is not touched by online_quant. Because online_quant is off, no target layer is re-quantized at runtime, so it cannot perturb draft/target agreement (acceptance length) or gsm8k. The DSpark draft head is read from the -0813 config.json and verifies against the unchanged MXFP4 target; the gsm8k numbers below reflect the offline MXFP4 checkpoint + DSpark only.

Measured results

• conc-128 (run 35810485934, 3627 s, 13917/15339 profiled, 0 errors): 464105 total tok/s (29007 tok/s/GPU over 16 GPUs), 3934 output tok/s, TTFT mean 9.75 s / p90 15.89, TPOT mean 12.98 ms / p90 14.43, interactivity mean 77.0 / p90 69.3, e2e mean 23.01 s, 3.84 QPS, 96.6% theoretical prefix-cache hit.
• conc-256 (run 35825955863, 3630 s, 21317/24162 profiled, 0 errors): 776348 total tok/s (48522 tok/s/GPU), 5794 output tok/s, TTFT mean 18.10 s / p90 32.78, TPOT mean 15.95 ms / p90 17.57, interactivity mean 62.7 / p90 56.9, e2e mean 33.89 s, 5.88 QPS, 96.8% theoretical prefix-cache hit. Not a clean scaling comparison vs conc-128 — this tier also swaps MTP for the DSpark draft model and adds the lmcache host KV tier.

Accuracy

conc-128 DP-attention cell (run 35851118405, lm-eval gsm8k 5-shot, eval concurrency 128, n=1319): em_strict 0.9719 ± 0.0045, em_flexible 0.9712 ± 0.0046. conc 1, 16, 64 are unmeasured on this image.

Notes

• server_atom.sh exports PORT before sourcing benchmark_lib.sh on the eval path: run_lm_eval's check_env_vars guard runs before it parses --port, and job.slurm's docker -e allowlist forwards ROUTER_PORT but never PORT, so every multi-node lm-eval cell aborted ~8 s in and hung the decode node in "Waiting until router closes..." until cancelled (run 35643358891).

中文

在 MI355X 上新增 DeepSeek-V4-Pro-0813 MXFP4 的 ATOM PD 分离式 agentic 分支 dsv4-fp4-mi355x-atom-disagg-agentic-lmcache-dspark，作为 dsv4-fp4-mi355x-sglang-disagg-agentic-hicache-mtp 的 ATOM 对照。1P1D TP8，mooncake RDMA KV 传输，atomesh PD 路由，镜像 rocm/atom-dev:nightly_202609221542。参考 ATOM recipes/DeepSeek-V4-Agentic-PD-Max.md。

搜索空间：TP8 并发 1,16 与 DP-attention 并发 64,128（MTP、GPU 常驻 KV、dram-utilization 0.80）；并发 256 使用 DSpark 草稿模型 + lmcache 主机 KV 档（dram-utilization 0.50，dp-sticky idx2idx 路由）。

online_quant 未启用：server_atom.sh 仅在 models_atom.yaml 设置了 online_quant_config/online_quant_dpa_config 时才输出 --online_quant_config；DeepSeek-V4-Pro-AgentX（及 -0813-AgentX 别名）两者都未设置，故 ONLINE_QUANT_ARG 为空，该标志不会传入。所服务的 DeepSeek-V4-Pro-0813 已是离线 MXFP4 量化，无需在线量化（在线量化用于 BF16/未量化 checkpoint）。因此 online_quant 不影响 DSpark 精度：不会在运行时重新量化 target 层，草稿/目标一致性（接受长度）与 gsm8k 均不受其影响。

[claude]

Additional findings (outside the current diff — GitHub can't attach inline comments there):

• 🔴 benchmarks/multi_node/amd_utils/server_atom.sh — The new DeepSeek-V4-Pro-AgentX model entry is silently never applied on ATOM: server_atom.sh looks up models_atom.yaml with the bare ${MODEL_NAME} and never appends -AgentX, so every agentic-only knob (attn-prefill-chunk-size, state-checkpoint-interval-tokens, level, spec-decode-acceptance-length, FP8 kv-cache, prefill-only TBO, block-size 256, DP session-affinity env) is dropped and the plain DeepSeek-V4-Pro throughput recipe runs instead. server_sglang.sh does this correctly: its own python heredoc computes is_agentic/model_key = f'{model_name}-AgentX' from IS_AGENTIC (server_sglang.sh:88-89); server_atom.sh has no equivalent logic. …

  Extended reasoning...

  …Fix: mirror that suffix resolution (with the same DI/AgentX/bare-name fallback) in server_atom.sh's yaml lookup before m = yaml.safe_load(f).get(...).

  job.slurm:50 computes MODEL_YAML_KEY="${MODEL_NAME}-AgentX" only to validate the key exists in models_atom.yaml (job.slurm:54-63); it never exports that suffixed value. MODEL_NAME itself stays bare and is exported unchanged at job.slurm:307. server_atom.sh:87 does yaml.safe_load(f).get('${MODEL_NAME}', {}), i.e. get('DeepSeek-V4-Pro', {}) regardless of IS_AGENTIC. Since 'DeepSeek-V4-Pro' also exists as a top-level key (the throughput entry), .get() returns that dict instead of {} or erroring, so the failure is silent: no crash, just wrong config. Every field the PR added under DeepSeek-V4-Pro-AgentX (models_atom.yaml lines ~48-75) becomes dead code for this recipe: MODEL_ENVS lacks MC_GID_INDEX/ATOM_DISABLE_MMAP/ATOM_PREFIX_CACHE_POLICY/ATOM_DP_SESSION_AFFINITY, prefill_tp_dp_flags lacks --enable-tbo, kv_cache_flags lacks fp8, block_size/mem_frac_static/max_num_batched_tokens fall back to throughput defaults, and…

  Verification: normal. The new ATOM agentic recipe silently never loads its DeepSeek-V4-Pro-AgentX YAML entry; it falls back to the plain throughput config. Reachable chain: - server.sh:18-20 dispatches ENGINE=atom-disagg -> sources server_atom.sh. - job.slurm:50 computes MODEL_YAML_KEY="${MODEL_NAME}-AgentX" only for existence validation (54-63); it is never exported. MODEL_NAME is exported bare at…

[claude]

🔴 The new changelog entry's pr-link is https://github.com/SemiAnalysisAI/InferenceX/pull/TBD, which fails the required changelog gate and blocks this PR from merging. validate_perf_changelog.py's validate_added_pr_link() only accepts the exact PR-number URL or the literal placeholders "XXX"/".../pull/XXX" (PR_LINK_PLACEHOLDERS, CANONICAL_PR_LINK regex requires \d+); "TBD" matches neither. Fix: set pr-link to the real PR number URL, or use the canonical "XXX" placeholder if the number isn't known yet.

Extended reasoning...

infx/workflows/validate_perf_changelog.py:144 validate_added_pr_link(link, pr_number) is run by the changelog gate CI check on every PR touching perf-changelog.yaml. For an open PR, pr_number is not None, so it computes expected = '.../pull/<pr_number>' and raises ChangelogValidationError unless link is in PR_LINK_PLACEHOLDERS ({'XXX', '.../pull/XXX'}) or equals expected. The added entry (perf-changelog.yaml:7819) has link '.../pull/TBD', which is neither, so the check raises 'new PR entry must use or an XXX placeholder; found .../pull/TBD'. No code in the repo rewrites 'TBD' before that check runs (grep for TBD outside perf-changelog.yaml finds nothing). Same failure occurs on the main-branch path (pr_number is None) which requires CANONICAL_PR_LINK.fullmatch(link) — also fails on the literal string 'TBD'. Net effect: CI blocks merge until the author replaces the placeholder.

Verification: normal. The appended changelog entry ends with pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/TBD (perf-changelog.yaml, appended block in the diff). The changelog gate rejects this value. In infx/workflows/validate_perf_changelog.py, CANONICAL_PR_LINK (lines 21-23) is .../pull/\d+, and TBD is not digits; PR_LINK_PLACEHOLDERS (lines 24-27) = {"XXX", ".../pull/XXX"}, which…

[github-actions]

View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=36159874837

View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=36159874837

[seungrokj]

full sweep: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/35855544139

[seungrokj]

@billishyahao @chunfangamd plz review this

[chunfangamd]

As a PR reviewer and CODEOWNER, I have reviewed this and have:

• Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
• Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
• Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/35855544139
• Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/35855544139
• Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
• Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. See Draft-model precision for what counts as the default and the MLPerf comparison.
• For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
• Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
• Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
• If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
• If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
• Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
  • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
• Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
• If this PR uses append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it.
• If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
• Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.

Additional detail section:

• insert any additional info here

Signed: @chunfangamd

[github-actions]

❌❌❌ REJECTED ❌❌❌

⚠️ Pareto coverage needs additional review: @functionstackx @cquil11 @Oseltamivir @adibarra. At least 5 points per affected throughput-versus-E2EL frontier are highly recommended. Below 5, or when coverage cannot be verified, merge only with an explicit, recorded admin bypass for the assessed commit; this advisory comment does not grant or enforce a bypass.

@chunfangamd — blocking: the sign-off's "Additional detail section" is the untouched template placeholder ("insert any additional info here"), so the draft-precision evidence that the checklist's draft-as-shipped item requires was never recorded, and Check 13 cannot be verified from the sign-off. Please edit the existing review to add it (checkpoint/head, stored precision, pinned-image default handling, effective serving precision, and a statement on AITER_BF16_FP8_MOE_BOUND=0), then re-verify.

❌ Check 13 (Draft runs as shipped): FAIL — Draft precision could not be verified: the sign-off records no draft evidence (no checkpoint/head identification, stored precision, pinned-image default handling, or effective serving precision). Independent inspection: the draft is the in-checkpoint DSpark head (mtp.0..2, target layers 58-60) of deepseek-ai/DeepSeek-V4-Pro-0813 (revision 72e1d32), stored as FP8 E4M3 + E8M0 scales for attention/dense/shared-expert weights and packed FP4 (expert_dtype: fp4) routed experts; the served command runs --method dspark --num-speculative-tokens 3 on that same checkpoint with online_quant_config: None, hf_overrides: None, and FP8 KV shared by target and draft (allowed). Unresolved lead the sign-off must address: the model env in benchmarks/multi_node/amd_utils/models_atom.yaml sets AITER_BF16_FP8_MOE_BOUND=0; in the pinned aiter fused_moe.py this bound defaults to 256 and, on gfx950 with interleaved MXFP4 experts (ATOM_MOE_GU_ITLV=1), selects bf16 MoE activations below the bound and fp8 at or above it, so 0 forces fp8 MoE activations for every batch, and the draft's mtp.*.ffn.experts share that MoE path. Whether this lowers effective draft MoE activation precision below the pinned image's default is not excluded by the diff and must be documented (the ATOM PD-Max recipe setting it does not override the rule). Evidence: run 35855544139 c1 server log, checkpoint config.

⚠️ Check 14 (Pareto coverage): WARN — curve dsv4 / agentic-coding / cluster:mi355x-amds / fp4 / atom-disagg / rocm/atom-dev:nightly_202609221542 / p90_e2el / run 35855544139 (source SHA 1ed4ab45): 4/5 frontier points from 5 measured bmk_agentic_* artifacts (c16, c64, c128, c256 on the frontier; c1 is dominated by c16 at higher p90 E2EL and lower tput_per_gpu). Helper infx.workflows.pareto_coverage reproduces 4/5, no canonical-flag restriction stamped. No admin-exception comment exists on the PR; admin bypass not requested and not verified. Request an explicit admin bypass for SHA 5499d1cc and this curve, or add a measured point (for example one more concurrency between c16 and c256).

Passed and not applicable checks

✅ Check 0 (CODEOWNER): PASS — @chunfangamd is a named owner of configs/amd-master.yaml; the other four changed paths fall under the * catch-all, which any CODEOWNER satisfies.

✅ Check 1 (Passing sweep on in-PR commit): PASS — commit 1ed4ab45 is in the PR and carries success for all five multi-node agentic / cells (c1, c16, c64, c128, c256) and all three multi-node agentic eval / cells in run 35855544139; head 5499d1cc unchanged since the sign-off.

✅ Check 2 (Evals pass): PASS — gsm8k 5-shot, n=1319, chat-completions, real acceptance (no forced AL on eval-only cells): c16 em_strict 0.978, c128 0.976, c256 (DSpark + lmcache tier) 0.922; all above the dsv4 bar of 0.91 in infx/evals/thresholds.yaml, on image rocm/atom-dev:nightly_202609221542 matching the PR config. Note the c256 cell sits about 5 points below the other two.

➖ Check 3 (Recipe linked/merged): N/A — disaggregated/multi-node submission (benchmarks/multi_node/**, multinode: true, disagg: true, framework: atom-disagg); the recipe-link requirement applies to single-node recipes only.

✅ Check 4 (Reuse command): PASS — /reuse-sweep-run 35855544139 posted as a whole-line comment by seungrokj (COLLABORATOR) on 2026-09-23.

✅ Check 5 (Latest checklist template): PASS — every item of the current docs/PR_REVIEW_CHECKLIST.md template has a corresponding checked item in the sign-off.

✅ Check 6 (Upstream images / engine-first): PASS — entry dsv4-fp4-mi355x-atom-disagg-agentic-lmcache-dspark uses framework: atom-disagg, so the vLLM/SGLang upstream-image rule does not apply to it; engine-first ordering is satisfied by existing dsv4-fp4-mi355x-vllm-agentic-mtp (vLLM) and dsv4-fp4-mi355x-sglang-agentic-mtp (SGLang) entries for dsv4 on cluster:mi355x-amds.

✅ Check 7 (No deprecated models/scenarios): PASS — dsv4 agentic-coding is active in MODELS.md on 2026-09-23, and the DSpark arm on DeepSeek-V4-Pro-0813 is the agreed plan of record.

✅ Check 8 (No architecture hacks): PASS — engine kwargs show hf_overrides: None; no layer/expert/indexer trimming; --index-cache-dtype fp4, FP8 KV, --level 3, and the prefill-chunk/state-checkpoint knobs change precision or scheduling, not model FLOPs, and match the merged single-node ATOM dsv4 recipe.

✅ Check 9 (Spec-decode via chat template): PASS — trace replay runs aiperf with --endpoint /v1/chat/completions --endpoint-type chat (benchmark_lib build_replay_cmd) against the atomesh router; evals also hit /v1/chat/completions.

✅ Check 10 (No engine patches): PASS — no .patch, git apply, sed -i, heredoc rewrites, or engine wheel installs; setup_deps.sh installs nothing for ENGINE=atom-disagg, and only client-side agentic deps are installed.

✅ Check 11 (Agentic spec-decode golden AL): PASS — throughput cells run --spec-decode-acceptance-length 3.01 with --method dspark --num-speculative-tokens 3, equal to golden_al_distribution/dsv4-pro-0813-dspark.yaml thinking_on level 3 (3.01); server logs show 3.01 toks/fwd achieved, and eval-only cells omit the flag (real acceptance 3.25-3.62). Informational: the master-config labels the c1-c128 tiers spec-decoding: "mtp" although the served method is DSpark, and the launcher's SPEC_DECODE_AL=2.49 default is overridden by the YAML value.

➖ Check 12 (Append-only): N/A — the new perf-changelog.yaml entry does not set append-only: true.

Assessed commit: 5499d1cc4a5c428c9489493382ef4f493423f468.

[chunfangamd]

As a PR reviewer and CODEOWNER, I have reviewed this and have:

• Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
• Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
• Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/35855544139
• Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/35855544139
• Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
• Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. See Draft-model precision for what counts as the default and the MLPerf comparison.
• For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
• Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
• Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
• If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
• If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
• Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
  • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
• Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
• If this PR uses append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it.
• If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
• Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.

Additional detail section:

• insert any additional info here
  • "online_quant is not used on this arm: server_atom.sh only emits --online_quant_config when models_atom.yaml sets online_quant_config/online_quant_dpa_config (read at server_atom.sh:65-66, assembled at :156-172, injected into the prefill/decode server commands at :317/561/652), and the DeepSeek-V4-Pro-AgentX entry and its -0813-AgentX alias set neither, so ONLINE_QUANT_ARG stays empty and the flag is never passed. The served DeepSeek-V4-Pro-0813 checkpoint is already offline MXFP4, so runtime quantization is unnecessary (online_quant targets BF16/unquantized checkpoints, e.g. the ptpc_fp8 configs in deprecated/models_atom.yaml). DSpark accuracy is therefore not touched by online_quant: no target layer is re-quantized at runtime, so acceptance length and the gsm8k numbers above reflect the offline MXFP4 target plus the DSpark draft head only."

Signed: @chunfangamd

[github-actions]

❌❌❌ REJECTED ❌❌❌

⚠️ Pareto coverage needs additional review: @functionstackx @cquil11 @Oseltamivir @adibarra. At least 5 points per affected throughput-versus-E2EL frontier are highly recommended. Below 5, or when coverage cannot be verified, merge only with an explicit, recorded admin bypass for the assessed commit; this advisory comment does not grant or enforce a bypass.

@chunfangamd — blocking: Check 13 (draft runs as shipped) fails. The updated additional detail section only argues that online_quant is off; it still does not record the draft head's stored precision, the pinned image's default handling, or the effective serving precision, and it does not address AITER_BF16_FP8_MOE_BOUND=0, which the previous verdict flagged. Independent tracing shows that env var lowers the DSpark draft's MoE activation precision below the pinned image's default. Please either drop AITER_BF16_FP8_MOE_BOUND=0 from the DeepSeek-V4-Pro-AgentX env in benchmarks/multi_node/amd_utils/models_atom.yaml (re-sweep) or document, with pinned-implementation evidence, why it does not reach the draft, then edit the existing review and re-verify.

❌ Check 13 (Draft runs as shipped): FAIL — Draft served below its shipped baseline, and draft evidence is missing from the sign-off. Baseline: the embedded DSpark head mtp.0..2 of deepseek-ai/DeepSeek-V4-Pro-0813 stores routed experts as packed MXFP4 (expert_dtype: fp4, E8M0 scales) and attention/shared-expert/main_proj weights as FP8 E4M3; in ATOM DSparkLayer inherits the V4 Block, so those experts run through the same FusedMoE → aiter fused_moe path (per_1x32, gate_mode=INTERLEAVE under ATOM_MOE_GU_ITLV=1, no explicit activation dtype), where aiter's resolve_activation_dtype picks BF16 activations for M < AITER_BF16_FP8_MOE_BOUND (default 256) and FP8 at or above it on gfx950. The recipe env AITER_BF16_FP8_MOE_BOUND=0 (models_atom.yaml DeepSeek-V4-Pro-AgentX.env, exported in the run 35855544139 c1 server log) forces FP8 activations at every batch size, including the draft's small decode batches where the default is BF16. That is submission-side online quantization of draft activations via an environment variable (CONTRIBUTING.md "Draft-model precision", forbidden list). The ATOM PD-Max recipe setting the same value does not override the rule, and ATOM source does not set it by default. Note: verified against aiter fused_moe.py at commit 12bb8056 (2026-09-22T15:31Z, immediately before the nightly_202609221542 build); the image contents themselves were not inspected. Other draft settings are compliant: online_quant_config: None, hf_overrides: None, --method dspark --num-speculative-tokens 3 on the same checkpoint, FP8 KV shared by target and draft.

⚠️ Check 14 (Pareto coverage): WARN — curve dsv4 / agentic-coding / cluster:mi355x-amds / fp4 / atom-disagg / rocm/atom-dev:nightly_202609221542 / p90_e2el / run 35855544139 attempt 1 (source SHA 1ed4ab45): 4/5 frontier points from 5 measured bmk_agentic_* artifacts (tput_per_gpu vs p90_e2el: c16, c64, c128, c256 on the frontier; c1 is dominated by c16 at higher p90 E2EL and lower throughput). infx.workflows.pareto_coverage reproduces 4/5 with no canonical-flag restriction. No admin-exception comment exists on the PR; admin bypass not requested and not verified (@chunfangamd and @seungrokj hold write, not admin). Request an explicit admin bypass for SHA 5499d1cc and this curve, or add a measured point.

Passed and not applicable checks

✅ Check 0 (CODEOWNER): PASS — @chunfangamd is a named owner of configs/amd-master.yaml; the other four changed paths fall under the * catch-all, which any CODEOWNER satisfies.

✅ Check 1 (Passing sweep on in-PR commit): PASS — commit 1ed4ab45 is in the PR and carries success for all five multi-node agentic / cells (c1, c16, c64, c128, c256) and all three multi-node agentic eval / cells in run 35855544139; head 5499d1cc is unchanged since the sign-off.

✅ Check 2 (Evals pass): PASS — gsm8k 5-shot, n=1319, on image rocm/atom-dev:nightly_202609221542 matching the PR config: c16 em_strict 0.978, c128 0.976, c256 (DSpark + lmcache tier) 0.922; all above the dsv4 bar of 0.91 in infx/evals/thresholds.yaml. Eval-only cells run real acceptance (no forced AL).

➖ Check 3 (Recipe linked/merged): N/A — disaggregated/multi-node submission (benchmarks/multi_node/**, multinode: true, disagg: true, framework: atom-disagg); the recipe-link requirement applies to single-node recipes only.

✅ Check 4 (Reuse command): PASS — /reuse-sweep-run 35855544139 posted as a whole-line comment by seungrokj (COLLABORATOR).

✅ Check 5 (Latest checklist template): PASS — every item of the current docs/PR_REVIEW_CHECKLIST.md template has a corresponding checked item in the sign-off.

✅ Check 6 (Upstream images / engine-first): PASS — the only entry, dsv4-fp4-mi355x-atom-disagg-agentic-lmcache-dspark, uses framework: atom-disagg, so the vLLM/SGLang upstream-image rule does not apply to it; engine-first ordering is satisfied by the existing dsv4-fp4-mi355x-vllm-agentic-mtp and dsv4-fp4-mi355x-sglang-agentic-mtp entries on cluster:mi355x-amds.

✅ Check 7 (No deprecated models/scenarios): PASS — dsv4 agentic coding is active in MODELS.md on 2026-09-23, and the DSpark arm on DeepSeek-V4-Pro-0813 is the agreed plan of record.

✅ Check 8 (No architecture hacks): PASS — server logs show hf_overrides: None and online_quant_config: None; no layer/expert/indexer trimming. --index-cache-dtype fp4, FP8 KV, --level 3 and the prefill-chunk/state-checkpoint knobs change precision or scheduling, not model FLOPs.

✅ Check 9 (Spec-decode via chat template): PASS — trace replay runs aiperf with --endpoint /v1/chat/completions --endpoint-type chat (build_replay_cmd in benchmarks/benchmark_lib.sh) against the atomesh router.

✅ Check 10 (No engine patches): PASS — no .patch, git apply, sed -i, heredoc rewrites of engine sources, or engine wheel installs anywhere in the diff.

✅ Check 11 (Agentic spec-decode golden AL): PASS — throughput cells run --spec-decode-acceptance-length 3.01 with --method dspark --num-speculative-tokens 3, equal to golden_al_distribution/dsv4-pro-0813-dspark.yaml thinking_on level 3 (3.01); the server log confirms forced acceptance at 3.01, and eval-only cells omit the flag. Informational: the master config labels the c1–c128 tiers spec-decoding: "mtp" although the served method is DSpark.

➖ Check 12 (Append-only): N/A — the new perf-changelog.yaml entry does not set append-only: true.

Assessed commit: 5499d1cc4a5c428c9489493382ef4f493423f468.

[adibarra]

/reuse-sweep-run 35855544139