[Klaud Cold] minimaxm3-fp8-gb300-dynamo-vllm-mtp: day-zero GB300 MXFP8 EAGLE3 MTP + FULL_DECODE_ONLY CG / 新增 GB300 MXFP8 EAGLE3 MTP 配方，解码启用 FULL_DECODE_ONLY 图模式 - #2486
Conversation
…DECODE_ONLY CG on decode
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| max-num-seqs: 1024 | ||
| max-num-batched-tokens: 16384 | ||
| max-cudagraph-capture-size: 2048 | ||
| cudagraph_mode: FULL_DECODE_ONLY |
There was a problem hiding this comment.
🔴 All 11 new decode vllm_config blocks in this PR set cudagraph_mode: FULL_DECODE_ONLY as a bare top-level key (e.g. line 104 in 1p1d-dep2-tp4-eagle3-c1-8k1k.yaml, and the same line in the other 10 sibling files), but every other recipe in this repo enables that mode by embedding it inside the compilation-config JSON blob (e.g. compilation-config: '{"cudagraph_mode":"FULL_DECODE_ONLY",...}'). Since srt-slurm passes top-level vllm_config keys through as --<key> CLI flags, this likely emits an unrecognized --cudagraph_mode flag (crashing decode server startup) or is silently dropped — either way defeating the PR's sole stated purpose of enabling FULL_DECODE_ONLY CUDA graphs on decode.
Extended reasoning...
The bug: every one of the 11 new decode vllm_config blocks added by this PR sets cudagraph_mode: FULL_DECODE_ONLY as a bare, top-level recipe key sitting alongside max-cudagraph-capture-size, kv-cache-dtype, etc. For example, in 1p1d-dep2-tp4-eagle3-c1-8k1k.yaml:
decode:
...
max-cudagraph-capture-size: 2048
cudagraph_mode: FULL_DECODE_ONLYWhy this deviates from the established pattern: grepping the repo shows cudagraph_mode is enabled in 100+ places across dozens of recipe files, and in every single one of them it is embedded inside a compilation-config JSON string, never set as a bare key. For instance, the sibling minimax-m3/b200-fp4/8k1k/*.yaml recipes (same model family) do:
compilation-config: '{"cudagraph_mode":"FULL_DECODE_ONLY","custom_ops":["+rms_norm"],"pass_config":{}}'The same pattern holds for the deepseek-v4 and kimi-k2.5-fp4 recipe families. This is because cudagraph_mode is a field of vLLM's CompilationConfig, not a standalone engine-args flag — there is no --cudagraph-mode/--cudagraph_mode CLI argument in vLLM's argparse; it can only be set via --compilation-config (JSON) or the -O dot-notation shorthand. That's precisely why every other recipe author routed it through the JSON blob instead of a bare key.
How this breaks at runtime: the other 20+ keys in these same vllm_config blocks (no-enable-flashinfer-autotune, kv-transfer-config, max-cudagraph-capture-size, tensor-parallel-size, etc.) are all kebab-case, matching srt-slurm's convention of passing top-level vllm_config keys through verbatim as --<key> CLI flags to vllm serve. cudagraph_mode is the only snake_case key in these files — it looks like the author copy-pasted the JSON field name but forgot to wrap it inside compilation-config. Under that passthrough convention, this key gets emitted as --cudagraph_mode FULL_DECODE_ONLY. Since vLLM has no such CLI flag, this either (a) is rejected by argparse as an unrecognized argument, crashing decode-server startup, or (b) is silently dropped by the recipe-to-CLI translation, in which case FULL_DECODE_ONLY is simply never applied.
Step-by-step proof:
- PR description states the sole functional change is: 'in all 11 decode vllm_config sections, add cudagraph_mode: FULL_DECODE_ONLY.'
- Every established recipe in the repo that has ever enabled this mode does so via
compilation-config: '{"cudagraph_mode":"FULL_DECODE_ONLY",...}'— confirmed by grep across 70+ files, including the same-modelminimax-m3/b200-fp4sibling recipes. - These 11 new files instead add a bare top-level
cudagraph_mode: FULL_DECODE_ONLYkey at decode-block scope (e.g. line 104 of1p1d-dep2-tp4-eagle3-c1-8k1k.yaml), with none of the 11 decode blocks touchingcompilation-configat all. - srt-slurm's recipe schema maps top-level
vllm_configkeys to--<key>vLLM CLI flags 1:1 (evidenced by every other kebab-case key in the same block mapping directly to a real vLLM flag). - vLLM's CLI has no top-level
--cudagraph-mode/--cudagraph_modeengine argument — that field only exists insideCompilationConfig, settable via--compilation-configJSON. - Therefore this key either crashes the decode vLLM server at startup (unrecognized argument) or is silently ignored, and in neither case is FULL_DECODE_ONLY CUDA graph mode actually enabled — defeating the entire stated purpose of the PR.
The fix: merge the key into the decode compilation-config JSON blob, consistent with every other recipe, e.g. add compilation-config: '{"cudagraph_mode":"FULL_DECODE_ONLY"}' to each of the 11 decode blocks (or fold it into an existing compilation-config entry if one is later added) instead of the current bare cudagraph_mode: FULL_DECODE_ONLY key.
|
|
||
| - config-keys: | ||
| - minimaxm3-fp8-gb300-dynamo-vllm-mtp | ||
| scenario-type: | ||
| - fixed-seq-len | ||
| description: | ||
| - "Add MiniMax-M3-MXFP8 GB300 8k1k disaggregated Dynamo-vLLM EAGLE3 MTP Pareto" | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2478 | ||
|
|
||
| - config-keys: | ||
| - minimaxm3-fp8-gb300-dynamo-vllm-mtp | ||
| scenario-type: | ||
| - fixed-seq-len | ||
| description: | ||
| - "Bump MiniMax-M3-MXFP8 GB300 EAGLE3 MTP recipes to vLLM nightly 5e35a6f4f and enable CUTLASS MSA sparse decode backend on decode servers" | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2478 |
There was a problem hiding this comment.
🟡 Both new perf-changelog.yaml entries for minimaxm3-fp8-gb300-dynamo-vllm-mtp link to PR #2478 instead of #2486, which is the PR actually landing this config-key on main (#2478 is still open/unmerged). Every other entry in the file points to its own introducing PR, so this reads as a stale reference carried over from the source branch; it should point to #2486. The entries also don't mention the cudagraph_mode: FULL_DECODE_ONLY addition, which is this PR's headline change.
Extended reasoning...
What's wrong: The two new perf-changelog.yaml entries (lines 5435 and 5443) for minimaxm3-fp8-gb300-dynamo-vllm-mtp both set pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2478. This PR is #2486. Verifiers checked via the GitHub REST API and confirmed #2478 is still open/unmerged, with head branch minimaxm3-fp8-gb300-dynamo-vllm-mtp — i.e. it's the exact source branch this PR describes itself as a "verbatim port" of. So #2478 will never itself land on main; #2486 is the PR that actually introduces minimaxm3-fp8-gb300-dynamo-vllm-mtp into configs/nvidia-master.yaml.
Why this is inconsistent with the file's convention: Scanning the sibling entries in perf-changelog.yaml shows a uniform pattern — every pr-link points to the PR that actually merged the corresponding change into main (e.g. #2473, #2471, #2483, #2470, #2468, and #2446 for the minimaxm3-fp* family all self-reference their own introducing PR). These two new entries break that pattern by pointing at a PR that is superseded by this one and will likely be closed unmerged once #2486 lands, leaving a dangling/misleading link.
Why existing tooling doesn't catch it: pr-link is purely informational — per AGENTS.md, only config-keys drives CI/benchmark triggering, so nothing in the pipeline validates or dereferences this field. It's a silent documentation drift that only a human (or link-checker) tracing history would notice.
Step-by-step proof:
- This PR ([Klaud Cold] minimaxm3-fp8-gb300-dynamo-vllm-mtp: day-zero GB300 MXFP8 EAGLE3 MTP + FULL_DECODE_ONLY CG / 新增 GB300 MXFP8 EAGLE3 MTP 配方，解码启用 FULL_DECODE_ONLY 图模式 #2486) adds the
minimaxm3-fp8-gb300-dynamo-vllm-mtpconfig-key toconfigs/nvidia-master.yamlfor the first time (no such key existed in main before this diff). perf-changelog.yamlgets two new entries for that config-key (lines 5428-5443), both withpr-link: .../pull/2478.- Querying GitHub for PR Add MiniMax-M3-MXFP8 GB300 disaggregated Dynamo-vLLM EAGLE3 MTP #2478 shows it is OPEN, not merged, with head ref
minimaxm3-fp8-gb300-dynamo-vllm-mtp— the literal source branch this PR's description says was "ported onto current main." - Therefore, once [Klaud Cold] minimaxm3-fp8-gb300-dynamo-vllm-mtp: day-zero GB300 MXFP8 EAGLE3 MTP + FULL_DECODE_ONLY CG / 新增 GB300 MXFP8 EAGLE3 MTP 配方，解码启用 FULL_DECODE_ONLY 图模式 #2486 merges, the config-key exists on main because of [Klaud Cold] minimaxm3-fp8-gb300-dynamo-vllm-mtp: day-zero GB300 MXFP8 EAGLE3 MTP + FULL_DECODE_ONLY CG / 新增 GB300 MXFP8 EAGLE3 MTP 配方，解码启用 FULL_DECODE_ONLY 图模式 #2486, but the changelog says Add MiniMax-M3-MXFP8 GB300 disaggregated Dynamo-vLLM EAGLE3 MTP #2478 — a PR that isn't merged and (per the port framing) isn't expected to be.
- Contrast with e.g. the entry for
minimaxm3-fp4-b300-dynamo-vllm-mtp-legacy-dep4a few lines above, whosepr-linkcorrectly points at feat(minimaxm3-fp4-b300-dynamo-vllm-mtp-legacy-dep4): bump vLLM nightly to 5e35a6f4, enable cutlass MSA decode / 更新 minimaxm3-fp4-b300-dynamo-vllm-mtp-legacy-dep4 vLLM 镜像至 5e35a6f4，启用 cutlass MSA 解码后端 #2483, the actual PR that merged that change.
On the refutation: One reviewer argued the pr-link might be intentional attribution to the "substantive originating PR" #2478 (crediting where the recipes and Pareto data were actually developed), pointing to the word "Bump" in the second entry's description as evidence it documents #2478's own history rather than this port. That's a plausible provenance argument, but it cuts against the file's own established convention — no other entry in perf-changelog.yaml uses pr-link as an "origin/attribution" pointer; every other entry uses it as "the PR that merged this into main," including entries for other verbatim-port-style bumps. Given #2478 won't merge on its own, keeping it as the link produces a permanently-dangling reference to a closed/superseded PR, which is worse for future traceability than pointing at #2486, the PR that actually shipped the change.
Impact and fix: No functional impact — this doesn't affect CI, benchmark triggering, or recipe correctness, since pr-link is metadata only. It's a one-line fix: change both pr-link values to https://github.com/SemiAnalysisAI/InferenceX/pull/2486, and optionally add a short line noting the cudagraph_mode: FULL_DECODE_ONLY addition to decode configs, since that's the actual headline change of this PR and isn't otherwise documented in the changelog.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30878275937 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30913091341 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30933638204 |
|
/reuse-sweep-run |
# Conflicts: # perf-changelog.yaml
2 participants
Summary
Verbatim port of
minimaxm3-fp8-gb300-dynamo-vllm-mtpbranch onto current main, with one addition:cudagraph_mode: FULL_DECODE_ONLYadded to all 11 decode vllm_config sections.中文说明
将
minimaxm3-fp8-gb300-dynamo-vllm-mtp分支原样移植至当前 main，并在全部 11 个解码 vllm_config 中新增cudagraph_mode: FULL_DECODE_ONLY。