Repository navigation
[AgentX] MiniMax-M3.1-Flash GB300 SGLang: update config / [AgentX] MiniMax-M3.1-Flash GB300 SGLang：更新配置 - #4686
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the contribution!
中文感谢你的贡献！
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zijiexia
added
the
full-sweep-fail-fast
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| speculative-algorithm: DSPARK | ||
| speculative-dspark-block-size: 7 | ||
| speculative-use-block-verification: true | ||
| speculative-draft-kv-cache-dtype: fp8_e4m3 |
There was a problem hiding this comment.
🔴 This new config runs the DSpark draft with an FP8 KV cache while the target keeps its default bf16 KV, which CONTRIBUTING.md forbids. Both roles set speculative-draft-kv-cache-dtype: fp8_e4m3 and set no target kv-cache-dtype. FP8 draft KV is allowed only when it is the framework default or applies to target and draft alike. Fix: drop the draft-only override in both roles so the draft keeps the framework default, or set one upstream-supported KV dtype for target and draft. Same pattern at 2 sites (disagg.yaml:119, disagg.yaml:169).
Why this was flagged
CONTRIBUTING.md:64-66 allows only the framework's default draft KV dtype, or an upstream-supported dtype applied to both target and draft. CONTRIBUTING.md:90 forbids draft-only dtype overrides that lower precision below the default. The prefill (disagg.yaml:119) and decode (disagg.yaml:169) args set speculative-draft-kv-cache-dtype: fp8_e4m3 but leave the target kv-cache-dtype unset. The repo's own changelog (perf-changelog.yaml:11515) describes this flag as storing draft KV in FP8 "instead of the bf16 default". So the new config key publishes a lowered-precision draft that the precision checks (codeowner verify Check 13) should FAIL. The single-node minimaxm31flash recipes already use this override, but this PR extends it to a new multi-node config and its results. Matching vLLM's drafter is not an exemption under CONTRIBUTING.md:102-106.
Verification: The prefill args (disagg.yaml:116-119) and the decode args (disagg.yaml:166-169) both set speculative-draft-kv-cache-dtype: fp8_e4m3. Neither role sets a target kv-cache-dtype, so the override is aimed at the draft alone. perf-changelog.yaml:11515 says this flag replaces "the bf16 default", and CONTRIBUTING.md:90 forbids such draft-only dtype overrides.
| PYTHONUNBUFFERED: "1" | ||
| PYTHONNOUSERSITE: "1" | ||
| FMHA_SM100_ALLOW_JIT: "1" | ||
| SGLANG_TIMEOUT_KEEP_ALIVE: "1800" | ||
| SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT: "3600" | ||
| SGLANG_DISAGGREGATION_WAITING_TIMEOUT: "3600" | ||
| SGLANG_MOONCAKE_CUSTOM_MEM_POOL: "True" | ||
| SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK: "32768" | ||
| NCCL_MNNVL_ENABLE: "1" | ||
| NCCL_CUMEM_ENABLE: "1" | ||
| MC_FORCE_MNNVL: "1" | ||
| MC_ENABLE_DEST_DEVICE_AFFINITY: "1" | ||
| # Some GB300 nodes list a BMC interface with the same address on every node first. | ||
| MC_TCP_BIND_ADDRESS: "{node}" | ||
| WITH_NVIDIA_PEERMEM: "0" | ||
| args: |
There was a problem hiding this comment.
🔴 Turns can get 529 rejections under load if Dynamo 1.7's default backend admission limits apply. That happens when one DP rank fills before the engine queue. The prefill and decode env blocks never raise the DYN_BACKEND_ADMISSION_* limits. Every other Dynamo 1.7 SGLang AgentX recipe raises them. Fix: add DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT, DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT and DYN_BACKEND_ADMISSION_REQUEST_QUEUE_LIMIT to both role envs so the SGLang scheduler queues work instead of Dynamo rejecting it.
Why this was flagged
This applies if Dynamo 1.7.0.dev20261030 (identity.frameworks.dynamo, disagg.yaml:13) keeps the admission behaviour that dsv41flash/sglang/gb300-fp4/agentx/disagg-dep16-hicache-variants.yaml:87-91 describes: "Dynamo 1.7 backend admission rejects turns with 529 once a DP rank is full". dsv41flash agg-variants.yaml:69-73 says the same. Both recipes set the three limits to "10000", and the minimaxm31flash vLLM gb300 recipes raise them too. The new recipe sets none of them in the prefill env (disagg.yaml:76-91) or the decode env (disagg.yaml:133-148). Decode at conc 360 runs 48 slots per DP rank (max-running-requests 384 over 8 ranks), with load-balance-method total_tokens. That is not request-count balancing, so a single rank can fill while others still have room. The AgentX client would then see failed turns, which can fail the sweep or skew its results.
Verification: The new recipe pins dynamo: "1.7.0.dev20261030" (disagg.yaml:13). Neither the prefill env (disagg.yaml:74-89) nor the decode env (disagg.yaml:131-146) sets any DYN_BACKEND_ADMISSION_* variable. The dsv41flash disagg-dep16-hicache-variants.yaml:87-91 raises all three and notes 529s once a DP rank is full. Decode allows 48 per DP rank (disagg.yaml:162-163) against conc up to 360, about 45 per rank.
| - "Add disaggregated SGLang behind Dynamo for MiniMax-M3.1-Flash AgentX on GB300 with lmsysorg/sglang:nightly-cu134-20261030-e371881 (Dynamo 1.7.0.dev20261030, Mooncake transfer, minimax-msa kernels)." | ||
| - "6 DEP2 prefill workers with a DEP8 decode at conc 280-360 or a DEP16 decode at conc 300; MegaMoE, DSpark block 7 with fp8 draft KV, Dynamo KV router with fastokens." | ||
| - "Prefill: shortest-prefill-first, chunk 8192, mem-fraction-static 0.90, HiCache 200 GB per rank write_back." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXXX |
There was a problem hiding this comment.
🟡 (optional) The new changelog block links to a placeholder PR (pull/XXXX), so ingested results for this config key point to a nonexistent PR. The changelog is append-only and byte-sensitive, so fixing it after merge goes against the append-only rule. Fix: set pr-link to this PR's real number before merge.
Why this was flagged
perf-changelog.yaml:11714 sets pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXXX. Every other entry in the file uses a real PR number, and this is the only XXXX in the file. The tooling treats the placeholder as defective: infx/workflows/recover_failed_ingest.py:243-245 emits "entry {index} has an XXX pr-link". If merged as-is, results collected for minimaxm31flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg are tied to a link that resolves nowhere. AGENTS.md:88 makes the changelog append-only and byte-sensitive, so a later fix means editing bytes that are already merged.
Verification: perf-changelog.yaml:11714 adds pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXXX. validate_perf_changelog.py:21-24 accepts only "XXX" and ".../pull/XXX" (PR_LINK_PLACEHOLDERS). As a result the merge helper fails on this PR. PR CI does not catch this: run-sweep.yml:105 runs validate_perf_changelog main(), which only calls infx.matrix.plan and never compare_entries.
| @@ -0,0 +1,202 @@ | |||
| schema: 2 | |||
There was a problem hiding this comment.
🟡 nit (optional): a two-row bundle is named disagg.yaml, but the recipe guide requires <agg|disagg>[-<axis>]-variants.yaml for multi-row entries. Every sibling sglang AgentX disagg bundle follows that rule. Fix: rename to disagg-variants.yaml and update both srt-recipe: selectors in nvidia-master.yaml.
Why this was flagged
RECIPES.md:23 says an entry with two or more rows of one design keeps them in one <agg|disagg>[-<axis>]-variants.yaml. AGENTS.md:90 repeats that rows of one design share a *-variants.yaml. The master entry selects two rows from this file: disagg.yaml:override_6p_dep2_dep8 and disagg.yaml:override_6p_dep2_dep16. Sibling sglang AgentX bundles, for example dsv41flash/sglang/gb300-fp4/agentx/disagg-dep16-hicache-variants.yaml and glm5.2/sglang/gb300-fp4/agentx/disagg-variants.yaml, use the -variants suffix. The plain name is reserved for single-row recipes, so this bundle breaks the convention that tells the two apart.
Verification: nit. The new master entry takes two rows of one design from the bundle, srt-recipe: disagg.yaml:override_6p_dep2_dep8 and srt-recipe: disagg.yaml:override_6p_dep2_dep16. AGENTS.md:90 says: "Rows of one design share a *-variants.yaml". The sibling glm5.2 has disagg-variants.yaml. No code in the repo checks the filename, so nothing fails at runtime.
| # MiniMax-M3.1-Flash AgentX on GB300: SGLang prefill and decode behind Dynamo. | ||
| # Six DEP2 prefill workers feed a DEP8 decode (conc 280-360) or a DEP16 decode | ||
| # (conc 300). The cu134 nightly at e371881 carries sgl-project/sglang#46851 | ||
| # (DSpark decode under attention DP) and the minimax-msa kernels. |
There was a problem hiding this comment.
🟡 (optional) Maintainers get header, topology and history comments that the repo's mandatory comment rule (AGENTS.md:56-57) forbids in every file. These describe the file, topology or overrides, or cite where the image fix came from (sgl-project/sglang#46851). Fix: delete every header, section and provenance comment and keep only one-line why-comments on specific values, which covers the 6 sites listed. Same pattern at 6 sites (disagg.yaml:3-6, disagg.yaml:68, disagg.yaml:127, disagg.yaml:187, disagg.yaml:191, nvidia-master.yaml:7580-7583).
Why this was flagged
AGENTS.md:54-58 makes comments mandatory-style in every file: no header comments describing a file, topology, config, recipe, override or section, and no history or provenance. The new recipe opens with a four-line header at disagg.yaml:3-6 that describes topology and upstream PR provenance. Topology section comments sit at disagg.yaml:68 ("# DEP2 prefill: attention DP2 + MoE EP2.") and disagg.yaml:127 ("# DEP8 decode across two trays."). The second one is also wrong for override_6p_dep2_dep16, which uses 4 nodes. Override section comments sit at disagg.yaml:187 and :191. The master entry repeats the header at nvidia-master.yaml:7580-7583. No sibling entry or tooling enforces this, so the comments would merge against the stated repo rule.
Verification: nit. The trigger is that the PR merges as written. AGENTS.md, which CLAUDE.md hands all guidance to, reads at lines 54-57: "No header comments describing a file, topology, config, recipe, override, or section." Lines 3-6 of the new disagg.yaml are a file and topology header with upstream provenance. Nothing breaks at runtime: comments do not change what the benchmark runs.
… trim comments Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=41185040680 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=41185040680 |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
/use 41154407006 |
|
@zijiexia staged run 41154407006. This run remains available across future |
…-sglang-pd # Conflicts: # inferencex-e2e/perf-changelog.yaml
|
/reuse-sweep-run 41154407006 |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zijiexia
added
full-sweep-fail-fast
|
/use 41185040680 |
|
@zijiexia staged run 41185040680. This run remains available across future |
…-sglang-pd # Conflicts: # inferencex-e2e/perf-changelog.yaml
|
/reuse-sweep-run 41185040680 |
edwingao28
left a comment
edwingao28
left a comment
There was a problem hiding this comment.
Reviewed: the reused sweep run 41185040680 matches the final recipe, all 5 points completed with no request errors, GSM8K is 0.958/0.959, and the changelog conflict is resolved.
|
Ingest report — ✅ Results ingested into the Ingest workflow · Merge Ingest run · Benchmark source run
benchmarkFiles 5, rawBenchmarkRows 5, coveredByPerJobFiles 0, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 2638, gpuMetricSamplesNew 0, powerPublicationPoints 5, changelogEntriesWritten 1 Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun. 中文Ingest 报告 — ✅ 结果已写入 上表仅统计本次运行写入的数据，各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。 |
Update config.
中文
更新配置。
AI model disclosure
Prepared with Claude Opus 5.5 (claude-opus-5-5).
🤖 Generated with Claude Code