[AgentX] Tune DSV4 GB300 MTP / 调优 DSV4 GB300 MTP - #2571
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
aa7d626 to
39a457c
Compare
ivanium
added
full-sweep-enabled
agentx
|
Claude finished @ivanium's task in 3m 51s —— View job Review of PR #2571
LGTM - no blocking issues found Verified: |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
LGTM — the reported issues are minor nits (changelog blank-line separator, a duplicable search-space entry, and a missing env var on one sibling config) that don't affect correctness of the sweep.
Extended reasoning...
Overview
This PR tunes GB300 AgentX MTP benchmark recipes for DeepSeek-V4-Pro: scheduler/CUDA-graph/memory settings across four existing recipe YAMLs, a new disagg 2P1D DEP8/DEP16 c1024 recipe, wiring changes in configs/nvidia-master.yaml's search-space definitions, and an appended perf-changelog.yaml entry. All changes are benchmark-harness config (SLURM recipes, env vars, vLLM engine args, sweep matrix entries) — no application/serving code paths are touched.
Security risks
None identified. These are benchmark configuration files consumed by an internal SLURM-based sweep harness; there's no user input handling, auth, or crypto surface in scope.
Level of scrutiny
This warrants light scrutiny: it's config-only tuning following the repo's well-established recipe pattern (topology, env vars, gpu-memory-utilization, cudagraph capture sizes), consistent with many prior similar PRs in this repo's history (e.g. the recent Kimi K3 DSpark and GLM-5.2 tuning PRs). The three findings from the bug-hunting pass are all nits: a missing blank-line separator in the changelog, a mergeable duplicate search-space entry, and a missing env var on one sibling config that only affects benchmark-harness flakiness (not served-model correctness). None of them block the sweep from running correctly or produce incorrect results.
Other factors
The PR description states 236 tests passed and changelog validation passed. No CODEOWNER-sensitive code is touched, and there are no unresolved reviewer comments in the timeline. Given the nature and low severity of the findings, human review is not required to safely merge this.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31583491445 |
15557ef to
ce61fdc
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31585588203 |
ce61fdc to
7e5fb67
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31585956705 |
Consolidate the GB300 AgentX MTP bring-up, runner selection, Dynamo transport hardening, AIPerf integration, and six-point aggregate/P-D sweep. Keep DeepGEMM AMX FP4 Mega MoE on expert-parallel P/D engines while restoring automatic MoE backend selection for pure aggregate TP. Disable the vLLM cumem allocator for aggregate TP, retain the 8192-token budget, size max-num-seqs as max(16, 4*concurrency) with matching MTP CUDA graph coverage, and use gpu-memory-utilization 0.94. 中文：整合 GB300 AgentX MTP 的启动流程、运行节点选择、Dynamo 传输加固、AIPerf 集成和六点聚合/P-D 扫描。在专家并行 P/D 引擎上保留 DeepGEMM AMX FP4 Mega MoE，并为纯聚合 TP 恢复自动 MoE 后端选择；聚合 TP 禁用 vLLM cumem 分配器并保留 8192-token 预算，同时按 max(16, 4*并发) 设置 max-num-seqs、配置对应的 MTP CUDA Graph 覆盖范围，并将 gpu-memory-utilization 设为 0.94。
Route GB300 multi-node AgentX recipes through the shared throughput-only synthetic acceptance injector and add regression coverage for eval-only behavior.\n\n中文：让 GB300 多节点 AgentX 配方使用仅吞吐量启用的共享合成接受率注入器，并添加仅评估模式的回归测试。
1f70b31 to
9926c05
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31624709550 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31633154542 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31633154542 |
|
/stage-results 31624709550 |
|
@ivanium staged run 31624709550: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-08-12~r31624709550 This run remains available across future |
|
/stage-results 31633154542 |
|
@ivanium staged run 31633154542: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-08-12~r31633154542 This run remains available across future |
|
/reuse-sweep-run 31633154542 |
中文：将最新 main 合并到 GB300 DSV4 AgentX 提交，并保留双方的性能变更日志。
Two fixes in one: 1. Restore PR #2571's entry (dsv4-fp4-gb300-dynamo-vllm-agentic-mtp-agg / -disagg). My automated merge-conflict resolution dropped it while reconciling the append-only tail, which the changelog gate correctly rejected -- deletions are not permitted. 2. Drop #2602's entry, which arrived with the code merge and pulled its whole vllm-simple sweep (c1..c20) onto this PR. That is not a deletion relative to main, since the entry only exists on #2602's branch. Rebuilding as 'main verbatim + this PR's entry' makes both correct by construction: zero deletions vs main, one addition.
…orkers Adds a GB200 counterpart to the GB300 AgentX DSV4 MTP recipes tuned in PR #2571. Every setting is copied unchanged except workers that used 4 GPUs on GB300 now use 8 on GB200 (agg TP4->TP8, disagg prefill DEP4->DEP8); worker sizes that were already 8/12/16 GPUs are untouched. New recipe files and dsv4-fp4-gb200-dynamo-vllm-agentic-mtp2-{agg,disagg} master-config keys are kept separate from the existing GB200 AgentX MTP recipes/keys, which have unrelated tuning and are left untouched. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…4-GPU->8-GPU workers (SemiAnalysisAI#2619) * [AgentX] Mirror GB300 MTP tuning (SemiAnalysisAI#2571) to GB200 with 4-GPU->8-GPU workers Adds a GB200 counterpart to the GB300 AgentX DSV4 MTP recipes tuned in PR SemiAnalysisAI#2571. Every setting is copied unchanged except workers that used 4 GPUs on GB300 now use 8 on GB200 (agg TP4->TP8, disagg prefill DEP4->DEP8); worker sizes that were already 8/12/16 GPUs are untouched. New recipe files and dsv4-fp4-gb200-dynamo-vllm-agentic-mtp2-{agg,disagg} master-config keys are kept separate from the existing GB200 AgentX MTP recipes/keys, which have unrelated tuning and are left untouched. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Fill in pr-link for GB200 mtp2 changelog entry Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Switch GB200 mtp2 recipes to stock nightly image + regular MegaMoE The GB300-pinned nightly-dev-arm64-cu13.0.1-426e59f tag is an ephemeral dev-branch build that has already been pruned from Docker Hub, and it exists specifically to carry deep_gemm_amxf4_mega_moe (W4A4 MegaMoE) support that isn't in stock vLLM nightly. Since these GB200 recipes don't need to stay on that dev branch, switch to vllm/vllm-openai:nightly and drop back to the regular deep_gemm_mega_moe backend that stock nightly supports. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Revert "Switch GB200 mtp2 recipes to stock nightly image + regular MegaMoE" This reverts commit 9749e1ef489e06744ff661e5eb5f97036e613ea3. * GB200 mtp2 disagg: lower Mooncake segment to 160GB, halve conc points Job 23562 (disagg-gb200-2p1d-dep8-dep16-c1024-mtp2-agentic, conc=1024) died silently mid-run: one prefill worker process was killed with no CUDA-OOM warning or traceback (exit code: None), which cascaded into an EngineDeadError and aborted the whole disagg job. No host-RAM telemetry was captured to confirm the cause, but the recipe reserves a 180GB host Mooncake KV-store segment per node on top of whatever vLLM/Dynamo itself uses, so as a mitigation: drop global_segment_size to 160GB across all three GB200 mtp2 disagg recipes, and halve each large-conc file's target concurrency (and rename/re-point accordingly) to reduce memory pressure: c256 -> c128 (disagg-gb200-1p1d-dep8-dep8) c1024 -> c512 (disagg-gb200-2p1d-dep8-dep16) c1152 -> c576 (disagg-gb200-2p1d-dep8-dep12) configs/nvidia-master.yaml's mtp2-disagg search-space conc-list and CONFIG_FILE references are updated to match. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Fix decode CUDA-graph capture size on GB200 c128 disagg recipe Job 23607 (disagg-gb200-1p1d-dep8-dep8-c128-mtp2-agentic) OOM'd during decode CUDA-graph capture: 183.24 GiB already in use on a 184 GiB GB200 GPU, then the capture-size warmup forward pass needed another 768 MiB it didn't have. The decode block's max-num-seqs (256) and max-cudagraph-capture-size (1024) were still sized for the pre-rename c256 concurrency, not the current c128 target. max-cudagraph-capture-size should track (num_speculative_tokens + 1) * max-num-seqs for MTP-3 decode graphs. Set decode max-num-seqs to 128 (matching the file's concurrency) and max-cudagraph-capture-size to 512 (4 * 128). The other two GB200 disagg mtp2 decode blocks (c576, c512) already satisfy this (64 * 4 = 256); prefill blocks are unaffected. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Lower GB200 c128 prefill gpu-memory-utilization to leave OOM headroom Job 23635 got past startup (decode capture-size fix from the previous commit held) but crashed ~4 min into serving: three prefill DP ranks hit torch.OutOfMemoryError inside the MTP speculator's draft-token prefill pass (speculator.py:_prefill -> wq_b block-scaled matmul), with the GPU already at 183.03/184 GiB in use. prefill's gpu-memory-utilization: 0.92 (copied from the GB300 source) reserves too much of the 184 GiB GB200 GPU for vLLM's static pool, leaving too little headroom for transient MTP draft-prefill activations on large-batch requests. Drop prefill gpu-memory-utilization to 0.85; decode is untouched since this failure was prefill-only. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * GB200 mtp2 disagg: 140GB Mooncake segment, add 1p1d c256 point Job 23715 (c576, global_segment_size 160GB) still died silently after ~1h51m of steady serving with no CUDA-OOM/cuBLAS error, matching the same host-RAM-exhaustion signature as job 23562 (180GB). Lower global_segment_size to 140GB across all three GB200 mtp2 disagg recipes as a further mitigation. Job 23714 (c128, fixed decode/prefill sizing) completed successfully. Its GPU KV-cache usage peaked at ~80% on prefill and ~39% on decode -- not maxed out -- so add the next concurrency point (256) on the same 1p1d dep8/dep8 topology rather than jumping to 2p1d: create disagg-gb200-1p1d-dep8-dep8-c256-mtp2-agentic.yaml as a clone of the working c128 file (decode max-num-seqs/max-cudagraph-capture-size and prefill gpu-memory-utilization left unchanged, since DEP8 replication already gives 8x the c128 file's per-rank max-num-seqs budget, well above a conc=256 target) and add its conc-list: [256] search-space entry to dsv4-fp4-gb200-dynamo-vllm-agentic-mtp2-disagg. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Update perf-changelog.yaml --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Cameron Quilici <cjquilici@gmail.com>
* [AgentX] Mirror GB300 MTP tuning (#2571) to GB200 with 4-GPU->8-GPU workers Adds a GB200 counterpart to the GB300 AgentX DSV4 MTP recipes tuned in PR #2571. Every setting is copied unchanged except workers that used 4 GPUs on GB300 now use 8 on GB200 (agg TP4->TP8, disagg prefill DEP4->DEP8); worker sizes that were already 8/12/16 GPUs are untouched. New recipe files and dsv4-fp4-gb200-dynamo-vllm-agentic-mtp2-{agg,disagg} master-config keys are kept separate from the existing GB200 AgentX MTP recipes/keys, which have unrelated tuning and are left untouched. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Fill in pr-link for GB200 mtp2 changelog entry Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Switch GB200 mtp2 recipes to stock nightly image + regular MegaMoE The GB300-pinned nightly-dev-arm64-cu13.0.1-426e59f tag is an ephemeral dev-branch build that has already been pruned from Docker Hub, and it exists specifically to carry deep_gemm_amxf4_mega_moe (W4A4 MegaMoE) support that isn't in stock vLLM nightly. Since these GB200 recipes don't need to stay on that dev branch, switch to vllm/vllm-openai:nightly and drop back to the regular deep_gemm_mega_moe backend that stock nightly supports. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Revert "Switch GB200 mtp2 recipes to stock nightly image + regular MegaMoE" This reverts commit 9749e1ef489e06744ff661e5eb5f97036e613ea3. * GB200 mtp2 disagg: lower Mooncake segment to 160GB, halve conc points Job 23562 (disagg-gb200-2p1d-dep8-dep16-c1024-mtp2-agentic, conc=1024) died silently mid-run: one prefill worker process was killed with no CUDA-OOM warning or traceback (exit code: None), which cascaded into an EngineDeadError and aborted the whole disagg job. No host-RAM telemetry was captured to confirm the cause, but the recipe reserves a 180GB host Mooncake KV-store segment per node on top of whatever vLLM/Dynamo itself uses, so as a mitigation: drop global_segment_size to 160GB across all three GB200 mtp2 disagg recipes, and halve each large-conc file's target concurrency (and rename/re-point accordingly) to reduce memory pressure: c256 -> c128 (disagg-gb200-1p1d-dep8-dep8) c1024 -> c512 (disagg-gb200-2p1d-dep8-dep16) c1152 -> c576 (disagg-gb200-2p1d-dep8-dep12) configs/nvidia-master.yaml's mtp2-disagg search-space conc-list and CONFIG_FILE references are updated to match. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Fix decode CUDA-graph capture size on GB200 c128 disagg recipe Job 23607 (disagg-gb200-1p1d-dep8-dep8-c128-mtp2-agentic) OOM'd during decode CUDA-graph capture: 183.24 GiB already in use on a 184 GiB GB200 GPU, then the capture-size warmup forward pass needed another 768 MiB it didn't have. The decode block's max-num-seqs (256) and max-cudagraph-capture-size (1024) were still sized for the pre-rename c256 concurrency, not the current c128 target. max-cudagraph-capture-size should track (num_speculative_tokens + 1) * max-num-seqs for MTP-3 decode graphs. Set decode max-num-seqs to 128 (matching the file's concurrency) and max-cudagraph-capture-size to 512 (4 * 128). The other two GB200 disagg mtp2 decode blocks (c576, c512) already satisfy this (64 * 4 = 256); prefill blocks are unaffected. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Lower GB200 c128 prefill gpu-memory-utilization to leave OOM headroom Job 23635 got past startup (decode capture-size fix from the previous commit held) but crashed ~4 min into serving: three prefill DP ranks hit torch.OutOfMemoryError inside the MTP speculator's draft-token prefill pass (speculator.py:_prefill -> wq_b block-scaled matmul), with the GPU already at 183.03/184 GiB in use. prefill's gpu-memory-utilization: 0.92 (copied from the GB300 source) reserves too much of the 184 GiB GB200 GPU for vLLM's static pool, leaving too little headroom for transient MTP draft-prefill activations on large-batch requests. Drop prefill gpu-memory-utilization to 0.85; decode is untouched since this failure was prefill-only. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * GB200 mtp2 disagg: 140GB Mooncake segment, add 1p1d c256 point Job 23715 (c576, global_segment_size 160GB) still died silently after ~1h51m of steady serving with no CUDA-OOM/cuBLAS error, matching the same host-RAM-exhaustion signature as job 23562 (180GB). Lower global_segment_size to 140GB across all three GB200 mtp2 disagg recipes as a further mitigation. Job 23714 (c128, fixed decode/prefill sizing) completed successfully. Its GPU KV-cache usage peaked at ~80% on prefill and ~39% on decode -- not maxed out -- so add the next concurrency point (256) on the same 1p1d dep8/dep8 topology rather than jumping to 2p1d: create disagg-gb200-1p1d-dep8-dep8-c256-mtp2-agentic.yaml as a clone of the working c128 file (decode max-num-seqs/max-cudagraph-capture-size and prefill gpu-memory-utilization left unchanged, since DEP8 replication already gives 8x the c128 file's per-rank max-num-seqs budget, well above a conc=256 target) and add its conc-list: [256] search-space entry to dsv4-fp4-gb200-dynamo-vllm-agentic-mtp2-disagg. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * chore: prepare pr 2619 ingest recovery 中文：准备 PR 2619 推理结果写入恢复。 * fix: recover pr 2619 production ingest 中文：恢复 PR 2619 的生产环境推理结果写入。 --------- Co-authored-by: Xin Li <xinli@nvidia.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2 participants
Tune the GB300 AgentX MTP configs.
调优 GB300 AgentX MTP 配置。
Tests / 测试: 236 passed; changelog validation passed.