[WIP] Test DeepSeek-V4 GB300 Dynamo AgentX recipes / 测试 DeepSeek-V4 GB300 Dynamo AgentX 配置 - #2302
[WIP] Test DeepSeek-V4 GB300 Dynamo AgentX recipes / 测试 DeepSeek-V4 GB300 Dynamo AgentX 配置#2302ivanium wants to merge 1 commit into
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
Claude finished @ivanium's task in 6m 24s —— View job PR Review
SummaryFound 1 blocking issue and 2 warnings (details in inline comments):
The rest looks good: new recipe YAMLs are well-documented, the |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
ivanium
changed the title
There was a problem hiding this comment.
This PR adds new multi-node GB200 AgentX recipes and modifies shared launcher/routing logic, including a change to configs/nvidia-master.yaml (a CODEOWNER-protected file), so it's worth a human look before merging.
What was reviewed: the two new srt-slurm recipe YAMLs (aggregate TP8 and disaggregated 3P1D DEP8/DEP16), the launch_gb200-nv.sh model-path routing and srt-slurm source switch for agentic runs, the benchmark_lib.sh aiperf conv-aware-routing opt-out, and the new nvidia-master.yaml/perf-changelog.yaml entries.
Extended reasoning...
Overview
The PR adds two new DeepSeek-V4 GB200 Dynamo AgentX recipes (aggregate TP8 and disaggregated 3P1D DEP8/DEP16), registers them in configs/nvidia-master.yaml and perf-changelog.yaml, and modifies two shared scripts: benchmarks/benchmark_lib.sh (adds an opt-out env var for aiperf's dynamo conv-aware-routing flag) and runners/launch_gb200-nv.sh (switches the agentic srt-slurm clone from a pinned fork to NVIDIA/srt-slurm:main, and branches MODEL_PATH resolution for DSV4 FP4 based on IS_AGENTIC).
Security risks
No security-sensitive surface — this is internal benchmark/CI infrastructure (Slurm job launching, recipe YAML, shell env plumbing) with no auth, crypto, or externally-reachable inputs. The git clone URLs and container images are pinned to specific orgs/tags rather than user-controlled input.
Level of scrutiny
Higher than typical for this repo: the diff touches shared launcher logic (launch_gb200-nv.sh) and the aiperf routing default in benchmark_lib.sh, both of which affect existing non-agentic GB200 runs in addition to the new recipes. It also modifies configs/nvidia-master.yaml, which is CODEOWNER-protected (@ankur-singh @kedarpotdar-nv @InferenceX/core). Combined with the size and multi-file nature of the change (new multi-node recipes with many tuned env vars, a repo-source switch for the agentic srt-slurm path, and conditional model-path logic), this warrants a human pass rather than auto-approval.
Other factors
The bug-hunting system found no bugs, and several candidate issues (leftover NCCL symm-mem env var, an odd tokenizer string, --no-preflight usage, combining kv-cache-memory with gpu-memory-utilization) were raised and ruled out as not real bugs. The PR description/comments show no CODEOWNER sign-off yet per the standard recipe-PR checklist, which is itself a signal that human review is expected before merge.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29960434846 |
baca859 to
99a28ab
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29976615750 |
ivanium
changed the title
99a28ab to
985c29e
Compare
985c29e to
dfc774f
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30154534054 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30170531787 |
Add four GB300 MTP3 AgentX topologies using the bedf79e image: aggregate TP8, 1P/1D DEP4/DEP8, 1P/1D DEP8/DEP8, and 2P/1D DEP8/DEP16. Follow PR #2157's aggregate/disaggregated organization, PR #2269's CI rack networking, and PR #2302's upstream srt-slurm per-node launch path with automatic hybrid load balancing. Align all disaggregated recipes on Dynamo 1.3.0.dev20260720, preserve prefill-only connector prefetch and breakable CUDA graph settings, and configure decode-side MooncakeStore as a synchronous KV consumer with lookup disabled. 中文：新增四个使用 bedf79e 镜像的 GB300 MTP3 AgentX 拓扑：聚合式 TP8、1P/1D DEP4/DEP8、1P/1D DEP8/DEP8 和 2P/1D DEP8/DEP16。配置组织遵循 PR #2157 的聚合式/分离式结构，网络设置对齐 PR #2269 的 CI 机架环境，并依据 PR #2302 使用上游 srt-slurm 的单节点单进程启动模式及其自动启用的混合负载均衡。所有分离式配置统一使用 Dynamo 1.3.0.dev20260720，连接器预取和可中断 CUDA Graph 设置仅保留在预填充侧；解码侧 MooncakeStore 设为同步 KV 消费端并关闭查找功能。
8a83d53 to
64f87af
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30217145037 |
Add three GB300 MTP3 AgentX topologies using the bedf79e image: aggregate TP8, 1P/1D DEP4/DEP8, and 1P/1D DEP8/DEP8. Follow PR #2157's aggregate/disaggregated organization, PR #2269's CI rack networking, and PR #2302's upstream srt-slurm per-node launch path with automatic hybrid load balancing. Align both disaggregated recipes on Dynamo 1.3.0.dev20260720, preserve prefill-only connector prefetch and breakable CUDA graph settings, and configure decode-side MooncakeStore as a synchronous KV consumer with lookup disabled. 中文：新增三个使用 bedf79e 镜像的 GB300 MTP3 AgentX 拓扑：聚合式 TP8、1P/1D DEP4/DEP8 和 1P/1D DEP8/DEP8。配置组织遵循 PR #2157 的聚合式/分离式结构，网络设置对齐 PR #2269 的 CI 机架环境，并依据 PR #2302 使用上游 srt-slurm 的单节点单进程启动模式及其自动启用的混合负载均衡。两个分离式配置统一使用 Dynamo 1.3.0.dev20260720，连接器预取和可中断 CUDA Graph 设置仅保留在预填充侧；解码侧 MooncakeStore 设为同步 KV 消费端并关闭查找功能。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32193014157 |
修复 c4 客户端放置，将 c256 预填充改为 DEP8，并以 c1728 的 3P DEP8/1D DEP16 拓扑替换 c1792。将 PR #2302 的测试矩阵限制为这三个修正点。
c0fce8a to
1620f3f
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32626309062 |
延长 c1728 的 AIPerf TCP 用户超时，并将 DEP8/DEP32 c256 的解码限制调整为 4/16/16。将 PR #2302 的测试矩阵限制为 c1728 和 c256 两个点。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32665164619 |
修复 c4 客户端放置，将 c256 预填充改为 DEP8，并以 c1728 的 3P DEP8/1D DEP16 拓扑替换 c1792。将 PR #2302 的测试矩阵限制为这三个修正点。
延长 c1728 的 AIPerf TCP 用户超时，并将 DEP8/DEP32 c256 的解码限制调整为 4/16/16。将 PR #2302 的测试矩阵限制为 c1728 和 c256 两个点。
383673a to
3fb763c
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32703790540 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32703790540 |
将 PR #2302 测试矩阵限制为单个 DEP4 预填充、DEP16 解码并发 256 点位。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32766964279 |
修复 c4 客户端放置，将 c256 预填充改为 DEP8，并以 c1728 的 3P DEP8/1D DEP16 拓扑替换 c1792。将 PR #2302 的测试矩阵限制为这三个修正点。
延长 c1728 的 AIPerf TCP 用户超时，并将 DEP8/DEP32 c256 的解码限制调整为 4/16/16。将 PR #2302 的测试矩阵限制为 c1728 和 c256 两个点。
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 2bfb951. Configure here.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32795506266 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33918546400 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33928040782 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34320654714 |
Use DeepSeek-V4-Pro-0813 with probabilistic DSpark6, synthetic acceptance length 3.77 for throughput, and real verification for eval. Preserve the TP8/DEP8 SimpleCPU sweep, use vLLM v0.28.0, reduce DRAM utilization to 0.90, update the MegaMoE backend name, and allow two hours for engine startup. 中文：使用 DeepSeek-V4-Pro-0813 和概率采样 DSpark6，吞吐测试采用 3.77 的合成接受长度，评测采用真实验证；保留 TP8/DEP8 SimpleCPU sweep，使用 vLLM v0.28.0，将 DRAM 利用率降至 0.90，更新 MegaMoE 后端名称，并允许引擎启动耗时最长两小时。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34329045612 |
2 participants
Summary
mainand re-append this PR's changelog entry at the tailAIPerf dependency fix: SemiAnalysisAI/agentx-harness#38
Validation
bash -n runners/launch_gb300-nv.sh benchmarks/benchmark_lib.sh中文说明
flock；同时让 AIPerf 的自动生成预热阶段真正应用已配置的 60 秒并发爬升，防止快照预热请求压垮 Dynamo 预填充入口main，并按追加式变更日志规则将本 PR 条目重新放到文件末尾AIPerf 依赖修复：SemiAnalysisAI/agentx-harness#38
Note
Medium Risk
Benchmark and matrix changes alter model, vLLM version, speculative decoding, GPU memory budget, and scheduler token limits, which directly affects reported AgentX throughput and comparability with prior MTP runs.
Overview
Migrates the B200 AgentX vLLM recipe from MTP (3 draft tokens, AL 2.49) to DSpark6 on DeepSeek-V4-Pro-0813 with vLLM v0.28.0, probabilistic drafting, and synthetic acceptance length 3.77 for throughput sweeps (real verification when
EVAL_ONLY).The benchmark script updates speculative
--speculative-config, CUDA graph token multiples fornum_speculative_tokens=6, MoE backend todeep_gemm_mega_moe, engine ready timeout to 7200s, and DEP scheduling:--max-num-batched-tokensis computed as 8192 + MAX_NUM_SEQS × (1+N) so DSpark verification slots do not shrink the prefill budget. Comments now describe TP8 + DEP8 SimpleCPU KV offload.configs/nvidia-master.yamlfordsv4-fp4-b200-vllm-agentic-mtppins the new image/model, lowers dram-utilization to 0.90, switchesspec-decodingfrom mtp to draft_model, and adds an explicit TP8 search-space arm alongside DEP8.perf-changelog.yamlrecords the change.Reviewed by Cursor Bugbot for commit 628cfeb. Bugbot is set up for automated code reviews on this repo. Configure here.