perf(gb300): Add more AgentX vLLM MTP aggregate and P/D recipes / 增加更多 GB300 AgentX vLLM MTP 聚合与 P/D 配方 - #2665
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
185f2e3 to
ad2c1a8
Compare
ivanium
changed the title
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Beyond the inline finding on the checked-in synthetic MTP config, I also checked the new 1P3D GB300 recipe (disagg-gb300-1p3d-dep4-tp8-c3-mtp-agentic.yaml) for missing multi-node NCCL/UCX env vars on the cross-node TP8 decode workers — this was examined and ruled out.
Extended reasoning...
Bugs were found in this run and are already posted as inline comments; this note only records an additional item that was examined and ruled out, not a full re-review.
be4e8ef to
023bf49
Compare
ivanium
added
agentx
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32226635322 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32228682229 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32228916968 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32231829727 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32239067095 |
65b0ed7 to
4c06b82
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32240961682 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32244949807 |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
2f96942 to
7f1d3ae
Compare
7f1d3ae to
1ef7058
Compare
55e4a1e to
ad2e946
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32607159092 |
ad2e946 to
9dce474
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32701852339 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32701852339 |
仅追加已验证的 GB300 AgentX P/D 点：DEP8/DEP16 c512，以及 DEP4 配合四个 TP8 解码 worker 的 c4。保留所有现有点和配方不变，并将性能变更记录标记为 append-only。
追加已验证的 GB300 AgentX DEP4 预填充、DEP16 解码并发 128 点位。
追加已验证的 GB300 AgentX DEP4 预填充、DEP16 解码并发 256 点位，并使用解码侧 8/32/32 限制。
将 PR #2665 的性能变更记录合并为一个简洁的 append-only 条目。
9dce474 to
54d9faf
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32809502132 |
|
/stage-results 32809502132 |
|
@cquil11 staged run 32809502132: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-08-25~r32809502132 This run remains available across future |
|
/reuse-sweep-run 32809502132 |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit ace786e. Configure here.
| no-enable-flashinfer-autotune: true | ||
| block-size: 256 | ||
| compilation-config: '{"cudagraph_mode":"FULL_DECODE_ONLY","mode":0}' | ||
| max-cudagraph-capture-size: 16 |
There was a problem hiding this comment.
Decode limits undersized after topology halved
High Severity
Halving DEP32 decode to DEP16 kept the original per-rank max-num-seqs, max-num-batched-tokens, and max-cudagraph-capture-size. Each decode rank now sees twice the load at the same concurrency, so c128 capacity is 64 sequences and c256 capacity is 128. The advertised points will queue half their in-flight requests and publish incorrect AgentX latency and throughput.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit ace786e. Configure here.
2 participants
Summary / 摘要
English
中文
Validation / 验证
Note
Low Risk
Benchmark and Slurm recipe YAML plus master matrix wiring only; no runtime service or library code changes.
Overview
Adds four GB300 DeepSeek-V4 AgentX disaggregated MTP benchmark recipes and wires them into
nvidia-master.yamlunderdsv4-fp4-gb300-dynamo-vllm-agentic-mtp-disagg, with a matchingperf-changelog.yamlentry.Three 1P1D recipes scale proven DEP8/DEP32-style points down to DEP4 prefill + DEP16 decode at concurrency 128 and 256 (Mooncake + NIXL KV, decode disables Mooncake prefix lookup). A fourth 1P1D point keeps DEP8 + DEP16 for c512.
A separate 1P4D recipe targets high interactivity at c4: one DEP4 prefill feeds four TP8 decode workers via NIXL only, Dynamo KV router, prefill KV events, FlashInfer allreduce on multi-node decode, and benchmark client colocated on first decode.
Throughput-oriented matrix entries use synthetic MTP acceptance at launch; the c4 1P4D recipe is positioned for eval runs with real MTP output.
Reviewed by Cursor Bugbot for commit ace786e. Bugbot is set up for automated code reviews on this repo. Configure here.