Repository navigation
perf(glm5.2): refresh the B200 FP4 Dynamo SGLang AgentX aggregate curve / perf(glm5.2)：刷新 B200 FP4 Dynamo SGLang AgentX 聚合曲线 - #4286
Conversation
Move glm5.2-fp4-b200-dynamo-sglang-agentic-agg to SGLang nightly-dev-cu13-20260901-07c8f729 and Dynamo 1.5.0.dev20260903 (KV router), the image and Dynamo build of the B300 aggregated key, with one aggregated engine per point, EAGLE speculative decoding (5 steps, 6 draft tokens) and HiCache write-back host-DRAM offload. HiCache takes 170 GB per rank, so the KV and DSA indexer host pools of the four ranks on each NUMA node fit that node. agg-variants.yaml is rewritten for the new points and keeps the checkpoint revision pin. Sweep six points in place of the three TP8 points: TP8 and TP4 at concurrency 1, TP4 at concurrency 4, 8 and 12, and TP8 MoE with DP8 attention at concurrency 32. At concurrency 1, admission and decode CUDA graphs are capped at 2 for TP8 and at 1 for TP4, so fan-out subagents queue instead of widening the decode batch. The DP8 attention point runs attention TP1 per rank with the DP LM head and the prefill delayer, mem-fraction-static 0.82, and 8192-token prefill chunks and 16 running requests per rank. The disaggregated key is unchanged. 将 glm5.2-fp4-b200-dynamo-sglang-agentic-agg 迁移到 SGLang nightly-dev-cu13-20260901-07c8f729 与 Dynamo 1.5.0.dev20260903（KV 路由）， 即 B300 聚合配置所用的镜像和 Dynamo 版本；每个测试点使用一个聚合引擎， 采用 EAGLE 投机解码（5 步，6 个草稿 token）和 HiCache write-back 主机 内存卸载。HiCache 每个 rank 170 GB，使每个 NUMA 节点上四个 rank 的 KV 与 DSA indexer 主机缓存池能够放入该节点的内存。agg-variants.yaml 按新的测试 点重写，并保留检查点的 revision 固定。 以 6 个测试点替换原有的 3 个 TP8 测试点：TP8 与 TP4 并发 1，TP4 并发 4、 8、12，以及 TP8 MoE 配合 DP8 attention 的并发 32。在并发 1 下，TP8 的准 入请求数与解码 CUDA graph 上限为 2，TP4 为 1，使 fan-out 子 agent 排队等 待，而不是扩大解码批次。DP8 attention 测试点每个 rank 的 attention TP 为 1，启用 DP LM head 与 prefill delayer，mem-fraction-static 为 0.82，每个 rank 的预填充分块为 8192 token、运行请求数上限为 16。分离式配置保持不 变。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the contribution!
中文感谢你的贡献！
|
Ankur-singh
added
the
full-sweep-enabled
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
I reviewed this PR and found no bugs. These are CODEOWNER-owned NVIDIA benchmark paths, and the PR changes a lot of engine tuning (TP4 base, DP8 attention point, new image and Dynamo build), so a human owner should still look it over.
What was reviewed:
- Each of the six master search-space points against its recipe override: the override names, tp (4 vs 8 through
gpus/tensor-parallel-size), dp-attn and conc all match. - The perf-changelog entry is added only at the tail, existing bytes are unchanged, and it covers the one affected key. Its pr-link is still the
pull/XXXplaceholder. - The new recipe comments explain specific lines; they are not header or history comments.
Extended reasoning...
The change rewrites the GLM-5.2 NVFP4 B200 Dynamo+SGLang AgentX aggregate srt-slurm recipe, updates the matching nvidia-master.yaml key (image, router version, dram-utilization, and six search points instead of three), and adds one perf-changelog entry. It touches no security-sensitive surface; it is benchmark configuration only. Both the recipe directory and nvidia-master.yaml are covered by CODEOWNERS entries, which rules out auto-approval. The tuning choices also need a domain owner's judgment, and the description says the PR depends on an external SGLang cookbook change merging first.
This review covers commit dd11e6f, which is no longer the latest commit on this pull request; later commits are not covered by it.
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=39166018772 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=39166018772 |
|
[by Claude Code] The SGLang cookbook integration of this PR's AgentX config ( This PR has no authorized |
|
[by Claude Code] sgl-project/sglang#45111 has been merged. This PR's AgentX config ( This PR has no authorized |
|
/use 39166018772 |
|
@nvpohanh staged run 39166018772. This run remains available across future |
|
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Additional detail section:
Signed: |
✅✅✅ Verdict: PASS ✅✅✅Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (Passing sweep on in-PR commit): PASS — Run Sweep 39166018772 ( ✅ Check 2 (Evals pass): PASS — ✅ Check 3 (Upstream recipe): PASS — sgl-project/sglang#45111 is MERGED (2026-10-19T07:24:09Z). The ✅ Check 4 (Reuse command): PASS — ✅ Check 5 (Latest checklist): PASS — every item in the current ✅ Check 6 (Upstream image / ordering): PASS — the entry uses upstream ✅ Check 7 (Deprecated models): PASS — ✅ Check 8 (No architecture hacks): PASS — the diff contains no ✅ Check 9 (Spec-decode chat template): PASS — the AgentX client drives the OpenAI ✅ Check 10 (No engine patches): PASS — no patches, inline source rewrites or forked engine wheels; the SGLang image runs as shipped, and Dynamo is the published ✅ Check 11 (Golden AL): PASS — the job log shows ➖ Check 12 (Append-only): N/A — the new changelog entry does not set ✅ Check 13 (Draft precision): PASS — the embedded MTP head (layer 78) of ✅ Check 14 (Pareto coverage): PASS — the curve GLM-5.2 / agentic-coding / B200 dynamo-sglang / FP4 / p90 from run 39166018772 ✅ Check 15 (PR description): PASS — the old/new image, Dynamo build, six-point topology, unchanged ✅ Check 16 (NVIDIA overview): PASS — the NVIDIA overview Framework Recipes table has a GLM-5.2 row with a populated SGLang cell. ➖ Check 17 (Metrics publication): N/A — the PR affects only a Assessed commit: |
同步 main 新增的 #4286 性能记录，消除 changelog 校验中的历史删除差异；保留不经过 canary 的全量 sweep 标签。
3 participants
[by Claude Code]
Summary
GLM-5.2 NVFP4 on B200 with Dynamo and SGLang is published from
glm5.2-fp4-b200-dynamo-sglang-agentic-agg(TP8 at c1, c4 and c8) andglm5.2-fp4-b200-dynamo-sglang-agentic-disagg(1P4D at c48, 1P1D DEP8 at c64), onlmsysorg/sglang:nightly-dev-20260910-00840301. This PR refreshes the aggregate key in place: it movesglm5.2-fp4-b200-dynamo-sglang-agentic-aggto the image and Dynamo build ofglm5.2-fp4-b300-dynamo-sglang-agentic-agg-tp4(lmsysorg/sglang:nightly-dev-cu13-20260901-07c8f729, Dynamo1.5.0.dev20260903) and replaces its three TP8 points with six points from the low-latency end to the throughput end.agg-variants.yamlis rewritten for these points and keeps the checkpoint revision pin. The 2026-09-01 nightly is older than the key's current image; it is the image the B300 key already runs these settings on, including the DP-attention path.AgentX draws one B200 Dynamo SGLang curve for this model across the aggregated and disaggregated keys. The
-disaggkey is unchanged and not swept, so this PR's points replace the published ones. SGLang cookbook coverage for these settings must merge before this PR does.AI model disclosure
claude-opus-5-5(Claude Opus 5.5, run through Claude Code)claude-opus-5-5, and assisted with these steps.PR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entriesinferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs):full-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
GLM-5.2 NVFP4 在 B200 上基于 Dynamo 与 SGLang 的已发布曲线来自
glm5.2-fp4-b200-dynamo-sglang-agentic-agg（TP8，并发 1、4、8）和glm5.2-fp4-b200-dynamo-sglang-agentic-disagg（并发 48 的 1P4D 与并发 64 的 1P1D DEP8），使用lmsysorg/sglang:nightly-dev-20260910-00840301镜像。本 PR 原地刷新聚合配置：将glm5.2-fp4-b200-dynamo-sglang-agentic-agg迁移到与glm5.2-fp4-b300-dynamo-sglang-agentic-agg-tp4相同的镜像和 Dynamo 版本（lmsysorg/sglang:nightly-dev-cu13-20260901-07c8f729，Dynamo1.5.0.dev20260903），并以 6 个测试点替换原有的 3 个 TP8 测试点，覆盖从低延迟端到高吞吐量端的整条曲线。agg-variants.yaml按这些测试点重写，并保留检查点的 revision 固定。2026-09-01 nightly 镜像早于该配置当前使用的镜像；B300 配置已在该镜像上使用这些设置（包括 DP attention 路径）。AgentX 将该模型在 B200 上的 Dynamo SGLang 聚合与分离式配置绘制为同一条曲线。
-disagg配置保持不变且不参与本次扫描，因此本 PR 的测试点将替换已发布的测试点。对应设置的 SGLang cookbook 文档须在本 PR 合并前合并。AI 模型使用说明：使用
claude-opus-5-5（Claude Opus 5.5，通过 Claude Code 运行）准备配置和 perf-changelog 条目、运行本地配置检查，并起草提交和本说明；委派的工作流 agent 使用同一模型claude-opus-5-5，协助完成上述工作。改动类型为配置变更。本地测试仅包括配置生成、matrix 和 changelog 检查，GPU 结果来自本 PR 的扫描。检查清单已在上方填写。