Update DeepSeek-V4-Pro (DSV4) FP4 GB300 Dynamo-SGLang AgentX MTP configs / 更新 DeepSeek-V4-Pro（DSV4）FP4 GB300 Dynamo-SGLang AgentX MTP 配置 - #2644
Conversation
Add TP8 concurrency 1/4 and TP4 concurrency 8 aggregate profiles with phase-specific decode CUDA graph limits and throughput-only synthetic acceptance. 中文：优化 SGLang AgentX 低并发吞吐，新增 TP8 并发 1/4 和 TP4 并发 8 的聚合配置，并使用解码阶段 CUDA Graph 上限及仅吞吐测试启用的模拟验收率。
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
Use SGLang nightly-dev-cu13-20260811-d59c1ddf and the Dynamo 1.4.0 release across the GB300 aggregate and disaggregated recipe family. 中文：将 GB300 聚合与分离式配置族统一更新到 SGLang nightly-dev-cu13-20260811-d59c1ddf 和 Dynamo 1.4.0 正式版。
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
|
||
| - config-keys: | ||
| - dsv4-fp4-gb300-dynamo-sglang-agentic-agg | ||
| scenario-type: | ||
| - agentic-coding | ||
| description: | ||
| - "Tune the GB300 SGLang AgentX aggregate low-concurrency frontier with TP8 concurrency 1/4 and TP4 concurrency 8 profiles." | ||
| - "Update the aggregate and disaggregated configurations to SGLang nightly-dev-cu13-20260811-d59c1ddf and the Dynamo 1.4.0 release." | ||
| - "Use phase-specific decode CUDA graph limits and throughput-only synthetic MTP acceptance; eval-only runs retain real target verification." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2644 |
There was a problem hiding this comment.
🔴 The new changelog entry (perf-changelog.yaml:6053-6062) only lists config-keys: [dsv4-fp4-gb300-dynamo-sglang-agentic-agg], but this same PR also bumps the image (to nightly-dev-cu13-20260811-d59c1ddf) and router (to 1.4.0) for dsv4-fp4-gb300-dynamo-sglang-agentic-disagg in configs/nvidia-master.yaml, plus the Dynamo wheel to 1.4.0 in all 5 disagg recipe YAMLs. Since run-sweep.yml/process_changelog.py only schedules a master-config change for keys listed in the changelog entry, these disagg version bumps will merge without being validated by this PR's sweep. Please add dsv4-fp4-gb300-dynamo-sglang-agentic-disagg to config-keys.
Extended reasoning...
What the bug is: The tail changelog entry this PR appends (perf-changelog.yaml:6053-6062) declares only one config key:
- config-keys:
- dsv4-fp4-gb300-dynamo-sglang-agentic-agg
scenario-type:
- agentic-coding
description:
- "Tune the GB300 SGLang AgentX aggregate low-concurrency frontier with TP8 concurrency 1/4 and TP4 concurrency 8 profiles."
- "Update the aggregate and disaggregated configurations to SGLang nightly-dev-cu13-20260811-d59c1ddf and the Dynamo 1.4.0 release."
- ...Note the description's own second line explicitly claims both the aggregate and disaggregated configs are being updated — but the config-keys list only selects the agg key.
The code path that triggers it: Per docs/architecture.md (Stage 1, 'configuration and trigger selection'): 'A master entry is inert until selected. On the main sweep path, additions to perf-changelog.yaml select exact config keys or key patterns... Editing a master entry without a matching changelog addition does not schedule that change through run-sweep.yml, whose path trigger watches perf-changelog.yaml.' utils/process_changelog.py's get_config_keys_from_master() only expands the config-keys listed in the added changelog lines, and the matrix generator is subsequently invoked with only those expanded keys. There is no separate mechanism that detects 'this master entry byte-diffed, so schedule it regardless of changelog selection.'
Why nothing else prevents it: validate_append_only_scope (the guard that would reject a master edit outside the entry's declared append-only scope) only fires when an entry is marked append-only: true. Neither this new entry nor the historical entries touching these keys carry that flag, so nothing rejects — or even flags — an unselected master-file edit. It silently passes review and CI.
Concrete proof this actually happens in the diff:\n1. configs/nvidia-master.yaml diff shows dsv4-fp4-gb300-dynamo-sglang-agentic-disagg changes image: lmsysorg/sglang:nightly-dev-cu13-20260719-99f5a6f4 → nightly-dev-cu13-20260811-d59c1ddf and router: { ..., version: "1.3.0.dev20260718" } → { ..., version: "1.4.0" }.\n2. All five disagg-gb300-*-mtp-kvoffload.yaml recipe files bump dynamo.wheel from "1.3.0.dev20260718" to "1.4.0".\n3. The new changelog entry's config-keys is only [dsv4-fp4-gb300-dynamo-sglang-agentic-agg].\n4. When run-sweep.yml fires off this changelog diff, process_changelog.py expands only the agg key into sweep points; the disagg key is never passed to generate_sweeps, so no disagg job runs and the new image + Dynamo 1.4.0 combination for the disagg family reaches main completely unvalidated by this PR.
Impact: The disaggregated recipe family (the most complex, highest-node-count topology here — up to 12P4D across 16 nodes) is left running an SGLang nightly image and Dynamo release it has never actually been benchmarked or eval'd against by this PR's own gate — exactly the class of regression the changelog-trigger split exists to catch.
Fix: Add dsv4-fp4-gb300-dynamo-sglang-agentic-disagg to the new entry's config-keys list so run-sweep.yml schedules and validates both the agg and disagg sweeps this PR actually touches.
| - config-keys: | ||
| - dsv4-fp4-gb300-dynamo-sglang-agentic-agg | ||
| - dsv4-fp4-gb300-dynamo-sglang-agentic-disagg | ||
| - dsv4-fp4-gb300-dynamo-sglang-agentic-disagg | ||
| description: | ||
| - "Add GB300 DeepSeek-V4 dynamo-sglang agentic recipes (agg-tp4 + dep8 disagg pareto)" | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2157 |
There was a problem hiding this comment.
🔴 This PR's changelog diff accidentally duplicates the dsv4-fp4-gb300-dynamo-sglang-agentic-disagg line inside the pre-existing PR #2157 entry (perf-changelog.yaml ~4885-4891), turning its config-keys into [agg, disagg, disagg]. Historical entries must stay byte-for-byte immutable except for pr-link fixes, so this will break the changelog validation gate — please drop the duplicate line and leave the #2157 entry untouched.
Extended reasoning...
The diff for perf-changelog.yaml adds a stray + - dsv4-fp4-gb300-dynamo-sglang-agentic-disagg line inside the pre-existing PR #2157 changelog entry, rather than only appending a brand-new entry at the end of the file. Before this PR, that entry's config-keys list was [dsv4-fp4-gb300-dynamo-sglang-agentic-agg, dsv4-fp4-gb300-dynamo-sglang-agentic-disagg]; after this PR it becomes [agg, disagg, disagg] — an accidental duplicate, almost certainly a copy/paste slip while adding the new tail entry for this PR (#2644). The entry's own description text ("agg-tp4 + dep8 disagg pareto") only ever described two config keys, confirming the third is unintentional.
This matters because perf-changelog.yaml historical entries are contractually immutable in this repo — only pr-link corrections are allowed. utils/validate_perf_changelog.py's compare_entries() diffs each historical entry against its base-branch counterpart via without_pr_link(); since config-keys differs here (not just pr-link), it raises \"entry N changed; existing entries are immutable except for pr-link-only corrections\". Separately, validate_raw_change() requires head_raw.startswith(base_raw) whenever new entries are appended (which they are, at EOF for PR #2644) — because the byte stream is now mutated mid-file rather than purely appended-to, that prefix check fails too, raising \"appended entries changed historical perf-changelog.yaml bytes; restore the base file byte-for-byte and append at the end\".
It's also worth noting this doesn't get filtered out by YAML parsing quirks: config-keys is a YAML sequence, not a mapping, so the UniqueKeyLoader used elsewhere in the validation pipeline (which only rejects duplicate mapping keys) does not catch or dedupe it — the duplicate list item survives untouched into the immutability comparison. Additionally, one verifier traced the PR-time check-changelog CI path specifically (validate_perf_changelog.py main() → validate_generated_config → process_changelog.py), which collects only the added + lines and feeds them to yaml.safe_load(). Because the orphaned mid-entry - dsv4-...-disagg fragment (indented 4 spaces) precedes the new top-level - config-keys: entry in that concatenated added-lines blob, the YAML parse itself fails with a ParserError (expected <document start>, but found <block sequence start>), independently breaking the same CI gate before it even reaches the immutability check.
Proof, step by step:
- Base (main)
perf-changelog.yamlentry for PR Add GB300 DeepSeek-V4 Dynamo-SGLang AgentX aggregated and disaggregated recipes / 新增 GB300 DeepSeek-V4 Dynamo-SGLang AgentX 聚合式与分离式配方 #2157 hasconfig-keys: [dsv4-fp4-gb300-dynamo-sglang-agentic-agg, dsv4-fp4-gb300-dynamo-sglang-agentic-disagg]. - This PR's diff inserts
+ - dsv4-fp4-gb300-dynamo-sglang-agentic-disaggas a third list item under that same entry (confirmed at perf-changelog.yaml lines ~4885-4891 in the current tree). compare_entries()parses both base and head versions of that entry;without_pr_link(base) != without_pr_link(head)becauseconfig-keysnow has 3 items vs. 2 — it raises the immutability error.- Independently,
validate_raw_change()checks that all bytes preceding the newly appended tail entries are unchanged (head_raw.startswith(base_raw)); since a line was inserted mid-file, this is false, raising a second, distinct error. - Either failure alone is sufficient to fail the
check-changelogCI job and block the PR from merging.
The fix is a one-line deletion: remove the erroneously duplicated - dsv4-fp4-gb300-dynamo-sglang-agentic-disagg line so the #2157 entry's config-keys reverts to its original two-item list, leaving that historical entry byte-for-byte identical to main (the new tail entry for PR #2644, appended at EOF, is unaffected and should remain).
中文：合并 main 并解决性能变更记录冲突。
完善 GB300 SGLang AgentX 配置：同步配方镜像与身份信息，将模拟验收率限定到吞吐任务，并统一 Slurm 账户与作业时限。
合并最新 main，并保留其完整变更日志后将本 PR 条目追加到文件末尾。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32154816722 |
RohitNagraj
changed the title
更新 GB300 电源通道契约测试，使其验证显式的 Slurm 账户和分区参数。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32170134901 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32175285831 |
4 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32175285831 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32175285831 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32175285831 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32175285831 |
…pe / 让 DSV4 GB300 聚合配置与其他配置采用相同预热 agg-gb300-tp4-mtp-lowlatency and agg-gb300-tp8-mtp-lowlatency were the only recipes in srt-slurm-recipes/ setting AIPERF_WARMUP_REQUESTS_PER_LANE, and they set it to 1. That is the value fast mode uses to minimise setup (benchmark_lib.sh:1979-1982), but without fast mode's matching cut to the profiling duration, so the full measurement window ran off a barely-primed live state. The override arrived in 80eb81a, whose message covers CUDA graph limits and synthetic acceptance but never mentions warmup, and the kvoffload recipe these two replace did not set it. The five disagg recipes in this PR do not set it either, so the aggregate and disaggregated arms of the same model and SKU were being warmed differently and published on the same chart. Drop the override so both fall back to the standard ten requests per lane. Requires a fresh sweep for the two aggregate profiles: warmup depth changes the live state the profiling window starts from.
Root cause: The PR still pinned the August 11 SGLang nightly, and the concurrency-1536 disaggregated point retained six DEP8 prefill workers after the validated topology moved to five. Fix: Pin all GB300 DSV4 SGLang AgentX recipes and master entries to nightly-dev-cu13-20260821-f825d729. Rename the c1536 recipe from 12P4D to 10P4D and reduce its prefill nodes/workers from 12/6 to 10/5 while retaining one DEP16 decode worker. Validation: Verified the multi-arch Docker tag, generated all eight matrix points with c1536 selecting five prefill workers, passed 231 matrix tests, and passed srtctl dry-run for all seven recipes. 中文：将全部 GB300 DSV4 SGLang AgentX 配置升级到 nightly-dev-cu13-20260821-f825d729，并将并发 1536 的分离式拓扑从 6 个 DEP8 预填充 worker 调整为 5 个；对应 recipe 从 12P4D 改为 10P4D。已验证镜像架构、矩阵生成、231 项单测及 7 个 recipe 的 dry-run。
Keep model.container on the dynamo-sglang logical alias so the runner resolves it through the master image and generated squash mapping. Preserve the exact nightly tag only in identity metadata. 让 model.container 保持 dynamo-sglang 逻辑别名，由 runner 通过 master image 和生成的 squash 映射解析。实际 nightly tag 仅保留在 identity 元数据中。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32462813609 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32462813609 |
kedarpotdar-nv
left a comment
kedarpotdar-nv
left a comment
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Full sweep workflow
- Verified that this PR passes evals. Full sweep and eval workflow
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
Additional detail section:
- Reviewed exact PR head
81c6e3590cba307e3bfe18a995df2a94f1189c2a; all eight multi-node AgentX throughput points and all five configured eval jobs passed in the linked workflow. - The recipes use three speculative MTP tokens with throughput-only synthetic acceptance fixed to the committed DeepSeek-V4-Pro
thinking_ongolden AL of2.49. Eval-only jobs retain real target-model verification. - This submission uses the upstream
lmsysorg/sglang:nightly-dev-cu13-20260821-f825d729image and does not modify the model architecture. - These are multi-node Dynamo–SGLang recipes, so the single-node recipe-publication requirement is not applicable. This PR does not use
append-only: true. - No inference-engine or serving-stack patch is introduced.
Signed: kedarpotdar-nv
|
/reuse-sweep-run 32462813609 |
合并 main，并解决 GB300 启动脚本和性能变更日志冲突。
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| SGLANG_OPT_USE_ONLINE_COMPRESS: "0" | ||
| SGLANG_OPT_USE_JIT_INDEXER_METADATA: "1" | ||
| SGLANG_OPT_USE_JIT_NORM: "1" | ||
| SGLANG_OPT_USE_TOPK_V2: "True" |
There was a problem hiding this comment.
TP8 aggregate missing multi-node NCCL
High Severity
The new TP8 low-latency aggregate recipe places one tp-size: 8 worker across agg_nodes: 2, but aggregated_environment omits the GB300 multi-node interconnect settings used by sibling SGLang recipes (NCCL_MNNVL_ENABLE, NCCL_CUMEM_ENABLE, SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK). Cross-node tensor parallel init can fail or hang, so the concurrency 1/4 aggregate matrix points may never serve traffic.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit c36d371. Configure here.
✅✅✅ Verdict: PASS ✅✅✅✅ Check 0 (CODEOWNER): PASS — |
|
/stage-results 32462813609 |
|
@cquil11 staged run 32462813609: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-08-21~r32462813609 This run remains available across future |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 6219021. Configure here.
| RESULT_DIR: "/logs/agentic" | ||
| PORT: "8000" | ||
| IS_MULTINODE: "false" | ||
| TP: "8" |
There was a problem hiding this comment.
TP8 aggregate forces single-node path
Medium Severity
The new TP8 low-latency recipe spans two GB300 nodes (agg_nodes: 2) but sets IS_MULTINODE to false. That routes AgentX result aggregation and power collection through the single-node path, so published results omit multinode topology fields and skip the multinode measurement-window power path. The matching vLLM GB300 TP8 aggregate recipe keeps IS_MULTINODE true so decode worker count 0 is handled correctly.
Reviewed by Cursor Bugbot for commit 6219021. Configure here.
7 participants
Description
nightly-dev-cu13-20260821-f825d729and the released Dynamo1.4.0wheels/router.Validation
python3 -m pytest utils/matrix_logic/ -q(232 passed)srtctl dry-runpassed for all seven affected aggregate/disaggregated recipes.utils/validate_perf_changelog.pypassed against the latestmain.arm64andamd64manifests.中文说明
nightly-dev-cu13-20260821-f825d729和 Dynamo1.4.0正式版 wheel/router。本地验证
python3 -m pytest utils/matrix_logic/ -q（232 项通过）srtctl dry-run。utils/validate_perf_changelog.py基于最新main验证通过。arm64与amd64manifest。Related Issue / 相关问题
N/A / 不适用
Type of Change / 变更类型
Checklist / 检查清单
perf-changelog.yamland have not edited historical entries. / 对于每项可能影响基准性能的变更以及每项 recipe 新增或修改，我已在perf-changelog.yaml的物理末尾追加新条目，且未修改历史条目。/reuse-sweep-runon this PR after a final full sweep is green with evals passing. / 在通过复用方式合并前，授权维护者将在最终完整扫描和评估全部通过后评论/reuse-sweep-run。Note
Medium Risk
Changes published benchmark topology, software pins, and throughput measurement (synthetic MTP acceptance), which directly affects reported AgentX performance and cluster job accounting—not production app code.
Overview
Retunes the GB300 DeepSeek-V4-Pro Dynamo-SGLang AgentX benchmark matrix to SGLang
nightly-dev-cu13-20260821-f825d729and Dynamo 1.4.0, with recipeidentityblocks pinning those versions.Aggregate path: removes the TP4 HiCache/offload recipe (
agg-gb300-tp4-mtp-kvoffload) and replaces the old wide concurrency sweep with low-latency profiles—TP8 at conc 1/4 and TP4 at conc 8—using tighter decode CUDA graphs (cuda-graph-max-bs-decode), no hierarchical KV offload, and simpler infra (no dedicated etcd/NATS node, single frontend).Disaggregated path: all Pareto recipes pick up the same image/Dynamo bump; decode drops in-container
SGLANG_SIMULATE_ACC_*in favor of matrixSYNTHETIC_ACCEPTANCE(throughput runs). The c1536 point shrinks from 12P→10P (5 prefill workers, recipe rename todisagg-gb300-10p4d-...).Runner/CI:
launch_gb300-nv.shapplies an 8h default Slurm limit for DSV4 agentic dynamo-sglang (not only TRT) and passes--accounton image squash validationsruncalls;perf-changelog.yamldocuments the performance intent.Reviewed by Cursor Bugbot for commit 6219021. Bugbot is set up for automated code reviews on this repo. Configure here.