[NV] Add GB300 AgentX Qwen3.5 recipes - #2121
Conversation
csahithi
commented
Jul 8, 2026
csahithi
commented
- 9 Qwen3.5-397B-A17B-NVFP4 GB300 sglang AgentX recipes (agg + disagg pareto)
- configs/nvidia-master.yaml: qwen3.5-fp4-gb300-dynamo-sglang{,-agentic-agg,-agentic-disagg}
- perf-changelog.yaml: agentic pareto entry
- runners/launch_gb300-nv.sh: qwen3.5 launcher branch + dynamo-wheels cache
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
2 similar comments
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
b90cee7 to
7dc6bac
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28975134400 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Additional findings (outside current diff — PR may have been updated during review):
-
🔴
configs/nvidia-master.yaml:10940-10969— 9 new sweep entries in configs/nvidia-master.yaml (agentic-agg + agentic-disagg, lines 10940–11085) declarespec-decoding: none, but every one of the 9 corresponding recipe files enables NEXTN MTP (speculative-algorithm: NEXTN+speculative-num-steps: 3+ eagle-topk/num-draft-tokens; the recipe filenames also encode-mtp-). The engine runs MTP correctly because srtctl applies the recipe directly, butSPEC_DECODINGpropagates verbatim intoRESULT_FILENAME(_spec-none_…in artifact names, benchmark-multinode-tmpl.yml:225) and into the aggregated result JSON'sspec_decodingfield (utils/agentic/aggregation/process_agentic_result.py:139), so downstream dashboards / Pareto comparisons will silently misclassify these MTP runs as non-MTP. Fix: changespec-decoding: none→spec-decoding: "mtp"on all 9 entries in this block (matches the file's existing MTP convention — every other qwen3.5/dsr1/dsv4 MTP entry usesmtp; the parallel dsv4 agentic entry usingnoneis self-consistent because its vLLM recipes have nospeculative-*fields).Extended reasoning...
What the bug is
All 9 newly added sweep entries under
qwen3.5-fp4-gb300-dynamo-sglang-agentic-aggandqwen3.5-fp4-gb300-dynamo-sglang-agentic-disagg(configs/nvidia-master.yaml lines 10943, 10957, 10986, 11000, 11014, 11029, 11044, 11059, 11074) set:spec-decoding: none
But every one of the 9 corresponding recipe files under
benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/gb300-fp4/agentic/*.yamlenables NEXTN MTP:speculative-algorithm: NEXTN speculative-num-steps: 3 speculative-eagle-topk: 1 speculative-num-draft-tokens: 4
The recipe filenames themselves also encode
-mtp-(e.g.agg-gb300-tp4-c1-mtp-hicache-jid2191933.yaml,disagg-gb300-1p1d-tp4-tp4-c32-mtp-hicache-convaware-jid2212783.yaml).Why the engine still runs MTP but the metadata is wrong
srtctl applyreads the recipe file directly, so the engine gets--speculative-algorithm NEXTN …on its command line and NEXTN MTP runs as intended. Thespec-decodingfield innvidia-master.yamlis not a live engine setting — it feedsSPEC_DECODINGinto the workflow env, which has two downstream consumers, and both are silently corrupted by this mislabel:-
Artifact filenames —
.github/workflows/benchmark-multinode-tmpl.yml:225bakes_spec-${{ env.SPEC_DECODING }}_directly intoRESULT_FILENAME. So uploaded result JSONs land as…_spec-none_conc*.jsondespite the run actually using MTP. -
Aggregated result JSON body —
utils/agentic/aggregation/process_agentic_result.py:139writes"spec_decoding": os.environ.get("SPEC_DECODING", "none")verbatim into the aggregated result JSON. Downstream dashboards / Pareto plots that filter or facet onspec_decodingwill silently misclassify these MTP runs as non-MTP.
(Some verifiers also flagged
utils/compare_results.py:51readingresult['spec_decoding']for baseline lookup — worth double-checking, but the two consumers above are sufficient on their own.)Convention check
Every other MTP recipe in
nvidia-master.yaml(dsr1-mtp, dsv4-b200-vllm-mtp, qwen3.5 fixed-seq-len MTP tiers, etc.) usesspec-decoding: "mtp". The one other agentic entry that legitimately usesspec-decoding: none—dsv4-fp4-gb300-dynamo-vllm-agentic— is self-consistent because its vLLM recipes atbenchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/agentic/*.yamlcontain nospeculative-*fields. So this is a real inconsistency introduced by this PR, not the file's existing style.Step-by-step proof (agentic-agg conc=1)
- Sweep matrix expands the
spec-decoding: none/conc-list: [1]entry into a GHA job withSPEC_DECODING=none,CONFIG_FILE=recipes/sglang/qwen3.5/gb300-fp4/agentic/agg-gb300-tp4-c1-mtp-hicache-jid2191933.yaml. srtctl apply -f $CONFIG_FILEreads the recipe, which containsspeculative-algorithm: NEXTN+speculative-num-steps: 3— the sglang server starts with NEXTN MTP enabled. ✅ Engine correct.benchmark-multinode-tmpl.yml:225computesRESULT_FILENAME=…_spec-none_conc1_…fromenv.SPEC_DECODING=none. ❌ Artifact filename claims non-MTP.process_agentic_result.py:139readsos.environ.get("SPEC_DECODING", "none")→"none"and writes"spec_decoding": "none"into the aggregated result JSON. ❌ Result body claims non-MTP.- A downstream MTP-vs-non-MTP Pareto plot that groups by
spec_decodingputs this run in the non-MTP bucket, distorting the frontier.
Fix
Mechanical 9-line change — replace
spec-decoding: nonewithspec-decoding: "mtp"on all 9 new entries in this block (lines 10943, 10957, 10986, 11000, 11014, 11029, 11044, 11059, 11074). Nothing else in the recipe files needs to change; the change is purely to keep the master-config label truthful to what the recipe actually runs. -
| - config-keys: | ||
| - qwen3.5-fp4-gb300-dynamo-sglang-agentic-agg | ||
| - qwen3.5-fp4-gb300-dynamo-sglang-agentic-disagg | ||
| description: | ||
| - "Add Qwen3.5-397B-A17B-NVFP4 FP4 GB300 SGLang AgentX Pareto-frontier benchmarks via the srtctl/dynamo stack" | ||
| - "agg: single aggregated node (TP4, MTP/NEXTN, hierarchical KV cache), concurrencies [1, 96]" | ||
| - "disagg: 1P1D-4P1D (TP4/DEP4 prefill, TP4/DEP4/DEP8 decode, MTP/NEXTN, hicache, conv-aware routing) over 7 Pareto points at concurrency 32-384" | ||
| - "Image: lmsysorg/sglang:nightly-dev-cu13-20260624-b2c8f7a2; runner: gb300-nv" | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2121 |
There was a problem hiding this comment.
🔴 The new perf-changelog entry for qwen3.5-fp4-gb300-dynamo-sglang-agentic-{agg,disagg} (perf-changelog.yaml:4366-4373) will block the changelog gate on two independent grounds: (1) it's missing the required pr-link field, and (2) it's inserted mid-file rather than appended to the end. Fix: move the 8-line block to the end of the file and add pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2121 (or the XXX placeholder while WIP).
Extended reasoning...
What breaks. utils/validate_perf_changelog.py (invoked from .github/workflows/run-sweep.yml on every PR push) enforces three independent checks that this new entry violates. Any one of them is enough to fail the changelog gate; the entry fails all three.
1. Missing pr-link — schema-level rejection. In utils/matrix_logic/validation.py:683, ChangelogEntry declares pr_link: str = Field(alias="pr-link") with no default and extra="forbid". parse_changelog (validate_perf_changelog.py:128-133) calls ChangelogEntry.model_validate(entry) on every entry; Pydantic raises ValidationError on the new entry, which is re-raised as ChangelogValidationError('… fails ChangelogEntry validation'). Even if that were bypassed, validate_added_pr_link (lines 144-160) rejects it a second time: str(entry.get('pr-link') or '') == '', which is neither in PR_LINK_PLACEHOLDERS = {'XXX', 'https://github.com/SemiAnalysisAI/InferenceX/pull/XXX'} nor equal to the canonical PR URL, so it raises ChangelogValidationError('new PR entry must use …').
2. Mid-file insertion — positional rejection. The new 8-line block sits at lines 4366-4373, between an existing pull/1931 entry and the pre-existing minimaxm3-fp4-mi355x-vllm-mtp entry — with ~30 more historical entries after it. compare_entries (lines 163-208) walks base_entries and head_entries in lockstep by index; at the insertion index, head_entries[i] is the new qwen3.5 entry but base_entries[i] is the pre-existing minimaxm3 entry, and without_pr_link differs, so it raises ChangelogValidationError('entry N changed; existing entries are immutable except for pr-link-only corrections').
3. Byte-level prefix check. validate_raw_change (lines 211-235) additionally requires head_raw.startswith(base_raw) when there are additions. A mid-file insertion breaks that even before compare_entries runs, raising 'appended entries changed historical perf-changelog.yaml bytes; restore the base file byte-for-byte and append at the end'. AGENTS.md line 78 codifies the same rule: 'new entries MUST be appended to the END, never inserted in the middle or prepended.'
Step-by-step proof.
- CI runs
python3 utils/validate_perf_changelog.py --base-ref origin/main --head-ref HEAD. parse_changelogreads perf-changelog.yaml and iterates entries. When it reaches the qwen3.5 entry,ChangelogEntry.model_validate({config-keys: [...], description: [...]})fails becausepr-linkis required and absent →ChangelogValidationError→ exit non-zero → theValidate perf-changelogcheck turns red → PR cannot merge. If pr-link were the only issue, addingpr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2121at the end of the block would clear check 1.- Even then,
validate_raw_changecomputesbase_raw = git show origin/main:perf-changelog.yamlandhead_raw = the current file; because ~30 entries follow the insertion point in head_raw but not at the same byte offset in base_raw,head_raw.startswith(base_raw)is false → raises. Only after moving the 8-line block to the file end (after the current last entry) does this pass. compare_entriesthen compares base and head entries index-by-index; index-of-first-difference is now len(base_entries), the appended entry is treated as an addition,validate_added_pr_linkre-validates the pr-link, and the gate passes.
Fix. Two edits:
- Move the qwen3.5-fp4-gb300-dynamo-sglang-agentic-{agg,disagg} entry (perf-changelog.yaml:4366-4373) to the end of perf-changelog.yaml, after the current last entry.
- Add
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2121to the moved block (orpr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXXfor early WIP).
There was a problem hiding this comment.
Resolved both the above comments
80d1f5c to
ae49f13
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28977280171 |
| speculative-algorithm: NEXTN | ||
| speculative-num-steps: 3 | ||
| speculative-eagle-topk: 1 | ||
| speculative-num-draft-tokens: 4 |
There was a problem hiding this comment.
started about discussion about AgentX MTP +viz @kedarpotdar-nv
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28977625116 |
cquil11
added
the
agentx
ChatGPT Codex Connector
bd940fb to
e571715
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29055853595 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29056354557 |
csahithi
changed the title
|
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Additional detail section:
Signed: |
✅✅✅ Verdict: PASS ✅✅✅✅ Check 0 (CODEOWNER): PASS — |
|
@cquil11 @functionstackx please review and sign off, thanks! |
# Conflicts: # configs/nvidia-master.yaml # perf-changelog.yaml # runners/launch_gb300-nv.sh
…-per-node) The agentic DRAM-offload matrix logic requires available-cpu-dram-mib for any runner used by a kv-offloading: dram (hicache) config. gb300-nv had no hardware entry, so generate_sweep_configs test-config crashed for the gb300 agentic keys (qwen3.5 here, and dsv4 already on main). GB300 NVL72 uses the same 2x Grace / 4-GPU-per-node topology and LPDDR5X capacity as GB200, so mirror gb200-nv's available host DRAM (860_160 MiB) and gpus-per-node (4).
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29773407433 |
# Conflicts: # perf-changelog.yaml
Agentic sweep results are ingested by the dedicated trigger-agentic-ingest job (repository dispatch to InferenceX-app, main-push only), not through collect-results, whose gate intentionally excludes the agentic sweeps. This reverts the qwen branch's addition so run-sweep.yml matches main.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30113721602 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30114330697 |
…erArgs fix) SGLang 20260724 (>= #31811) makes ServerArgs read-only after resolution, which crashed the Dynamo worker at startup (bare server_args.incremental_streaming_output assignment). Pin Dynamo to the ai-dynamo/dynamo#12081 head commit (8e4bfa69), a source build that sets incremental_streaming_output on parsed_args *before* resolution, so no post-resolution mutation is attempted. Swap the 7 disagg recipes' dynamo wheel 1.3.0.dev1 -> hash 8e4bfa69 and bump the disagg router version to match.
# Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30230509317 |
…n routing Switch the 7 disagg recipes from the legacy nvext conv-aware routing (which the newer Dynamo no longer honors) to header-based session affinity: - benchmark.env: AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING=1 -> AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID=true - frontend: add nginx_session_affinity + nginx_session_affinity_header X-Dynamo-Session-ID - frontend args: router-reset-states -> router-session-affinity-ttl-secs 3600
# Conflicts: # perf-changelog.yaml
Skip the legacy nvext conv-aware CLI routing when a recipe opts into the X-Dynamo-Session-ID header path (AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID=true); the existing AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING opt-out and default are unchanged.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30291719028 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30291719028 |
…6 -> 512 The c128 (1P1D DEP4/DEP4) decode server crashed on a DeepEP low-latency dispatch assertion (x.size(0) <= num_max_dispatch_tokens_per_rank, deep_ep.cpp:1262): under DP-attention load imbalance one DP rank's token count (amplified by MTP draft tokens) exceeded the 256 buffer. Double the DeepEP dispatch buffer to 512 (still well under sglang's 1024 cap) to give headroom for the imbalanced+MTP worst case.
# Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30391410523 |
|
/reuse-sweep-run 30391410523 |
|
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Additional detail section:
Signed: |
✅✅✅ Verdict: PASS ✅✅✅✅ Check 0 (CODEOWNER): PASS — |
|
/reuse-sweep-run |
# Conflicts: # perf-changelog.yaml
✅✅✅ Verdict: PASS ✅✅✅✅ Check 0 (CODEOWNER): PASS — |
7 participants
