Qwen3.5 FP4 GB300 disaggregated Dynamo-SGLang: update SGLang to v0.5.14-cu130 and NVFP4-V2 / Qwen3.5 FP4 GB300 分离式 Dynamo-SGLang：将 SGLang 更新至 v0.5.14-cu130 并切换到 NVFP4-V2

[hshrivastava-droid]

Summary

• Updates qwen3.5-fp4-gb300-dynamo-sglang from lmsysorg/sglang:nightly-dev-cu13-20260624-b2c8f7a2 to the upstream lmsysorg/sglang:v0.5.14-cu130 image.
• Switches the checkpoint and served model name from nvidia/Qwen3.5-397B-A17B-NVFP4 to nvidia/Qwen3.5-397B-A17B-NVFP4-V2, and updates the GB300 launcher model path accordingly.
• Keeps the existing scope: FP4, GB300, multi-node disaggregated Dynamo-SGLang, STP (no speculative decoding), fixed 8k input / 1k output. This PR does not add AgentX workloads.

Topologies

• 1 prefill worker (TP4) × 1 decode worker (TP4), Mooncake, 8 GPUs, concurrency 1/4/8/16/32/64/256.
• 5 prefill workers (TP4/EP4/DP-attention) × 1 decode worker (TP16/EP16/DP-attention), NIXL, 36 GPUs, concurrency 2048.
• 6 prefill workers (TP4/EP4/DP-attention) × 1 decode worker (TP16/EP16/DP-attention), Mooncake, 40 GPUs, concurrency 5120.
• 7 prefill workers (TP4/EP4/DP-attention) × 1 decode worker (TP16/EP16/DP-attention), Mooncake, 44 GPUs, concurrency 5120.

Changed paths

• configs/nvidia-master.yaml
• runners/launch_gb300-nv.sh
• Four recipe YAMLs under benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/gb300-fp4/8k1k/disagg/stp/
• perf-changelog.yaml

Validation

• Run Sweep 29473335540 completed successfully on head 38d77d67b74131d77a2ec4ffeda6f200820a1bbf; all four applicable multi-node throughput jobs and all four matching multi-node eval jobs ran successfully.
• The aggregated GSM8K strict scores are 0.9659–0.9682, above the Qwen3.5 threshold of 0.94. The job evidence identifies lmsysorg/sglang:v0.5.14-cu130 and nvidia/Qwen3.5-397B-A17B-NVFP4-V2.

Open review item

• The current CHANGES_REQUESTED review asks for customer-traction evidence and the rationale for prioritizing the V2 quantization checkpoint. That rationale is not present in the current PR body or GitHub review thread.

中文说明

• 将 qwen3.5-fp4-gb300-dynamo-sglang 使用的镜像从 lmsysorg/sglang:nightly-dev-cu13-20260624-b2c8f7a2 更新为上游镜像 lmsysorg/sglang:v0.5.14-cu130。
• 将检查点及服务模型名称从 nvidia/Qwen3.5-397B-A17B-NVFP4 切换为 nvidia/Qwen3.5-397B-A17B-NVFP4-V2，并同步更新 GB300 启动器中的模型路径。
• 保持现有范围不变：FP4、GB300、多节点分离式 Dynamo-SGLang、STP（不使用投机解码）、固定 8k 输入 / 1k 输出。本 PR 不新增 AgentX 工作负载。

拓扑

• 1 个预填充 worker（TP4）× 1 个解码 worker（TP4），使用 Mooncake，共 8 张 GPU，并发度为 1/4/8/16/32/64/256。
• 5 个预填充 worker（TP4/EP4/DP-attention）× 1 个解码 worker（TP16/EP16/DP-attention），使用 NIXL，共 36 张 GPU，并发度为 2048。
• 6 个预填充 worker（TP4/EP4/DP-attention）× 1 个解码 worker（TP16/EP16/DP-attention），使用 Mooncake，共 40 张 GPU，并发度为 5120。
• 7 个预填充 worker（TP4/EP4/DP-attention）× 1 个解码 worker（TP16/EP16/DP-attention），使用 Mooncake，共 44 张 GPU，并发度为 5120。

变更路径

• configs/nvidia-master.yaml
• runners/launch_gb300-nv.sh
• benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/gb300-fp4/8k1k/disagg/stp/ 下的 4 个配方 YAML
• perf-changelog.yaml

验证

• Run Sweep 29473335540 已在 head 38d77d67b74131d77a2ec4ffeda6f200820a1bbf 上成功完成；4 个适用的多节点吞吐量任务和 4 个对应的多节点评估任务均已实际运行并通过。
• 汇总的 GSM8K strict 分数为 0.9659–0.9682，高于 Qwen3.5 的 0.94 门槛。任务证据确认使用了 lmsysorg/sglang:v0.5.14-cu130 和 nvidia/Qwen3.5-397B-A17B-NVFP4-V2。

待确认审阅问题

• 当前 CHANGES_REQUESTED 审阅要求补充客户使用情况，以及优先投入 V2 量化检查点的依据。现有 PR 描述和 GitHub 审阅线程中尚未提供相关说明。

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[functionstackx]

what is difference between nvidia/Qwen3.5-397B-A17B-NVFP4 & nvidia/Qwen3.5-397B-A17B-NVFP4-v2

[functionstackx]

Are yall seeing lots of customer traction for qwen3.5 hence why modelopt team has pritotized and used their limited engineering bandwidth on an v2 quant on qwen3.5?

+viz @Ankur-singh

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29473335540
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29473335540

[Ankur-singh]

/reuse-sweep-run 29473335540

[Ankur-singh]

As a PR reviewer and CODEOWNER, I have reviewed this and have:

• Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
• Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
• Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. Run Sweep 29473335540
• Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. Run Sweep 29473335540
• Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
• For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
• Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
• If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
• If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
• Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
  • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
• Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
• If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.

Additional detail section:

• Head signed: a87e7fb779e353d6030f68a66c0f139f855af01f.
• Benchmark and eval evidence, and why it is reused. The full sweep run 29473335540 executed on in-PR commit 38d77d67 and concluded success: all 4 multi-node 8k1k / jobs and all 4 multi-node eval / jobs ran non-skipped and succeeded. Its job matrix confirms it exercised this PR's new image lmsysorg/sglang:v0.5.14-cu130 and the new nvidia/Qwen3.5-397B-A17B-NVFP4-V2 checkpoint. agg_eval_all.json from that run has 4 GSM8K rows for nvidia/Qwen3.5-397B-A17B-NVFP4-V2, em_strict 0.9659–0.9682 at n_eff 1319, against the qwen3.5 override bar of 0.94 in utils/evals/thresholds.yaml.
• The current head is that same tree plus a main merge only. 38d77d67 → a87e7fb7 is a single Merge origin/main and resolve perf changelog commit, authorized for reuse by /reuse-sweep-run 29473335540 posted before the push, which is why the sweep at this head reports reuse-sweep-gate: success with benchmark jobs skipped. I verified the benchmark-relevant surface is unchanged across that merge: the qwen3.5-fp4-gb300-dynamo-sglang config block is byte-identical, and all four GB300 recipes (8k1k_stp_lowlat_0, 8k1k_stp_maxtpt_0/1/2) are byte-identical. In runners/launch_gb300-nv.sh the MODEL_PATH=/scratch/models/Qwen3.5-397B-A17B-NVFP4-V2 line and the non-agentic dynamo-sglang/qwen3.5 srt-slurm branch both survive unchanged; the only launcher change main contributed is a new agentic qwen3.5 branch, and this config is fixed-seq-len, not agentic, so it does not take that path.
• Item 2 (general code quality). The earlier revision of this PR stripped the two trailing spaces from the blank separator line that PR Add GB300 DeepSeek-V4 Dynamo-SGLang AgentX aggregated and disaggregated recipes / 新增 GB300 DeepSeek-V4 Dynamo-SGLang AgentX 聚合式与分离式配方 #2157 left as " " in perf-changelog.yaml, which AGENTS.md forbids ("Never delete or modify whitespace in perf-changelog.yaml"). That is fixed at this head: the conflict was resolved by taking main's perf-changelog.yaml byte-for-byte and re-appending this PR's entry at the tail, so the separator is restored (35 trailing-space lines, matching main) and the diff against main adds only this PR's 7-line entry with no deletions. utils/process_changelog.py validates clean against origin/main and regenerates exactly the four benchmark and four eval configurations that the reused run covered.
• Item 5 (spec-decode chat templates) — not applicable: the config's search space sets spec-decoding: "none" and none of the four recipes contains any speculative-decoding key.
• Item 6 (agentic golden AL) — not applicable: no agentic workloads; the only scenario is fixed-seq-len 8k1k, and no simulated-acceptance knobs appear anywhere in the diff.
• Items 10 / 10a (single-node recipe documentation) — not applicable: this is a multi-node disaggregated submission (multinode: true, disagg: true, framework dynamo-sglang, GB300); the recipe-documentation requirement is scoped to single-node vLLM/SGLang recipes.
• Item 7 (architecture): the diff changes only the pinned image tag, the checkpoint name in four served-model-name fields plus the config, and one MODEL_PATH. No --hf-overrides, no layer skipping, no change to parallelism or any FLOPs-affecting knob.
• Items 8 / 9 (upstream image and framework ordering): the image is lmsysorg/sglang:v0.5.14-cu130 from the upstream lmsysorg Docker org, and this PR moves the config off the nightly-dev-cu13-20260624-b2c8f7a2 build onto a tagged upstream release, improving reproducibility. This entry is itself the SGLang submission.
• Item 11 (no engine patches): none of the four GB300 recipes declares setup_script or any patch key, and the diff contains no .patch files, git apply/patch calls, sed or heredoc rewrites of engine sources, site-packages edits, or forked engine wheel installs. No waiver is required. Noted for transparency and not introduced by this PR: the fp4 path checks out the srt-slurm sa-submission-q2-2026 branch rather than a pinned tag, which is a mutable reference.
• Open reviewer thread (non-checklist): functionstackx has an outstanding CHANGES_REQUESTED review asking what differs between the NVFP4 and NVFP4-V2 checkpoints and about Qwen3.5 customer traction motivating the V2 quant. That thread is theirs to resolve and this sign-off does not clear it.

Signed: Ankur-singh

[Klaud-Cold]

✅✅✅ Verdict: PASS ✅✅✅

✅ Check 0 (CODEOWNER): PASS — Ankur-singh is a named owner of configs/nvidia-master.yaml; all other changed paths fall to the catch-all, which a recognized CODEOWNER satisfies.
✅ Check 1 (passing sweep on in-PR commit): PASS — in-PR commit 38d77d67 has all 4 multi-node 8k1k / and all 4 multi-node eval / check-runs green (executed, non-skipped) in run 29473335540.
✅ Check 2 (evals pass): PASS — agg_eval_all.json from that run has 4 GSM8K rows for nvidia/Qwen3.5-397B-A17B-NVFP4-V2, em_strict 0.9659–0.9682 (n_eff 1319) vs the 0.94 qwen3.5 bar; check-run matrix confirms the run used this PR's image lmsysorg/sglang:v0.5.14-cu130.
➖ Check 3 (recipe link): N/A — disaggregated/multi-node submission (multinode: true, disagg: true, dynamo-sglang, all benchmark files under benchmarks/multi_node/**); the recipe-link requirement applies to single-node recipes only.
✅ Check 4 (reuse command): PASS — /reuse-sweep-run 29473335540 posted by Ankur-singh (COLLABORATOR).
✅ Check 5 (latest checklist): PASS — every current-template item is present; the three unchecked items (spec-decode chat templates, agentic golden AL, single-node recipe link) are each explained as N/A in the additional detail section.
✅ Check 6 (upstream image / engine-first): PASS — image is upstream lmsysorg/sglang:v0.5.14-cu130 (moves the config off a nightly-dev build); the PR adds no new non-vLLM/SGLang entry, it updates an existing dynamo-sglang entry running the upstream SGLang engine.
✅ Check 7 (no architecture hacks): PASS — diff changes only the image tag, checkpoint name (NVFP4 → NVFP4-V2), and MODEL_PATH; no --hf-overrides or FLOPs-affecting knobs.
➖ Check 8 (spec-decode chat templates): N/A — spec-decoding: "none"; no speculative-decoding flags in any changed recipe.
✅ Check 9 (no engine patches): PASS — no .patch/git apply/heredoc rewrites/site-packages edits/forked wheels in the diff or the four recipes (only *_DISPATCH env toggles, which are runtime settings, not patches).
➖ Check 10 (agentic golden AL): N/A — no agentic speculative-decoding changes; the only scenario is fixed-seq-len 8k1k with spec-decoding none.

[functionstackx]

per above

[xinli-sw]

recipes PR merged, should this go in?