Switch Qwen3.5 FP4 SGLang configs to NVFP4 V2 / 将 Qwen3.5 FP4 SGLang 配置切换到 NVFP4 V2

[hshrivastava-droid]

Summary

Update the four aggregated Qwen3.5 FP4 SGLang configs (qwen3.5-fp4-b200-sglang and qwen3.5-fp4-b300-sglang, with and without MTP) to nvidia/Qwen3.5-397B-A17B-NVFP4-V2.

Scope:

• Hardware: NVIDIA B200 and B300
• Precision/framework: FP4 with SGLang lmsysorg/sglang:v0.5.14-cu130
• Workload: aggregated, single-node, fixed-sequence throughput and regular evals; no disaggregated or AgentX changes
• Sequence lengths: 1k1k and 8k1k
• Topologies: B200 TP4 and TP2; B300 TP4 and TP2/EP2; both STP and MTP

V1 versus V2

The published NVIDIA model cards describe the following differences:

• V1 was released on 2026-02-17 and quantized with NVIDIA Model Optimizer 0.42.0. Its card describes NVFP4 quantization of linear operators within the MoE blocks.
• V2 was released on 2026-06-29 as NVFP4 2.0 and quantized with NVIDIA Model Optimizer 0.45.0.dev173+g52f1ccbee. Its card says routed experts use NVFP4 with MSE-based scale setting, while attention and shared experts use per-tensor FP8.
• Both checkpoints retain the same Qwen3.5-397B-A17B architecture (397B total parameters, 17B activated); this PR changes the quantized checkpoint, not the model architecture.

Changes

• configs/nvidia-master.yaml: switch the four B200/B300 SGLang STP/MTP config keys from V1 to V2.
• runners/launch_b200-dgxc.sh: resolve Qwen3.5 FP4 to /scratch/fsw/models/Qwen3.5-397B-A17B-NVFP4-V2.
• runners/launch_b300-nv.sh: add Qwen3.5-397B-A17B-NVFP4-V2 to STAGED_MODELS.
• perf-changelog.yaml: append the four config keys with the PR link.

The B200 TRT-LLM configs and GB300 Dynamo-SGLang disaggregated configs intentionally remain on V1 for follow-up validation.

Validation

Exact-head Run Sweep 29371470246, attempt 3 completed successfully on cac471c051825d2c5a8099978f205c22b7beb267:

• 74 aggregated throughput result rows across B200/B300, 1k1k/8k1k, STP/MTP, and the declared TP/EP variants
• 12 non-skipped GSM8K eval jobs; strict exact-match scores range from 0.9629 to 0.9704, above the Qwen3.5 threshold of 0.94
• collect-results, collect-evals, and compare-results all succeeded; compare-results matched 74/74 result rows
• Eval jobs used the same lmsysorg/sglang:v0.5.14-cu130 image declared by the configs

The currently published SGLang Qwen3.5 cookbook still names the V1 checkpoint. Under the live InferenceX checklist, a merged/published upstream recipe update that names V2 and matches the major server arguments is still required before merge.

中文说明

概要

将四个 Qwen3.5 FP4 SGLang 聚合式配置（qwen3.5-fp4-b200-sglang 和 qwen3.5-fp4-b300-sglang，分别包含 STP 与 MTP 版本）更新为 nvidia/Qwen3.5-397B-A17B-NVFP4-V2。

范围：

• 硬件：NVIDIA B200 和 B300
• 精度/框架：FP4，使用 SGLang 镜像 lmsysorg/sglang:v0.5.14-cu130
• 工作负载：聚合式、单节点、固定序列长度吞吐量基准测试及常规评估；不涉及分离式（推理）或 AgentX
• 序列长度：1k1k 和 8k1k
• 拓扑：B200 使用 TP4 与 TP2；B300 使用 TP4 与 TP2/EP2；同时覆盖 STP 和 MTP

V1 与 V2 的差异

NVIDIA 发布的模型卡说明了以下差异：

• V1 发布于 2026-02-17，使用 NVIDIA Model Optimizer 0.42.0 量化。其模型卡说明，MoE 模块中的线性算子采用 NVFP4 量化。
• V2 发布于 2026-06-29，是 NVFP4 2.0 版本，使用 NVIDIA Model Optimizer 0.45.0.dev173+g52f1ccbee 量化。其模型卡说明，路由专家采用基于均方误差（MSE）的 NVFP4 缩放设置，而注意力层和共享专家采用逐张量 FP8。
• 两个检查点均保持相同的 Qwen3.5-397B-A17B 架构（总参数量 397B、激活参数量 17B）；本 PR 仅更换量化检查点，不改变模型架构。

变更内容

• configs/nvidia-master.yaml：将四个 B200/B300 SGLang STP/MTP 配置键从 V1 切换到 V2。
• runners/launch_b200-dgxc.sh：将 Qwen3.5 FP4 解析到 /scratch/fsw/models/Qwen3.5-397B-A17B-NVFP4-V2。
• runners/launch_b300-nv.sh：将 Qwen3.5-397B-A17B-NVFP4-V2 加入 STAGED_MODELS。
• perf-changelog.yaml：在文件末尾追加四个配置键及当前 PR 链接。

B200 TRT-LLM 配置和 GB300 Dynamo-SGLang 分离式（推理）配置在本 PR 中有意保留 V1，待后续验证后再迁移。

验证

基于当前 head cac471c051825d2c5a8099978f205c22b7beb267 的 Run Sweep 29371470246（第 3 次尝试） 已成功完成：

• 共聚合 74 条吞吐量结果，覆盖 B200/B300、1k1k/8k1k、STP/MTP 及声明的 TP/EP 变体
• 12 个未跳过的 GSM8K 评估任务；严格精确匹配得分为 0.9629–0.9704，高于 Qwen3.5 的 0.94 阈值
• collect-results、collect-evals 和 compare-results 均成功；compare-results 匹配 74/74 条结果
• 评估任务使用与配置一致的 lmsysorg/sglang:v0.5.14-cu130 镜像

当前发布的 SGLang Qwen3.5 cookbook 仍使用 V1 检查点。根据 InferenceX 最新检查清单，在合并前仍需让指向 V2 且主要服务参数一致的上游 recipe 更新完成合并或正式发布。

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[claude]

LGTM — straightforward checkpoint swap.

Extended reasoning...

Overview

This PR swaps the checkpoint used for all 7 Qwen3.5 FP4 benchmark configs (b200 sglang/trt ±mtp, b300 sglang ±mtp, gb300 dynamo-sglang disagg) from nvidia/Qwen3.5-397B-A17B-NVFP4 to nvidia/Qwen3.5-397B-A17B-NVFP4-V2. Changes are confined to: the model: field for all 7 relevant keys in configs/nvidia-master.yaml, served-model-name in the 4 gb300 disagg recipe YAMLs, MODEL_PATH filename strings in launch_b200-dgxc.sh/launch_gb300-nv.sh, a new entry in the STAGED_MODELS array in launch_b300-nv.sh, and a matching perf-changelog.yaml entry.

Security risks

None — this is a benchmark config/recipe change with no code paths touching auth, secrets, or user input. The checkpoint name is a static string pointing to a pre-staged local path.

Level of scrutiny

Low. This is a mechanical, well-scoped model-checkpoint version bump following the exact pattern of prior similar PRs in this repo's history (e.g. recent H200 FP8 refresh, other model swaps). I cross-checked that all references to the old checkpoint name are updated consistently across the master config, the gb300/b300/b200 launcher scripts, and the recipe YAMLs, and that the aliasing scheme (served-model-name vs. model-path alias vs. STAGED_MODELS basename matching) stays internally consistent after the swap — no path or config was missed or left mismatched.

Other factors

No bugs were surfaced by the bug hunting system, and the two candidate issues it raised (missing Chinese translation, placeholder PR link) were checked and are non-issues — the changelog entry has a real PR link and translation isn't required for this template. No CODEOWNER-sensitive or business-logic code is touched.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29353750144
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29353750144

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29370174289
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29370174289

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29370502845
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29370502845

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29371470246
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29371470246

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29371470246
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29371470246

[functionstackx]

what is difference between nvidia/Qwen3.5-397B-A17B-NVFP4-V2 & nvidia/Qwen3.5-397B-A17B-NVFP4

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29371470246
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29371470246

[Ankur-singh]

Revoking the standing `/reuse-sweep-run` authorization on this PR (removing the bare command comment from 2026-07-23).

A bare `/reuse-sweep-run` is not one-shot — `utils/find_reusable_sweep_run.py` re-reads it on every synchronize, so it sets `skip-pr-sweep=true` for every future push. Run 30499419692 at the current head is that firing: setup skipped, zero benchmark jobs, and reuse-source-run-id empty — it neither ran fresh nor bound reuse artifacts.

Why reuse is no longer safe here: the evidence run 29371470246 is from 2026-07-14, and .github/workflows/benchmark-tmpl.yml has since gained SALLOC_EXCLUDE: 'b300-005,b300-006' ("These b300 nodes are currently broken", added 2026-07-16). The B300 half of that evidence may have been measured on hardware since flagged bad. The 8k1k code path itself is unchanged, so the B200 half looks sound — but the B300 half should be re-measured.

Re-authorize with an explicit run ID once a fresh sweep lands.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30506346629
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=30506346629

[Ankur-singh]

/reuse-sweep-run 30506346629

[Ankur-singh]

As a PR reviewer and CODEOWNER, I have reviewed this and have:

• Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
• Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
• Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. — https://github.com/SemiAnalysisAI/InferenceX/actions/runs/30506346629
• Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. — https://github.com/SemiAnalysisAI/InferenceX/actions/runs/30506346629
• Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
• For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
• Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
• If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
• If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
• Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
  • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
• Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
• If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.

Additional detail section:

• Scope. Switches the four single-node Qwen3.5 FP4 SGLang config keys — qwen3.5-fp4-b200-sglang, -b200-sglang-mtp, -b300-sglang, -b300-sglang-mtp — from nvidia/Qwen3.5-397B-A17B-NVFP4 to the mixed-precision nvidia/Qwen3.5-397B-A17B-NVFP4-V2 checkpoint, on lmsysorg/sglang:v0.5.14-cu130. Reviewed at head 99e3a3fdaee4f2822ff43442509915f9a3a1bae0.
• Validation and evals. Run Sweep 30506346629 ran on the exact current head. It is a real run, not a reuse-gated no-op: setup = success, and the benchmark lanes executed. Non-skipped successes: 36/36 single-node 8k1k and 12/12 eval /, plus canary, collect-results, collect-evals, compare-results and calc-success-rate all green. Zero failures. Multi-node and agentic lanes are skipped because these keys are single-node fixed-seq-len only.
• Why this is a fresh sweep rather than a reuse. The prior evidence run (29371470246, 2026-07-14) predates SALLOC_EXCLUDE: 'b300-005,b300-006' ("These b300 nodes are currently broken", added 2026-07-16), so the B300 half of that evidence may have been measured on hardware since flagged bad. The standing bare /reuse-sweep-run on this PR was therefore revoked and the sweep re-run from scratch at this head.
• Recipe — already published, linked here. The matching recipe is the published SGLang cookbook page for Qwen3.5: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.5 (source: docs_new/cookbook/autoregressive/Qwen/Qwen3.5.mdx). It documents this exact serving shape — --quantization modelopt_fp4, --kv-cache-dtype fp8_e4m3, --attention-backend trtllm_mha, --moe-runner-backend flashinfer_trtllm, B200/B300 Blackwell TP guidance, and EAGLE speculative decoding for the MTP variants. The serve configuration is unchanged by this PR: NVFP4-V2 is a ModelOpt MIXED_PRECISION checkpoint (per-tensor FP8 linears, NVFP4 routed experts) that SGLang detects from the checkpoint config and routes through modelopt_mixed automatically, so the documented command still passes --quantization modelopt_fp4 and the only delta is the --model-path checkpoint ID. Engine support for this path is merged upstream (for example [NVIDIA] Allow modelopt_mixed quantization with flashinfer_cutedsl MoE runner sgl-project/sglang#30443, allowing modelopt_mixed with the flashinfer_cutedsl MoE runner) and is present in the pinned v0.5.14-cu130 image — empirically confirmed by this sweep loading and serving the V2 checkpoint with 12/12 evals passing.
  • Reviewer note, stated plainly: the published page currently names the non-V2 checkpoint in its model table. The judgement recorded here is that the recipe — the serving configuration the cookbook documents — is what the requirement covers, and that configuration is byte-identical for both checkpoints because the quantization path is auto-detected. No separate upstream PR is required on that basis.
• Chat template (spec decode). Verified directly at head rather than assumed: both MTP scripts pass --use-chat-template to run_benchmark_serving (benchmarks/single_node/fixed_seq_len/qwen3.5_fp4_b200_mtp.sh, qwen3.5_fp4_b300_mtp.sh). The two non-MTP scripts have no speculative decoding at all.
• Agentic golden AL (unchecked). Not applicable. These are single-node fixed-seq-len 8k1k benchmarks, not agentic workloads, and the MTP variants run real EAGLE speculative decoding — no SGLANG_SIMULATE_ACC_LEN or synthetic acceptance anywhere in the four scripts — so there is no injected acceptance target to reconcile against golden_al_distribution/.
• Model architecture. No --hf-overrides in any of the four scripts, and no layer-skipping or FLOPs-reducing knobs. NVFP4-V2 raises the dense linears from NVFP4 to per-tensor FP8, i.e. higher precision on those layers, and the configuration clears the eval gate 12/12.
• Upstream image and framework ordering. lmsysorg/sglang:v0.5.14-cu130 from the official https://hub.docker.com/u/lmsysorg org — no fork or vendor rebuild. This is the SGLang submission itself, so the engine-first ordering requirement is satisfied by construction.
• No engine patching. No .patch files, git apply, sed -i, site-packages edits, monkey-patching, or rebuilt engine wheels — the pinned image runs as shipped. No waiver required or claimed.
• Code quality — one defect found and fixed during review. runners/launch_b200-dgxc.sh resolved MODEL_PATH on MODEL_PREFIX + PRECISION alone, but two checkpoints now share qwen3.5 + fp4. Since the launcher later does export MODEL="$MODEL_PATH", the unrelated qwen3.5-fp4-b200-trt / -trt-mtp keys would have served V2 weights while publishing under the old checkpoint name — and only on b200-dgxc_* runners, since launch_b200-cw.sh / launch_b200-nb.sh pin no path. Fixed in fix(runners): scope b200-dgxc qwen3.5 fp4 model path to the V2 checkpoint / 修复：将 b200-dgxc 的 qwen3.5 fp4 模型路径限定到 V2 检查点 #2414 (merged into this branch) by branching on the checkpoint; the old-checkpoint arm restores the exact /lustre path used on main, so TRT behaviour is unchanged.

Signed: Ankur-singh

[Klaud-Cold]

❌❌❌ REJECTED ❌❌❌

@Ankur-singh — blocking: no merged/published upstream recipe documents nvidia/Qwen3.5-397B-A17B-NVFP4-V2, the checkpoint this PR benchmarks. The linked SGLang cookbook page names only the V1 checkpoint.

• ✅ Check 0 (CODEOWNER): PASS — Ankur-singh is a named owner of configs/nvidia-master.yaml; all other changed paths carry only the catch-all.
• ✅ Check 1 (passing sweep on in-PR commit): PASS — head 99e3a3f has 36/36 non-skipped single-node 8k1k / and 12/12 eval / successes in run 30506346629; the 1k1k lane skipped because these four configs define only 8k1k scenarios.
• ✅ Check 2 (evals pass): PASS — 12/12 GSM8K rows on the NVFP4-V2 checkpoint, min em_strict 0.9621 ≥ the 0.94 qwen3.5 bar (utils/evals/thresholds.yaml); eval jobs ran lmsysorg/sglang:v0.5.14-cu130, matching the configs.
• ❌ Check 3 (recipe merged & matches): FAIL — the model checkpoint is a MAJOR arg, and the published cookbook (source docs_new/cookbook/autoregressive/Qwen/Qwen3.5.mdx on sgl-project/sglang main) names only nvidia/Qwen3.5-397B-A17B-NVFP4; NVFP4-V2 appears nowhere upstream. V2 is a different quantization scheme (per-tensor FP8 attention/shared experts vs NVFP4), so a community member following the cookbook reproduces a different benchmark. The sign-off's judgment call that "no separate upstream PR is required" does not meet this standard — the PR's own description records that a merged/published upstream update naming V2 is required before merge. Other MAJOR flags do match (modelopt_fp4, fp8_e4m3 KV, trtllm_mha, flashinfer_trtllm, TP guidance, spec-decode steps/topk/draft-tokens); chunked-prefill and scheduler-recv diffs are InferenceX tuning, informational only.
• ✅ Check 4 (reuse command): PASS — /reuse-sweep-run 30506346629 posted 2026-07-30 by Ankur-singh (COLLABORATOR).
• ✅ Check 5 (latest template): PASS — all current-template items present; the single unchecked item (agentic golden AL) is explained as not applicable in the detail section.
• ✅ Check 6 (upstream image / engine-first): PASS — lmsysorg/sglang:v0.5.14-cu130 from the upstream lmsysorg org on established B200/B300; this is itself the SGLang submission.
• ✅ Check 7 (no architecture hacks): PASS — only the checkpoint ID changes; no --hf-overrides or FLOPs-reducing knobs in the diff.
• ✅ Check 8 (spec-decode chat template): PASS — both MTP scripts pass --use-chat-template to run_benchmark_serving.
• ✅ Check 9 (no engine patches): PASS — no patch files, heredoc rewrites, site-packages edits, or engine wheel installs in the diff.
• ➖ Check 10 (agentic golden AL): N/A — no agentic spec-decode changes; the MTP configs are fixed-seq-len and correctly run real (unsimulated) acceptance.

[Ankur-singh]

As a PR reviewer and CODEOWNER, I have reviewed this and have:

• Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
• Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
• Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. — https://github.com/SemiAnalysisAI/InferenceX/actions/runs/30506346629
• Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. — https://github.com/SemiAnalysisAI/InferenceX/actions/runs/30506346629
• Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
• For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
• Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
• Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
• If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
• If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
• Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
  • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
• Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
• If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.

Additional detail section:

• Scope. Switches the four single-node Qwen3.5 FP4 SGLang config keys — qwen3.5-fp4-b200-sglang, -b200-sglang-mtp, -b300-sglang, -b300-sglang-mtp — from nvidia/Qwen3.5-397B-A17B-NVFP4 to the mixed-precision nvidia/Qwen3.5-397B-A17B-NVFP4-V2 checkpoint, on lmsysorg/sglang:v0.5.14-cu130. Reviewed at head 99e3a3fdaee4f2822ff43442509915f9a3a1bae0.
• This re-signs an earlier sign-off. The 2026-07-30 sign-off was written against checklist blob 79f52725; the live template is now 8fae7c37 and has gained the MODELS.md deprecation item, so the earlier sign-off is stale on Check 5 regardless of its substance. The head SHA is unchanged.
• Recipe — the blocker from the previous verdict is now cleared. The earlier sign-off was rejected on Check 3: the model checkpoint is a MAJOR arg, and at that time the published SGLang cookbook named only the V1 checkpoint. That verdict was accepted rather than re-argued, and the item was left unchecked pending upstream.
  • The upstream recipe PR — sgl-project/sglang#32945, "Qwen3.5 NVFP4 V2" — is now MERGED: merged at 2026-08-07T22:26:07Z by ishandhanani, merge commit 115cd7bde1b5e5b5d444a7bdd6c21b534f871bfe. It merged before this InferenceX PR, which is what the checklist requires; an open or draft upstream PR would not have satisfied it.
  • Verified on sgl-project/sglang main rather than taken from the PR description: docs/src/snippets/autoregressive/qwen35-deployment.jsx now resolves Blackwell to 'nvidia/Qwen3.5-397B-A17B-NVFP4-V2', with the accompanying comment "AMD MI355X uses the MXFP4 checkpoint; Blackwell uses NVFP4-V2", and the merged change also updates docs/cookbook/autoregressive/Qwen/Qwen3.5.mdx. The published Qwen3.5 cookbook page therefore now documents the exact checkpoint this PR benchmarks, on the same --quantization modelopt_fp4 / --kv-cache-dtype fp8_e4m3 / --attention-backend trtllm_mha / --moe-runner-backend flashinfer_trtllm serving shape, for B200 and B300.
• Validation and evals. Run Sweep 30506346629 ran on the exact current head and is a real run, not a reuse-gated no-op: setup = success and the benchmark lanes executed. Non-skipped successes: 36/36 single-node 8k1k and 12/12 eval /, plus canary, collect-results, collect-evals, compare-results and calc-success-rate all green. Zero failures. Multi-node and agentic lanes skipped because these keys are single-node fixed-seq-len only.
• MODELS.md. Qwen3.5-397B-A17B is active for Single-turn 8k1k on fp8/fp4. This PR touches only fp4 8k1k keys. The three current qwen3.5 deprecations are all out of scope: bf16 recipes (this is fp4), Single-turn 1k1k (these four configs define only an 8192/1024 scenario — which is why the single-node 1k1k lane skipped in the sweep), and the non-MTP agentic arm (not yet enacted, and no agentic scenario here).
• Chat template (spec decode). Re-verified at head: both MTP scripts pass --use-chat-template to run_benchmark_serving (benchmarks/single_node/fixed_seq_len/qwen3.5_fp4_b200_mtp.sh:78, qwen3.5_fp4_b300_mtp.sh:113). The two non-MTP scripts contain no speculative decoding at all.
• Agentic golden AL (unchecked). Not applicable. These are single-node fixed-seq-len 8k1k benchmarks, not agentic workloads, and the MTP variants run real EAGLE speculative decoding (--speculative-algorithm EAGLE, 3 steps / topk 1 / 4 draft tokens) with no SGLANG_SIMULATE_ACC_LEN anywhere in the four scripts — so there is no injected acceptance target to reconcile against golden_al_distribution/.
• Model architecture. No --hf-overrides in any of the four scripts, and no layer-skipping or FLOPs-reducing knobs. NVFP4-V2 raises the dense linears from NVFP4 to per-tensor FP8 — higher precision on those layers — and the configuration clears the eval gate 12/12.
• Upstream image and framework ordering. lmsysorg/sglang:v0.5.14-cu130 from the official https://hub.docker.com/u/lmsysorg org — no fork or vendor rebuild. This is the SGLang submission itself, so the engine-first ordering requirement is satisfied by construction.
• No engine patching. No .patch files, git apply, sed -i, site-packages edits, monkey-patching, or rebuilt engine wheels; no setup_script on any of the four config keys. The pinned image runs as shipped. No waiver required or claimed.
• Code quality. runners/launch_b200-dgxc.sh resolved MODEL_PATH on MODEL_PREFIX + PRECISION alone, but two checkpoints now share qwen3.5 + fp4. Since the launcher later does export MODEL="$MODEL_PATH", the unrelated qwen3.5-fp4-b200-trt / -trt-mtp keys would have served V2 weights while publishing under the old checkpoint name — and only on b200-dgxc_* runners, since launch_b200-cw.sh / launch_b200-nb.sh pin no path. Fixed in fix(runners): scope b200-dgxc qwen3.5 fp4 model path to the V2 checkpoint / 修复：将 b200-dgxc 的 qwen3.5 fp4 模型路径限定到 V2 检查点 #2414 (merged into this branch) by branching on the checkpoint; the old-checkpoint arm restores the exact /lustre path used on main, so TRT behaviour is unchanged. runners/launch_b300-nv.sh adds Qwen3.5-397B-A17B-NVFP4-V2 to the staged-model allow-list.
• Mergeability (separate from this review). The PR is currently CONFLICTING. The sole conflict is perf-changelog.yaml — the append-at-end entry versus newer entries on main; configs/nvidia-master.yaml and runners/launch_b200-dgxc.sh both auto-merge cleanly.

Signed: Ankur-singh

[Klaud-Cold]

✅✅✅ Verdict: PASS ✅✅✅

The blocker from the previous verdict is cleared: the upstream recipe naming NVFP4-V2 is now merged. Head unchanged at 99e3a3f.

✅ Check 0 (CODEOWNER): PASS — @ankur-singh owns configs/nvidia-master.yaml; remaining paths are catch-all only.
✅ Check 1 (sweep on in-PR commit): PASS — head 99e3a3f carries 36/36 single-node 8k1k / and 12/12 eval / check-runs, all success (non-skipped), from run 30506346629.
✅ Check 2 (evals pass): PASS — 12/12 GSM8K entries on Qwen3.5-397B-A17B-NVFP4-V2, em_strict 0.9621–0.9697 vs 0.94 threshold (utils/evals/thresholds.yaml), eval jobs on the PR's lmsysorg/sglang:v0.5.14-cu130 image.
✅ Check 3 (recipe merged & matches): PASS — sgl-project/sglang#32945 is MERGED (2026-08-07T22:26:07Z); the live qwen35-deployment.jsx resolves Blackwell FP4 to nvidia/Qwen3.5-397B-A17B-NVFP4-V2 with matching major args (--quantization modelopt_fp4, --kv-cache-dtype fp8_e4m3, --attention-backend trtllm_mha, --moe-runner-backend flashinfer_trtllm, TP4, EAGLE/NEXTN MTP 3 steps · topk 1 · 4 draft tokens). Informational only: the TP2 (B200) / TP2-EP2 (B300) sweep lanes and scheduler/chunked-prefill knobs are InferenceX harness tuning.
✅ Check 4 (reuse command): PASS — authorized /reuse-sweep-run 30506346629 posted by Ankur-singh (COLLABORATOR), superseding their earlier revocation of the bare command.
✅ Check 5 (latest template): PASS — all current PR_REVIEW_CHECKLIST.md items present; the one unchecked item (agentic golden AL) is explained as N/A in the additional detail section.
✅ Check 6 (upstream image / engine-first): PASS — lmsysorg/sglang:v0.5.14-cu130 from the upstream org; this is itself the SGLang submission.
✅ Check 7 (deprecations): PASS — as of 2026-08-07, qwen3.5 is active for Single-turn 8k1k fp8/fp4; these four fp4 keys define only 8k1k, and the deprecated bf16/1k1k/agentic arms are untouched.
✅ Check 8 (no architecture hacks): PASS — no --hf-overrides or FLOPs-reducing knobs in the diff or the four scripts; V2 raises dense linears to per-tensor FP8.
✅ Check 9 (spec-decode chat template): PASS — both MTP scripts call run_benchmark_serving with --use-chat-template (qwen3.5_fp4_b200_mtp.sh:78, qwen3.5_fp4_b300_mtp.sh:113).
✅ Check 10 (no engine patches): PASS — no .patch/git apply/sed -i/heredoc rewrites or engine reinstalls; only client-side pip install datasets pandas.
➖ Check 11 (agentic golden AL): N/A — no agentic spec-decode changes; the fixed-seq-len MTP configs run real acceptance with no simulated-AL knobs, as Check 9 requires.