feat: add Qwen3.5 RTX PRO 6000 SGLang recipe / 新增 Qwen3.5 RTX PRO 6000 SGLang 基准配置 - #2312
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # Qwen3.5-397B-A17B NVFP4 single-node vLLM sweep using 4 of 8 RTX PRO | ||
| # 6000 Blackwell GPUs. Both arms use ordinary NCCL collectives on PCIe. | ||
| qwen3.5-fp4-rtx6000pro-vllm: | ||
| image: vllm/vllm-openai:v0.25.1@sha256:e4f88a835143cd22aee2397a26ec6bb80b3a4a6fe0c882bcbc63822904766089 | ||
| model: nvidia/Qwen3.5-397B-A17B-NVFP4 | ||
| model-prefix: qwen3.5 | ||
| runner: rtx6000pro-lat | ||
| precision: fp4 | ||
| framework: vllm | ||
| multinode: false | ||
| scenarios: | ||
| fixed-seq-len: | ||
| - isl: 8192 | ||
| osl: 1024 | ||
| search-space: | ||
| - { tp: 4, conc-list: [1, 4, 16, 64] } | ||
| - { tp: 4, ep: 4, conc-list: [1, 4, 16, 64] } | ||
|
|
There was a problem hiding this comment.
🔴 This PR adds the new config-key qwen3.5-fp4-rtx6000pro-vllm to configs/nvidia-master.yaml but doesn't append a matching entry to perf-changelog.yaml. Per AGENTS.md, the changelog is the append-only benchmark trigger log, so without an entry the new recipe won't get picked up/triggered for benchmarking — please add a config-keys: [qwen3.5-fp4-rtx6000pro-vllm] block at the end of perf-changelog.yaml describing the new RTX PRO 6000 recipe, with a link to this PR. / 此 PR 在 configs/nvidia-master.yaml 中新增了配置项 qwen3.5-fp4-rtx6000pro-vllm，但没有在 perf-changelog.yaml 中添加对应条目。根据 AGENTS.md，该变更日志是仅追加的基准测试触发日志，缺少条目会导致新配方无法被触发进行基准测试——请在 perf-changelog.yaml 文件末尾添加一个 config-keys: [qwen3.5-fp4-rtx6000pro-vllm] 区块，描述这个新的 RTX PRO 6000 配方，并附上此 PR 的链接。
Extended reasoning...
What's missing: configs/nvidia-master.yaml gains a brand-new config-key, qwen3.5-fp4-rtx6000pro-vllm (lines 1346-1363 of the diff), defining a 4-GPU RTX PRO 6000 vLLM sweep for Qwen3.5-397B-A17B NVFP4. The PR's four changed files are benchmarks/single_node/fixed_seq_len/qwen3.5_fp4_rtx6000pro.sh, configs/nvidia-master.yaml, configs/runners.yaml, and runners/launch_rtx6000pro-lat.sh — perf-changelog.yaml is not touched at all, and grep -n rtx6000pro perf-changelog.yaml returns zero matches.
Why this matters: AGENTS.md is explicit on this point in two places. It describes perf-changelog.yaml as the "benchmark trigger log; append-only" (Directory Structure section), and the "Adding a benchmark configuration" workflow spells out the exact sequence: "Add entries to configs/nvidia-master.yaml or amd-master.yaml ..., append to perf-changelog.yaml, then validate with generate_sweep_configs.py full-sweep." This isn't a style nit — the repo's own CI encodes it as a hard requirement: .github/workflows/claude-pr-review.yml has a dedicated check that fires exactly on this condition ("If either master config file was edited AND perf-changelog.yaml was NOT edited in the same PR" → comment that it must be updated), and .github/workflows/run-sweep.yml runs a "Validate perf-changelog matrix" step against `--changelog-file perf-changelog.yaml" as part of the sweep pipeline.
Concrete example / proof: Looking at the tail of perf-changelog.yaml, every recent config-key addition follows the same pattern — e.g. PR #2301 (kimik2.5-fp4-mi355x-vllm-disagg), PR #2249 (kimik2.5-fp4-b200-dynamo-trt), PR #2258/#2259 (the MTP variants) — each has a config-keys: block naming the new/changed key, a description: list, and a pr-link:. Doing the same grep for qwen3.5-fp4-rtx6000pro-vllm (or just rtx6000pro) against perf-changelog.yaml on this branch comes back empty, confirming the new recipe has no changelog entry — unlike every comparable precedent in the file.
Impact if merged as-is: Since the changelog is what triggers/documents which configs get benchmarked, the new qwen3.5-fp4-rtx6000pro-vllm recipe would sit in nvidia-master.yaml without ever being picked up by the changelog-driven trigger path, and the sweep-matrix validation step in CI would have nothing to validate this key against. That defeats the actual purpose of the PR (getting this recipe benchmarked and its results published), even though the recipe script, runner, and hardware wiring are otherwise correctly added.
Fix: Append a new entry to the end of perf-changelog.yaml with config-keys: [qwen3.5-fp4-rtx6000pro-vllm], a short description of the new RTX PRO 6000 recipe, and a pr-link to this PR, matching the format of the most recent entries in the file.
There was a problem hiding this comment.
The changelog is still not touched. configs/nvidia-master.yaml now defines qwen3.5-fp4-rtx6000pro-sglang (the recipe was switched from vLLM to SGLang in a later commit), but perf-changelog.yaml has no entry for qwen3.5-fp4-rtx6000pro-sglang (or the old -vllm key) anywhere in the file — a full grep for rtx6000pro against perf-changelog.yaml returns nothing. Please append a config-keys: [qwen3.5-fp4-rtx6000pro-sglang] entry at the end of perf-changelog.yaml with a description of the RTX PRO 6000 SGLang recipe and a pr-link to this PR, matching the format of the most recent entries.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30022003696 |
Replace the vLLM benchmark script with an SGLang one and rename the config key to qwen3.5-fp4-rtx6000pro-sglang. Pin lmsysorg/sglang:v0.5.16-cu130, whose auto backend resolution knows that trtllm-gen MoE is SM100-only and routes modelopt_fp4 routed experts to FlashInfer CUTLASS on SM120. 96 GiB per GPU forces two deviations from the B300 SGLang recipe: mem-fraction-static drops to 0.7 with a 2-request prefill chunk (0.8 with a 32k chunk sized the KV pool at 2.2M tokens and OOM'd on the first 8k prefill), and the six tokenizer workers are dropped because each pins a ~0.7 GiB CUDA context on GPU 0. The launcher now prefers a framework-tagged benchmark script, matching the B200/B300 launchers, so vLLM and SGLang recipes for the same model can coexist. Validated on the node through the real launcher at all four corners of the sweep with every request successful: TEP4 conc 64 640/640 at 6198 tok/s, TP4 conc 64 640/640 at 5999 tok/s, TEP4 conc 1 10/10 at 672 tok/s, TP4 conc 1 10/10 at 658 tok/s. 中文：将 Qwen3.5 RTX PRO 6000 基准配置从 vLLM 切换为 SGLang。配置键更名为 qwen3.5-fp4-rtx6000pro-sglang，镜像固定为 lmsysorg/sglang:v0.5.16-cu130 ——该版本的自动后端选择已知 trtllm-gen MoE 仅支持 SM100，会在 SM120 上将 modelopt_fp4 路由专家切换到 FlashInfer CUTLASS。受限于每卡 96 GiB 显存， 相比 B300 配方做了两处调整：mem-fraction-static 降到 0.7 并将预填充分块 缩小到两个请求（0.8 配 32k 分块会把 KV 池撑到 220 万 token，首次 8k 预填充 即 OOM），并移除六个 tokenizer worker（每个都会在 GPU 0 上占用约 0.7 GiB CUDA 上下文）。启动器改为优先选择带框架后缀的基准脚本，与 B200/B300 启动器 一致，使同一模型的 vLLM 与 SGLang 配方可以共存。已在节点上通过真实启动器 验证扫描的四个角落，全部请求成功。
Oseltamivir
changed the title
…ro-vllm # Conflicts: # perf-changelog.yaml
|
Claude finished @Oseltamivir's task in 5m 59s —— View job Review complete
Summary1 blocking issue (inline comment on Everything else checks out:
CI: PR checks pass; the full sweep run for the SGLang head is currently queued (run 30238542970) — that run will be the real validation of the new image tag and server flags on the RTX PRO 6000 node. Note: I could not verify the |
The merge with main resolved perf-changelog.yaml by taking main's copy and re-appending this PR's entry, but only main's copy was staged, so the merge commit dropped the entry. Without it the PR diff no longer touches perf-changelog.yaml and run-sweep.yml's path filter never matches. 中文：与 main 合并时对 perf-changelog.yaml 采用了「取 main 版本再追加本 PR 条目」的方式，但只暂存了 main 的版本，导致合并提交丢失了该条目。缺少该条目 后 PR 的改动不再涉及 perf-changelog.yaml，run-sweep.yml 的路径过滤器便不会 匹配。此提交恢复该条目。
|
|
||
| # Qwen3.5-397B-A17B NVFP4 single-node SGLang sweep using 4 of 8 RTX PRO | ||
| # 6000 Blackwell GPUs. Both arms use ordinary NCCL collectives on PCIe. | ||
| qwen3.5-fp4-rtx6000pro-sglang: |
There was a problem hiding this comment.
🔴 BLOCKING: Master config was modified but perf-changelog.yaml was not updated. This is the same issue flagged in the previous review — it now applies to the renamed key qwen3.5-fp4-rtx6000pro-sglang (the changelog still has no rtx6000pro entry at all).
Why it matters: Per AGENTS.md, perf-changelog.yaml is the append-only benchmark trigger log. Without an entry, the new recipe won't be picked up for benchmarking after merge, which defeats the purpose of the PR.
Fix: Append this to the end of perf-changelog.yaml (it's read chronologically, newest at the bottom):
- config-keys:
- qwen3.5-fp4-rtx6000pro-sglang
description:
- "Add Qwen3.5-397B-A17B NVFP4 single-node SGLang 8k/1k fixed-seq-len sweep on 4x RTX PRO 6000 Blackwell (TP4 and TP4/EP4 arms, concurrency 1-64) using lmsysorg/sglang:v0.5.16-cu130"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2312🔴 阻断性问题：修改了 master 配置但没有更新 perf-changelog.yaml。这与上次审阅指出的问题相同——现在适用于重命名后的配置项 qwen3.5-fp4-rtx6000pro-sglang（变更日志中仍然没有任何 rtx6000pro 条目）。根据 AGENTS.md，该文件是仅追加的基准测试触发日志；缺少条目会导致合并后新配方不会被触发进行基准测试。请将上述条目追加到 perf-changelog.yaml 文件末尾（按时间顺序读取，最新的在底部）。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30238542970 |
Drive the checkpoint's built-in MTP draft head through SGLang's EAGLE path (3 steps, top-k 1, 4 draft tokens) over the same 8k/1k grid as the non-MTP config. mem-fraction-static bounds weights plus the KV and Mamba pools, so the 4.05 GiB per-rank draft head has to be paid for by raising the fraction, not lowering it: 0.65 and 0.75 both left the pools empty, and 0.85 came up with only 4.56 GiB free per rank. 0.80 lands at 1.08M KV tokens with 9.6 GiB free. max-running-requests now tracks the tested concurrency rather than a fixed 128, since the client never opens more connections and the oversized Mamba state pool was what starved the draft head. Validated on the node through the real launcher, every request successful: TEP4 conc 64 640/640 at 6895 tok/s (+11% over the non-MTP arm, mean TTFT 27.2s to 9.6s) and TEP4 conc 1 10/10 at 1233 tok/s (+83%, mean TPOT 12.6ms to 6.5ms). 中文：为 Qwen3.5 RTX PRO 6000 SGLang 配方新增 MTP 分支，通过 SGLang 的 EAGLE 路径驱动检查点内置的 MTP 草稿头（3 步、top-k 1、4 个草稿 token），扫描网格与 非 MTP 配置相同。mem-fraction-static 同时约束权重与 KV／Mamba 池，因此每卡 4.05 GiB 的草稿头必须通过调高该比例来容纳，而非调低：0.65 与 0.75 都会让池为 空，0.85 虽能启动但每卡仅剩 4.56 GiB。0.80 可得到 108 万 KV token 且剩余 9.6 GiB。max-running-requests 改为跟随被测并发数而非固定 128，因为客户端不会 建立更多连接，过大的 Mamba 状态池正是草稿头显存不足的原因。已在节点上通过真实 启动器验证，全部请求成功。
…ro-vllm # Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30252251058 |
|
/reuse-sweep-run |
Resolve two conflicts that merge_with_reuse.sh does not auto-resolve: - configs/runners.yaml: both sides appended to the hardware-metadata map; keep both entries, with cluster:gb300-nv continuing the GB family and cluster:rtx6000pro-lat after it, before the AMD block. - perf-changelog.yaml: take main's file wholesale and re-append this PR's two entries (qwen3.5-fp4-rtx6000pro-sglang and -mtp) at the tail, per the documented recipe. No lines deleted relative to main. 中文：解决 merge_with_reuse.sh 无法自动处理的两处冲突：(1) configs/runners.yaml — 双方都向硬件元数据表追加了条目，保留两者，cluster:gb300-nv 紧随 GB 系列， cluster:rtx6000pro-lat 置于其后、AMD 区块之前；(2) perf-changelog.yaml —— 按既定 做法完整采用 main 的版本，再将本 PR 的两条条目追加到文件末尾，相对 main 无任何删除。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
/reuse-sweep-run |
2 participants
No description provided.