Repository navigation
Add Qwen3.5 NVFP4 V2 GB200 Dynamo-SGLang MTP / 新增 Qwen3.5 NVFP4 V2 GB200 Dynamo-SGLang MTP 配置 - #2594
Conversation
新增 Qwen3.5 FP4 GB200 Dynamo-SGLang MTP 配置。
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
1 similar comment
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
RohitNagraj
added
the
full-sweep-enabled
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| - "Add nine checked-in srt-slurm recipes covering TP and data-parallel-attention topologies" | ||
| - "Use chat-formatted benchmark inputs with Mooncake KV transfer" | ||
| - "Image: lmsysorg/sglang:nightly-dev-cu13-20260709-074bb928" | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX |
There was a problem hiding this comment.
🟡 The new perf-changelog.yaml entry (line 5966) uses a placeholder pr-link: .../pull/XXX instead of a real PR number. Since this is PR #2594, it should read pull/2594, matching every other entry in the file.
Extended reasoning...
The last block appended to perf-changelog.yaml in this diff ends with:
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXXXXX is a literal, unfilled placeholder rather than an actual PR number. Every other entry in the file follows the same schema and always points to its own real PR — for example the immediately preceding entries link to pull/2577, pull/2578, and pull/2580. This PR is #2594, so the correct value is pull/2594.
Why existing checks don't catch it: the PR description states that a 'changelog newline' check passed, but that only verifies the file ends with a trailing newline / is well-formed YAML — it does not validate that pr-link contains a real, resolvable PR number. There is no schema or CI validation cross-checking the pr-link field against the PR number that is actually opened, so a placeholder like XXX sails through untouched.
Proof / step-by-step:
- Open
perf-changelog.yamlat the new entry (last block in the diff, starting at line ~5956). - Read the
pr-linkfield:https://github.com/SemiAnalysisAI/InferenceX/pull/XXX. - Compare against the file's own convention: scroll up to the entries directly above it (lines 5940-5955) — they use
pull/2577,pull/2578,pull/2580, each matching the PR that introduced that entry. - Note the current PR's actual number is 2594 (per PR metadata).
- Following the file's established convention, the link should be
https://github.com/SemiAnalysisAI/InferenceX/pull/2594.
Impact: this is a documentation/metadata-only bug — the changelog is not consumed by any runtime code path, so nothing breaks functionally. However, anyone who later clicks the link while investigating this changelog entry (e.g., during a regression bisect or audit) will hit a 404 or an unrelated PR, making it harder to trace which PR introduced the entry. It should be fixed before merge since it is a one-line, unambiguous fix.
Fix: replace pull/XXX with pull/2594 in the new changelog block.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31746204670 |
kedarpotdar-nv
removed
the
full-sweep-enabled
|
Temporarily removing full sweep label |
hshrivastava-droid
added
the
full-sweep-enabled
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34389014274 |
|
/reuse-sweep-run |
|
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Additional detail section:
Signed: |
✅✅✅ Verdict: PASS ✅✅✅✅ Check 0 (CODEOWNER): PASS — |
|
Sorry, over the weekend, there was 2 major refactors to clean up the technical debt accumalated over the past 11 months of moving at the speed of light. We don't see any major refactors in the forthseeable future besides cleaning up AMD multinode AgentX pile of bash. As much, due to the refactors, u would need to ask your agent to rebase from remote main@latest. Thank you in advance for ur understanding |
|
/use 34389014274 Select the successful full sweep, attempt 1, on 中文选择成功的完整 sweep：运行 |
Applied srtctl migrate from the srt-slurm commit pinned by main; resolved prefill/decode nodes, workers, env, engine args, frontend, model and benchmark settings are unchanged.
|
Since this is disagg config, I don't think SGLang recipe update is needed. Please ping me if there is any issue. |
|
I will integrate this into sgl-project/sglang#43083 anyways |
|
@functionstackx This is 8k1k, though, not AgentX |
|
oh nvm |
|
@functionstackx can we merge this? sgl-project/sglang#43083 was merged |
functionstackx
left a comment
functionstackx
left a comment
There was a problem hiding this comment.
lgtm, @Ankur-singh feel free to merge it
|
/reuse-sweep-run 34389014274 |
# Conflicts: # inferencex-e2e/perf-changelog.yaml
…t-recipe (#3805) * refactor(matrix): select srt-slurm recipes with srt-recipe-dir and srt-recipe Master configs name each row's recipe with srt-recipe (and eval-srt-recipe) relative to a per-entry srt-recipe-dir instead of CONFIG_FILE and EVAL_CONFIG_FILE additional-settings. Validation requires them on every srt-slurm row, checks the file and selected variant exist at load time, and rejects the old settings; llmd-vllm entries keep CONFIG_FILE. The generator emits project-relative paths, counts nodes from the row's recipe, and the launcher reads SRT_RECIPE/EVAL_SRT_RECIPE, staging the same recipes/<path>:<selector> argument for srtctl. Also drops the unread SYNTHETIC_ACCEPTANCE* and SLURM_PARTITION settings. * chore(recipes): drop stale CONFIG_FILE header comments Master rows now select bundle variants with srt-recipe, so the recipe headers no longer describe how to pick one. * fix(matrix): resolve srt recipes under the loaded config's project Recipe existence, selector checks and recipe node counts read files from the project that holds the loaded configs/ (or the root a caller passes) instead of the tooling checkout's repository_root(). The generator CLI and planner pass config_root(config files), Klaud live families pass the checkout root, and Klaud's canonical matrix fetches the candidate's recipes from the candidate head. * test(operatorx): give the generator fixture a project-layout srt recipe Master configs now name an srt-recipe resolved under the project that holds configs/, so the fixture places its master in configs/ beside a recipe. * fix(configs): select the DSv4.1-Flash ATOM AgentX recipe with srt-recipe-dir srt-recipe paths, which validation rejects now that every srt-slurm entry needs srt-recipe-dir. The entry sets srt-recipe-dir and its rows select agentic.yaml; the generated rows are unchanged. * chore(recipes): drop stale CONFIG_FILE headers from the DSv4.1-Flash GB300 AgentX recipes #3598 added the GB300 Dynamo+SGLang recipes with headers that say to select a variant with CONFIG_FILE. Master rows select them with srt-recipe now, so the headers stop at the variant layout like the other AgentX bundles. * refactor(matrix): deslop srt-recipe selection Set an AgentX row's srt-recipe fields once after the topology branch, and fetch each recipe file a Klaud family references once rather than once per row. * test: deslop srt-recipe selection tests The installed-package test no longer checks recipe node counts, which take an explicit root since srt-recipe and so no longer exercise caller-repository resolution. Drop unused monkeypatch fixtures and a history comment. * docs: deslop srt-recipe selection guidance Show srt-recipe in the CONFIGS.md search-space example, shorten its selection note, name srt-slurm recipes in the revision snapshot description, and shorten the debug-runs recipe path. * fix(configs): select each GB200 DSR1 FP8 low-latency variant explicitly The low-latency row named disagg-tp8-stp-variants.yaml without a selector, so one job submitted all three zip variants under a 4-node, 1-decode-worker label. Each variant now runs only the concurrencies it is tuned for, with its selector and its 7 TP8 decode workers on 16 nodes: zip_override_lowlat[0] (c8) at 4 and 8, zip_override_lowlat[1] (c16-32) at 16. zip_override_lowlat[2] is tuned for concurrency 64, which the entry never ran, so it has no row. * fix(configs): select the Qwen3.8 and Qwen3.5 GB300 AgentX recipes with srt-recipe-dir #3816 and #3840 added qwen3.8next-fp4-gb300-sglang-agentic-mtp and the Qwen3.5 FP8 GB300 midcurve entries with a project-relative srt-recipe path and CONFIG_FILE settings, which validation rejects now that every srt-slurm entry needs srt-recipe-dir. The entries set srt-recipe-dir and their rows select the same recipes and DEP16 overrides; the generated rows are unchanged. * fix(matrix): report a malformed srt-recipe as a validation error A recipe that fails to parse raised a raw yaml.YAMLError out of validate_master_config, so Klaud's canonical_matrix crashed on a malformed candidate recipe instead of rejecting it like other bad srt-recipe references. It now raises the same ValueError, naming the recipe file. * fix(configs): select the Qwen3.5 GB200 FP4 V2 MTP recipes with srt-recipe-dir #2594 added qwen3.5-fp4-gb200-dynamo-sglang-mtp with CONFIG_FILE settings, which validation rejects now that every srt-slurm entry needs srt-recipe-dir. The entry sets srt-recipe-dir and each row selects the same recipe; the generated rows are unchanged.
… the matrix point (#3809) * refactor(srt): compose fixed-sequence recipes from fragments and bind the point Active fixed-sequence srt-slurm recipes keep only recipe-specific settings. At launch the lane's shared block (configs/srt-recipes/) is merged under the fragment, the row's variant is selected, and the binder writes the matrix point: model path, image, precision, ISL/OSL, single-node client values (MODEL, CONC, RANDOM_RANGE_RATIO, USE_CHAT_TEMPLATE) and telemetry concurrencies. A fragment that sets a bound key fails before submission. The hf:<MODEL> model path maps to the cluster's staged checkpoint and the master image to its staged container, so recipes stop naming aliases. The mi355x-amds DeepSeek-R1 MXFP4 checkpoint moves into models.entries. Six b300-fp8 trtllm recipes declared fp4 precision; it is now bound to fp8. * feat(srt): add infx generate for bound fixed-sequence recipes `infx generate --config-key <key|glob> --output-dir <dir>` writes the bound recipe of every fixed-sequence point and variant, validated by the pinned srtctl, with a manifest mapping files to matrix points. srtctl's import dependencies sit behind the new `recipes` extra. RECIPES.md documents fragments, composition and the command. * refactor(srt): set the multi-node fixed-sequence TOKENIZER in the shared block Every active multi-node fixed-sequence fragment repeated benchmark.env.TOKENIZER: /model. The shared block configs/srt-recipes/fixed-sequence-multi.yaml now sets it and the fragments drop it, along with the benchmark blocks it was the only entry of. The three mi355x fragments that lacked it (DeepSeek-R1 FP8 disagg variants, Qwen3.5 FP4 and FP8 1p1d) now tokenize with the mounted checkpoint too instead of the served model id. Every other bound recipe is unchanged. * fix(srt): bind exactly one multi-node variant in infx generate infx generate bound every variant a multi-node selector matched, while the launcher's binder refuses anything but one, so it wrote recipes for points that launch rejects, such as dsr1-fp8-gb200-dynamo-sglang. Both now share bind_multinode, and generate fails such a point as launch does. * refactor(srt): deslop the fixed-sequence binder The binder parses CONC_LIST with the launcher's parse_concurrencies, now in workload.py, so telemetry concurrencies get its canonical and unique checks. Define the bound recipe file name once, drop generate's MODEL_PREFIX and RUN_EVAL inputs that nothing on its path reads, name what _validated does, and drop a model guard the binder makes redundant. Fragment checks drop two cases that repeat a covered branch. * test(srt): drop assertions on where the bound recipe lives The content checks that follow already prove srtctl ran the bound variant. * docs: deslop fixed-sequence fragment guidance Drop restatements of the shared blocks and alias history from RECIPES.md, and correct which keys add-model-hardware says fragments omit. * build: drop test pins the recipes extra already declares CI and the testing guide install every extra with the test group. * fix(srt): fail fast when a fixed-sequence point's cluster stages no checkpoint Fragments no longer name model.path aliases, so the alias guard never fired for fixed-sequence recipes: an unstaged MODEL left model_paths empty and srtctl resolved the bound hf:<MODEL> from the Hub on the compute nodes. model_paths now raises before submission instead, unless an Override row sets hub to load the Hub snapshot. The mi355x-amds DeepSeek-R1 FP8 and Qwen3.5 FP4 SGLang disagg lanes, whose fragments named hf: paths before they were bound, set it and keep loading the Hub snapshot. * refactor(srt): compose the Qwen3.5 GB200 FP4 V2 MTP recipes from fragments #2594 added nine standalone recipes for qwen3.5-fp4-gb200-dynamo-sglang-mtp that name a model alias, the image and the precision, and run srt-slurm's sa-bench. They become fragments: the binder writes the master model, image and precision, and the shared fixed-sequence-multi.yaml block runs the InferenceX client, as every other multi-node fixed-sequence recipe has since #3415. The client keeps sa-bench's settings: the point's lengths, the workflow's 0.8 range ratio, ten prompts and two warmups per concurrency, an unthrottled request rate, the chat template, ignore-EOS and the /model tokenizer. The c12288 point's sa-bench reuse_http_connections has no counterpart; the client opens a connection per request, as it does for every other point.
8 participants
Description
Add a Qwen3.5-397B-A17B NVFP4 V2 GB200 Dynamo-SGLang disaggregated MTP configuration for the fixed-sequence 8k/1k scenario.
Validation:
中文说明
新增 Qwen3.5-397B-A17B NVFP4 V2 在 GB200 上的 Dynamo-SGLang 分离式 MTP 配置，用于固定序列长度 8k/1k 场景。
验证：
Related Issue
N/A
Type of Change
Checklist
perf-changelog.yamlwithout editing historical entries/reuse-sweep-runonly after a final green sweep with passing evalsNote
Low Risk
Benchmark and master-config YAML only; no application or serving runtime code changes.
Overview
Adds
qwen3.5-fp4-gb200-dynamo-sglang-mtptonvidia-master.yamlfor the fixed 8k ISL / 1k OSL scenario, with nine sweep points that each point at a checked-in srt-slurm recipe undergb200-fp4-v2/8k1k/mtp/.The new recipes pin Dynamo + disaggregated SGLang on GB200 for
nvidia/Qwen3.5-397B-A17B-NVFP4-V2(nightly SGLang image), Mooncake KV transfer, NEXTN/MTP on decode, and sa-bench withuse_chat_template: true. Coverage spans small 1p1d TP4 baselines (conc 1/4/8) through scaled 2p2d/2p3d/3p5d layouts and high-concurrency DEP4 points—including 7p1d decode at EP16 with CuteDSL MoE and optional multi-frontend nginx at 5120/8192, plus 7p4d at 12288 with Dynamo install/NATS and tighter decode watchdog settings.perf-changelog.yamlrecords the new config key and recipe set.Reviewed by Cursor Bugbot for commit c633122. Bugbot is set up for automated code reviews on this repo. Configure here.