Repository navigation
[NVIDIA][AgentX] Refresh GB200 DeepSeek-V4.1-Flash SGLang aggregate curve / [NVIDIA][AgentX] 更新 GB200 DeepSeek-V4.1-Flash SGLang 聚合曲线 - #4112
Conversation
Move dsv41flash-fp4-gb200-sglang-agentic-dspark to lmsysorg/sglang:nightly-dev-cu13-20261014-684fea8a (multiarch digest pinned) and replace the eight points with ten tuned points: TP4/EP1 at C1/C4/C8/C32, TP4/EP4 at C16, and aggregated TP4/EP4 with DP attention behind sglang-router at C64/C96/C128/C160/C256 using the MegaMoE all-to-all backend and HiCache host-DRAM KV offload. The recipe carries the per-point schedule policy, attention backend, prefill-decode interval, prefill chunking, admission, memory and SWA prefix-tail settings. 将 dsv41flash-fp4-gb200-sglang-agentic-dspark 更新为 lmsysorg/sglang:nightly-dev-cu13-20261014-684fea8a（固定 multiarch digest），并以十个调优点取代原本的八个点：TP4/EP1 的 C1/C4/C8/C32、 TP4/EP4 的 C16，以及在 sglang-router 之后启用 DP attention 的聚合式 TP4/EP4 的 C64/C96/C128/C160/C256（使用 MegaMoE all-to-all 后端与 HiCache 主机 DRAM KV offload）。recipe 包含各点的调度策略、attention 后端、prefill-decode interval、 prefill chunk、运行请求上限、显存与 SWA 前缀尾部设置。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the contribution!
中文感谢你的贡献！
|
shyeh25
added
the
full-sweep-fail-fast
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.
Beyond the inline nit, I checked the candidate concern that TP4/EP4 C16 never gets a GSM8K accuracy eval — per infx/evals/EVALS.md, agentic eval groups are split by model/runner/framework/precision/spec-decoding/dp-attn/image but explicitly not by TP/EP, so C16 (EP4) shares a group with the EP1 entries and the eval correctly lands on the group's highest concurrency (C32); this is documented behavior, not a bug.
Extended reasoning...
Config-only change (image bump + search-space retune) across nvidia-master.yaml, agentic.yaml, and perf-changelog.yaml for one SGLang benchmark entry; no auth/crypto/permissions surface. The one open item is the already-reported comment-style nit (restating dict values); separately verified against EVALS.md that the per-point eval-coverage concern raised by the ruled-out candidate is expected behavior given how eval groups are keyed, not a correctness bug.
| # TP4 with tensor-parallel experts (EP1) at C1-C8 and C32; the KV cache stays GPU-resident. | ||
| - { tp: 4, ep: 1, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 4, 8, 32], srt-recipe: agentic.yaml } | ||
| # TP4/EP4 at C16; the KV cache stays GPU-resident. | ||
| - { tp: 4, ep: 4, kv-offloading: none, spec-decoding: mtp, conc-list: [16], srt-recipe: agentic.yaml } | ||
| # TP4/EP4 with DP attention and MegaMoE behind sglang-router; HiCache extends the | ||
| # prefix cache into host DRAM. |
There was a problem hiding this comment.
🟡 nit (optional): the three new per-entry comments restate the dict's own tp/ep/conc-list values ("TP4 with tensor-parallel experts (EP1) at C1-C8 and C32", "TP4/EP4 at C16", "TP4/EP4 with DP attention and MegaMoE behind sglang-router") instead of giving a succinct, line-specific technical reason. sweep:^\s*# TP4 AGENTS.md's "Code comments (mandatory)" section bans restating the code and bans header/overview-style comments above a config block. Fix: keep only the non-obvious technical justification (e.g. why the KV cache stays GPU-resident, or what HiCache's host-DRAM extension buys), and drop the topology/concurrency recap that duplicates the adjacent { tp:, ep:, conc-list: } literal.
Why this was flagged
inferencex-e2e/configs/nvidia-master.yaml:8605, 8607, and 8609-8610 each add a comment above a search-space entry that repeats values already in the same line (tp, ep, conc-list), which AGENTS.md:64-67 ("No header comments", "Do not restate the code") forbids for every file including configs. The diff's own header-comment shortening elsewhere shows the author is aware of this rule, yet these three new comments reintroduce the same overview-style restatement at per-entry scope. A reviewer or coding agent using AGENTS.md as the checklist gets comments that drift out of sync with the dict they restate (as already happened: override_tp4_c16 below was not renamed to match its sibling's new override_tp4ep1_c* naming) instead of a single durable technical note.
Verification: nit: style/convention issue, nothing breaks. The three new comments at nvidia-master.yaml:8605, 8607, and 8609-8610 each restate values already present on the immediately following search-space line, which AGENTS.md:61-67 ("No header comments... above a recipe, override, or section" / "Do not restate the code.") forbids. Each comment does carry one legitimately technical clause, so the fix is to keep only those clauses and drop the topology/concurrency restatement.
…ntry Remove the per-entry comments that restated the topology and concurrency of each search-space line. The configuration is unchanged. 删除 GB200 SGLang 条目中逐行复述拓扑与并发的 search-space 注释，配置不变。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=38749229680 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=38749229680 |
|
[by Claude Code] The SGLang cookbook integration of this PR's AgentX config ( This PR has no authorized |
|
[by Claude Code] sgl-project/sglang#44654 has been merged. This PR's AgentX config ( This PR has no authorized |
|
/use 38749229680 |
|
@nvpohanh staged run 38749229680. This run remains available across future |
|
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Additional detail section:
Signed: |
✅✅✅ Verdict: PASS ✅✅✅Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (Passing sweep + evals on in-PR commit): PASS — Run 38749229680 (attempt 2, ✅ Check 2 (Eval accuracy): PASS — ✅ Check 3 (Upstream recipe): PASS — sgl-project/sglang#44654 is ✅ Check 4 (Reuse command): PASS — ✅ Check 5 (Latest checklist template): PASS — every item in the current ✅ Check 6 (Upstream image / engine ordering): PASS — ✅ Check 7 (No deprecated models/scenarios): PASS — on 2026-10-16, ✅ Check 8 (No architecture hacks): PASS — the effective recipe has no ✅ Check 9 (Spec-decode chat template): PASS — AgentX replay drives ✅ Check 10 (No engine patches): PASS — the diff only changes YAML recipe/config/changelog; it adds no patches, heredoc rewrites, or engine wheel installs. ✅ Check 11 (Agentic golden AL): PASS — ➖ Check 12 (Append-only): N/A — the new changelog entry does not set ✅ Check 13 (Draft precision): PASS — the bundled DSpark head of ✅ Check 14 (Pareto coverage): PASS — curve ✅ Check 15 (PR description): PASS — the body's image, 10-point topology (TP4/EP1 C1/4/8/32, TP4/EP4 C16, DEP4+HiCache C64–C256), router 0.3.2 and C256 ✅ Check 16 (NVIDIA overview): PASS — the NVIDIA overview Framework Recipes table lists DeepSeek-V4.1-Flash with an SGLang ✅ linking to the DeepSeek-V4.1 cookbook page. ➖ Check 17 (Dynamo metrics): N/A — Assessed commit: |
…agg-sglang-agentx-1014 Resolve the inferencex-e2e/perf-changelog.yaml conflict with infx.workflows.prepare_perf_changelog_merge resolve-conflict: main's bytes are kept unchanged and this PR's single entry is re-appended at the end, with the PR link filled in. [skip-sweep] Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nvpohanh
removed
the
full-sweep-fail-fast
3 participants
Summary
Refresh the DeepSeek-V4.1-Flash FP4 GB200 SGLang AgentX aggregate curve (
dsv41flash-fp4-gb200-sglang-agentic-dspark):lmsysorg/sglang:nightly-dev-cu13-20261014-684fea8a, pinned by multiarch digest;kv-offloading: dramwith thehicachebackend), and route them through sglang-router 0.3.2 with cache-aware, correlation-ID-affine routing.The deployment remains single-node and aggregate. No cluster-route or runner configuration changes are included. DSpark remains the checkpoint-bundled draft at its shipped precision with block size 5.
AI model disclosure
claude-opus-5-5)PR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entriesinferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs):full-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
更新 DeepSeek-V4.1-Flash FP4 GB200 SGLang AgentX 聚合曲线（
dsv41flash-fp4-gb200-sglang-agentic-dspark）：lmsysorg/sglang:nightly-dev-cu13-20261014-684fea8a，并以 multiarch digest 固定；kv-offloading: dram、后端hicache），并通过 sglang-router 0.3.2 进行缓存感知及 correlation-ID 亲和路由。部署仍为单节点聚合式部署。本 PR 不包含集群路由或 runner 配置改动。DSpark 继续使用 checkpoint 自带的 draft 及其原始精度，block size 为 5。
AI 模型披露：本 PR 使用 Claude Opus 5.5（
claude-opus-5-5）进行 benchmark 调优与分析、recipe/矩阵整合、验证与 PR 准备。🤖 Generated with Claude Code