[NVIDIA][AgentX] Move GB300 DSV4.1-Flash Dynamo + SGLang to the bonus-fixed nightlies with prefill-HiCache disagg points and TP4 c1 megakernel / [NVIDIA][AgentX] GB300 DSV4.1-Flash Dynamo + SGLang 升级到含 bonus 修复的 nightly，分离式改用 prefill HiCache，TP4 c1 启用 megakernel by weireweire · Pull Request #4595 · SemiAnalysisAI/InferenceX

weireweire · 2026-10-28T02:59:47Z

Move the GB300 DeepSeek-V4.1-Flash Dynamo + SGLang AgentX curve to nightlies that include sgl-project/sglang#46404: the disaggregated points to digest-pinned lmsysorg/sglang:nightly-dev-cu13-20261028-586355a8 and the aggregated points to nightly-dev-cu13-20261029-c10e0e17 (below). The 586355a8 image bundles Dynamo 1.7.0.dev20261027 and SGLang 0.0.0.dev1+g586355a83. #46404 draws the DSpark simulated-acceptance bonus at the simulated row instead of the real mismatch row. No SGLang engine patch is added.

The disaggregated points move to a new key, dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg-hicache, whose prefill uses a local host-DRAM HiCache tier instead of the Mooncake external linker. The existing dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg key (nightly-dev-cu13-20261025-65633129, which predates #46404) and its disagg-variants.yaml recipe are retired.

Prefill	Decode	Concurrency	Decode CUDA graph max BS (per rank)	Decode max-running-requests
5 × DEP2	1 × DEP16	2884	40	640
8 × DEP2	1 × DEP16	4096	64	1024
9 × DEP2	1 × DEP16	7480	160	2560

Removed points. The retired key's published points were measured on 65633129, before #46404. Each one is covered by this PR's points on the same Dynamo + SGLang curve (tput_per_gpu @ mean_intvty; head values are our 1200 s runs, which understate the sweep):

Retired point	Published	Covered by
c4096	654,247 @ 154.6	8 × DEP2 c4096, 692,348 @ 189.7 (dominates)
c2884	573,579 @ 194.6	5 × DEP2 c2884, 619,465 @ 220.8 (dominates)
c1228	367,046 @ 247.5	the segment from agg TP4 c16 (published 29,353 @ 561.8) to 5 × DEP2 c2884, which is about 573,000 at 247.5

c1228 is not carried over. It also sat under the published curve's own line from agg TP2/EP2 c16 (58,177 @ 401.6) to c2884 (about 442,000 at 247.5).

Why this recipe:

Prefill uses a local host-DRAM HiCache tier (hicache-ratio 1.2, write-back, direct I/O, mem-fraction-static 0.85, 64 running requests per DP rank) instead of the Mooncake external linker.
- With the linker, an L3 hit is read layer by layer at forward time with one blocking get per layer, so the forward waits on it.
- At c7480, the client's turn burst in minutes 6-8 builds deep prefill queues. Device-radix prefixes then age out, L3 reloads grow, and prefill stalls (inter-batch gaps were about 40 % of wall time) and does not recover.
- With the host tier, evicted prefixes reload from local DRAM, off the forward path. Prefill drains after the burst, and the prompt hit rate reaches 0.966 (theoretical 0.971).
Memory fractions follow the image's KV sizing. On this image the DSv4 KV pool sizes at about 890 B/token, so the same fraction holds about 1.8× more tokens than on earlier nightlies.
- At hicache-ratio 1.8, the host mirror of that larger pool exceeds node memory.
- At decode mem-fraction-static 0.85, decode runs out of memory during CUDA graph capture; 0.75 leaves the headroom capture needs.
Decode admits transferred requests every step, keeping SGLang's default disaggregation-decode-polling-interval of 1; the retired key set 8. At 8, each admission batch stalled the decode loop about 2 ms.
Dynamo 1.7 admission limits are raised to 10000 on both roles. Without that, Dynamo 1.7 rejects turns with HTTP 529 once a DP rank is full, and the warmup fails. With the raised limits, waiting turns queue in the SGLang scheduler as they do on Dynamo 1.6. The readiness wait on Dynamo 1.7 needs no patch: main's srt-slurm pin (v2.65.2) includes fix(dynamo): fall back to /health?instances=true when /health is status-only NVIDIA/srt-slurm#629.
The Mooncake store and master services are not started; without the linker nothing reads them. Mooncake still carries the prefill-to-decode KV transfer.

Aggregated row: dsv41flash-fp4-gb300-dynamo-sglang-agentic-agg moves to digest-pinned nightly-dev-cu13-20261029-c10e0e17 (Dynamo 1.7.0.dev20261028, SGLang 0.0.0.dev1+gc10e0e177).

That image bundles sgl-nvidia-mega 54b428921749 ([Fix] docker: update NVIDIA megakernel pin to tested commit sgl-project/sglang#47145), which implements megakernel API 2, so TP4 c1 runs with --enable-nvidia-megakernel again. Target and draft megakernels are active on all four TP ranks.
TP4 c1's 16 megakernel runtime-optimization variables (the DSpark fold, metadata-glue and graph-capture switches from [Perf] [NVIDIA] dspark: Move the host work between DSv4.1 decode graphs into the graphs for the megakernel path sgl-project/sglang#45450) are replaced by SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION=1 ([Feat] [NVIDIA] runtime: Add a master switch for megakernel runtime optimizations sgl-project/sglang#47105). The switch also turns on the Qwen3.5/EAGLE runtime group, most of which is inactive on this path. SGLANG_OPT_DSV41_DSPARK_SIM_DRAW_AHEAD, the simulation settings, SGLANG_NVIDIA_MEGAKERNEL_SLOTS, the Engram host table and the NCCL settings stay explicit.
The Dynamo 1.7 backend admission limits are raised as on the disaggregated rows.
The TP2/EP2 c16 point and its override are removed. It sits far below the line from TP4 c16 to the disaggregated points.
TP4 c4/c16 keep their settings.

Throughput gain at the same p90 interactivity (tok/s/GPU). Our points were measured on our GB300 cluster with the bound recipes of this PR and the AgentX client, not in this PR's sweep. The vLLM column linearly interpolates the currently published GB300 DeepSeek-V4.1-Flash Dynamo + vLLM and vLLM points on their Pareto frontier at our p90 interactivity. A blank cell means no gain.

This recipe	Run	Throughput	p90 interactivity	vs vLLM
5 × DEP2, c2884	1200 s	619,465	233.6
8 × DEP2, c4096	1200 s	692,348	192.4
9 × DEP2, BS 160, c7480	3600 s	1,104,326	107.1	+10.1 %

At p90 107.1, the vLLM frontier lies between 1,011,514 @ 75.5 and 993,521 @ 145.6, which gives 1,003,413. The 1200 s values understate a 3600 s run by a factor of about 1.17-1.20 at these concurrencies, but even after that, c2884 and c4096 stay below vLLM's c10888 point (954,146 @ 196.5).

Validation details

All runs used the image above through srt-slurm on GB300 with the AgentX client (reasoning_effort high, temperature 1) and the golden acceptance length 3.51. Each run's resolved recipe matches the recipe that infx generate binds for this PR, apart from cluster NIC names, scheduler directives and the unused Mooncake services. The 1200 s runs had 0 request errors; the 3600 s run had 3 empty responses out of about 1.05M requests.

Point	Run	Throughput @ p90	TTFT p90	Prompt hit rate (theoretical)
5 × DEP2, c2884	1200 s	619,465 @ 233.6	4.9 s	0.964 (0.968)
8 × DEP2, c4096	1200 s	692,348 @ 192.4	5.1 s	0.963 (0.967)
9 × DEP2, BS 160, c7480	3600 s	1,104,326 @ 107.1	8.7 s	0.966 (0.971)

The 9 × DEP2 point ran a decode batch of about 159 per rank with a 29.8-30.1 ms step and accept length 3.50.
10 × DEP2 at the same decode BS 160 gave 1,051,622 @ 106.5 (3600 s): the same interactivity with two more GPUs, so 9 × DEP2 is used.
10 × DEP2 at decode BS 128 gave 825,382 @ 122.4 in 1200 s, about 957-990k after the 1200 s → 3600 s factor, below the vLLM frontier at that interactivity (about 999k), so it is not included.

中文

将 GB300 DeepSeek-V4.1-Flash Dynamo + SGLang AgentX 曲线换到包含 sgl-project/sglang#46404（DSpark 模拟接受的 bonus 改在模拟位置采样）的 nightly：分离式点用固定 digest 的 lmsysorg/sglang:nightly-dev-cu13-20261028-586355a8（Dynamo 1.7.0.dev20261027、SGLang 0.0.0.dev1+g586355a83），聚合式点用 nightly-dev-cu13-20261029-c10e0e17（见下）。不添加 SGLang 引擎补丁。

分离式： 新 key dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg-hicache，DEP2 prefill 加一个 DEP16 decode。
- 5 × DEP2 跑 c2884（decode 每 rank CUDA graph max BS 40）。
- 8 × DEP2 跑 c4096（BS 64）。
- 9 × DEP2 跑 c7480（BS 160，max-running-requests 2560）。
退役旧 key： 旧的 dspark-disagg key（65633129，早于 #46404）及其 disagg-variants.yaml recipe 退役。它已发布的三个点都被本 PR 的点覆盖（按 tput_per_gpu 与 mean_intvty）：c4096 和 c2884 分别被 8 × DEP2 和 5 × DEP2 支配；c1228（367,046 @ 247.5）低于 agg TP4 c16 到 5 × DEP2 c2884 的连线（在 247.5 处约 573,000），因此不保留。
Prefill 缓存： 用本地主机内存 HiCache 层（hicache-ratio 1.2、write-back、direct I/O、mem-fraction-static 0.85、每个 DP rank 64 个运行请求）替代 Mooncake external linker。
- linker 在前向时逐层阻塞读取 L3，c7480 的突发请求之后 prefill 会停住，且无法恢复。
- 改用主机内存层后，被逐出的前缀从本地内存读回，不在前向路径上。突发后 prefill 能排空，命中率 0.966（理论值 0.971）。
显存比例： 该镜像的 KV 每 token 约 890 B，相同比例下 KV pool 约为之前的 1.8 倍。
- hicache-ratio 1.8 时主机镜像超出节点内存，改为 1.2。
- decode mem-fraction-static 0.85 时，CUDA graph capture 阶段 OOM，改为 0.75。
Decode 准入： decode 每步接收已传输的请求（默认 polling interval 1）；旧 key 设为 8 时，每批准入让 decode 停顿约 2 ms。
不再启动 Mooncake store 和 master 服务（没有 linker 就没有读取方）；prefill 到 decode 的 KV 传输仍走 Mooncake。
Dynamo 1.7 admission 上限： 两侧设为 10000，否则 DP rank 满时 Dynamo 1.7 返回 HTTP 529、warmup 失败；放开后等待的请求在 SGLang 调度器中排队。main 的 srt-slurm 版本（v2.65.2）已包含 fix(dynamo): fall back to /health?instances=true when /health is status-only NVIDIA/srt-slurm#629，Dynamo 1.7 的 readiness 等待不再需要补丁。
聚合式： 聚合式 key 换到固定 digest 的 nightly-dev-cu13-20261029-c10e0e17（Dynamo 1.7.0.dev20261028）。该镜像自带 sgl-nvidia-mega 54b428921749（megakernel API 2，[Fix] docker: update NVIDIA megakernel pin to tested commit sgl-project/sglang#47145），TP4 c1 重新启用 --enable-nvidia-megakernel，4 个 TP rank 上 target 和 draft megakernel 均生效。TP4 c1 原先逐个设置的 16 个 megakernel 运行时优化变量改为 SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION=1（[Feat] [NVIDIA] runtime: Add a master switch for megakernel runtime optimizations sgl-project/sglang#47105）；该开关同时打开 Qwen3.5/EAGLE 那组优化，在此路径上大多不生效。SIM_DRAW_AHEAD、模拟设置、megakernel slots、Engram 和 NCCL 设置仍显式指定。放开 admission 上限；删除 TP2/EP2 c16，TP4 c4/c16 配置不变。

同交互性（p90 interactivity）下相对 vLLM 的吞吐提升： 数据在我们的 GB300 集群上用本 PR 的配方测得，不是本 PR 的 sweep；在这些并发下，3600 s 结果约为 1200 s 的 1.17-1.20 倍。vLLM 用已发布的 Dynamo + vLLM 与 vLLM 点的 Pareto 前沿，在相同交互性处线性插值。

点	时长	吞吐 @ p90	相对 vLLM
9 × DEP2、BS 160、c7480	3600 s	1,104,326 @ 107.1	+10.1%（vLLM 插值 1,003,413）
5 × DEP2、c2884	1200 s	619,465 @ 233.6	无提升
8 × DEP2、c4096	1200 s	692,348 @ 192.4	无提升

c2884 和 c4096 即使换算到 3600 s，也低于 vLLM 的 c10888 点（954,146 @ 196.5）。

AI model disclosure

Claude Opus 5.5 (runtime identifier aws/anthropic/eccn-claude-opus-5-5) investigated the bottleneck, ran the cluster experiments through delegated agents on the same model, wrote the recipe, master config, changelog entry and this description, and ran an independent review of the diff through a separate delegated agent on the same model.

…HiCache Root cause: at c7480 the Mooncake external linker reads L3 hits layer by layer on the forward path; after the AgentX turn burst, prefill stalls on those reads (about 40 % of wall time), device prefixes age out, and the deployment does not recover. Fix: a new config key on nightly-dev-cu13-20261027-5884e4fe (Dynamo 1.7.0.dev20261026) with 9 and 10 DEP2 prefill workers and one DEP16 decode. Prefill keeps evicted prefixes in a local host-DRAM HiCache tier (ratio 1.8, write-back, direct I/O); decode keeps SGLang's default polling interval of 1. Dynamo 1.7 backend admission limits are raised so turns queue in the scheduler, and NVIDIA/srt-slurm#629 is backported so the readiness wait asks /health?instances=true when /health is status-only. The agg and dspark-disagg keys are listed so they rerun in the same sweep. Validation: on GB300 with the same recipe and AgentX client, 10 x DEP2 batch 128 gave 924,652 tok/s/GPU at p90 117.2 tok/s/user over 3600 s.

…6-hicache # Conflicts: # inferencex-e2e/perf-changelog.yaml

claude · 2026-10-28T03:05:58Z

+      - spec-decoding: mtp
+        conc-list: [7480]
+        srt-recipe: disagg-dep16-hicache-variants.yaml:override_10p1d_dep2_dep16_b160_c7480
+        kv-offloading: dram
+        kv-offload-backend: { name: hicache }
+        prefill:
+          num-worker: 10
+          tp: 2
+          ep: 2
+          dp-attn: true
+        decode:
+          num-worker: 1
+          tp: 16
+          ep: 16
+          dp-attn: true


🔴 If either 10-prefill c7480 job of a sweep is rerun, the other 10-prefill job's result is dropped. Klaud's coverage check then fails the sweep. Rows override_10p1d_dep2_dep16_c7480 and override_10p1d_dep2_dep16_b160_c7480 differ only by recipe. They share every field of agentic_key (infx/results/artifacts.py:123), so superseded_attempt_files treats one as a retry of the other. Fix: give each row its own result identity, e.g. key superseded_attempt_files on recipe_fingerprint plus conc, the identity check_coverage already uses.

Why this was flagged

The two rows at nvidia-master.yaml:9752-9781 have the same prefill (10 x tp2 ep2 dpa), decode (1 x tp16 ep16 dpa), conc 7480, spec mtp and kv-offload-backend {name: hicache}. agentic_key (infx/results/artifacts.py:123-168) is built only from those fields, so both point results get the same key. superseded_attempt_files (artifacts.py:197-215) drops any file whose rows all have an older run_attempt than the latest attempt for that key. Klaud recovery POSTs rerun-failed-jobs (infx/klaud/lifecycle.py:705) after a failure or preemption, and only the failed job reruns. Say the b160 job reruns as attempt 2 while the b128 point finished in attempt 1. agentic_point_files (artifacts.py:188-194) then discards the b128 file. check_coverage (infx/klaud/validation.py:307-320) expects (recipe_fingerprint, 7480, image) for both rows and raises "Full-sweep result coverage or consistency failed". RESULT_FILENAME adds the recipe fingerprint, which keeps artifact names distinct, but the superseding logic never reads that fingerprint.

Verification: The rows override_10p1d_dep2_dep16_c7480 and override_10p1d_dep2_dep16_b160_c7480 in configs/nvidia-master.yaml differ only in srt-recipe. agentic_key (infx/results/artifacts.py:123-168) leaves out recipe_fingerprint. On the base branch these rows do not exist, so the failure is new.

github-actions · 2026-10-28T03:53:53Z

Thanks for the contribution!

Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.

中文

感谢你的贡献！

**审阅：**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER，请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，再通过 Slack 联系核心维护者。必须严格遵循模板，并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have，才能触发签核验证。
**PR 验证：**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目，并且恰好带有一个主标签时，才会运行扫描：full-sweep-fail-fast（强烈推荐；canary 加逐矩阵 fail-fast）、full-sweep-enabled（有 canary；矩阵任务在失败后继续运行）或 non-canary-full-sweep-enabled（无 canary，也无 fail-fast）。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
**合并后：**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复；参见重新运行失败任务的说明。

…6-hicache # Conflicts: # inferencex-e2e/perf-changelog.yaml

…ightly and drop TP2/EP2 c16 nightly-dev-cu13-20261028-586355a8 includes sgl-project/sglang#46404. Raise the Dynamo 1.7 backend admission limits as on the disagg rows. TP2/EP2 c16 sits far below the line from TP4 c16 to the disaggregated points, so it is removed with its override.

nvpohanh · 2026-10-28T08:42:55Z

[by Claude Code] The SGLang cookbook integration of this PR's AgentX configs (dsv41flash-fp4-gb300-dynamo-sglang-agentic-agg, dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg-hicache) will happen in sgl-project/sglang#47027, in the Agentic Long-Context Deployment section of the DeepSeek-V4.1 cookbook page.

This PR has no authorized /use <run_id> comment yet, so the cookbook shows the recipe at this PR's head (e480515ce) as "in progress". A later cookbook PR switches it to the measured run once a run is pinned or this PR merges.

nvpohanh · 2026-10-28T09:18:41Z

[by Claude Code] sgl-project/sglang#47027 has been merged. This PR's AgentX configs (dsv41flash-fp4-gb300-dynamo-sglang-agentic-agg, dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg-hicache) are now in the Agentic Long-Context Deployment section of the SGLang DeepSeek-V4.1 cookbook: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4_1#agentic-long-context-deployment

This PR has no authorized /use <run_id> comment yet, so the cookbook shows the recipe at this PR's head (e480515ce) as "in progress". A later cookbook PR switches it to the measured run once a run is pinned or this PR merges.

…6-hicache

… 20261028 nightly and retire the Mooncake-linker disagg key The hicache key now runs on nightly-dev-cu13-20261028-586355a8, which includes sgl-project/sglang#46404. On that image the KV pool sizes at 890 B/token, so hicache-ratio drops to 1.2 (1.8 exceeds host memory) and decode mem-fraction-static to 0.75 (0.85 runs out of memory at graph capture). Points: 5P c2884 (bs40), 8P c4096 (bs64) and 10P c7480 (bs160). The 9P and 10P bs128 c7480 points are dropped; bs160 is higher at the same interactivity. The dspark-disagg key on 65633129 and its recipe are retired.

At decode batch 160, 9 x DEP2 reaches the same p90 interactivity as 10 x DEP2 with two fewer GPUs, so its throughput per GPU is higher.

…recipe in the changelog

…6-hicache # Conflicts: # inferencex-e2e/perf-changelog.yaml

…ightly and use the megakernel runtime-optimization switch at TP4 c1 nightly-dev-cu13-20261029-c10e0e17 bundles sgl-nvidia-mega 54b428921749 (megakernel API 2, sgl-project/sglang#47145), so TP4 c1 starts with --enable-nvidia-megakernel again. SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION (sgl-project/sglang#47105) replaces the 16 runtime-optimization variables the recipe set one by one.

nvpohanh · 2026-10-29T07:15:20Z

[by Claude Code] The SGLang cookbook update of this PR's AgentX configs (dsv41flash-fp4-gb300-dynamo-sglang-agentic-agg, dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg-hicache) will happen in sgl-project/sglang#47214, in the Agentic Long-Context Deployment section of the DeepSeek-V4.1 cookbook page. The configs have been live in that section since sgl-project/sglang#47027; sgl-project/sglang#47214 refreshes them with the pin below.

This PR has no authorized /use <run_id> comment yet, so the cookbook shows the recipe at this PR's head (1b0252fcc) as "in progress". A later cookbook PR switches it to the measured run once a run is pinned or this PR merges.

nvpohanh · 2026-10-29T07:56:23Z

[by Claude Code] sgl-project/sglang#47214 has been merged. This PR's AgentX configs (dsv41flash-fp4-gb300-dynamo-sglang-agentic-agg, dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg-hicache) are now in the Agentic Long-Context Deployment section of the SGLang DeepSeek-V4.1 cookbook: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4_1#agentic-long-context-deployment

This PR has no authorized /use <run_id> comment yet, so the cookbook shows the recipe at this PR's head (1b0252fcc) as "in progress". A later cookbook PR switches it to the measured run once a run is pinned or this PR merges.

…6-hicache # Conflicts: # inferencex-e2e/perf-changelog.yaml

…2.65.2 Main bumped inferencex-e2e/utils/srt-slurm to v2.65.2 (#4601), which includes NVIDIA/srt-slurm#629, so runners/srt-slurm/patches/629-dynamo-health-instances.patch and its README row are removed and the changelog entry no longer mentions it.

github-actions · 2026-10-30T02:04:20Z

View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=40989481011

View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=40989481011

nvpohanh · 2026-10-30T11:12:37Z

[by Claude Code] The SGLang cookbook update of this PR's AgentX configs (dsv41flash-fp4-gb300-dynamo-sglang-agentic-agg, dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg-hicache) will happen in sgl-project/sglang#47464, in the Agentic Long-Context Deployment section of the DeepSeek-V4.1 cookbook page. The configs have been live in that section since sgl-project/sglang#47027; sgl-project/sglang#47464 refreshes them with the pin below.

This PR has no authorized /use <run_id> comment yet, so the cookbook shows the recipe at this PR's head (4145e263b) as "in progress". A later cookbook PR switches it to the measured run once a run is pinned or this PR merges.

nvpohanh · 2026-10-30T11:15:47Z

/use 40989481011

github-actions · 2026-10-30T11:16:02Z

@nvpohanh staged run 40989481011.

DeepSeek-V4.1-Flash · agentic-traces · FP4 · gb300_dynamo-sglang · 2026-10-30

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

nvpohanh · 2026-10-30T11:28:54Z

[by Claude Code] sgl-project/sglang#47464 has been merged. This PR's AgentX configs (dsv41flash-fp4-gb300-dynamo-sglang-agentic-agg, dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg-hicache) are now in the Agentic Long-Context Deployment section of the SGLang DeepSeek-V4.1 cookbook: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4_1#agentic-long-context-deployment

This PR's /use 40989481011 comment came after this cookbook round was synced, so the cookbook shows the recipe at 4145e263b as "in progress". That is the commit run 40989481011 measured, so the commands are already the measured ones; a later cookbook PR marks them verified.

nvpohanh · 2026-10-30T11:33:35Z

As a PR reviewer and CODEOWNER, I have reviewed this and have:

Additional detail section:

Assessed head: 4145e263b55bbb071885a6c17bb40c5463f6f70e (current PR head). The pinned run is Run Sweep 40989481011 (attempt 1, pull_request, head 4145e263, conclusion success), named by /use 40989481011. Its results_bmk, eval_results_all and bmk_agentic_* artifacts are unexpired.
Code quality: The diff is config-only.
- It updates agg-variants.yaml and renames disagg-variants.yaml to disagg-dep16-hicache-variants.yaml.
- In nvidia-master.yaml, it updates the agg entry and replaces the -dspark-disagg key with -dspark-disagg-hicache.
- It appends one perf-changelog.yaml entry at the end of the file and edits no existing entry.
- Config generation has no errors at base or head, and the master worker counts match the recipe overrides: 5/8/9 DEP2 prefill and 1 DEP16 decode give 26/32/34 GPUs in the results.
PR validation: In run 40989481011, all 6 benchmark jobs succeeded. These are the TP4 c1 canary, multi-node agentic / TP4 c4/c16, and disagg 5P c2884, 8P c4096 and 9P c7480. All 6 multi-node agentic eval / jobs also succeeded, as did check-changelog, collect-results and collect-evals. The skipped jobs are scenario lanes this PR does not generate. No commits follow 4145e263.

Evals: gsm8k em_strict, n=1319. The threshold is the default 0.90 in infx/evals/thresholds.yaml, which has no dsv41flash override.

Config	Image	em_strict
agg TP4 c1	`20261029-c10e0e17`	0.9727
agg TP4 c4	`20261029-c10e0e17`	0.9727
agg TP4 c16	`20261029-c10e0e17`	0.9765
disagg 5P c2884	`20261028-586355a8`	0.9773
disagg 8P c4096	`20261028-586355a8`	0.9719
disagg 9P c7480	`20261028-586355a8`	0.9735

Every row is ok against its best sibling and against the exact-topology cohort (run 40153045152). All rows report infrastructure_success.
The evals ran the changed paths. The TP4 c1 eval server log (*conc1*_lm-eval_ server-log artifact) shows enable_nvidia_megakernel: True, megakernel: active for both DeepSeek-V4.1 and deepseek_v41_dspark_draft_nvidia, and megakernel: draft proposal engaged: batch 1, 5 LM-head rows.
The c2884 eval prefill log shows enable_hierarchical_cache: True, hicache_ratio: 1.2, hicache_io_backend: direct, hicache_attached=True and enable_unified_cache_external_linker: False. The decode log shows enable_fp32_lm_head: True, enable_eplb: True and ep_num_redundant_experts: 16.

Chat template: The AgentX client drives the Dynamo chat frontend. Both recipes set AIPERF_EXTRA_INPUTS={"chat_template_kwargs":{"reasoning_effort":"high","thinking":true},"temperature":1}.
Draft precision:
- The draft is the DSpark head embedded in deepseek-ai/DeepSeek-V4.1-Flash. Eval server logs resolve speculative_draft_model_path: deepseek-ai/DeepSeek-V4.1-Flash, so no checkpoint is substituted.
- Shipped precision: attention, shared experts and main_proj are FP8 E4M3 with E8M0 scales (block 32). Routed experts are FP4 with E8M0 scales. Norms, gate, markov and confidence heads are BF16, and hc_*/attn_sink are F32.
- The pinned image's default handling loads the draft wo_a from FP8 to BF16. deepseek_v4_dspark.py sets wo_a_fp8=False at both pinned SGLang commits, c10e0e177 and 586355a83. This is the baseline and does not lower precision.
- Effective precision is the same as shipped. Agg c1, disagg prefill and disagg decode all show speculative_draft_model_quantization: None, speculative_draft_kv_cache_dtype: None (inherits the target) and quantization: None. KV dtype fp8_e4m3 is the image default for this checkpoint (auto resolves to fp8_e4m3) and applies to both target and draft. The c1 megakernel draft_* slots reuse the weights SGLang loaded.
- None of the new settings lowers draft precision:
  - SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION=1 only defaults the 16 removed DSpark fold, metadata-glue and graph switches, plus inactive Qwen3.5/EAGLE/GDN graph toggles (environ.py at c10e0e177).
  - SGLANG_OPT_DEEPGEMM_MEGA_GATE runs the BF16 gate in a fused BF16 kernel.
  - SGLANG_DSV4_KV_LAYOUT=v41_new is a byte layout.
  - ENGRAM_PEER_SHARDS changes where Engram rows are read from.
  - enable-fp32-lm-head raises precision.
- SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not enabled. It is absent from both recipes, the generated configs, all 12 job logs and the eval server logs. It is EnvBool(False) in environ.py at c10e0e177 and 586355a83.
Golden AL: Every worker Env: dump in all 6 benchmark jobs shows SGLANG_SIMULATE_ACC_LEN=3.51, SGLANG_SIMULATE_ACC_METHOD=match-expected and SGLANG_SIMULATE_ACC_TOKEN_MODE=real-draft-token. That value matches dsv41flash_dspark.yaml deepseek-v4.1-flash.thinking_on.5 = 3.51: the recipes use speculative-dspark-block-size: 5 and send thinking: true. The eval-only jobs run real acceptance.
MODELS.md: dsv41flash lists Agentic coding (SGLang DSpark) as active, with no deprecation.
Benchmark hacks and integer all-reduce:
- There is no --hf-overrides, --json-model-override-args, model-config edit or indexer skipping.
- EPLB with 16 redundant experts on decode only re-places experts and does not change FLOPs.
- SGLANG_OPT_DSV41_BF16_INDEX_LOGITS, now also set on prefill, only changes the storage dtype of the target indexer's top-k scores. The evals above cover it.
- This is an NVIDIA CUDA path, and no *QUICK_REDUCE_QUANTIZATION or other integer all-reduce control is set. dsv41flash is also an existing model.
Dynamo TRT-LLM metrics: N/A. These are dynamo-sglang configs, not Dynamo TRT-LLM.
Upstream images and main-branch provenance:
- agg: lmsysorg/sglang:nightly-dev-cu13-20261029-c10e0e17@sha256:78142f9f…, with Dynamo 1.7.0.dev20261028 bundled (dynamo.install: false).
- disagg: lmsysorg/sglang:nightly-dev-cu13-20261028-586355a8@sha256:b4f09a2c…, with Dynamo 1.7.0.dev20261027 bundled.
- Both are upstream lmsysorg nightlies. Their SGLang commits c10e0e177 and 586355a83 are ancestors of sgl-project/sglang main, and both digests resolve now.
Engine-first ordering: The ordering rule does not apply. dynamo-sglang is an open-source engine entry, and this PR adds no vendor-specific framework entry.
Upstream recipe:
- Merged cookbook PR sgl-project/sglang#47464 (MERGED 2026-10-30T11:23:25Z) publishes both lanes at this PR's head 4145e263 in the SGLang DeepSeek-V4.1 cookbook, Agentic Long-Context Deployment. The lanes are dsv41flash-fp4-gb300-dynamo-sglang-agentic-agg and -dspark-disagg-hicache; the PR also removes the retired -dspark-disagg lane. Earlier revisions were in #47027 and #47214, both merged.
- The deepseek-v4_1.jsx data on sglang main covers both images and the etcd/NATS and Dynamo frontend commands.
- Agg: the TP4 c1 megakernel launch, with SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION=1, SGLANG_NVIDIA_MEGAKERNEL_SLOTS, the Dynamo 1.7 DYN_BACKEND_ADMISSION_* limits and the TP4 c4/c16 settings.
- Disagg prefill (DEP2): host-DRAM HiCache (--enable-hierarchical-cache, --hicache-ratio 1.2, --hicache-io-backend direct, --hicache-write-policy write_back), --enable-encoder-swa-bounded-replay, --max-prefill-tokens 32768, SGLANG_DSV4_KV_LAYOUT, BF16 index logits and SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=65536.
- Disagg decode (DEP16): --enable-fp32-lm-head, EPLB (--enable-eplb, --ep-num-redundant-experts 16, --eplb-rebalance-num-iterations 15000, --expert-distribution-recorder-buffer-size 1000), --mem-fraction-static 0.75, SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS, SGLANG_OPT_DEEPGEMM_MEGA_GATE, SGLANG_OPT_DSV41_MEGA_MHC_PREFILL, DYN_SGL_GATEWAY_WORKERS=16, and the per-point max-running-requests, CUDA-graph BS and extra slots, including c7480 (2560, 160).
- @fabric.rdma-devices and {node} appear in the cookbook as $IB_DEVICES and $NODE_IP.
- Excluded InferenceX-only settings have no production serving effect. They are harness, collection or synthetic benchmark controls:
  - logging (PYTHONUNBUFFERED, PYTHONNOUSERSITE)
  - cluster plumbing (HF_HUB_CACHE, Slurm/srun/sbatch options, placement, health checks, the tachometer and otel-collector services)
  - the AIPERF_* client env
  - the SGLANG_SIMULATE_ACC_* synthetic-acceptance controls
No engine patches: At 4145e263 there is no setup script, .patch, overlay, sitecustomize or wheel install (dynamo.install: false in both recipes). The megakernel plugin ships in the pinned image, and no waiver is needed.
Append-only: N/A. The new changelog entry does not use append-only: true.
Removed points (Check 18g):
- The PR removes agg TP2/EP2 c16 and retires the -dspark-disagg key (c1228/c2884/c4096). Their last measurements come from the bmk_agentic_* artifacts of run 40153045152, attempt 1.
- I ran signoff_facts dominance on the combined Dynamo + SGLang curve (tput_per_gpu against 1/mean TPOT). Every removed point is covered, and undominated_removed is empty:
  - Old c2884 (573,579 @ 195.7) and old c4096 (654,247 @ 156.0) are dominated by the new 5P c2884 (714,534 @ 218.8).
  - Old c1228 (367,046 @ 248.1) and TP2/EP2 c16 (58,177 @ 404.9) lie under the segment from the new agg TP4 c16 (29,428 @ 546.4) to the new 5P c2884.
- The per-key facts list TP2/EP2 c16 as undominated only because they compare it with agg points alone.

Pareto coverage:

There is one affected curve: dsv41flash / agentic-coding / gb300_dynamo-sglang / fp4 / run 40989481011. Agg and disagg points form one series.
Metrics are per_gpu.total_tput_tps against E2EL, from results_bmk. Counts come from infx.workflows.pareto_coverage.
P90 E2EL: 5/6 frontier points. TP4 c4 is dominated by TP4 c16.

P75 E2EL: 5/6 frontier points. TP4 c1 is dominated by TP4 c4.

Point	tput/GPU	P90 E2EL (s)	P75 E2EL (s)
agg TP4 c1	8,255	5.12	2.70
agg TP4 c4	10,530	5.99	2.56
agg TP4 c16	29,428	5.93	2.79
disagg 5P c2884	714,534	12.83	6.54
disagg 8P c4096	803,198	14.57	7.31
disagg 9P c7480	1,094,347	26.91	14.68

Signed: nvpohanh

github-actions · 2026-10-30T11:36:47Z

✅✅✅ Verdict: PASS ✅✅✅

Passed and not applicable checks

✅ Check 0 (CODEOWNER): PASS — @nvpohanh is a named owner of inferencex-e2e/configs/nvidia-master.yaml and of the srt-slurm-recipes/**/gb300-*/ recipes; perf-changelog.yaml falls only under the catch-all.

✅ Check 1 (Sweep on in-PR commit): PASS — Pinned head 4145e263 is the PR tip and carries green multi-node agentic /, multi-node canary / (TP4 c1) and 6/6 multi-node agentic eval / check-runs from run 40989481011 (attempt 1, success); no commits follow it.

✅ Check 2 (Evals): PASS — All 6 gsm8k rows in eval_results_all (run 40989481011, head 4145e263) ran on the PR images, with em_strict 0.9719–0.9773 at n=1319 and infrastructure_success; every row is ok against its best sibling and the exact-topology cohort in run 40153045152, with the largest gap 0.0053 against a 0.0123 threshold. The eval launch commands carry the changed flags (--enable-hierarchical-cache, --hicache-ratio 1.2, --enable-fp32-lm-head, --enable-eplb, SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION=1). The c1 megakernel runtime lines come from the sign-off's server-log citation; the verifier sandbox could not decompress that tarball.

✅ Check 3 (Upstream recipe): PASS — sgl-project/sglang#47464 is MERGED (2026-10-30T11:23:25Z). Its deepseek-v4_1.jsx on main, rendered in the DeepSeek-V4.1 cookbook, matches both images and every production-useful agg TP4 c1/c4/c16, prefill DEP2 (HiCache), decode DEP16 (fp32 LM head, EPLB, per-point batch sizes including c7480 2560/160) and Dynamo frontend, etcd and NATS setting. Only metrics flags and a default-valued flag differ.

✅ Check 4 (Reuse command): PASS — nvpohanh (COLLABORATOR) posted /use 40989481011 on its own line, naming the Check 1 run.

✅ Check 5 (Checklist template): PASS — All 20 current template items, plus the upstream-PR sub-item, are present and checked.

✅ Check 6 (Upstream images): PASS — Both images are upstream lmsysorg/sglang nightlies pinned by digest. Their SGLang commits c10e0e177 and 586355a83 are ancestors of sgl-project/sglang main (ahead_by 0), and the PR adds only dynamo-sglang entries, so the engine-first rule does not apply.

✅ Check 7 (Deprecated models): PASS — On 2026-10-30, MODELS.md lists dsv41flash Agentic coding (SGLang DSpark) as active, with no deprecation.

✅ Check 8 (Benchmark hacks): PASS — There is no --hf-overrides, model-config edit or indexer skipping. EPLB only re-places experts, the BF16 index-logits change is covered by the passing evals, the path is NVIDIA CUDA with no *QUICK_REDUCE_QUANTIZATION control, and dsv41flash is grandfathered.

✅ Check 9 (Chat template): PASS — Both recipes drive the Dynamo chat frontend with AIPERF_EXTRA_INPUTS chat_template_kwargs (thinking: true, reasoning_effort: high).

✅ Check 10 (Engine patches): PASS — At 4145e263 the affected recipes have no setup script, .patch, overlay or wheel install, and dynamo.install: false.

✅ Check 11 (Golden AL): PASS — Job logs for the c1 canary and c7480 show SGLANG_SIMULATE_ACC_LEN=3.51 with match-expected and real-draft-token, matching dsv41flash_dspark.yaml thinking_on block 5 = 3.51 (signoff_facts golden-al matches: true).

➖ Check 12 (Append-only): N/A — The new changelog entry does not set append-only: true.

✅ Check 13 (Draft precision): PASS — The draft is the embedded DSpark head of deepseek-ai/DeepSeek-V4.1-Flash. No draft quantization, dtype or checkpoint override is set, and FP8 KV applies to both target and draft. SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is absent from the recipes and is EnvBool(False) in environ.py at c10e0e177. SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION only defaults scheduling and graph switches.

✅ Check 14 (Pareto coverage): PASS — There is one curve, dsv41flash / agentic-coding / gb300_dynamo-sglang / fp4 / run 40989481011 (results_bmk, tput_per_gpu against E2EL). The pareto_coverage helper gives 5/6 frontier points at P90 (TP4 c4 dominated) and 5/6 at P75 (TP4 c1 dominated).

✅ Check 15 (PR description): PASS — Images, Dynamo versions, worker counts, batch sizes, removed points and retired-key measurements match the pinned code and run 40153045152. The 1200 s and 3600 s cluster figures are labeled as out-of-sweep.

✅ Check 16 (NVIDIA overview): PASS — The NVIDIA overview Framework Recipes table has a DeepSeek-V4.1-Flash row with a populated SGLang cell linking to the cookbook.

➖ Check 17 (Metrics publication): N/A — These are dynamo-sglang configs; the Dynamo TRT-LLM metrics controls do not apply.

✅ Check 18 (Recipe/image/sweep consistency): PASS — There are no generation errors, master worker, TP, EP and DPA values match the recipe overrides (26/32/34 GPUs), and both digests resolve. The changelog matches the field inventory. Using run 40153045152 measurements, old c2884 (573,579 @ 195.5) and old c4096 (654,247 @ 156.1) are dominated by 5P c2884 (714,534 @ 218.3), and old c1228 (367,046 @ 248.0) and TP2/EP2 c16 (58,177 @ 401.6) lie under the TP4 c16 → 5P c2884 segment. Re-tuned deltas for agg TP4 c1/c4/c16: tput −0.65% / +0.66% / +0.26%, interactivity −1.9% / −3.66% / −3.78%.

Assessed commit: 4145e263b55bbb071885a6c17bb40c5463f6f70e.

cquil11 · 2026-10-30T14:28:41Z

Hi Po-Han
For 4595, I am wondering if you can please open up a follow up append-only PR to add back the TEP2 and 3p1d points so that we don't have such a large interactivity/e2e gap. Or you can tune other points to fill the gap but these should suffice. Once the PR is opened and referenced we can merge this one.

cquil11 · 2026-10-30T15:24:02Z

follow up #4679

# Conflicts: # inferencex-e2e/perf-changelog.yaml

cquil11 · 2026-10-30T15:26:09Z

/use 40989481011

github-actions · 2026-10-30T15:26:26Z

@cquil11 staged run 40989481011.

DeepSeek-V4.1-Flash · agentic-traces · FP4 · gb300_dynamo-sglang · 2026-10-30

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

github-actions · 2026-10-30T15:44:16Z

Ingest report — ✅ Results ingested into the production database.

Ingest workflow · Merge Ingest run · Benchmark source run

Category	new	duplicate	skipped
benchmark	6	0	6
eval	6	6	0
trace	6	0	0
power	0	0	6

_{benchmarkFiles 7, rawBenchmarkRows 12, coveredByPerJobFiles 6, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 7914, gpuMetricSamplesNew 0, powerPublicationPoints 6, changelogEntriesWritten 1}

Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun.

中文

Ingest 报告 — ✅ 结果已写入 production 数据库。

上表仅统计本次运行写入的数据，各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。

…6 and 3P1D c1228 points (append-only, follows #4595) / [NVIDIA][AgentX] 追加 GB300 DSV4.1-Flash Dynamo + SGLang TP2/EP2 c16 和 3P1D c1228 点（仅追加，接续 #4595） (#4679) * Add GB300 DSV4.1-Flash SGLang DEP16 disagg c7480 points with prefill HiCache Root cause: at c7480 the Mooncake external linker reads L3 hits layer by layer on the forward path; after the AgentX turn burst, prefill stalls on those reads (about 40 % of wall time), device prefixes age out, and the deployment does not recover. Fix: a new config key on nightly-dev-cu13-20261027-5884e4fe (Dynamo 1.7.0.dev20261026) with 9 and 10 DEP2 prefill workers and one DEP16 decode. Prefill keeps evicted prefixes in a local host-DRAM HiCache tier (ratio 1.8, write-back, direct I/O); decode keeps SGLang's default polling interval of 1. Dynamo 1.7 backend admission limits are raised so turns queue in the scheduler, and NVIDIA/srt-slurm#629 is backported so the readiness wait asks /health?instances=true when /health is status-only. The agg and dspark-disagg keys are listed so they rerun in the same sweep. Validation: on GB300 with the same recipe and AgentX client, 10 x DEP2 batch 128 gave 924,652 tok/s/GPU at p90 117.2 tok/s/user over 3600 s. * Set perf-changelog pr-link to #4595 * Restore historical perf-changelog entries; set only this entry's pr-link * Move the GB300 DSV4.1-Flash Dynamo + SGLang agg row to the 20261028 nightly and drop TP2/EP2 c16 nightly-dev-cu13-20261028-586355a8 includes sgl-project/sglang#46404. Raise the Dynamo 1.7 backend admission limits as on the disagg rows. TP2/EP2 c16 sits far below the line from TP4 c16 to the disaggregated points, so it is removed with its override. * Move the GB300 DSV4.1-Flash Dynamo + SGLang HiCache disagg key to the 20261028 nightly and retire the Mooncake-linker disagg key The hicache key now runs on nightly-dev-cu13-20261028-586355a8, which includes sgl-project/sglang#46404. On that image the KV pool sizes at 890 B/token, so hicache-ratio drops to 1.2 (1.8 exceeds host memory) and decode mem-fraction-static to 0.75 (0.85 runs out of memory at graph capture). Points: 5P c2884 (bs40), 8P c4096 (bs64) and 10P c7480 (bs160). The 9P and 10P bs128 c7480 points are dropped; bs160 is higher at the same interactivity. The dspark-disagg key on 65633129 and its recipe are retired. * Use 9 prefill workers for the c7480 HiCache point At decode batch 160, 9 x DEP2 reaches the same p90 interactivity as 10 x DEP2 with two fewer GPUs, so its throughput per GPU is higher. * Name the KV layout, precision and Engram changes against the retired recipe in the changelog * Move the GB300 DSV4.1-Flash Dynamo + SGLang agg row to the 20261029 nightly and use the megakernel runtime-optimization switch at TP4 c1 nightly-dev-cu13-20261029-c10e0e17 bundles sgl-nvidia-mega 54b428921749 (megakernel API 2, sgl-project/sglang#47145), so TP4 c1 starts with --enable-nvidia-megakernel again. SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION (sgl-project/sglang#47105) replaces the 16 runtime-optimization variables the recipe set one by one. * Drop the srt-slurm #629 backport patch now that main pins srt-slurm v2.65.2 Main bumped inferencex-e2e/utils/srt-slurm to v2.65.2 (#4601), which includes NVIDIA/srt-slurm#629, so runners/srt-slurm/patches/629-dynamo-health-instances.patch and its README row are removed and the changelog entry no longer mentions it. * Add back GB300 DSV4.1-Flash Dynamo+SGLang TP2/EP2 c16 and 3P1D c1228 points (append-only) Fill the interactivity/E2EL gap between agg TP4 c16 and disagg 5P1D c2884 on each config's existing image: agg TP2/EP2 c16 with its previous per-point settings, and a 3 x DEP2 / 1 x DEP16 c1228 point on the prefill HiCache disagg recipe with the previous c1228 decode sizing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Set the append-only changelog pr-link to #4679 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore: drop history clauses from DSV4.1-Flash hicache decode comments --------- Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: cquil11 <60715037+cquil11@users.noreply.github.com>

weireweire requested a review from a team October 28, 2026 02:59

github-project-automation Bot added this to InferenceX Board Oct 28, 2026

weireweire requested review from Ankur-singh, RohitNagraj, cquil11, kedarpotdar-nv, nvpohanh and wzhao18 as code owners October 28, 2026 02:59

weireweire added 3 commits October 28, 2026 10:59

Set perf-changelog pr-link to #4595

7095a6b

Restore historical perf-changelog entries; set only this entry's pr-link

4b5e7d9

Merge remote-tracking branch 'origin/main' into dsv41-sgl-disagg-dep1…

c0fec90

…6-hicache # Conflicts: # inferencex-e2e/perf-changelog.yaml

claude Bot reviewed Oct 28, 2026

View reviewed changes

Steven0706 approved these changes Oct 28, 2026

View reviewed changes

Oseltamivir requested a review from Steven0706 October 28, 2026 03:53

Oseltamivir removed the request for review from Steven0706 October 28, 2026 04:02

weireweire added 2 commits October 28, 2026 15:17

Merge remote-tracking branch 'origin/main' into dsv41-sgl-disagg-dep1…

51247f6

…6-hicache # Conflicts: # inferencex-e2e/perf-changelog.yaml

nvpohanh mentioned this pull request Oct 28, 2026

[Doc] cookbook: Drop purged SemiAnalysis AgentX runs and refresh to InferenceX main 253c38bf sgl-project/sglang#47027

Merged

weireweire added 4 commits October 28, 2026 21:33

Merge remote-tracking branch 'origin/main' into dsv41-sgl-disagg-dep1…

4b1a7a5

…6-hicache

Use 9 prefill workers for the c7480 HiCache point

eb4a618

At decode batch 160, 9 x DEP2 reaches the same p90 interactivity as 10 x DEP2 with two fewer GPUs, so its throughput per GPU is higher.

Name the KV layout, precision and Engram changes against the retired …

bb5e630

…recipe in the changelog

weireweire added 2 commits October 29, 2026 10:32

Merge remote-tracking branch 'origin/main' into dsv41-sgl-disagg-dep1…

0c4309b

…6-hicache # Conflicts: # inferencex-e2e/perf-changelog.yaml

weireweire added the full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) label Oct 29, 2026

nvpohanh mentioned this pull request Oct 29, 2026

[Doc] cookbook: Refresh SemiAnalysis AgentX launches to InferenceX main 70a76017 sgl-project/sglang#47214

Merged

weireweire added 2 commits October 30, 2026 10:01

Merge remote-tracking branch 'origin/main' into dsv41-sgl-disagg-dep1…

c63fac3

…6-hicache # Conflicts: # inferencex-e2e/perf-changelog.yaml

weireweire mentioned this pull request Oct 30, 2026

[NVIDIA][AgentX] Retune GB300 DSV4.1-Flash Dynamo + SGLang HiCache disagg points on the 20261030 nightly / [NVIDIA][AgentX] 在 20261030 nightly 上重新调整 GB300 DSV4.1-Flash Dynamo + SGLang HiCache 分离式点位 #4673

Merged

nvpohanh mentioned this pull request Oct 30, 2026

[Doc] cookbook: Refresh SemiAnalysis AgentX launches to InferenceX main ea8b3439 sgl-project/sglang#47464

Merged

nvpohanh approved these changes Oct 30, 2026

View reviewed changes

cquil11 approved these changes Oct 30, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into pr4595

dead055

# Conflicts: # inferencex-e2e/perf-changelog.yaml

chore: refresh PR #4595 for sweep reuse [skip-sweep]

c6ef640

cquil11 merged commit b649419 into main Oct 30, 2026
26 checks passed

cquil11 deleted the weireweire/dsv41flash-gb300-sgl-disagg-hicache branch October 30, 2026 15:27

github-project-automation Bot moved this to Done in InferenceX Board Oct 30, 2026

This was referenced Nov 2, 2026

[Doc] cookbook: Refresh SemiAnalysis AgentX launches to InferenceX main c5e8b9b3 and revert #47560 sgl-project/sglang#47929

Merged

[Doc] cookbook: Refresh SemiAnalysis AgentX launches to InferenceX main 57116fff sgl-project/sglang#48392

Merged

Conversation

weireweire commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

claude Bot Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 28, 2026

Uh oh!

nvpohanh commented Oct 28, 2026

Uh oh!

nvpohanh commented Oct 28, 2026

Uh oh!

nvpohanh commented Oct 29, 2026

Uh oh!

nvpohanh commented Oct 29, 2026

Uh oh!

github-actions Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nvpohanh commented Oct 30, 2026

Uh oh!

nvpohanh commented Oct 30, 2026

Uh oh!

github-actions Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nvpohanh commented Oct 30, 2026

Uh oh!

nvpohanh commented Oct 30, 2026

Additional detail section:

Uh oh!

github-actions Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅✅✅ Verdict: PASS ✅✅✅

Uh oh!

cquil11 commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cquil11 commented Oct 30, 2026

Uh oh!

cquil11 commented Oct 30, 2026

Uh oh!

github-actions Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Oct 30, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

weireweire commented Oct 28, 2026 •

edited

Loading

github-actions Bot commented Oct 30, 2026 •

edited

Loading

github-actions Bot commented Oct 30, 2026 •

edited

Loading

github-actions Bot commented Oct 30, 2026 •

edited

Loading

cquil11 commented Oct 30, 2026 •

edited

Loading

github-actions Bot commented Oct 30, 2026 •

edited

Loading