Update DSV4 B300 SGLang AgentX image and HiCache concurrency grid / 更新 DSV4 B300 SGLang AgentX 镜像和 HiCache 并发网格

[yhyang201]

Description

Updates dsv4-fp4-b300-sglang-agentic-hicache-mtp in two ways:

• Replaces the one-off staging image lmsysorg/sglang-staging:dev-cu13-pr-35880 with the published upstream nightly lmsysorg/sglang:nightly-dev-cu13-20260827-20621aa1.
• Removes concurrency 64 and 128 from the DEP8 HiCache row. Concurrency 128 failed with a CUDA out-of-memory error in run 33051183882; concurrency 64 was still unmeasured when the row was cut and used the same mem-fraction-static tier. The resulting submission contains ten points: TP8 at concurrency 1, 4, 8, 16, and 32, plus DEP8 HiCache at concurrency 32, 256, 384, 512, and 576.

The serving flags, dram-utilization, HiCache write policy, router configuration, and speculative-decoding settings remain unchanged. --prefill-decode-interval 20 is present in the published nightly because its upstream PR is merged.

Exact-head validation: Run Sweep 33145139961, attempt 2 completed successfully on fb85931b1edec09f9498509835a8c814bebe3c65; all ten applicable AgentX benchmark leaves, the applicable AgentX eval leaf, and the eval collector succeeded.

The HiCache concurrent load-back pin fix is not included in this image: sgl-project/sglang#34975 closed without merge, and #35880 remains an open draft. Concurrency 384, 512, and 576 all passed under --hicache-write-policy write_back in the exact-head sweep, so the previously predicted failure did not reproduce. The underlying timing-dependent risk remains until an upstream fix is merged and included in a later image.

中文说明

本 PR 对 dsv4-fp4-b300-sglang-agentic-hicache-mtp 做出两项更新：

• 将一次性 staging 镜像 lmsysorg/sglang-staging:dev-cu13-pr-35880 替换为已发布的上游 nightly 镜像 lmsysorg/sglang:nightly-dev-cu13-20260827-20621aa1。
• 从 DEP8 HiCache 配置中移除并发度 64 和 128。并发度 128 在运行 33051183882 中因 CUDA 显存不足而失败；并发度 64 在裁剪搜索空间时尚未完成测量，并且使用相同的 mem-fraction-static 档位。更新后的提交包含十个数据点：TP8 的并发度 1、4、8、16、32，以及 DEP8 HiCache 的并发度 32、256、384、512、576。

服务参数、dram-utilization、HiCache 写入策略、路由器配置和推测解码设置均保持不变。--prefill-decode-interval 20 对应的上游 PR 已合并，因此该参数已包含在此公开 nightly 镜像中。

精确 head 验证：Run Sweep 33145139961，attempt 2 已在 fb85931b1edec09f9498509835a8c814bebe3c65 上成功完成；十个适用的 AgentX 基准任务、一个适用的 AgentX 评测任务以及评测汇总任务均成功。

该镜像尚未包含 HiCache 并发 load-back pin 修复：sgl-project/sglang#34975 已关闭且未合并，#35880 仍为开放的草稿 PR。在本次精确 head sweep 中，并发度 384、512 和 576 均在 --hicache-write-policy write_back 下通过，因此此前预测的故障未能复现。在上游修复合并并进入后续镜像之前，相关的时序风险仍然存在。

Related Issue

N/A

Type of Change

• Bug fix
• New feature
• Configuration change
• Documentation update
• Other (please describe)

Checklist

• I have tested my changes locally — the B300 benchmark is not runnable locally; the exact-head full sweep is linked above.
• I have updated documentation if necessary (perf-changelog.yaml).
• For every change that can affect benchmark performance and every recipe addition or modification, I have appended a new entry to the physical end of perf-changelog.yaml and have not edited historical entries
• Before merging via reuse, an authorized maintainer (OWNER/MEMBER/COLLABORATOR) has commented /reuse-sweep-run on this PR. The planned pinned authorization is /reuse-sweep-run 33145139961.

---

Note

Low Risk
Benchmark configuration and container pin only; no application or auth logic changes, though published numbers will reflect the new image and fewer HiCache concurrency points.

Overview
Updates the dsv4-fp4-b300-sglang-agentic-hicache-mtp benchmark recipe: swaps the SGLang container from the one-off staging image to lmsysorg/sglang:nightly-dev-cu13-20260827-20621aa1, and narrows the DEP8 HiCache conc-list by dropping 64 and 128 (OOM at 128; 64 unmeasured on the same memory tier).

Serving flags, HiCache/router/MTP settings, and other search-space rows are unchanged. perf-changelog.yaml records the image bump and concurrency grid change for this config key.

^{Reviewed by Cursor Bugbot for commit 1416480. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 4960d11. Configure here.}

[claude]

Beyond the inline pr-link finding, I also checked whether this bump reintroduces the HiCache load-back hang for the DEP8 conc 384/512/576 points in the recipe's search-space — that search-space line is unchanged by this diff and the risk is already explicitly disclosed in the new changelog entry, so it's not a new/hidden regression introduced silently by this PR.

Extended reasoning...

A single confirmed finding exists (the placeholder pull/PENDING pr-link in perf-changelog.yaml), which will be posted inline and is enough reason to hold off on approval until it's replaced with a real PR link. I additionally verified the candidate concern that the image bump reintroduces a HiCache load-back hang for the DEP8 conc 384/512/576 search-space points: that search-space entry (line 1170) is untouched by this diff — it predates the PR — and the exact regression mechanism (missing #34975/#35880 fix, write_back default, assertion in unified_tree_core.py) is transparently documented in the new changelog entry itself rather than being a silently introduced or hidden issue. That rules it out as a new undisclosed bug worth a separate inline comment.

[claude]

🟡 pr-link is a placeholder 'pull/PENDING' instead of a real PR number, unlike every other entry in the file.

Extended reasoning...

The changelog entry merges with a dead/invalid link (github.com/SemiAnalysisAI/InferenceX/pull/PENDING resolves to nothing), breaking the append-only historical record's traceability to the actual PR and diverging from the convention every prior entry follows.

Verification: nit. Line 6549 of perf-changelog.yaml reads pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/PENDING. Every other entry in the file uses a real numeric PR ID (e.g. line 6539 .../pull/2756, 6528 .../pull/2758, 6429 .../pull/2701). The placeholder PENDING is not a valid PR number, so the URL resolves to nothing, breaking the append-only changelog's traceability to the merging…

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33051183882
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=33051183882

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33051183882
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=33051183882

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33051183882
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=33051183882

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33145139961
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=33145139961

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33145139961
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=33145139961

[Ankur-singh]

/reuse-sweep-run 33145139961

[Ankur-singh]

As a PR reviewer and CODEOWNER, I have reviewed this and have:

• Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
• Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
• Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33145139961
• Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33145139961
• Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
• For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
• Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
• Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
• If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
• If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
• Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
  • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
• Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
• If this PR uses append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it.
• If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.

Additional detail section:

• Published SGLang cookbook recipe: https://github.com/sgl-project/sglang/blob/main/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx

Signed: Ankur-singh

[Klaud-Cold]

✅✅✅ Verdict: PASS ✅✅✅

✅ Check 0 (CODEOWNER): PASS — Ankur-singh is a listed owner of configs/nvidia-master.yaml; perf-changelog.yaml falls under the catch-all, covered by any recognized CODEOWNER.
✅ Check 1 (Sweep on in-PR commit): PASS — run 33145139961 executed on PR head fb85931; all 10 agentic / benchmark leaves and the agentic eval / leaf concluded success (the ten points exactly match the new config grid; generic single-node/eval lanes were skipped because this PR selects only an agentic config).
✅ Check 2 (Eval accuracy): PASS — downloaded agg_eval_all.json from run 33145139961: dsv4 gsm8k em_strict 0.959 vs threshold 0.91 (utils/evals/thresholds.yaml), run on this PR's exact image lmsysorg/sglang:nightly-dev-cu13-20260827-20621aa1.
✅ Check 3 (Recipe merged & matches): PASS — sign-off links the published SGLang cookbook DeepSeek-V4 page (on sglang main). Its verified B300+Pro+FP4 cells match all major args: TP8 + flashinfer_mxfp4 + EAGLE 3-1-4 (low-latency cell) and TP8/DP8 + --enable-dp-attention + --moe-a2a-backend megamoe + SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320 (high-throughput cell); HiCache and the EAGLE/MTP 3-1-4 preset are documented cookbook cards. Informational: mem-fraction tiers, chunked-prefill scaling, router/warmup plumbing are InferenceX-side tuning, as expected.
✅ Check 4 (Reuse command): PASS — /reuse-sweep-run 33145139961 posted by Ankur-singh (COLLABORATOR).
✅ Check 5 (Latest checklist): PASS — sign-off contains every item of the current docs/PR_REVIEW_CHECKLIST.md template, all checked.
✅ Check 6 (Upstream image & ordering): PASS — lmsysorg/sglang:nightly-dev-cu13-20260827-20621aa1 is from the upstream lmsysorg org (replacing a staging tag — an improvement); framework is SGLang itself, so engine-first ordering is inherently satisfied.
✅ Check 7 (Deprecations): PASS — dsv4 agentic coding with MTP is the active, published arm per MODELS.md (2026-08-28); nothing deprecated is submitted.
✅ Check 8 (No architecture hacks): PASS — diff touches only image tag and concurrency list; no --hf-overrides or FLOPs-reducing knobs anywhere in the config's launch path.
✅ Check 9 (Spec-decode chat template): PASS — the agentic replay serves through --chat-template deepseek_v4_thinking.jinja with tool-call/reasoning parsers (chat-completions style), not raw completions.
✅ Check 10 (No engine patches): PASS — no .patch/heredoc/sed/monkey-patching; the pinned image runs as shipped (this PR moves OFF a patched staging image).
✅ Check 11 (Golden simulated AL): PASS — script pins SGLANG_SIMULATE_ACC_LEN=2.49 with match-expected/real-draft-token; golden dsv4_mtp.yaml thinking_on at num_speculative_tokens=3 (--speculative-num-steps 3) is exactly 2.49; eval leaves run real acceptance (EVAL_ONLY guard), and no synthetic knobs appear on non-agentic configs.
➖ Check 12 (Append-only): N/A — neither new perf-changelog.yaml entry sets append-only: true (the removal of conc 64/128 was correctly submitted as a regular change with a full rerun of all 10 points).

[yhyang201]

Is there anything else we need to do before merging? Thanks! @Ankur-singh

[Ankur-singh]

@functionstackx @cquil11 waiting SA approval, please review