[hshrivastava-droid]

Summary

Update qwen3.8next-fp4-b300-sglang-agentic-mtp to the NVIDIA Qwen3.8-Flash-Next NVFP4 checkpoint on digest-pinned SGLang nightly-dev-cu13-20261028-586355a8.

The AgentX matrix covers TP4/EP4 at concurrency 1 and 2, TP1/EP1 GPU-only at concurrency 1, 2, 12 and 24, and TP1/EP1 with HiCache at concurrency 48 and 80. TP1 c1 shares the existing c2 tuning; TP4 c2 follows TP4 c1 with a four-request admission cap. The higher-concurrency variants are unchanged by this addition.

Image: lmsysorg/sglang:nightly-dev-cu13-20261028-586355a8@sha256:b4f09a2ccbcfcc262d49600c3cd7f8581a793ea7d104947257b6a95500953b90

Validation

Full sweep 41117066222 passed at d06564bc533cc54584ef6a35837336a2b4b84421: all eight AgentX throughput points, both selected evaluation jobs, and result collection succeeded. Concurrency 1 and 2 passed at both TP1 and TP4. A performance delta against published main results has not been verified.

Run evidence

Performance preview and accuracy preview.

AI model disclosure

• Claude Sonnet 5.5 (claude-sonnet-5-5): prepared the original PR.
• GPT-6 Astra: added low-concurrency coverage, updated this description, and monitored CI.

中文

将 qwen3.8next-fp4-b300-sglang-agentic-mtp 更新为 NVIDIA Qwen3.8-Flash-Next NVFP4 检查点，并使用固定 digest 的 SGLang nightly-dev-cu13-20261028-586355a8 镜像。

AgentX 测试矩阵覆盖 TP4/EP4 并发 1、2，TP1/EP1 GPU-only 并发 1、2、12、24，以及启用 HiCache 的 TP1/EP1 并发 48、80。TP1 c1 复用现有 c2 调优设置；TP4 c2 沿用 TP4 c1 设置，并将请求准入上限设为 4。本次补充未修改更高并发配方。

Claude Sonnet 5.5 (claude-sonnet-5-5) 准备了原始 PR；GPT-6 Astra 补充了低并发覆盖、更新了描述并监控 CI。

完整 sweep 41117066222 已在 d06564bc533cc54584ef6a35837336a2b4b84421 通过：八个 AgentX 吞吐测试点、两个选定评测任务及结果收集均成功。TP1 和 TP4 的并发 1、2 均已通过。尚未验证相对 main 已发布结果的性能变化。

[github-actions]

Thanks for the contribution!

• Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
• PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
• After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.

中文

感谢你的贡献！

• **审阅：**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER，请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，再通过 Slack 联系核心维护者。必须严格遵循模板，并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have，才能触发签核验证。
• **PR 验证：**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目，并且恰好带有一个主标签时，才会运行扫描：full-sweep-fail-fast（强烈推荐；canary 加逐矩阵 fail-fast）、full-sweep-enabled（有 canary；矩阵任务在失败后继续运行）或 non-canary-full-sweep-enabled（无 canary，也无 fail-fast）。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
• **合并后：**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复；参见重新运行失败任务的说明。

[github-actions]

View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=41117066222

View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=41117066222

[claude]

I reviewed this PR and didn't find any bugs. The recipe rewrite involves many engine-tuning choices on a nightly image, and the changed paths have NVIDIA CODEOWNERS, so a human review is still worthwhile.

What was reviewed:

• The recipe's YAML anchor/merge-key overrides. The GPU-only rows (tp4_c1, c2, c12, c24) null out the base HiCache and Mamba settings and set KV_OFFLOADING=none / TOTAL_CPU_DRAM_GB=0. The c48 and c80 rows keep the base HiCache with fp8 KV.
• Whether the master-config search-space rows (TP4/EP4 at c1, TP1 at c2/12/24, TP1 with HiCache at c48/80) match the override names. Also whether the recipe leaves image, model, precision and benchmark type/command to the binder.
• The new perf-changelog entry. It is appended at the tail and covers the one changed config key. Its pr-link is still the pull/XXX placeholder.

Extended reasoning...

The change rewrites one single-node srt-slurm AgentX recipe for Qwen3.8-Flash-Next NVFP4 on B300. It also updates the matching nvidia-master.yaml entry (image, model and a three-row search space) and appends one perf-changelog entry. It touches no security-sensitive code, only benchmark configuration. I didn't approve because nvidia-master.yaml and the b300-* recipe directory are covered by NVIDIA CODEOWNERS entries. Also, the recipe adds dozens of new server args and feature-flag env vars whose performance trade-offs need a domain owner to judge.

This review covers commit f06a819, which is no longer the latest commit on this pull request; later commits are not covered by it.

[hshrivastava-droid]

/use 41117066222

[github-actions]

@hshrivastava-droid staged run 41117066222.

• Qwen3.8-Flash-Next · agentic-traces · FP4 · b300_sglang · 2026-10-30

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

[nvpohanh]

[by Claude Code] The SGLang cookbook integration of this PR's AgentX config (qwen3.8next-fp4-b300-sglang-agentic-mtp) will happen in sgl-project/sglang#47929, in the Agentic Long-Context Deployment section of the Qwen3.8-Flash-Next cookbook page.

The commands mirror the launch measured in run 41117066222, pinned by this PR's /use comment.

[RohitNagraj]

As a PR reviewer and CODEOWNER, I have reviewed this and have:

Checklist items

• Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
• Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
• Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
• Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
• Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
• Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison.
• For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
• Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
• Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
• Verified that new models disable integer all-reduce, with SGLang ROCm explicitly setting ROCM_QUICK_REDUCE_QUANTIZATION=NONE. Existing models are grandfathered per MODELS.md.
• For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
• If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
• For official InferenceX (non-preview), verified that established models use upstream engine images built from master (or the equivalent default development branch), with new models transitioning within a reasonable time after release. See MODELS.md.
• If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
• Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
  • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
• Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
• If this PR uses append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it.
• If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
• Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.

Additional detail section:

• Validation: run 41117066222, attempt 1 passed its benchmark and selected eval jobs on d06564bc533cc54584ef6a35837336a2b4b84421. The subsequent change at 6f2f785247ec1c87dc25da3c2d718f9c5513e154 updates the changelog description only.
• The selected eval jobs passed. Their greedy decoding does not exercise the enabled sparse verification sampling branch, so branch-specific accuracy coverage remains unverified.
• The embedded MTP draft uses nvidia/Qwen3.8-Flash-Next-NVFP4 at fc694b54fb0174e0913e6adf86691ef85a4ead47. Its experts are block-scaled FP8 with BF16 scales; other MTP tensors are BF16. The pinned image uses the checkpoint's modelopt_mixed handling, with no draft-weight precision override; the prohibited conversion flag is disabled and draft KV dtype matches the target. Checkpoint metadata; pinned MTP loader.
• Agentic MTP uses the committed golden acceptance curve. The AgentX client uses the chat endpoint. Qwen3.8-Flash-Next and agentic coding are active; the model is grandfathered under integer all-reduce policy. Dynamo TRT-LLM metrics and ROCm-specific requirements do not apply. The SGLang image build used the main branch.
• The SGLang cookbook includes the affected B300 variants; sglang#47560 is merged. Benchmark harness and result collection settings do not configure production serving.
• Pareto coverage: Qwen3.8-Flash-Next / agentic-coding / B300 SGLang / FP4 has 6 measured throughput/E2EL frontier points out of 8 measured points, using p90 E2EL, in run 41117066222, attempt 1.
• The existing /use 41117066222 selects the tested run.

中文

验证任务及所选评测在 d06564bc 上通过；后续提交仅更新 changelog 描述。所选评测任务已通过，但贪心评测未覆盖已启用的稀疏验证采样分支，该分支的准确性覆盖仍未验证。MTP 草稿沿用检查点存储精度，禁止的转换设置已关闭。Agentic MTP 使用已提交的黄金接受长度曲线。SGLang cookbook 已收录受影响的 B300 配置。该曲线的 8 个实测点中有 6 个位于吞吐量与 p90 E2EL 的 Pareto 前沿。现有 /use 41117066222 已选择该次运行。

Signed: RohitNagraj

[github-actions]

❌❌❌ REJECTED ❌❌❌

@RohitNagraj Blocking: the recipe enables the conditional sparse verify-sampling branch (SGLANG_Q35MF_EDGE_SPARSE_VERIFY_SAMPLING: '1') for every variant, but no eval exercised it; the sign-off itself records this coverage as unverified.

❌ Check 2 (Evals pass): FAIL — No eval covers the enabled sparse verify-sampling path: SGLANG_Q35MF_EDGE_SPARSE_VERIFY_SAMPLING: '1' (agentic.yaml base env) switches the pinned SGLang 586355a8 to the compact top-k verify sampling only when every request samples with a finite top-k (the AgentX/golden-AL traffic uses top_k 20), yet both gsm8k evals in run 41117066222 ran greedy (temperature: 0, do_sample: false). Scores are otherwise fine: TP1 c2 and TP1 HiCache c80 both 0.9773, cohort 0.9765, no gaps. Add a sampled eval (finite top-k, non-zero temperature) whose logs show the branch ran, or drop the flag and rerun.

Passed and not applicable checks

✅ Check 0 (CODEOWNER): PASS — RohitNagraj is a named owner of inferencex-e2e/configs/nvidia-master.yaml and the srt-slurm-recipes/**/b300-*/ recipe; perf-changelog.yaml falls under the catch-all only.

✅ Check 1 (In-PR sweep + evals): PASS — In-PR commit d06564bc has green executed agentic /, canary / (TP4 c1) and both agentic eval / jobs in run 41117066222 (attempt 1). The only later commit, 6f2f7852, adds one changelog description line (non-material).

✅ Check 3 (Upstream recipe): PASS — sglang#47560 is MERGED (2026-10-31). Its qwen3.8-flash-next-b300.jsx is rendered on the published cookbook page and matches the TP4/EP4, TP1 GPU-KV and TP1 HiCache args and env for each point. Differences are logging/seed/metrics only. Open #47929 is a later refresh and does not affect this.

✅ Check 4 (Reuse command): PASS — /use 41117066222 was posted on its own line by hshrivastava-droid (COLLABORATOR).

✅ Check 5 (Latest template): PASS — All 20 items in the current PR_REVIEW_CHECKLIST.md template are present and checked.

✅ Check 6 (Upstream images): PASS — lmsysorg/sglang:nightly-dev-cu13-20261028-586355a8@sha256:b4f09a2c… is upstream lmsysorg, built by sglang's scheduled main build at 586355a8. It is an SGLang entry, so the ordering rule does not apply.

✅ Check 7 (Deprecated models): PASS — MODELS.md lists qwen3.8next (added 2026-08-26) with agentic coding active; nothing is deprecated as of 2026-11-02.

✅ Check 8 (No arch hacks): PASS — No --hf-overrides or config edits. strip-thinking-cache, mamba-track-interval and mamba-max-states-per-path only change cache retention. The FP8 KV at c12+ is covered by the c80 eval. This is a CUDA path with no quick-reduce/integer all-reduce control, and flashinfer-allreduce-fusion-backend: trtllm is non-integer.

✅ Check 9 (Spec-decode chat template): PASS — The AgentX replay client drives /v1/chat/completions; the harness is unchanged by this PR.

✅ Check 10 (No engine patches): PASS — No setup script, patch, overlay or wheel install. Every SGLANG_Q35MF_* and SGLANG_ENABLE_EAGLE_POST_VERIFY_CUDA_GRAPH / SGLANG_OPT_CUDA_GRAPH_FILL_FAST_PATH switch is defined in the pinned upstream environ.py at 586355a8.

✅ Check 11 (Agentic golden AL): PASS — The connector injected SGLANG_SIMULATE_ACC_LEN=2.32, match-expected and real-draft-token (job log, TP1 c2). That equals qwen3.8next_mtp.yaml thinking_on at 3 speculative tokens (2.32). The recipe hard-codes no AL.

➖ Check 12 (Append-only): N/A — The new changelog entry is not append-only.

✅ Check 13 (Draft precision): PASS — The embedded MTP head of nvidia/Qwen3.8-Flash-Next-NVFP4@fc694b54 stores mtp.layers.0.mlp.experts as FP8_BLOCK_SCALES and the rest unquantized, loaded via modelopt_mixed. There is no draft quantization or dtype override. SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is explicitly 0 (default EnvBool(False)). Draft KV dtype matches the target (BF16, or FP8 at c12+).

✅ Check 14 (Pareto coverage): PASS — Qwen3.8-Flash-Next / agentic-coding / b300_sglang / fp4 / p90 E2EL: 6 of 8 measured points are on the frontier (pareto_coverage helper), from results_bmk of run 41117066222 attempt 1 at d06564bc. TP4 c1 and TP1 c2 are dominated.

✅ Check 15 (PR description): PASS — The body's search space, low-concurrency tuning claims, digest-pinned image and validation (8 throughput points, 2 evals at d06564bc) match the pinned code and run.

✅ Check 16 (NVIDIA overview): PASS — Qwen3.8-Flash-Next has an SGLang checkmark in the Framework Recipes table of the NVIDIA overview.

➖ Check 17 (Metrics publication): N/A — This is an SGLang aggregated config, not Dynamo TRT-LLM.

✅ Check 18 (Recipe/image/sweep consistency): PASS — There are no generation errors. Master TP4/EP4, TP1 and HiCache rows map to the override_tp4_*, override_c* and HiCache overrides. The image manifest resolves. All five removed v0.5.21 points are dominated by head points. The changelog discloses the checkpoint, image and FP8 KV change, and the prior KV default was BF16 (the RadixArk checkpoint has no KV quant). Re-tune deltas: c12 tput +16.2% / intvty +12.6%; c1 tput +95.5% / intvty +533.7%.

Assessed commit: 6f2f785247ec1c87dc25da3c2d718f9c5513e154.

[nvpohanh]

[by Claude Code] sgl-project/sglang#47929 has been merged. This PR's AgentX config (qwen3.8next-fp4-b300-sglang-agentic-mtp) is now in the Agentic Long-Context Deployment section of the SGLang Qwen3.8-Flash-Next cookbook: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.8-Flash-Next#agentic-long-context-deployment

The commands mirror the launch measured in run 41117066222, pinned by this PR's /use comment.

[functionstackx]

Approved at Oren’s request, conditional on your confirmation that this optimization is used in production. The greedy-only eval coverage gap described in the discussion is accepted on that basis; this approval does not establish non-greedy correctness or independently verify production use.

[github-actions]

Ingest report — ✅ Results ingested into the production database.

Ingest workflow · Merge Ingest run · Benchmark source run

Category	new	duplicate	skipped	unmapped	failed
benchmark	8	0	8	0	0
eval	2	2	2	0	0
trace	8	0	0	0	0
power	8	0	0	0	0

_{benchmarkFiles 9, rawBenchmarkRows 16, coveredByPerJobFiles 8, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 2638, gpuMetricSamplesNew 87243, powerPublicationPoints 8, changelogEntriesWritten 1}

Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun.

中文

Ingest 报告 — ✅ 结果已写入 production 数据库。

上表仅统计本次运行写入的数据，各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。