[AgentX] GB300 DSV4.1 Flash SGLang: 20261014 nightly, fastokens, DEP2 bounded replay and c160, TP4 c1 single admission / [AgentX] GB300 DSV4.1 Flash SGLang：升级 20261014 nightly、启用 fastokens、DEP2 开 bounded replay 并新增 c160、TP4 c1 单请求准入

[weireweire]

Description

Config: dsv41flash-fp4-gb300-sglang-agentic-dspark (aggregated, single node). Image, recipe, master config and changelog only; follows #3859.

1. Image bump to lmsysorg/sglang:nightly-dev-cu13-20261014-684fea8a, pinned by digest. It ships fastokens 0.3.4 ([Tokenizer] Require fastokens>=0.3.2 and include it in the all extra sgl-project/sglang#43104 requires fastokens>=0.3.2) and allows decoder SWA bounded replay with DP attention ([DSV4.1] Support decoder SWA bounded replay with CUDA DP attention sgl-project/sglang#43812).
2. tokenizer-backend: fastokens on every point except DEP2 c160. The chat endpoint re-encodes the whole conversation each turn, 300K–600K tokens on AgentX. HF tokenizers takes about 0.6 s per request on Grace, on the TTFT path and the API server event loop; fastokens takes about 0.03 s.
3. DEP2 c64 and c128 run decoder SWA bounded replay.
4. New DEP2 c160 point with the c128 settings, a prefill-decode interval of 8 and the HuggingFace tokenizer. At this load fastokens admits short subagent prefills faster, and shortest-prefill-first then holds back long main-agent turns, so fewer long-context requests complete and tok/s/GPU drops.
5. TP4 c1 admits one request at a time (was two) and runs decoder SWA bounded replay with the prefill CUDA graph disabled. A subagent request that arrives while another request decodes now waits instead of stalling that decode with its prefill.

Local results

One-hour AgentX runs on GB300, CI harness, synthetic acceptance 3.51. Fastokens: 20261012 without fastokens (#3859) → 20261013 with fastokens. Throughput in tok/s/GPU, interactivity is P90 tok/s/user.

point	throughput	interactivity	TTFT p50 / p90
TP4 c1	7,314 → 7,850	625.7 → 610.0	0.87 / 1.38 s → 0.24 / 0.28 s
TP2 c32	113,613 → 115,868	183.1 → 190.9
DEP2 c128	271,364 → 271,676	81.6 → 86.6

DEP2 c128 with fastokens, one-hour runs: 271,676 tok/s/GPU at 86.6 P90 on 20261013 → 280,752 at 88.5 on 20261014 → 283,868 at 91.3 with bounded replay; TTFT p90 3.43 → 3.02 → 2.62 s, TTFT p99 74 → 60 → 56 s. DEP2 c64 uses the same server settings with bounded replay and is validated by this sweep.

DEP2 c160, one-hour runs: 258,332 tok/s/GPU at 85.8 P90 with fastokens (20261013) against 296,603 at 79.2 (20261012) and 287,471 at 78.2 (20261013) with the HuggingFace tokenizer. On 20261014 with bounded replay, interval 16 measured 268,187 at 90.9 in CI, below c128 on both axes; interval 8 measures 294,159 at 61.1 locally, since AgentX throughput is almost all prefill tokens. Dropping bounded replay at interval 16 gave 259,506.

TP4 c1 admission change (20261013 image with fastokens, one-hour runs): 7,851 → 7,848 tok/s/GPU, P90 interactivity 625.1 → 654.4 (median 671.3 → 673.8), TTFT p50/p90 0.24/0.28 → 0.23/0.27 s. The cost is in the tail: TTFT p99 0.97 → 1.13 s, and the few requests that queue behind a long decode wait up to 17 s.

AI model disclosure

• Model/version: Claude Opus 5.5 (claude-opus-5-5)
• Role: ran the local experiments, ported the change onto main and drafted this description.

Checklist

• Configuration change
• I have completed the AI model disclosure and kept it current
• I have tested my changes locally
• I have appended a new entry to the physical end of inferencex-e2e/perf-changelog.yaml and have not edited historical entries
• This PR carries exactly one primary sweep label (full-sweep-fail-fast recommended)

中文

配置：dsv41flash-fp4-gb300-sglang-agentic-dspark（聚合、单节点），只改镜像、配方、master 配置和 changelog，接续 #3859。

1. 镜像升级到 lmsysorg/sglang:nightly-dev-cu13-20261014-684fea8a，按 digest 固定。该镜像带 fastokens 0.3.4（[Tokenizer] Require fastokens>=0.3.2 and include it in the all extra sgl-project/sglang#43104 要求 fastokens>=0.3.2），并支持 DP attention 下的 decoder SWA bounded replay（[DSV4.1] Support decoder SWA bounded replay with CUDA DP attention sgl-project/sglang#43812）。
2. 除 DEP2 c160 外所有点位开启 tokenizer-backend: fastokens。 chat 接口每轮都会重新编码整段对话，AgentX 上有 30 万到 60 万 token。HF tokenizers 在 Grace 上每个请求约 0.6 s，且位于 TTFT 路径和 API server 事件循环上；fastokens 约 0.03 s。

本地一小时 AgentX 测试（GB300，CI harness，synthetic acceptance 3.51），从 20261012 不开 fastokens 到 20261013 开 fastokens：

• TP4 c1：吞吐 7,314 → 7,850 tok/s/GPU，P90 交互性 625.7 → 610.0，TTFT p50/p90 从 0.87/1.38 s 降到 0.24/0.28 s。
• TP2 c32：113,613 → 115,868，P90 交互性 183.1 → 190.9。
• DEP2 c128：271,364 → 271,676，P90 交互性 81.6 → 86.6。

3. DEP2 c64、c128 开启 decoder SWA bounded replay。 DEP2 c128（开 fastokens）一小时测试：271,676 tok/s/GPU、P90 86.6（20261013）→ 280,752、88.5（20261014）→ 283,868、91.3（加 bounded replay）；TTFT p90 3.43 → 3.02 → 2.62 s。
4. 新增 DEP2 c160 点，配置与 c128 相同，但 prefill-decode interval 为 8，并使用 HuggingFace tokenizer。在这个负载下 fastokens 让短的 subagent prefill 更快进入调度，shortest-prefill-first 会把长的主 agent 请求往后压，完成的长上下文请求变少，吞吐下降。c160 一小时测试：开 fastokens 258,332 tok/s/GPU、P90 85.8（20261013）；HuggingFace tokenizer 296,603、79.2（20261012），287,471、78.2（20261013）。20261014 开 bounded replay 时，interval 16 在 CI 上为 268,187、P90 90.9，两项都不如 c128；interval 8 本地为 294,159、61.1，因为 AgentX 吞吐几乎全是 prefill token。interval 16 去掉 bounded replay 为 259,506。
5. TP4 c1 一次只接纳一个请求（原为两个），并开启 decoder SWA bounded replay（prefill CUDA graph 关闭）。在另一个请求 decode 时到达的 subagent 请求改为排队，不再用自己的 prefill 打断对方的 decode。同镜像加 fastokens 的一小时测试：吞吐 7,851 → 7,848 tok/s/GPU，P90 交互性 625.1 → 654.4（中位数 671.3 → 673.8），TTFT p50/p90 0.24/0.28 → 0.23/0.27 s。代价在长尾：TTFT p99 0.97 → 1.13 s，少数排队的请求最长等 17 s。

[github-actions]

Thanks for the contribution!

• Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
• PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
• After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.

中文

感谢你的贡献！

• **审阅：**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER，请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，再通过 Slack 联系核心维护者。必须严格遵循模板，并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have，才能触发签核验证。
• **PR 验证：**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目，并且恰好带有一个主标签时，才会运行扫描：full-sweep-fail-fast（强烈推荐；canary 加逐矩阵 fail-fast）、full-sweep-enabled（有 canary；矩阵任务在失败后继续运行）或 non-canary-full-sweep-enabled（无 canary，也无 fail-fast）。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
• **合并后：**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复；参见重新运行失败任务的说明。

[github-actions]

View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=38614048439

View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=38614048439

[nvpohanh]

[by Claude Code] The SGLang cookbook integration of this PR's AgentX config (dsv41flash-fp4-gb300-sglang-agentic-dspark) will happen in sgl-project/sglang#44712, in the Agentic Long-Context Deployment section of the DeepSeek-V4.1 cookbook page.

This PR has no authorized /use <run_id> comment yet, so the cookbook shows the recipe at this PR's head (a61b7eba2) as "in progress". It switches to the measured run at the next cookbook sync after a run is pinned or this PR merges.

[nvpohanh]

[by Claude Code] sgl-project/sglang#44712 has been merged. This PR's AgentX config (dsv41flash-fp4-gb300-sglang-agentic-dspark) is now in the Agentic Long-Context Deployment section of the SGLang DeepSeek-V4.1 cookbook: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4_1#agentic-long-context-deployment

This PR has no authorized /use <run_id> comment yet, so the cookbook shows the recipe at this PR's head (a61b7eba2) as "in progress". A later cookbook PR switches it to the measured run once a run is pinned or this PR merges.

[nvpohanh]

[by Claude Code] I've started babysitting the pipeline of this PR. I'll keep monitoring it until it passes. If a job fails, I may rerun it when the cause is infrastructure, or push a fix or a merge of the latest main when the cause is in this PR.

[nvpohanh]

/use 38614048439

[github-actions]

@nvpohanh staged run 38614048439.

• DeepSeek-V4.1-Flash · agentic-traces · FP4 · gb300_sglang · 2026-10-14

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

[nvpohanh]

As a PR reviewer and CODEOWNER, I have reviewed this and have:

• Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
• Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
• Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
• Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
• Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
• Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison.
• For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
• Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
• Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
• For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
• If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
• If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
• Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
  • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
• Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
• If this PR uses append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it.
• If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
• Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.

Additional detail section:

• Recipe publication (merged): every point in this PR is documented in the SGLang cookbook by sgl-project/sglang#44712, which is MERGED (mergedAt: 2026-10-16T11:31:09Z, merge commit 5e240a200dd04060e4d47bf967c409611949b82f). It is the latest commit touching the DeepSeek-V4.1 AgentX data on SGLang main.
  • Published page: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4_1#agentic-long-context-deployment (hardware GB300, checkpoint deepseek-ai/DeepSeek-V4.1-Flash, image 684fea8a, provenance PR 4033 at a61b7eba, the measured commit; the recipe is unchanged at the current head 86ace5cf). The live page carries these GB300 entries, and the NVIDIA overview links the same page.
  • Data: docs/src/snippets/agentx/deepseek-ai/deepseek-v4_1.jsx has three GB300 deployments from dsv41flash-fp4-gb300-sglang-agentic-dspark: "Agg TP4" (C1/C4/C16), "Agg TP2·EP2" (C16/C32) and "Agg DEP2" (C64/C128/C160, with sglang-router).
  • Image: lmsysorg/sglang:nightly-dev-cu13-20261014-684fea8a@sha256:ddae0e399d3d94f7827f16183301cf8dac4cf0eae0ef25590b7dac5881d713d8. sglang-router 0.3.2 and fastokens==0.3.4 ship in the image: the worker fingerprint's container package inventory lists sglang-router @ file:///tmp/gateway_wheels/sglang_router-0.3.2-…whl and fastokens==0.3.4, and nothing is installed at launch. The cookbook has no install step.
  • Worker, all 8 points: python3 -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V4.1-Flash --served-model-name deepseek-ai/DeepSeek-V4.1-Flash --trust-remote-code --speculative-algorithm DSPARK --speculative-dspark-block-size 5 --reasoning-parser auto --tool-call-parser auto --watchdog-timeout 3600, plus the per-point settings below.
    • Worker environment: NCCL_CUMEM_ENABLE=1, SGLANG_DEFAULT_THINKING=1, SGLANG_DSV41_REASONING_EFFORT=high, SGLANG_DSPARK_OPT_MARKOV_W2_BF16=True, SGLANG_DSPARK_FOLDED_SAMPLING=2, SGLANG_TIMEOUT_KEEP_ALIVE=900.
    • NCCL_CUMEM_ENABLE=1 is not set by the recipe. The GB300-NV cluster exports it into every worker container (worker fingerprint of every point), so the cookbook carries it.
    • SGLANG_DSPARK_FOLDED_SAMPLING is deprecated without replacement in this image (environ.py#L2007-L2011; every worker logs "Environment variable SGLANG_DSPARK_FOLDED_SAMPLING is deprecated"), so it has no effect. Recipe and cookbook both carry it.
    • "Agg TP4", all three points: --tensor-parallel-size 4 --expert-parallel-size 1 --chunked-prefill-size 16384 --mem-fraction-static 0.8 --tokenizer-backend fastokens and SGLANG_RAGGED_VERIFY_MODE=static.
      • C1: --max-running-requests 1 --cuda-graph-max-bs-decode 64 --prefill-decode-interval 0 --enable-decoder-swa-bounded-replay --cuda-graph-backend-prefill disabled, with SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=0.
      • C4: --max-running-requests 128 --cuda-graph-max-bs-decode 128 --prefill-decode-interval 0 --swa-prefix-tails 4096, with SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1 and SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT=per_rank.
      • C16: --max-running-requests 128 --cuda-graph-max-bs-decode 128 --prefill-decode-interval 192 --swa-prefix-tails 4096 --tokenizer-worker-num 4, with SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1, SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT=per_rank and SGLANG_NUMA_BIND_V2=0.
    • "Agg TP2·EP2", both points: --tensor-parallel-size 2 --expert-parallel-size 2 --chunked-prefill-size 16384 --enable-decoder-swa-bounded-replay --cuda-graph-backend-prefill disabled --cuda-graph-max-bs-decode 64 --mem-fraction-static 0.8 --tokenizer-backend fastokens --tokenizer-worker-num 4, with SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1, SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT=per_rank and PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True.
      • C16: --prefill-decode-interval 96 --swa-prefix-tails 1024 --max-running-requests 32, with SGLANG_NUMA_BIND_V2=0.
      • C32: --prefill-decode-interval 48 --swa-prefix-tails 2048 --max-running-requests 64.
    • "Agg DEP2", all three points:
      • Arguments: --tensor-parallel-size 2 --data-parallel-size 2 --expert-parallel-size 2 --enable-dp-attention --enable-dp-lm-head --enable-dp-attention-local-control-broadcast --load-balance-method total_requests --moe-a2a-backend megamoe --schedule-policy shortest-prefill-first --tokenizer-worker-num 4 --chunked-prefill-size 16384 --swa-prefix-tails 4096 --enable-decoder-swa-bounded-replay.
      • Environment: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1, SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT=per_rank, SGLANG_RAGGED_VERIFY_MODE=static, SGLANG_NUMA_BIND_V2=0, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True.
      • C64: --prefill-decode-interval 24 --max-running-requests 128 --cuda-graph-max-bs-decode 64 --mem-fraction-static 0.8 --tokenizer-backend fastokens.
      • C128: --prefill-decode-interval 16 --max-running-requests 256 --cuda-graph-max-bs-decode 128 --mem-fraction-static 0.85 --tokenizer-backend fastokens.
      • C160: the C128 settings with --prefill-decode-interval 8 and --tokenizer-backend huggingface.
    • Router, "Agg DEP2": python3 -m sglang_router.launch_router --policy cache_aware --request-id-headers x-correlation-id --dp-aware --connect-timeout-secs 900 --request-timeout-secs 14400 --disable-health-check --disable-retries --worker-urls <worker>.
  • I resolved the cookbook commands for each of the 8 points, following the cookbook renderer's ref/from/diff/patch semantics. I diffed them against the logged worker command, the logged Env: line, the full worker-container environment (fingerprint) and the logged sglang-router command in run 38614048439. Every recipe argument and recipe environment variable matches, and the router arguments match exactly. The only differences are the InferenceX-only settings below and placeholders (ports, IP addresses, worker URL).
    • The pinned image auto-resolves backends the recipe does not set: attention_backend=dsv4 and dsv4_attn_backend=auto everywhere, and moe_runner_backend=flashinfer_mxfp4 on the TP4 and TP2·EP2 points ("Use flashinfer_mxfp4 as MoE runner backend for DeepseekV4ForCausalLM"). The DEP2 points use MegaMoE through --moe-a2a-backend megamoe. The KV cache is fp8_e4m3, the image default for this checkpoint.
    • Under DP attention, this image maps --data-parallel-size 2 to attn_dp_size=2 and splits --max-running-requests and --chunked-prefill-size across the 2 attention-DP ranks. The logs show 64 / 128 / 128 and 8192 per rank. The recipe and the cookbook pass the same CLI values.
  • Excluded as InferenceX-only:
    • --enable-metrics. This is the Prometheus export that AIPerf scrapes for result collection (AIPERF_REQUIRED_SERVER_METRIC_PREFIX=sglang:); it does not affect scheduling or kernels.
    • --host, --port, --nccl-port, the router's --worker-urls, CUDA_VISIBLE_DEVICES and HEAD_NODE_IP. These are srt-slurm process placement.
    • PYTHONUNBUFFERED, PYTHONNOUSERSITE, SGLANG_CUDA_COREDUMP_BEFORE_CRASH=0 and SGLANG_PYSPY_DUMP_BEFORE_CRASH=0. These are log buffering, Python path isolation and crash-dump toggles.
    • SGLANG_SIMULATE_ACC_*, the synthetic acceptance control. The cookbook discloses it (synthetic-acceptance) with the golden AL of 3.51.
    • The rest of the GB300-NV container environment in the worker fingerprint. NCCL_IB_*, NCCL_SOCKET_IFNAME, NCCL_MNNVL_ENABLE, NCCL_NET_GDR_C2C, NVSHMEM_*, UCX_NET_DEVICES and OMPI_MCA_* are the cluster's NIC names and fabric wiring, which a GB300 NVL72 operator sets for their own fabric. NCCL_DEBUG, UCX_LOG_LEVEL and HF_* are log levels and cache paths. SGLANG_BUILD_*, SGLANG_IMAGE_TAG and SGLANG_RUST_BUILD_MODE are image metadata. None of them is a serving setting.
    • The AIPerf client settings (CONC, AGENTIC_WARMUP_GRACE_PERIOD, AIPERF_*), the DRAM budget (dram-utilization), the srt-slurm health_check and observability blocks, and Slurm settings.
• PR validation: Run Sweep 38614048439 ran on commit a61b7eba2e536cce11ffe7f2620bbea4e8fbed85 (pull_request, full-sweep-enabled), which is still in the PR. It completed with success on attempt 2.
  • The current head, 86ace5cf836ad92bf302c3d39d3ff48b2f98af2c, adds one [skip-sweep] commit on top that only corrects the fastokens version in the changelog description. The recipe, master config and image are unchanged, and no sweep ran for it.
  • It is pinned with /use 38614048439 and staged (staged dashboard).
  • Its bmk_agentic_*, results_bmk, eval_results_all and server-log artifacts are unexpired. Attempt 2's expire 2027-01-14; C1's attempt-1 artifacts expire 2027-01-12.
  • All 8 benchmark jobs succeeded: TP4 C1 (canary), C4 and C16; TP2/EP2 C16 and C32; TP2/EP2 with DP2 attention C64, C128 and C160. Every point recorded 0 error-dropped requests.
  • Both agentic eval / jobs succeeded.
  • Attempt 1 measured only the C1 canary (2026-10-14 22:17-23:57Z). The other 7 benchmark jobs and both eval jobs were cancelled on 2026-10-15 at 04:33-04:36Z before any of them finished, and they uploaded no results: attempt 1's results_bmk holds only C1, and its eval_results_all is empty. Attempt 2 reran only those cancelled jobs. Its C1 canary entry is the carried-over attempt-1 job (same runner and timestamps), and its artifacts are attempt 1's.
  • The skipped jobs are the empty fixed-sequence and multi-node lanes.
  • Checkpoint revision: C1 loaded deepseek-ai/DeepSeek-V4.1-Flash at 5254448103b6254fc646dacbca1cbbf20e9216c1. The attempt-2 points loaded 314113f8a7dadb388215396ba5fbee2c610a04d6, pushed 2026-10-16 09:27Z ("fix(encoding): preserve tool image order and current-turn thinking"). Between the two revisions only the reference scripts encoding/encoding.py, encoding/test_encoding.py and encoding/README.md changed. Every weight shard, config.json, the tokenizer files and the chat template have identical blob/LFS ids, and the server does not load encoding/. All 8 points therefore serve the same model.
• Evals: gsm8k em_strict, both n=1319 with infrastructure_success: true, from eval_results_all:
  • 0.9750 at TP2/EP2 C32 (job).
  • 0.9742 at TP2/EP2 with DP2 attention, C160 (job).
  • Both clear the 0.90 default in infx/evals/thresholds.yaml. The lookup is by exact model prefix and has no dsv41flash override. They also clear the 0.91 dsv4 value.
  • Both ran on the config's image digest (ddae0e39…, SGLANG_BUILD_COMMIT=684fea8a…) with real DSpark verification. The eval server environment has no SGLANG_SIMULATE_ACC_*, and the measured mean accept length is 3.93 (C32) and 4.21 (C160).
• Image: upstream lmsysorg/sglang:nightly-dev-cu13-20261014-684fea8a, pinned by digest sha256:ddae0e399d3d94f7827f16183301cf8dac4cf0eae0ef25590b7dac5881d713d8.
  • SGLang 684fea8af3b11a5b51ec9966c02959a32ad3421c (SGLANG_BUILD_COMMIT in the logged environment; the logged package is sglang 0.0.0.dev1+g684fea8af). Both upstream changes the PR cites, [Tokenizer] Require fastokens>=0.3.2 and include it in the all extra sgl-project/sglang#43104 and #43812, are ancestors of that commit.
  • The config is framework: sglang, an open-source engine entry, so the engine-ordering item is satisfied by this submission itself.
• Chat template: AgentX replay runs agentx profile, and the AIPerf export of every point records endpoint type chat with streaming.
  • On every point, the only inference route in the worker logs is POST /v1/chat/completions (355 at TP4 C1 to 18,912 at DEP2 C160). There is no /v1/completions traffic.
  • The server applies the checkpoint's chat template (chat_template=None, no override). It logs "Auto-detected --reasoning-parser as 'deepseek-v41' from chat template" and the deepseekv41 tool-call parser.
  • Every job logs THINKING_MODE: thinking_on, with SGLANG_DEFAULT_THINKING=1 and reasoning effort high.
• Draft precision: the served draft is the DSpark head bundled in deepseek-ai/DeepSeek-V4.1-Flash (identical weights at both revisions above).
  • The head is mtp.0-mtp.2: 3 stages, block size 5, 128 routed experts per stage, and 2,401 tensors in shards 44-46.
  • Every point logs "DSpark draft weights are bundled in the target checkpoint; defaulting --speculative-draft-model-path to --model-path" and type=DeepseekV4ForCausalLMDSpark, quant=fp8.
  • Shipped (safetensors headers):
    • The attention projections (wq_a, wq_b, wkv, wo_a, wo_b), the shared experts and main_proj are F8_E4M3 with F8_E8M0 block scales.
    • The 384 routed experts are packed FP4 (I8 storage) with F8_E8M0 group-32 scales, which is MXFP4.
    • The norms, the gate weight, markov_head and confidence_head are BF16. attn_sink, the gate bias and hc_* are F32.
    • config.json has quant_method: fp8, dynamic activations, weight_block_size: [32, 32], scale_fmt: ue8m0 and expert_dtype: fp4.
  • Pinned image's default handling: --speculative-draft-model-quantization is not set, so the draft inherits the target's checkpoint-detected quantization (serving_hook.py#L655-L668). All 8 points log quantization=None, speculative_draft_model_quantization=None and _speculative_draft_quantization_explicitly_set=False.
    • wo_a: the image dequantizes the shipped FP8 wo_a to BF16 at load (deepseek_v4_dspark.py#L108 wo_a_fp8=False, #L1014 _dequant_fp8_wo_a_streaming). Every point logs "Finished streaming dequant fp8 wo_a". This is the default conversion that the verifier treats as the baseline.
    • Routed experts on the TP4 and TP2·EP2 points: the draft's experts stay MXFP4 on the image's auto-selected flashinfer_mxfp4 runner ("Shuffling FP4 expert weights for TRT-LLM MxFP4 kernel" for all 3 draft stages). On SM100 that runner uses MXFP8 activations (mxfp4.py#L414-L425).
    • Routed experts on the DEP2 points (MegaMoE): enable_w4a4_mxfp4_megamoe=False, and speculative_enable_w4a4_mxfp4_megamoe=None on every point. The draft therefore inherits the target's value (moe/utils.py#L609-L615, fused_moe_triton/layer.py#L404). Its MegaMoE MMA type is fp8xfp4, FP8 activations against the unrequantized MXFP4 weights (mega_moe.py#L56-L60). The W4A4 mxf4xmxf4 path is not used.
    • The Markov head stays BF16 as shipped: SGLANG_DSPARK_OPT_MARKOV_W2_BF16=True restates the EnvBool(True) default (environ.py#L542).
    • Draft sampling is at the image default. SGLANG_DSPARK_FOLDED_SAMPLING is the inert deprecated variable described above, and SGLANG_DSPARK_USE_GREEDY_DRAFT and SGLANG_DSPARK_FOLDED_PROPOSAL are unset.
  • Effective: as shipped.
    • FP8 attention, shared experts and main_proj; BF16 wo_a through the default conversion; MXFP4 routed experts with MXFP8/FP8 activations; BF16 heads.
    • The logged server args of all 8 points show dtype=auto, json_model_override_args='{}' and speculative_draft_kv_cache_dtype=None. The draft KV follows the target's default fp8_e4m3.
  • SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is absent from the recipe, the shared launchers and helpers under inferencex-e2e/benchmarks, runners and infx at the head, every point's worker environment and the logs.
    • It defaults to EnvBool(False) in environ.py#L1057.
    • It would only act on modelopt_fp4 checkpoints with the deep_gemm runner (deepseek_common/utils.py#L131-L134).
    • SGLANG_GLM_NEXTN_MOE_PTPC and SGLANG_NVFP4_CKPT_FP8_GEMM_IN_ATTN are unset for this config.
• Golden AL: infx/golden_al_distribution/dsv41flash_dspark.yaml gives 3.51 for thinking_on with 5 speculative tokens, matching --speculative-dspark-block-size 5.
  • Every benchmark point logged SGLANG_SIMULATE_ACC_LEN=3.51, SGLANG_SIMULATE_ACC_METHOD=match-expected and SGLANG_SIMULATE_ACC_TOKEN_MODE=real-draft-token. The values are injected by infx/srt_slurm/synthetic_acceptance.py from the golden curve ("Applied override: --set roles.agg.env.SGLANG_SIMULATE_ACC_LEN=3.51"), not set in the recipe.
  • The mean accept len in the decode logs is 3.507 to 3.512 on every point.
• MODELS.md: dsv41flash agentic coding is active (SGLang DSpark arms per SKU).
  • No KV cache offload is used: kv-offloading: none, and HiCache is off on every point (enable_hierarchical_cache=False). The Engram host table, enabled on all points except TP4 C1, holds the model's Engram tables row-sharded per rank in host DRAM, not KV cache.
• No architecture hacks: there is no --hf-overrides, json_model_override_args is '{}', and no model files are edited. All 3 DSpark stages run, and the target runs its native layers.
• Dynamo TRT-LLM metrics: N/A. This config is framework: sglang. It serves through sglang.launch_server and, at DEP2, sglang_router, with no Dynamo or TRT-LLM process in the serving path. enable-metrics is on.
• Code quality:
  • The diff retunes this recipe's overrides (fastokens, DEP2 bounded replay, TP4 C1 single admission, new DEP2 C160), bumps the image digest, adds C160 to the DEP2 search-space row, and appends one changelog entry (12 added lines, nothing changed or removed). check-changelog passed.
  • The only change after the measured commit a61b7eba is the one-line changelog wording fix in 86ace5cf, so there is no recipe drift.
  • The only merge conflict with current main is in perf-changelog.yaml, and this PR's change to it is append-only.
  • The bot review's pull/XXXX placeholder finding is fixed; the entry uses the canonical pull/4033 link.
  • Admission and graph batch cover the observed per-rank decode batches:
    • TP4: peak running requests 1 / 5 / 9 at C1 / C4 / C16, against max-running-requests of 1 / 128 / 128.
    • TP2/EP2: 11 / 22 at C16 / C32, against 32 / 64.
    • DEP2: per-rank peaks 29 / 46 / 57 at C64 / C128 / C160, against 64 / 128 / 128 per rank and a decode graph batch of 64 / 128 / 128.
  • Two non-blocking notes that do not change the served configuration:
    • The unchanged base environment still sets SGLANG_DSPARK_FOLDED_SAMPLING, which is inert in this image.
    • The PR description and changelog said the image "ships fastokens 0.3.2". The image ships fastokens==0.3.4, and [Tokenizer] Require fastokens>=0.3.2 and include it in the all extra sgl-project/sglang#43104 only sets the fastokens>=0.3.2 floor. Both now state 0.3.4 (description edited; changelog in 86ace5cf).
• No engine patches: the diff adds no patch files, git apply, heredoc rewrites, site-packages edits, monkey-patching or forked engine wheels. No serving-stack package is installed on top of the image; the only installs are the host-side srt-slurm orchestrator venv and the benchmark client.
  • runners/srt-slurm/patches/ on main holds only 573-participating-gpus.patch, an srt-slurm power-telemetry harness backport. It does not touch SGLang.
  • No waiver is needed.
• append-only: N/A, because the single new changelog entry does not use append-only: true (append-only: false in the run's changelog metadata).
• Pareto: one curve, dsv41flash / agentic-coding / gb300_sglang / fp4 / run 38614048439, counted with infx.workflows.pareto_coverage from upstream main (x = E2EL in seconds, y = total token throughput per GPU; 4 GPUs at TP4, 2 GPUs at TP2/EP2 and DEP2).
  • Source: the 8 bmk_agentic_* artifacts of the run (request_metrics.throughput.per_gpu.total_tput_tps and request_metrics.latency.e2el). Every point uses the same image and the measured commit.
  • P90 E2EL frontier: 7 of 8 points (TP4 C4 and C16, TP2/EP2 C16 and C32, DEP2 C64, C128 and C160). TP4 C1 is dominated by TP4 C4.
  • P75 E2EL frontier: 7 of 8 points, the same 7. TP4 C1 is dominated by TP4 C4.
  • Per point (tok/s/GPU, P90 E2EL s, P75 E2EL s): TP4 C1 7842.7, 8.094, 4.457; TP4 C4 10100.7, 6.199, 2.668; TP4 C16 29456.4, 6.712, 3.050; TP2/EP2 C16 56200.2, 9.275, 4.073; TP2/EP2 C32 114813.8, 13.925, 6.570; DEP2 C64 211867.0, 18.651, 9.291; DEP2 C128 279528.8, 30.553, 13.721; DEP2 C160 288554.3, 41.937, 18.522.
  • Helper output: dsv41flash/agentic-coding/gb300_sglang/fp4/run38614048439/p90_e2el is PASS with measuredPoints: 8, invalidPoints: 0, frontierPoints: 7; .../p75_e2el is PASS with measuredPoints: 8, invalidPoints: 0, frontierPoints: 7.
  • Dashboard: https://inferencex.semianalysis.com/inference?unofficialRun=38614048439
• NVIDIA overview (advisory): the NVIDIA recipes overview Framework Recipes table lists DeepSeek-V4.1-Flash with SGLang ✅, linking to the same DeepSeek-V4.1 cookbook page.

Signed: nvpohanh

[github-actions]

✅✅✅ Verdict: PASS ✅✅✅

Passed and not applicable checks

✅ Check 0 (CODEOWNER): PASS — @nvpohanh is a listed owner of inferencex-e2e/configs/nvidia-master.yaml and of **/gb300-*/ srt-slurm recipes; perf-changelog.yaml falls under the catch-all.

✅ Check 1 (Sweep on in-PR commit): PASS — commit a61b7eba (still in the PR) has all 8 agentic / and both agentic eval / check-runs at success in run 38614048439; the fixed-sequence single-node */ and eval / lanes are empty for this agentic-only config. Head 86ace5cf is unchanged and only edits the changelog wording.

✅ Check 2 (Evals): PASS — eval_results_all gives gsm8k em_strict 0.9750 (TP2/EP2 C32) and 0.9742 (DEP2 C160), n=1319, infrastructure_success: true, both on image digest ddae0e39…. Both are above the 0.90 bar.

✅ Check 3 (Upstream recipe): PASS — sgl-project/sglang#44712 is MERGED (2026-10-16T11:31:09Z). I resolved its deepseek-v4_1.jsx ref/diff/patch chain for all 8 GB300 points: Agg TP4 C1/C4/C16, TP2·EP2 C16/C32, and DEP2 C64/C128/C160 with the sglang-router command. Every argument and environment variable matches the pinned recipe, including fastokens, bounded replay, C1 single admission and C160 HF tokenizer/interval 8. The published page carries these entries.

✅ Check 4 (Reuse command): PASS — /use 38614048439 posted by nvpohanh (COLLABORATOR).

✅ Check 5 (Checklist template): PASS — all 18 current-template items are present and checked.

✅ Check 6 (Upstream image / ordering): PASS — framework: sglang with upstream lmsysorg/sglang:nightly-dev-cu13-20261014-684fea8a@sha256:ddae0e39…; this is an open-source engine entry, so the ordering rule does not apply.

✅ Check 7 (Deprecated models): PASS — dsv41flash agentic coding is active in MODELS.md as of 2026-10-16.

✅ Check 8 (No architecture hacks): PASS — no --hf-overrides, JSON model overrides, or model-file edits in the diff or recipe.

✅ Check 9 (Chat template): PASS — AgentX replay drives the chat endpoint, and the server uses the checkpoint's chat template with thinking on.

✅ Check 10 (No engine patches): PASS — the diff has no patch files, inline source rewrites, monkey-patching, or engine wheel installs.

✅ Check 11 (Golden AL): PASS — dsv41flash_dspark.yaml gives thinking_on, 5 tokens = 3.51. The job logs show SGLANG_SIMULATE_ACC_LEN=3.51 with match-expected / real-draft-token, injected by synthetic_acceptance.py.

➖ Check 12 (Append-only): N/A — the new changelog entry does not set append-only: true.

✅ Check 13 (Draft precision): PASS — the bundled DSpark head is served with no draft quantization or dtype override. In the pinned 684fea8a, the default FP8→BF16 wo_a conversion (deepseek_v4_dspark.py#L108,#L1014) is the baseline. SGLANG_DSPARK_OPT_MARKOV_W2_BF16 restates the EnvBool(True) default. SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is unset in the recipe and launchers, and defaults to EnvBool(False) (environ.py#L1057).

✅ Check 14 (Pareto coverage): PASS — one curve, dsv41flash/agentic-coding/gb300_sglang/fp4/run38614048439. Using the 8 bmk_agentic_* artifacts (same image) and pareto_coverage, P90 E2EL has 7/8 frontier points and P75 E2EL has 7/8. TP4 C1 is dominated in both.

✅ Check 15 (PR description): PASS — the body's claims (image, fastokens 0.3.4, per-point changes, new DEP2 C160) match the pinned recipe and master config.

✅ Check 16 (NVIDIA overview): PASS — the NVIDIA overview Framework Recipes table lists DeepSeek-V4.1-Flash with SGLang ✅, linking to the cookbook page.

➖ Check 17 (Metrics publication): N/A — direct framework: sglang serving with no Dynamo/TRT-LLM settings; enable-metrics: true is set.

Assessed commit: 86ace5cf836ad92bf302c3d39d3ff48b2f98af2c.