[weireweire]

Move dsv41flash-fp4-gb300-sglang-agentic-dspark (single-node SGLang AgentX on GB300) to digest-pinned lmsysorg/sglang:nightly-dev-cu13-20261030-302b78cc, and apply the low-latency settings tuned on the Dynamo + SGLang aggregated row.

• Image: from nightly-dev-cu13-20261021-c1c287a4 to nightly-dev-cu13-20261030-302b78cc.
  • It includes [Fix] dspark: Draw the simulated-acceptance bonus at the simulated row sgl-project/sglang#46404, which draws the DSpark simulated-acceptance bonus token at the simulated row. Before it, the bonus came from the real mismatch row and the committed text degenerated, which made simulated throughput look better than real.
  • It includes #46840, which keeps the real verify accept cost under simulated acceptance. The previous pin, nightly-dev-cu13-20261029-c10e0e17, was built before it.
  • It also includes #46891, which removes a per-step host sync in the DSpark draft-block metadata.
  • It bundles sgl-nvidia-mega 54b428921749 ([Fix] docker: update NVIDIA megakernel pin to tested commit sgl-project/sglang#47145), which implements megakernel API 2 as SGLang requires.
• TP4 c1: runs the batch-1 NVIDIA megakernel, as the Dynamo + SGLang TP4 c1 row does: --enable-nvidia-megakernel, fp8_e4m3 KV cache, mem-fraction-static 0.91, decode graph batch 1 and the Engram host table. Target and draft megakernels are active on all four TP ranks on this image.
  • The 16 runtime-optimization variables that recipe set one by one (the DSpark fold, metadata-glue and graph-capture switches from [Perf] [NVIDIA] dspark: Move the host work between DSv4.1 decode graphs into the graphs for the megakernel path sgl-project/sglang#45450) are replaced by SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION=1 ([Feat] [NVIDIA] runtime: Add a master switch for megakernel runtime optimizations sgl-project/sglang#47105). The switch also turns on the Qwen3.5/EAGLE runtime group, most of which is inactive on this path.
  • SGLANG_OPT_DSV41_DSPARK_SIM_DRAW_AHEAD stays explicit. It moves the simulated acceptance draw ahead of the draft launch and is inactive with real acceptance. The simulation settings, megakernel slots and NCCL settings also stay explicit.
• TP4 c1/c4/c16 and TP2/EP2 c16/c32: pin dsv4-attn-backend: flashmla, as the Dynamo + SGLang aggregated rows do. auto resolves to FlashMLA MegaAttention, which zero-pads per-rank attention heads to 64 (4x at TP4, 2x at TP2). The DP-attention points (c64/c128/c160) are unchanged.
• The search space is unchanged.

Risk: this image stores the DeepSeek-V4.1 KV cache in about 55 % of the previous bytes per token, so a given mem-fraction-static holds more tokens and commits more memory. On a DEP16 decode worker this left too little memory for CUDA graph capture at 0.85. The previous sweep of this PR on nightly-dev-cu13-20261028-586355a8 started every point except the megakernel one. If a point fails to start on this image, its memory setting will be lowered.

中文

将 dsv41flash-fp4-gb300-sglang-agentic-dspark（GB300 单节点 SGLang AgentX）更新到固定 digest 的 lmsysorg/sglang:nightly-dev-cu13-20261030-302b78cc，并采用在 Dynamo + SGLang 聚合式配置上调好的低延迟参数。

• 镜像： 从 nightly-dev-cu13-20261021-c1c287a4 升级到 nightly-dev-cu13-20261030-302b78cc。
  • 它包含 [Fix] dspark: Draw the simulated-acceptance bonus at the simulated row sgl-project/sglang#46404：DSpark 模拟接受的 bonus token 改为在模拟位置采样。此前 bonus 取自真实的不匹配位置，提交的文本会退化，使模拟吞吐好于真实情况。
  • 它包含 #46840：模拟接受下保留真实 verify accept 的开销。上一版固定的 nightly-dev-cu13-20261029-c10e0e17 构建于它合入之前。
  • 它还包含 #46891，去掉了 DSpark draft-block metadata 中每步的一次主机同步。
  • 它自带 sgl-nvidia-mega 54b428921749（[Fix] docker: update NVIDIA megakernel pin to tested commit sgl-project/sglang#47145），实现 SGLang 要求的 megakernel API 2。
• TP4 c1： 与 Dynamo + SGLang TP4 c1 一样运行 batch-1 NVIDIA megakernel：--enable-nvidia-megakernel、fp8_e4m3 KV cache、mem-fraction-static 0.91、decode graph batch 1、Engram host table。在该镜像上，4 个 TP rank 的 target 和 draft megakernel 均生效。
  • 原先逐个设置的 16 个运行时优化变量（来自 [Perf] [NVIDIA] dspark: Move the host work between DSv4.1 decode graphs into the graphs for the megakernel path sgl-project/sglang#45450 的 DSpark fold、metadata glue 和 graph capture 开关）改为 SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION=1（[Feat] [NVIDIA] runtime: Add a master switch for megakernel runtime optimizations sgl-project/sglang#47105）。该开关同时打开 Qwen3.5/EAGLE 那组优化，在此路径上大多不生效。
  • SGLANG_OPT_DSV41_DSPARK_SIM_DRAW_AHEAD 仍显式设置：它把模拟接受的采样提前到 draft launch 之前，真实接受时不起作用。模拟设置、megakernel slots 和 NCCL 设置也仍显式指定。
• TP4 c1/c4/c16 和 TP2/EP2 c16/c32： 与 Dynamo + SGLang 聚合式配置一样固定 dsv4-attn-backend: flashmla。auto 会选 FlashMLA MegaAttention，它把每个 rank 的 attention head 零填充到 64（TP4 为 4 倍，TP2 为 2 倍）。DP attention 的点（c64/c128/c160）不变。
• 搜索空间不变。

风险： 该镜像中 DeepSeek-V4.1 每个 token 的 KV 只占原来约 55% 的字节，同样的 mem-fraction-static 能放更多 token，实际占用的显存也更多。在 DEP16 decode worker 上，0.85 时剩余显存不够 CUDA graph capture。本 PR 上一轮在 nightly-dev-cu13-20261028-586355a8 上的 sweep 中，除 megakernel 点外的所有点都能启动；如在新镜像上有点无法启动，会调低其显存设置。

AI model disclosure

• Claude Opus 5.5 (runtime identifier aws/anthropic/eccn-claude-opus-5-5) chose the image, ported the tuned settings from the Dynamo + SGLang aggregated row, ran the changelog and recipe checks, and wrote this description. Cluster validation runs through a delegated agent on the same model.

[claude]

Code review found no issues

No high-confidence issues detected in this change.

This review covers commit 67af7c3, which is no longer the latest commit on this pull request; later commits are not covered by it.

[github-actions]

View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=41261902147

View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=41261902147

[nvpohanh]

[by Claude Code] The SGLang cookbook integration of this PR's AgentX config (dsv41flash-fp4-gb300-sglang-agentic-dspark) will happen in sgl-project/sglang#47214, in the Agentic Long-Context Deployment section of the DeepSeek-V4.1 cookbook page.

This PR has no authorized /use <run_id> comment yet, so the cookbook shows the recipe at this PR's head (9bd047ac7) as "in progress". A later cookbook PR switches it to the measured run once a run is pinned or this PR merges.

[nvpohanh]

[by Claude Code] sgl-project/sglang#47214 has been merged. This PR's AgentX config (dsv41flash-fp4-gb300-sglang-agentic-dspark) is now in the Agentic Long-Context Deployment section of the SGLang DeepSeek-V4.1 cookbook: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4_1#agentic-long-context-deployment

This PR has no authorized /use <run_id> comment yet, so the cookbook shows the recipe at this PR's head (9bd047ac7) as "in progress". A later cookbook PR switches it to the measured run once a run is pinned or this PR merges.

[nvpohanh]

[by Claude Code] The SGLang cookbook update of this PR's AgentX config (dsv41flash-fp4-gb300-sglang-agentic-dspark) will happen in sgl-project/sglang#47464, in the Agentic Long-Context Deployment section of the DeepSeek-V4.1 cookbook page. The config has been live in that section since sgl-project/sglang#47214; sgl-project/sglang#47464 refreshes it with the pin below.

This PR has no authorized /use <run_id> comment yet, so the cookbook shows the recipe at this PR's head (9bd047ac7) as "in progress". A later cookbook PR switches it to the measured run once a run is pinned or this PR merges.

[nvpohanh]

/use 40809149952

[github-actions]

@nvpohanh staged run 40809149952.

• DeepSeek-V4.1-Flash · agentic-traces · FP4 · gb300_sglang · 2026-10-29

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

[nvpohanh]

[by Claude Code] sgl-project/sglang#47464 has been merged. This PR's AgentX config (dsv41flash-fp4-gb300-sglang-agentic-dspark) is now in the Agentic Long-Context Deployment section of the SGLang DeepSeek-V4.1 cookbook: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4_1#agentic-long-context-deployment

This PR's /use 40809149952 comment came after this cookbook round was synced, so the cookbook shows the recipe at 9bd047ac7 as "in progress". That is the commit run 40809149952 measured, so the commands are already the measured ones; a later cookbook PR marks them verified.

[nvpohanh]

As a PR reviewer and CODEOWNER, I have reviewed this and have:

Checklist items

• Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
• Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
• Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
• Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
• Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
• Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison.
• For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
• Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
• Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
• Verified that new models disable integer all-reduce, with SGLang ROCm explicitly setting ROCM_QUICK_REDUCE_QUANTIZATION=NONE. Existing models are grandfathered per MODELS.md.
• For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
• If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
• For official InferenceX (non-preview), verified that established models use upstream engine images built from master (or the equivalent default development branch), with new models transitioning within a reasonable time after release. See MODELS.md.
• If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
• Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
  • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
• Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
• If this PR uses append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it.
• If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
• Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.

Additional detail section:

• Assessed head: 41d29b2834a85fcee09db6f9f52653b7c2f30ef3, which is the current PR head. The pinned run is Run Sweep 41261902147 (pull_request, head 41d29b28, conclusion success after attempt 2). /use 41261902147 is posted and is the latest /use on this PR. It replaces the earlier /use 40809149952. No commits follow 41d29b28. This sign-off replaces the one for 9bd047ac.
• Changes since the previous sign-off (9bd047ac):
  • 6bb31166 moves the image from nightly-dev-cu13-20261029-c10e0e17 to nightly-dev-cu13-20261030-302b78cc (digest d4b7701d…) so that the run includes [Fix] dspark: Keep the real verify accept cost under simulated acceptance sgl-project/sglang#46840 (simulated acceptance keeps the real verify accept cost). It also updates this PR's changelog entry and the master comment ("every point uses the host-DRAM Engram table").
  • cb232ed0 and 41d29b28 are clean merges of main.
  • The recipe overrides are unchanged since 9bd047ac.
• Code quality:
  • The diff against main is config-only. It touches the override_tp2_c16, override_tp2_c32, override_tp4_c1, override_tp4_c4 and override_tp4_c16 overrides in dsv41flash/sglang/gb300-fp4-mtp/agentic.yaml, and the image pin and comment of dsv41flash-fp4-gb300-sglang-agentic-dspark in nvidia-master.yaml.
  • It appends one perf-changelog.yaml entry at the end of the file and edits no existing entries. The search space is unchanged (8 points).
• PR validation:
  • In run 41261902147, all 8 agentic / benchmark jobs succeeded: TP4 c1/c4/c16, TP2/EP2 c16/c32, and TP2/EP2/DPA c64/c128/c160. All 6 agentic eval / jobs, check-changelog and collect-evals also succeeded.
  • Attempt 1 lost 3 benchmark jobs (TP4 c16, DPA c64/c128) and 2 eval jobs (TP4 c16, TP2/EP2 c16) to an infrastructure error. The AIPerf client failed to load the semianalysisai/cc-traces-weka-062126 dataset and the tokenizer from the HuggingFace Hub (DatasetLoaderError, "Network or API error communicating with HuggingFace Hub") after the server was up. Attempt 2 reran only those jobs on the same commit, and all of them passed.
  • The skipped jobs are the fixed-sequence, multi-node and non-agentic eval lanes, which this config does not generate. Every bmk_agentic_* and eval_* artifact is unexpired until 2027-01-30.
• Evals (gsm8k em_strict, n=1319; from eval_results_all on the PR image):
  • Scores: TP4 c1 0.9735, TP4 c4 0.9757, TP4 c16 0.9757, TP2/EP2 c16 0.9735, TP2/EP2 c32 0.9757, and TP2/EP2/DPA c160 0.9689.
  • Each score clears the 0.90 default in infx/evals/thresholds.yaml, which has no dsv41flash entry, and the stricter 0.91 dsv4 value.
  • The largest sibling gap is 0.0068 (DPA c160 against 0.9757), within 2σ = 0.0127. All rows are within 2σ of the previous run 40809149952 (0.9719–0.9750).
  • The TP4 c1 eval server ran the changed path: enable_nvidia_megakernel: True, --enable-nvidia-megakernel: DeepSeek-V4.1 on sm_103, dsv4_attn_backend: flashmla and kv_cache_dtype: fp8_e4m3.
• Chat template: The AgentX client replays through the chat endpoint with thinking on, which the server reinforces through SGLANG_DEFAULT_THINKING=1 and SGLANG_DSV41_REASONING_EFFORT=high (TP4 c1 fingerprint env). This is the thinking-on regime of the golden curve.
• Draft precision:
  • The draft is the DSpark head bundled in deepseek-ai/DeepSeek-V4.1-Flash. The server logs of TP4 c1 and DEP2 c64 show speculative_draft_model_path: deepseek-ai/DeepSeek-V4.1-Flash and Draft checkpoint bundles a DSpark head; loading draft arch DeepseekV4ForCausalLMDSpark, with no substituted checkpoint.
  • Shipped precision: The checkpoint has quant_method: fp8, weight_block_size: [32, 32], scale_fmt: ue8m0 and expert_dtype: fp4, which means MXFP8 dense weights and MXFP4 experts. By default, the pinned image loads the DSpark wo_a as BF16 and keeps markov_w2 in BF16 (SGLANG_DSPARK_OPT_MARKOV_W2_BF16 = EnvBool(True) in environ.py at the GB300 build commit 52604e1d). The recipe's 'True' restates that default.
  • Between the previously verified c10e0e17 and 52604e1d, the only change to deepseek_v4_dspark.py / dspark.py adds a vocab-shard width field. No draft load dtype changes.
  • Effective precision: It matches the shipped baseline. Both logs show dtype: auto, quantization: None, speculative_draft_model_quantization: None and speculative_draft_kv_cache_dtype: None, so the draft KV inherits the target's. TP4 c1 runs speculative_moe_runner_backend: flashinfer_mxfp4 on the stored MXFP4 experts.
  • TP4 c1 kv-cache-dtype: fp8_e4m3: This restates the image default. DEP2 c64, which does not set the flag, also resolves kv_cache_dtype: fp8_e4m3. It applies to both target and draft.
  • Megakernel draft path: The image bundles sgl-nvidia-mega 54b428921749 (the TP4 c1 fingerprint shows sgl-nvidia-mega.git@54b42892…), the same plugin commit verified for 9bd047ac. The draft_* slots reuse the weights SGLang already loaded.
  • SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION=1: It only flips scheduling and fusion defaults. None of them lowers precision.
  • SGLANG_SPEC_TP_SYNC=rng,-dspark-accept-sample (TP4 c1): This answers the open review question. At 52604e1d (speculative/spec_tp_sync.py), it only selects which speculative decisions rank 0 broadcasts to its TP group. rng keeps the init/memory and RNG sites, and -dspark-accept-sample drops the broadcast of the sampled accept length and bonus at the DSpark accept site. It does not touch weights, dtypes or the draft computation, and it does not change the simulated acceptance target. The measured TP4 c1 mean accept length is 3.510 over 5,107 decode log lines (golden 3.51), and the TP4 c1 eval passes at 0.9735.
  • SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE: It is not enabled. It is absent from the recipe, the generated config.yaml / sbatch_script.sh / recipe.lock.yaml, the fingerprint env and the server logs of TP4 c1 and DEP2 c64, and environ.py at 52604e1d defines it as EnvBool(False).
• Golden AL:
  • The generated config and fingerprint show SGLANG_SIMULATE_ACC_LEN=3.51, SGLANG_SIMULATE_ACC_METHOD=match-expected and SGLANG_SIMULATE_ACC_TOKEN_MODE=real-draft-token. This matches dsv41flash_dspark.yaml deepseek-v4.1-flash.thinking_on.5 = 3.51 for speculative_dspark_block_size: 5 (6 draft tokens) with thinking on.
  • SGLANG_SIMULATE_ACC_GREEDY: '0' on TP4 c1 restates the EnvBool(False) default.
  • SGLANG_OPT_DSV41_DSPARK_SIM_DRAW_AHEAD=1 only moves the simulated draw before the draft launch.
  • The image includes #46840, so simulated acceptance now keeps the real verify accept cost.
  • Measured mean accept length is 3.510 on TP4 c1 and 3.512 on DEP2 c64 (8,983 lines).
• MODELS.md: It lists dsv41flash Agentic coding (SGLang DSpark) as active, with no deprecated scenario.
• Benchmark hacks: There is no --hf-overrides, --json-model-override-args, model-config edit, layer skipping or indexer skipping. The dsv4-attn-backend: flashmla pin selects an upstream attention backend instead of auto (MegaAttention, which zero-pads heads). It runs the same attention computation, and the evals above cover it.
• Integer all-reduce: N/A. dsv41flash is an existing model on NVIDIA, and no integer all-reduce setting is used.
• Dynamo TRT-LLM metrics: N/A. This is single-node SGLang.
• Upstream image:
  • lmsysorg/sglang:nightly-dev-cu13-20261030-302b78cc@sha256:d4b7701dc94d287e4fc135dfb4c1a88df0778a936255021aeb26eb54d396e579 is an upstream lmsysorg nightly.
  • Its index holds amd64 built from 302b78ccf078 and arm64 built from 52604e1d46c1. GB300 runs the arm64 build, whose fingerprint shows SGLANG_BUILD_COMMIT=52604e1d… and SGLANG_BUILD_URL sgl-project/sglang actions run 40978821580.
  • Both commits are ancestors of sgl-project/sglang main (main is 319 and 325 commits ahead, 0 behind).
  • [Fix] dspark: Keep the real verify accept cost under simulated acceptance sgl-project/sglang#46840, #46404, #46891, #47145 and #47105 are all merged ancestors of 52604e1d.
• Engine-first ordering: This is an SGLang (in-tree engine) entry, so the ordering rule is satisfied.
• Upstream recipe:
  • Merged cookbook PR sgl-project/sglang#47929 (MERGED 2026-11-03T03:28:14Z) publishes this PR's head 41d29b28, run 41261902147 and the nightly-dev-cu13-20261030-302b78cc@sha256:d4b7701d… image in docs/src/snippets/agentx/deepseek-ai/deepseek-v4_1.jsx on main. It appears in the SGLang DeepSeek-V4.1 cookbook, Agentic Long-Context Deployment section. The config was first published there by sgl-project/sglang#47214 and refreshed by sgl-project/sglang#47464.
  • The cookbook covers the python3 -m sglang.launch_server TP4 c1 megakernel launch: --enable-nvidia-megakernel, --kv-cache-dtype fp8_e4m3, --mem-fraction-static 0.91, --dsv4-attn-backend flashmla, --moe-a2a-backend none, --startup-weight-load-mode serial, --context-length 1048576, --cuda-graph-max-bs-decode 1, --prefill-decode-interval 16, --schedule-policy lpm and --swa-prefix-tails 128.
  • The c1 environment is covered too: SGLANG_NVIDIA_MEGAKERNEL_SLOTS, SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION, SGLANG_OPT_DSV41_DSPARK_SIM_DRAW_AHEAD, SGLANG_SPEC_TP_SYNC, the Engram host table and layout, and NCCL_CUMEM_ENABLE / NCCL_MNNVL_ENABLE.
  • The TP4 c4/c16, TP2/EP2 c16/c32 (with flashmla) and DEP2 router points are covered as per-point patches.
  • Excluded InferenceX-only settings: the SGLANG_SIMULATE_ACC_* synthetic-acceptance controls, enable-metrics / enable-metrics-for-all-schedulers (server-metrics scraping for result collection), PYTHONNOUSERSITE / PYTHONUNBUFFERED, health-check timing, the observability/telemetry toggles, and the AIPERF_* / AGENTIC_WARMUP_GRACE_PERIOD / CONC client controls. These are harness, collection or synthetic benchmark controls with no production serving effect.
• No engine patches: At 41d29b28, the recipe has no setup script, .patch, overlay, git apply, sed -i, site-packages edit or wheel install. The PR does not touch runners/srt-slurm/patches/, and there is no docs/waiver/4603.md. The megakernel plugin ships in the pinned image.
• append-only: N/A. The new changelog entry does not use append-only: true.
• Pareto coverage:

  • There is one affected curve: dsv41flash / agentic-coding / GB300 sglang / fp4 / run 41261902147. Its points come from the 8 bmk_agentic_* artifacts (attempt 1 for TP4 c1/c4, TP2/EP2 c16/c32 and DPA c160; attempt 2 for TP4 c16 and DPA c64/c128). All 8 ran on the PR image, with 0–3 error-dropped records per point, far below the 10% failed-request threshold. They are plotted as per_gpu.total_tput_tps against E2EL.

  • Counts were computed with infx.workflows.pareto_coverage from main.

    Point	P90 E2EL (s)	P75 E2EL (s)	tput/GPU	Δ tput/GPU vs run 40809149952
    TP4 c1	5.67	3.17	8,105	+0.1%
    TP4 c4	6.29	2.79	9,425	−7.8%
    TP4 c16	6.25	3.01	29,242	−0.0%
    TP2/EP2 c16	7.64	3.55	57,957	+0.2%
    TP2/EP2 c32	12.07	5.79	118,105	−0.6%
    TP2/EP2/DPA c64	15.40	7.61	224,816	+0.3%
    TP2/EP2/DPA c128	23.44	10.72	321,183	+0.3%
    TP2/EP2/DPA c160	33.68	14.56	340,890	−3.5%

  • P90 E2EL: 7/8 frontier points. Only TP4 c4 is off the frontier; TP4 c16 dominates it.

  • P75 E2EL: 7/8 frontier points. Only TP4 c1 is off the frontier; TP4 c4 dominates it.

  • Both percentiles meet the 5-point recommendation.

Signed: nvpohanh

[github-actions]

✅✅✅ Verdict: PASS ✅✅✅

Passed and not applicable checks

✅ Check 0 (CODEOWNER): PASS — @nvpohanh is a named owner of inferencex-e2e/configs/nvidia-master.yaml and of the gb300-* srt-slurm recipe path; perf-changelog.yaml falls under the catch-all only.

✅ Check 1 (Sweep on in-PR commit): PASS — The pinned head 41d29b28 has Run Sweep 41261902147 (pull_request, attempt 2, success): all 8 agentic / and 6 agentic eval / jobs succeeded on image nightly-dev-cu13-20261030-302b78cc@sha256:d4b7701d…. No commits follow the pinned SHA.

✅ Check 2 (Evals): PASS — eval_results_all from run 41261902147 has 6 gsm8k rows (em_strict 0.9689–0.9757, n=1319) on the PR image. Every row is within 2σ of its best sibling (largest gap 0.0068 vs 0.0128) and of the exact ingested cohort (run 40059333495). The TP4 c1 eval launch command includes --enable-nvidia-megakernel, --kv-cache-dtype fp8_e4m3 and --dsv4-attn-backend flashmla, so the eval covers the changed path.

✅ Check 3 (Upstream recipe): PASS — sgl-project/sglang#47929 is MERGED (2026-11-03T03:28:14Z). On main, docs/src/snippets/agentx/deepseek-ai/deepseek-v4_1.jsx publishes this config at sha 41d29b28 / run 41261902147 with the pinned image. It covers the TP4 c1 megakernel launch (flashmla, fp8_e4m3 KV, mem-fraction 0.91, SGLANG_SPEC_TP_SYNC, megakernel slots, Engram host table, NCCL env), the per-point patches for TP4 c4/c16 and TP2/EP2, and the DEP2 router. The excluded settings (synthetic-acceptance, metrics, AIPerf/client) are harness-only and the sign-off justifies them.

✅ Check 4 (Reuse command): PASS — COLLABORATOR nvpohanh posted /use 41261902147, the newest reuse command; it names the Check 1 run on the pinned SHA.

✅ Check 5 (Checklist template): PASS — All 20 items of the current PR_REVIEW_CHECKLIST.md are present and checked.

✅ Check 6 (Upstream image/provenance/ordering): PASS — lmsysorg/sglang:nightly-dev-cu13-20261030-302b78cc is upstream. Its build (sgl-project/sglang run 40978821580) was a scheduled run on main at 302b78cc, and both 302b78cc and 52604e1d are ancestors of sglang main (0 behind). This is an SGLang entry, so the engine-first ordering rule does not apply.

✅ Check 7 (Deprecated models): PASS — MODELS.md lists dsv41flash Agentic coding (SGLang DSpark) as active on 2026-11-03.

✅ Check 8 (Benchmark hacks): PASS — No --hf-overrides, model-config edits, or layer/indexer skipping. The c1 kv-cache-dtype fp8_e4m3 matches the image default (DEP2 resolves the same) and the TP4 c1 eval covers it. No integer all-reduce controls on this NVIDIA path.

✅ Check 9 (Chat template): PASS — The AgentX client replays through the chat endpoint with thinking on (SGLANG_DEFAULT_THINKING=1, reasoning effort high).

✅ Check 10 (Engine patches): PASS — The recipe at 41d29b28 has no setup script, patch, sed/git apply, overlay or wheel install. The only srt-slurm patch (573-participating-gpus.patch) changes harness power telemetry, not the serving stack.

✅ Check 11 (Golden AL): PASS — The benchmark launch exports SGLANG_SIMULATE_ACC_LEN=3.51, METHOD=match-expected and TOKEN_MODE=real-draft-token. With speculative-dspark-block-size 5 and thinking on, this matches dsv41flash_dspark.yaml (signoff_facts golden-al matches=true).

➖ Check 12 (Append-only): N/A — The new changelog entry does not set append-only: true.

✅ Check 13 (Draft precision): PASS — The draft is the DSpark head bundled in deepseek-ai/DeepSeek-V4.1-Flash, with no draft quantization, dtype or path overrides. At build 52604e1d, environ.py defines SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE=EnvBool(False) (not set anywhere in the recipe or launch) and SGLANG_DSPARK_OPT_MARKOV_W2_BF16=EnvBool(True) (the recipe restates the default). The fp8 KV applies to target and draft alike, and the megakernel/runtime-optimization switches only change scheduling and fusion.

✅ Check 14 (Pareto coverage): PASS — One curve: dsv41flash / agentic-coding / GB300 sglang / fp4 / run 41261902147 (8 bmk_agentic_* artifacts, all on the PR image). pareto_coverage gives P90 E2EL 7/8 and P75 E2EL 7/8 frontier points.

✅ Check 15 (PR description): PASS — The body's claims (old/new image tags, flashmla on TP4 and TP2/EP2 points, TP4 c1 megakernel/fp8 KV/0.91/Engram host table, unchanged search space) match the pinned diff and run evidence.

✅ Check 16 (NVIDIA overview): PASS — The NVIDIA overview Framework Recipes table has a DeepSeek-V4.1-Flash row with the SGLang cell populated.

➖ Check 17 (Metrics publication): N/A — Single-node SGLang; no Dynamo TRT-LLM serving path is affected.

✅ Check 18 (Recipe/image/sweep consistency): PASS — No generation errors. Master TP/EP/DPA and concurrencies match the recipe overrides and GPU counts. The digest-pinned image resolves now. The changelog states the image, KV dtype, Engram and attention-backend changes accurately. No removed points: all 8 are re-tuned, not removed. Report-only deltas (tput/GPU, intvty): TP4 c1 +12.1%/+76.7%, TP4 c4 −6.4%/+20.4%, TP4 c16 +0.5%/+16.4%, DPA c160 −2.1%/−5.5%; the other points are within ±2.5%/±4.6%.

Assessed commit: 41d29b2834a85fcee09db6f9f52653b7c2f30ef3.

[weireweire]

i've updated the container, let's reuse the new sweep @nvpohanh

[nvpohanh]

/use 41261902147

[github-actions]

@nvpohanh staged run 41261902147.

• DeepSeek-V4.1-Flash · agentic-traces · FP4 · gb300_sglang · 2026-11-01

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

[nvpohanh]

[by Claude Code] The SGLang cookbook update of this PR's AgentX config (dsv41flash-fp4-gb300-sglang-agentic-dspark) will happen in sgl-project/sglang#47929, in the Agentic Long-Context Deployment section of the DeepSeek-V4.1 cookbook page. The config has been live in that section since sgl-project/sglang#47214; sgl-project/sglang#47929 refreshes it with the pin below.

The commands mirror the launch measured in run 41261902147, pinned by this PR's /use comment.

[nvpohanh]

[by Claude Code] sgl-project/sglang#47929 has been merged. This PR's AgentX config (dsv41flash-fp4-gb300-sglang-agentic-dspark) is now in the Agentic Long-Context Deployment section of the SGLang DeepSeek-V4.1 cookbook: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4_1#agentic-long-context-deployment

The commands mirror the launch measured in run 41261902147, pinned by this PR's /use comment.

[functionstackx]

Approved at Oren’s request, accepting Po-Han’s explanation of SPEC_TP_SYNC=rng,-dspark-accept-sample and its suitability for production use. This approval does not independently verify existing production adoption.

Dismissed at Oren’s explicit request following acceptance of Po-Han’s explanation of the optimization in the Slack discussion.

[github-actions]

Ingest report — ✅ Results ingested into the production database.

Ingest workflow · Merge Ingest run · Benchmark source run

Category	new	duplicate	skipped	unmapped	failed
benchmark	8	0	0	0	0
eval	6	6	10	0	0
trace	8	0	0	0	0
power	8	0	0	0	0

_{benchmarkFiles 8, rawBenchmarkRows 8, coveredByPerJobFiles 0, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 7914, gpuMetricSamplesNew 118366, powerPublicationPoints 8, changelogEntriesWritten 1}

Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun.

中文

Ingest 报告 — ✅ 结果已写入 production 数据库。

上表仅统计本次运行写入的数据，各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。