[NVIDIA][AgentX] GB300 DeepSeek-V4.1-Flash Dynamo+SGLang: TP4/TEP2 aggregated and DEP2/DEP16 disaggregated points / [NVIDIA][AgentX] GB300 DeepSeek-V4.1-Flash Dynamo+SGLang：TP4/TEP2 聚合与 DEP2/DEP16 分离式配置点

[nvpohanh]

Moves the GB300 DeepSeek-V4.1-Flash Dynamo + SGLang lane to lmsysorg/sglang:nightly-dev-cu13-20261015-a6a8c560 (multiarch digest pinned) and runs the ai-dynamo 1.6.0.dev20261014 that the image ships (dynamo.install: false, TCP request plane). It replaces the aggregated points, adds DEP2/DEP16 disaggregated points under a new key, and removes the TP2/EP2 1PxD disaggregated points.

Config key	Topology	Conc	Nodes	Key settings
dsv41flash-fp4-gb300-dynamo-sglang-agentic-agg	TP4/EP1	1	1	Engram in HBM, one running request, SWA bounded replay, no prefill-decode interval
	TP4/EP1	4, 16	1	host Engram tables, 4096 SWA tails, 128 graph batch; 192-step prefill-decode interval at c16
	TP2/EP2	16	1	host Engram tables, SWA bounded replay, 96-step prefill-decode interval
dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg (new)	4x DEP2 prefill + 1x DEP16 decode	1228	7	MegaMoE, FlashMLA, Mooncake, prefill HiCache write-back DRAM KV offload (ratio 1.02), 8 decode gateway workers, NCCL DP-batch sync, 24 requests per decode rank
	6x DEP2 prefill + 1x DEP16 decode	2884	8	as c1228 with HiCache ratio 1.1 and 40 requests per decode rank

• The aggregated engine args follow the matching standalone SGLang points of dsv41flash-fp4-gb300-sglang-agentic-dspark ([AgentX] GB300 DSV4.1 Flash SGLang: 20261014 nightly, fastokens, DEP2 bounded replay and c160, TP4 c1 single admission / [AgentX] GB300 DSV4.1 Flash SGLang：升级 20261014 nightly、启用 fastokens、DEP2 开 bounded replay 并新增 c160、TP4 c1 单请求准入 #4033). The Dynamo frontend tokenizes with fastokens, so SGLang's tokenizer-backend and tokenizer-worker-num are dropped, as are env settings that restate this image's defaults or only configure SGLang's own HTTP server.
• The Dynamo frontend renders the chat template, so the AgentX client sends thinking=true, reasoning_effort=high and temperature=1 with every request. Synthetic acceptance uses the golden thinking-on AL of 3.51 at DSpark block size 5 on every role.
• Each disaggregated point uses one dedicated CPU node for the frontend, etcd, NATS and the AgentX client. Two DEP2 prefill workers share each prefill node; the pinned srt-slurm v2.59.1 includes fix(sglang): Port collision when colocating 2 workers on same node with DP NVIDIA/srt-slurm#584, which reserves a KV-event port per DP rank so they do not collide.
• Removes dsv41flash-fp4-gb300-dynamo-sglang-agentic-disagg (six TP2/EP2 1PxD points, c16 to c256, on the 20260928 image) and its recipe overrides.

中文

将 GB300 DeepSeek-V4.1-Flash Dynamo + SGLang 配置迁移到 lmsysorg/sglang:nightly-dev-cu13-20261015-a6a8c560（固定 multiarch digest），并直接使用镜像自带的 ai-dynamo 1.6.0.dev20261014（dynamo.install: false，TCP request plane）。本 PR 替换聚合模式的配置点，在新的 config key 下新增 DEP2/DEP16 分离式配置点，并移除 TP2/EP2 1PxD 分离式配置点。

• dsv41flash-fp4-gb300-dynamo-sglang-agentic-agg（每个点 1 个节点）：
  • TP4/EP1 c1：Engram 放在 HBM，单个运行请求，启用 SWA bounded replay，无 prefill-decode interval。
  • TP4/EP1 c4、c16：Engram 表放在主机内存，SWA tails 4096，graph batch 128；c16 使用 192 步 prefill-decode interval。
  • TP2/EP2 c16：Engram 表放在主机内存，启用 SWA bounded replay，96 步 prefill-decode interval。
• dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg（新增）：
  • c1228：4 个 DEP2 prefill + 1 个 DEP16 decode，共 7 个节点；MegaMoE、FlashMLA、Mooncake，prefill HiCache write-back DRAM KV offload（ratio 1.02），8 个 decode gateway worker，NCCL DP batch 同步，每个 decode rank 24 个请求。
  • c2884：6 个 DEP2 prefill + 1 个 DEP16 decode，共 8 个节点；设置同 c1228，但 HiCache ratio 为 1.1，每个 decode rank 40 个请求。
• 聚合模式的引擎参数沿用 dsv41flash-fp4-gb300-sglang-agentic-dspark 中对应的独立 SGLang 配置点（[AgentX] GB300 DSV4.1 Flash SGLang: 20261014 nightly, fastokens, DEP2 bounded replay and c160, TP4 c1 single admission / [AgentX] GB300 DSV4.1 Flash SGLang：升级 20261014 nightly、启用 fastokens、DEP2 开 bounded replay 并新增 c160、TP4 c1 单请求准入 #4033）。分词由 Dynamo 前端使用 fastokens 完成，因此去掉 SGLang 的 tokenizer-backend 与 tokenizer-worker-num，并去掉与镜像默认值相同或仅作用于 SGLang 自身 HTTP 服务的环境变量。
• Chat template 由 Dynamo 前端渲染，因此 AgentX 客户端在每个请求中发送 thinking=true、reasoning_effort=high 与 temperature=1。所有角色的模拟接受长度均采用 DSpark block size 5 下 thinking-on 的 golden AL 3.51。
• 每个分离式配置点使用 1 个专用 CPU 节点运行前端、etcd、NATS 与 AgentX 客户端。每个 prefill 节点上运行两个 DEP2 prefill worker；当前固定的 srt-slurm v2.59.1 已包含 fix(sglang): Port collision when colocating 2 workers on same node with DP NVIDIA/srt-slurm#584，为每个 DP rank 预留独立的 KV event 端口，避免端口冲突。
• 移除 dsv41flash-fp4-gb300-dynamo-sglang-agentic-disagg（基于 20260928 镜像的六个 TP2/EP2 1PxD 配置点，c16 至 c256）及其 recipe override。

AI model disclosure

• Model/version: Claude Opus 5.5 (claude-opus-5-5), via Claude Code.
• Role: ported and converted the recipes and master-config entries, wrote the changelog entry and PR text, and ran the local schema, matrix, and changelog checks. No delegated agents.

🤖 Generated with Claude Code

[github-actions]

Thanks for the contribution!

• Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
• PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
• After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.

中文

感谢你的贡献！

• **审阅：**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER，请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，再通过 Slack 联系核心维护者。必须严格遵循模板，并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have，才能触发签核验证。
• **PR 验证：**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目，并且恰好带有一个主标签时，才会运行扫描：full-sweep-fail-fast（强烈推荐；canary 加逐矩阵 fail-fast）、full-sweep-enabled（有 canary；矩阵任务在失败后继续运行）或 non-canary-full-sweep-enabled（无 canary，也无 fail-fast）。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
• **合并后：**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复；参见重新运行失败任务的说明。

[claude]

Findings marked 🟡 are optional suggestions and need no follow-up push.

[nvpohanh]

[by Claude Code] I've started babysitting the pipeline of this PR. I'll keep monitoring it until it passes. If a job fails, I may rerun it when the cause is infrastructure, or push a fix or a merge of the latest main when the cause is in this PR.

[github-actions]

View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=38911006463

View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=38911006463

[nvpohanh]

[by Claude Code] The SGLang cookbook integration of this PR's AgentX configs (dsv41flash-fp4-gb300-dynamo-sglang-agentic-agg, dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg) will happen in sgl-project/sglang#45111, in the Agentic Long-Context Deployment section of the DeepSeek-V4.1 cookbook page.

This PR has no authorized /use <run_id> comment yet, so the cookbook shows the recipe at this PR's head (2bc599fc4) as "in progress". A later cookbook PR switches it to the measured run once a run is pinned or this PR merges.

[nvpohanh]

[by Claude Code] sgl-project/sglang#45111 has been merged. This PR's AgentX configs (dsv41flash-fp4-gb300-dynamo-sglang-agentic-agg, dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg) are now in the Agentic Long-Context Deployment section of the SGLang DeepSeek-V4.1 cookbook: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4_1#agentic-long-context-deployment

This PR has no authorized /use <run_id> comment yet, so the cookbook shows the recipe at this PR's head (2bc599fc4) as "in progress". A later cookbook PR switches it to the measured run once a run is pinned or this PR merges.

[nvpohanh]

/use 38911006463

[github-actions]

@nvpohanh staged run 38911006463.

• DeepSeek-V4.1-Flash · agentic-traces · FP4 · gb300_dynamo-sglang · 2026-10-16

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

[nvpohanh]

As a PR reviewer and CODEOWNER, I have reviewed this and have:

• Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
• Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
• Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
• Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
• Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
• Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison.
• For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
• Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
• Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
• For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
• If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
• If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
• Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
  • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
• Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
• If this PR uses append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it.
• If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
• Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.

Additional detail section:

• Assessed head: 2bc599fc4149e0d960c65d4deb528fd55cf282af. The checklist template matches main at d892f56a.
  • The diff is config-only. It rewrites agg-variants.yaml and disagg-variants.yaml, replaces one nvidia-master.yaml entry (-agentic-disagg becomes -agentic-dspark-disagg), updates the -agentic-agg entry, and appends one perf-changelog.yaml entry (13 added lines; nothing edited or removed). check-changelog passed.
  • The only merge conflict with current main is in perf-changelog.yaml, and this PR's change to it is append-only.
• PR validation: Run Sweep 38911006463 (pull_request, head 2bc599fc, full-sweep-fail-fast, attempt 1) completed success. Pinned with /use 38911006463.
  • All 6 multi-node agentic / benchmark jobs succeeded: TP4 c1, c4 and c16; TP2/EP2 c16; 4P1D DEP2/DEP16 c1228; and 6P1D DEP2/DEP16 c2884.
  • All 4 multi-node agentic eval / jobs succeeded. None of the benchmark or eval jobs was skipped. The skipped jobs are the empty fixed-sequence and single-node lanes.
  • Every point recorded 0 error-dropped requests; only warmup records are excluded.
  • The bmk_agentic_*, eval_results_all and server-log artifacts are unexpired (they expire 2027-01-14).
  • Every worker loaded deepseek-ai/DeepSeek-V4.1-Flash at HF revision 314113f8a7dadb388215396ba5fbee2c610a04d6.
• Evals: gsm8k em_strict from eval_results_all, each with n=1319 and infrastructure_success: true:
  • 0.9719 at TP4 c16 (job).
  • 0.9735 at TP2/EP2 c16 (job).
  • 0.9719 at 4P1D c1228 (job).
  • 0.9735 at 6P1D c2884 (job).
  • All four clear the 0.90 default in infx/evals/thresholds.yaml, which has no dsv41flash override. They also clear the 0.91 dsv4 value.
  • All four ran on the config's image, where every eval worker fingerprint has SGLANG_BUILD_COMMIT=a6a8c56015b8dbca4147634fa9ef22c7c836de05. They used real DSpark verification: the eval worker environment has no SGLANG_SIMULATE_ACC_*, and the measured mean accept length is 3.82 to 4.06.
• Image and Dynamo: upstream lmsysorg/sglang:nightly-dev-cu13-20261015-a6a8c560, pinned by digest sha256:0a1ba33aa4e032b8c5bcebdc69c45664074c032074add4f7db01629ec1fcc381, which is SGLang a6a8c56015b8dbca4147634fa9ef22c7c836de05.
  • Every worker fingerprint (agg, prefill and decode) lists sglang 0.0.0.dev1+ga6a8c5601, ai-dynamo==1.6.0.dev20261014, ai-dynamo-runtime==1.6.0.dev20261014 and fastokens==0.3.4.
  • Dynamo ships in the image. The recipes set dynamo.install: false, so nothing is installed at launch.
  • Both configs are framework: dynamo-sglang, an open-source engine entry, so the engine-ordering item is satisfied by this submission itself.
• Chat template: AgentX replay runs agentx profile. The AIPerf export of every point records endpoint type chat with streaming: true.
  • The Dynamo frontend logs only /v1/chat/completions traffic, from 1,412 lines at TP4 c1 to 1,568,325 at c2884. There is no /v1/completions traffic.
  • The checkpoint's chat template is used with no override (chat_template=None). Every worker logs "Auto-detected --reasoning-parser as 'deepseek-v41' from chat template".
  • Every job logs THINKING_MODE: thinking_on. The client sends chat_template_kwargs {"thinking":true,"reasoning_effort":"high"} and temperature: 1, which match the golden-curve settings.
• Draft precision: the served draft is the DSpark head bundled in deepseek-ai/DeepSeek-V4.1-Flash@314113f8 (mtp.0-mtp.2).
  • Every agg, prefill and decode worker logs "DSpark draft weights are bundled in the target checkpoint; defaulting --speculative-draft-model-path to --model-path" and type=DeepseekV4ForCausalLMDSpark, quant=fp8.
  • Shipped (safetensors headers of this revision): the attention projections, shared experts and main_proj are F8_E4M3 with F8_E8M0 block scales. The routed experts are MXFP4 (I8-packed FP4 with F8_E8M0 group-32 scales). The norms, gate weight, markov_head and confidence_head are BF16. config.json has quant_method: fp8 and expert_dtype: fp4.
  • Pinned image's default handling: --speculative-draft-model-quantization is unset, so the draft inherits the checkpoint-detected quantization (serving_hook.py#L655-L668). Every worker logs quantization=None, speculative_draft_model_quantization=None, _speculative_draft_quantization_explicitly_set=False, dtype='auto', json_model_override_args='{}' and speculative_draft_kv_cache_dtype=None.
    • wo_a: the image dequantizes the shipped FP8 wo_a to BF16 at load (deepseek_v4_dspark.py#L108 wo_a_fp8=False, #L1014). Every worker logs "Finished streaming dequant fp8 wo_a". This is the default conversion, so it is the baseline.
    • Routed experts at TP4 and TP2/EP2 (agg): the image auto-selects moe_runner_backend=flashinfer_mxfp4, so the draft experts stay MXFP4 with MXFP8 activations.
    • Routed experts at DEP2 prefill and DEP16 decode (MegaMoE): enable_w4a4_mxfp4_megamoe=False and speculative_enable_w4a4_mxfp4_megamoe=None on every worker. The draft therefore keeps the target's value (moe/utils.py#L609-L615), and the MMA type is fp8xfp4 (mega_moe.py#L60): FP8 activations against the unrequantized MXFP4 weights. The W4A4 mxf4xmxf4 path is not used.
    • The Markov head stays BF16 under the SGLANG_DSPARK_OPT_MARKOV_W2_BF16=EnvBool(True) default (environ.py#L544). No recipe sets any SGLANG_DSPARK_* variable.
  • Effective: as shipped. Attention, shared experts and main_proj are FP8; wo_a is BF16 through the default conversion; routed experts are MXFP4 with MXFP8/FP8 activations; the heads are BF16. The draft KV cache follows the target's default fp8_e4m3.
  • SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is absent from both recipes, the shared launchers and helpers at the head, the env fingerprint of every agg, prefill and decode worker (benchmark and eval), and all worker logs.
    • It defaults to EnvBool(False) (environ.py#L1059).
    • It would only act on modelopt_fp4 checkpoints with the deep_gemm runner (deepseek_common/utils.py#L125-L133).
    • SGLANG_GLM_NEXTN_MOE_PTPC is also unset.
• Golden AL: infx/golden_al_distribution/dsv41flash_dspark.yaml gives 3.51 for thinking_on with 5 speculative tokens, which matches --speculative-dspark-block-size 5 on every role.
  • infx/srt_slurm/synthetic_acceptance.py injects the value. The job logs show "Applied override: --set roles.{agg,prefill,decode}.env.SGLANG_SIMULATE_ACC_LEN=3.51", with SGLANG_SIMULATE_ACC_METHOD=match-expected and SGLANG_SIMULATE_ACC_TOKEN_MODE=real-draft-token.
  • The agg, every prefill and the decode worker fingerprints all carry SGLANG_SIMULATE_ACC_LEN=3.51, so prefill and decode use the same method.
  • The mean accept len in the benchmark logs is 3.507 to 3.513 on every agg and decode worker.
• MODELS.md: dsv41flash agentic coding is active (SGLang DSpark arms per SKU).
• No architecture hacks: there is no --hf-overrides, json_model_override_args is '{}' on every worker, and no model files are edited. All 3 DSpark stages run, and the target runs its native layers.
  • HiCache (prefill DRAM KV offload), the Engram host table, MegaMoE, FlashMLA and the prefill-decode interval change where data lives and how kernels are scheduled, not what is computed.
• Dynamo TRT-LLM metrics: N/A. Both configs are dynamo-sglang and have no Dynamo TRT-LLM path. enable-metrics is on for every SGLang worker.
• SGLang cookbook (merged): sgl-project/sglang#45111 is MERGED (mergedAt: 2026-10-19T07:24:09Z, merge commit af9bb2a32d4a313d20a629ac1b2cb8e6ef245b95). It is the latest commit touching the DeepSeek-V4.1 AgentX data on SGLang main.
  • Published page: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4_1#agentic-long-context-deployment. The NVIDIA overview links the same page.
  • Data: docs/src/snippets/agentx/deepseek-ai/deepseek-v4_1.jsx has four GB300 Dynamo cells with provenance PR 4180 at 2bc599fc, the assessed head and the measured commit. They are "Agg TP4" (c1/c4/c16) and "Agg TP2·EP2" (c16) from -agentic-agg, and "Disagg 4P1D · prefill DEP2 · decode DEP16" (c1228) and "Disagg 6P1D" (c2884) from -agentic-dspark-disagg. Every cell uses the same image digest, and none has an install step because Dynamo ships in the image.
  • Comparison method: I resolved the cookbook's Dynamo commands for each of the 6 points, following the renderer's ref/from/diff/patch semantics. I compared them role by role with the effective recipes (base plus override) at 2bc599fc and with the logged server_args of every worker.
  • Frontend: python3 -m dynamo.frontend --router-mode kv --router-session-affinity-ttl-secs 14400 --active-decode-blocks-threshold None --active-prefill-tokens-threshold None --active-prefill-tokens-threshold-frac None.
    • Environment: DYN_TOKENIZER=fastokens, DYN_TOKENIZER_CACHE=1, DYN_TOKENIZER_CACHE_BYTES=8000000000, DYN_ROUTER_TEMPERATURE=0, DYN_ROUTER_QUEUE_THRESHOLD=None, DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS=3600, DYN_TCP_CHANNEL_BUFFER=128, DYN_TCP_REQUEST_TIMEOUT=60 and DYN_REQUEST_PLANE=tcp.
    • The cookbook also includes the etcd and NATS commands. All of this matches the recipe's frontend args/env and request_plane: tcp exactly.
  • Agg workers: python3 -m dynamo.sglang.
    • Shared arguments: --trust-remote-code --mem-fraction-static 0.8 --chunked-prefill-size 16384 --speculative-algorithm DSPARK --speculative-dspark-block-size 5 --reasoning-parser auto --tool-call-parser auto --watchdog-timeout 3600.
    • TP4 c1: --tensor-parallel-size 4 --expert-parallel-size 1 --max-running-requests 1 --cuda-graph-max-bs-decode 64 --prefill-decode-interval 0 --enable-decoder-swa-bounded-replay --cuda-graph-backend-prefill disabled.
    • TP4 c4 and c16: --max-running-requests 128 --cuda-graph-max-bs-decode 128 --swa-prefix-tails 4096 with --prefill-decode-interval 0 at c4 and 192 at c16, and SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1 with SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT=per_rank. c16 adds SGLANG_NUMA_BIND_V2=0.
    • TP2/EP2 c16: --tensor-parallel-size 2 --expert-parallel-size 2 --max-running-requests 32 --cuda-graph-max-bs-decode 64 --prefill-decode-interval 96 --swa-prefix-tails 1024 --enable-decoder-swa-bounded-replay --cuda-graph-backend-prefill disabled, with the Engram host table, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and SGLANG_NUMA_BIND_V2=0.
  • Disagg prefill (4 or 6 DEP2 workers, 2 per node):
    • Arguments: --disaggregation-mode prefill --tensor-parallel-size 2 --data-parallel-size 2 --expert-parallel-size 2 --enable-dp-attention --enable-dp-lm-head --moe-a2a-backend megamoe --dsv4-attn-backend flashmla --disaggregation-transfer-backend mooncake --chunked-prefill-size 65536 --max-running-requests 128 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay --enable-hierarchical-cache --hicache-write-policy write_back --hicache-io-backend direct --hicache-mem-layout page_first_direct --hicache-ratio 1.02|1.1 --mem-fraction-static 0.85 --weight-loader-prefetch-checkpoints --model-loader-extra-config '{"enable_multithread_load":true}', plus DSpark and the parsers.
    • KV events: --kv-events-config (the recipe sets kv_events: true on prefill only).
    • Environment: the Engram host table, SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600, SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600, SGLANG_ENABLE_PREFILL_WAR_READ_DONE=1, SGLANG_MOONCAKE_CUSTOM_MEM_POOL=True, SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=32768, SGLANG_OPT_DSV41_MEGA_MHC_PREFILL=1, NCCL_MNNVL_ENABLE=1, NCCL_CUMEM_ENABLE=1 and MC_FORCE_MNNVL=1.
    • The cookbook pins the two workers on each node to GPUs 0,1 and 2,3.
  • Disagg decode (1 DEP16 worker over 4 nodes):
    • Arguments: --disaggregation-mode decode --nnodes 4 --tensor-parallel-size 16 --data-parallel-size 16 --expert-parallel-size 16 --enable-dp-attention --enable-dp-lm-head --moe-a2a-backend megamoe --dsv4-attn-backend flashmla --disaggregation-transfer-backend mooncake --mem-fraction-static 0.85 --weight-loader-prefetch-checkpoints --model-loader-extra-config '{"enable_multithread_load":true}', plus DSpark and the parsers.
    • Per point: --max-running-requests 384|640 --cuda-graph-max-bs-decode 24|40 --disaggregation-decode-extra-slots 36|80.
    • Environment: the Engram host table, DYN_SGL_GATEWAY_WORKERS=8, the two disaggregation timeouts, SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH=1, SGLANG_MOONCAKE_CUSTOM_MEM_POOL=True, SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=16384, NCCL_MNNVL_ENABLE=1, NCCL_CUMEM_ENABLE=1 and MC_FORCE_MNNVL=1.
  • Result: every recipe argument and recipe environment variable of every role and point matches. The only differences are the InferenceX-only settings below, the default-equivalent setting below, and placeholders (ports, IP addresses, node rank).
    • weight-loader-drop-cache-after-load: false restates the default False (fields/model.py#L304-L306), and every worker logs weight_loader_drop_cache_after_load=False. The cookbook omits it.
    • NCCL_CUMEM_ENABLE=1 is not in the agg recipe. The GB300-NV cluster exports it into every worker container (fingerprint env), so the cookbook carries it.
    • Under DP attention, this image splits --chunked-prefill-size across the attention-DP ranks (32768 per rank on prefill). The recipe and the cookbook pass the same CLI value.
  • Excluded as InferenceX-only:
    • --enable-metrics: the Prometheus export that AIPerf scrapes for result collection (AIPERF_REQUIRED_SERVER_METRIC_PREFIX=sglang:). It does not affect scheduling or kernels.
    • PYTHONUNBUFFERED and PYTHONNOUSERSITE: log buffering and Python path isolation.
    • HF_HUB_CACHE: the cluster's mounted cache path.
    • SGLANG_SIMULATE_ACC_*: synthetic acceptance. The cookbook discloses it (synthetic-acceptance) with the golden AL of 3.51.
    • The srt-slurm observability (Tachometer off), otel-collector (disabled), health_check, slurm, sbatch_directives and srun_options blocks, and the dedicated CPU node placement for the frontend, etcd, NATS and the benchmark client.
    • The AIPerf client settings (AIPERF_*, including the per-request chat_template_kwargs thinking/effort and temperature) and the DRAM budget (dram-utilization).
    • The rest of the GB300-NV container environment in the worker fingerprints (NCCL_IB_*, NCCL_SOCKET_IFNAME, NCCL_NET_GDR_C2C, NCCL_DEBUG, HF_*, SGLANG_BUILD_*, SGLANG_IMAGE_TAG). These are fabric wiring, log levels, cache paths and image metadata, not serving settings.
• No engine patches: the diff adds no patch files, git apply, heredoc rewrites, site-packages edits, monkey-patching or forked engine wheels, and nothing is installed into the serving image (dynamo.install: false). No waiver is needed.
  • runners/srt-slurm/patches/ on main holds only 573-participating-gpus.patch, an srt-slurm power-telemetry harness backport. It does not touch SGLang or Dynamo.
• append-only: N/A. The new changelog entry does not use append-only: true (append-only: false in the run's changelog-metadata).
• Pareto: one curve, dsv41flash / agentic-coding / gb300 dynamo-sglang / fp4 / run 38911006463. It combines both config keys, which are the only dsv41flash GB300 dynamo-sglang keys.
  • Method: infx.workflows.pareto_coverage from main, with x = E2EL in seconds and y = total token throughput per GPU (4 GPUs at TP4, 2 at TP2/EP2, 24 at 4P1D, 28 at 6P1D).
  • Source: the 6 bmk_agentic_* artifacts of the run (request_metrics.throughput.per_gpu.total_tput_tps and request_metrics.latency.e2el), all on the same image.
  • P90 E2EL: 5 of 6 frontier points (PASS), TP4 c4 and c16, TP2/EP2 c16, 4P1D c1228 and 6P1D c2884. TP4 c1 is dominated by TP4 c4.
  • P75 E2EL: 5 of 6 frontier points (PASS), the same 5. TP4 c1 is dominated.
  • Per point (tok/s/GPU, P90 E2EL s, P75 E2EL s): TP4 c1 7,846.0, 8.132, 4.502; TP4 c4 10,250.6, 6.356, 2.729; TP4 c16 29,122.0, 6.862, 3.120; TP2/EP2 c16 56,204.2, 9.340, 4.088; 4P1D c1228 312,581.8, 15.712, 8.431; 6P1D c2884 375,815.7, 43.756, 32.486.
  • Helper output: .../p90_e2el and .../p75_e2el are both PASS with measuredPoints: 6, invalidPoints: 0, frontierPoints: 5.
  • Dashboard: https://inferencex.semianalysis.com/inference?unofficialRun=38911006463
• NVIDIA overview (advisory): the NVIDIA recipes overview Framework Recipes table lists DeepSeek-V4.1-Flash with SGLang ✅, linking to the same DeepSeek-V4.1 cookbook page.

Signed: nvpohanh

[github-actions]

✅✅✅ Verdict: PASS ✅✅✅

Passed and not applicable checks

✅ Check 0 (CODEOWNER): PASS — @nvpohanh is a named owner of inferencex-e2e/configs/nvidia-master.yaml and of srt-slurm-recipes/**/gb300-*/; perf-changelog.yaml falls under the catch-all only.

✅ Check 1 (Passing sweep on in-PR commit): PASS — Run Sweep 38911006463 (pull_request, head 2bc599fc, which is the current PR tip) has all 6 multi-node agentic / and all 4 multi-node agentic eval / check-runs at success, none skipped.

✅ Check 2 (Evals): PASS — eval_results_all reports gsm8k em_strict of 0.9719 (TP4 c16), 0.9735 (TP2/EP2 c16), 0.9719 (4P1D c1228) and 0.9735 (6P1D c2884), each n=1319 with infrastructure_success: true. All clear the 0.90 default (dsv41flash has no override), and all eval jobs ran on the config's lmsysorg/sglang:nightly-dev-cu13-20261015-a6a8c560@sha256:0a1ba33a… image.

✅ Check 3 (Upstream recipe): PASS — sgl-project/sglang#45111 is MERGED (2026-10-19T07:24:09Z, af9bb2a3). Its deepseek-v4_1.jsx covers all 6 points at 2bc599fc and the same image digest: frontend, agg, DEP2 prefill and DEP16 decode dynamo.sglang args and env all match the recipes at the assessed SHA. Remaining differences (--enable-metrics, PYTHON*, HF_HUB_CACHE, SGLANG_SIMULATE_ACC_*, AIPerf/slurm blocks) are InferenceX-only and justified in the sign-off.

✅ Check 4 (Reuse command): PASS — /use 38911006463 was posted by nvpohanh (COLLABORATOR).

✅ Check 5 (Latest checklist template): PASS — every item of the current template, including the Dynamo metrics, draft-precision, append-only and Pareto items, is present and checked.

✅ Check 6 (Upstream images / engine ordering): PASS — both entries are framework: dynamo-sglang on upstream lmsysorg/sglang (digest-pinned), so they are open-source engine entries and the ordering rule does not apply.

✅ Check 7 (Deprecated models/scenarios): PASS — dsv41flash agentic coding is active in MODELS.md (SGLang DSpark arms per SKU) as of 2026-10-19.

✅ Check 8 (Architecture hacks): PASS — the diff adds no --hf-overrides, json-model-override-args or model-file edits. HiCache, Engram host tables, MegaMoE and FlashMLA change placement and kernels, not FLOPs.

✅ Check 9 (Spec-decode chat template): PASS — AgentX replay drives the Dynamo chat endpoint, and the recipes send chat_template_kwargs (thinking=true, reasoning_effort=high) via AIPERF_EXTRA_INPUTS.

✅ Check 10 (Engine patches): PASS — no patch files, git apply, heredoc rewrites, monkey-patching or wheel installs. dynamo.install changed from true to false, so the image runs as shipped.

✅ Check 11 (Agentic golden AL): PASS — dsv41flash_dspark.yaml gives thinking_on block 5 = 3.51, which matches speculative-dspark-block-size: 5. The job logs show SGLANG_SIMULATE_ACC_LEN=3.51 with match-expected / real-draft-token applied to every prefill and decode worker (4P1D job).

➖ Check 12 (Append-only): N/A — the new perf-changelog.yaml entry does not set append-only: true.

✅ Check 13 (Draft precision): PASS — the bundled DSpark head of deepseek-ai/DeepSeek-V4.1-Flash is used, and the recipes set no draft quantization, dtype or KV-dtype overrides. At pinned SGLang a6a8c560, the draft inherits the target quantization when the flag is unset (serving_hook.py), the W4A4 MegaMoE draft override is a no-op when unset (moe/utils.py), and SGLANG_DSPARK_OPT_MARKOV_W2_BF16 defaults to True. SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE defaults to EnvBool(False), would only act on modelopt_fp4 checkpoints, and is absent from the recipes and the logged worker env.

✅ Check 14 (Pareto coverage): PASS — there is one curve, dsv41flash / agentic-coding / gb300 dynamo-sglang / fp4 / run 38911006463, built from the 6 bmk_agentic_* artifacts on one image. pareto_coverage returns 5/5 frontier points for P90 E2EL and 5/5 for P75 E2EL; TP4 c1 is dominated in both. The removed -agentic-disagg curve is out of scope.

✅ Check 15 (PR description): PASS — the image/digest, Dynamo 1.6.0.dev20261014, topologies, concurrencies, node counts (7 and 8), HiCache ratios 1.02/1.1, 24/40 requests per decode rank and the srt-slurm v2.59.1 pin all match the code at 2bc599fc.

✅ Check 16 (NVIDIA overview): PASS — the NVIDIA overview Framework Recipes table lists DeepSeek-V4.1-Flash with SGLang ✅, linking to the DeepSeek-V4.1 cookbook page.

➖ Check 17 (Dynamo metrics): N/A — both configs are dynamo-sglang; there is no Dynamo TRT-LLM path.

Assessed commit: 2bc599fc4149e0d960c65d4deb528fd55cf282af.

[Oseltamivir]

lgtm, swa-bounded-replay