[hshrivastava-droid]

Summary

Move the DeepSeek-V4.1-Flash H200 FP4 SGLang AgentX config to a newer SGLang nightly, set TP4 concurrency 2 max-running-requests to 2, and sweep concurrency 1 to 16.

• Config: dsv41flash-fp4-h200-sglang-agentic-dspark
• Image: lmsysorg/sglang:nightly-dev-cu13-20261013-7224fce1

AI model disclosure

• Model/version: claude-sonnet-5-5
• Role: prepared the config change and PR text

PR checklist and change type

Type of Change

• Bug fix
• New feature
• Configuration change
• Documentation update
• Other (please describe)

Checklist

• I have completed the AI model disclosure and kept it current
• I have tested my changes locally
• I have updated documentation if necessary
• If python -m infx.matrix.changed --base origin/main lists affected config keys, I have appended one new entry covering them to the physical end of inferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entries
• If this PR appends an inferencex-e2e/perf-changelog.yaml entry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs): full-sweep-fail-fast (recommended), full-sweep-enabled, or non-canary-full-sweep-enabled. Optional modifiers all-evals, evals-only, and agentx-fast require a primary label; the last two block reuse while applied.
• Before merging via reuse, an authorized maintainer (OWNER/MEMBER/COLLABORATOR) has commented /use <run_id> (or the legacy /reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.

中文

将 DeepSeek-V4.1-Flash H200 FP4 SGLang AgentX 配置升级到更新的 SGLang nightly，将 TP4 并发 2 的 max-running-requests 设为 2，并将并发范围设为 1 到 16。

• 配置：dsv41flash-fp4-h200-sglang-agentic-dspark
• 镜像：lmsysorg/sglang:nightly-dev-cu13-20261013-7224fce1

AI 模型使用说明：claude-sonnet-5-5，负责准备配置改动与 PR 文本。

[claude]

Code review found no issues

No high-confidence issues detected in this change.

[github-actions]

View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=39318221321

View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=39318221321

[nvpohanh]

/use 39318221321

[github-actions]

@nvpohanh staged run 39318221321.

• DeepSeek-V4.1-Flash · agentic-traces · FP4 · h200_sglang · 2026-10-19

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

[nvpohanh]

[by Claude Code] The SGLang cookbook integration of this PR's AgentX config (dsv41flash-fp4-h200-sglang-agentic-dspark) will happen in sgl-project/sglang#45311, in the Agentic Long-Context Deployment section of the DeepSeek-V4.1 cookbook page.

The commands mirror the launch measured in run 39318221321, pinned by this PR's /use comment.

[nvpohanh]

[by Claude Code] sgl-project/sglang#45311 has been merged. This PR's AgentX config (dsv41flash-fp4-h200-sglang-agentic-dspark) is now in the Agentic Long-Context Deployment section of the SGLang DeepSeek-V4.1 cookbook: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4_1#agentic-long-context-deployment

The commands mirror the launch measured in run 39318221321, pinned by this PR's /use comment.

[nvpohanh]

As a PR reviewer and CODEOWNER, I have reviewed this and have:

• Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
• Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
• Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
• Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
• Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
• Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison.
• For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
• Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
• Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
• For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
• If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
• If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
• Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
  • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
• Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
• If this PR uses append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it.
• If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
• Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.

Additional detail section:

• Scope: one config, dsv41flash-fp4-h200-sglang-agentic-dspark (single-node aggregated SGLang, H200, agentic-coding). Image bump to lmsysorg/sglang:nightly-dev-cu13-20261013-7224fce1 (server fingerprint: SGLang commit 7224fce1b5e488c9916f20ec0556d696e595d512), TP4 C2 max-running-requests 4 -> 2, and concurrency 1-16 on TP4/EP1 and TP8/EP1. Assessed head: 5bfd55593e80a10bb9a0e6dd22ec33fd606cf574.
• PR validation: pinned by /use 39318221321. Run Sweep 39318221321 ran as a pull_request run on the current head and finished success. All 10 agentic / benchmark jobs (TP4 and TP8 at C1/2/4/8/16) and the agentic eval / job succeeded, with none skipped. The fixed-sequence single-node */ and eval / jobs are skipped because this config is agentic-only.
• Evals: agentic eval job (TP4 C16, eval-only, same image). GSM8K full set (n=1319): em_strict 0.9750 ± 0.0043, above the 0.90 default threshold in infx/evals/thresholds.yaml. The eval server runs real DSpark verification, and its logs contain no SGLANG_SIMULATE_ACC_*.
• Chat template: AgentX replays use the chat endpoint, with SGLANG_DEFAULT_THINKING=1 and SGLANG_DSV41_REASONING_EFFORT=high.
• Draft precision: the draft is the DSpark/MTP head bundled in deepseek-ai/DeepSeek-V4.1-Flash (mtp.0-2.*). The server logs DSpark draft weights are bundled in the target checkpoint; defaulting --speculative-draft-model-path to --model-path.
  • Shipped precision: the checkpoint quantization_config is quant_method: fp8 (block 32x32, ue8m0 scales) with expert_dtype: fp4. The mtp.* weights ship with block .scale tensors, i.e. FP8 linear layers and FP4 experts.
  • Effective precision: the draft loads as DeepseekV4ForCausalLMDSpark, quant=fp8 through the pinned image's default handling. The server args show speculative_draft_model_quantization=None, quantization=None (auto from the checkpoint), dtype=auto, json_model_override_args={} and speculative_draft_kv_cache_dtype=None. The draft KV inherits the target's fp8_e4m3, which is the upstream default.
  • There is no draft quantization, dtype override, checkpoint substitution or patch. SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE does not appear in the recipe, in inherited settings, or in the rendered server environment.
• Golden AL: the connector (infx/srt_slurm/synthetic_acceptance.py) injects the golden AL automatically for throughput runs. The value comes from dsv41flash_dspark.yaml, thinking_on, 5 speculative tokens (speculative-dspark-block-size: 5), which gives 3.51. The TP4 C2 and TP8 C16 server logs both show SGLANG_SIMULATE_ACC_LEN=3.51, SGLANG_SIMULATE_ACC_METHOD=match-expected and SGLANG_SIMULATE_ACC_TOKEN_MODE=real-draft-token.
• Model/scenario: dsv41flash agentic coding is active in MODELS.md. There is no --hf-overrides, and json_model_override_args is empty.
• Engine and order: upstream lmsysorg/sglang image, SGLang-only change.
• Patches: none. The recipe has no setup scripts, patch files, site-packages edits or wheel installs, so no waiver is needed.
• Dynamo TRT-LLM metrics: N/A. This is plain SGLang with no Dynamo TRT-LLM serving path.
• append-only: N/A. The new changelog entry does not use append-only: true.
• Recipe publication: sgl-project/sglang#45311 is MERGED (2026-10-20T06:48:49Z). It is published on the DeepSeek-V4.1 cookbook page, in the Agentic Long-Context Deployment section, entries dsv41flash-fp4-h200-sglang-agentic-dspark#0/#1. Those entries cover the same image, TP4/TP8 python3 -m sglang.launch_server commands, and every production serving flag:
  • --attention-backend dsv4, --moe-runner-backend marlin, --mem-fraction-static 0.7, --chunked-prefill-size 4096, --prefill-decode-interval 16, --enable-decoder-swa-bounded-replay, --swa-prefix-tails 1024, --speculative-algorithm DSPARK, --speculative-dspark-block-size 5, --cuda-graph-max-bs-decode 64, --watchdog-timeout 3600, and the reasoning/tool parsers.
  • Per-concurrency --max-running-requests, including TP4 C2 = 2.
  • Env SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1, SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT=per_rank, SGLANG_DEFAULT_THINKING=1, SGLANG_DSV41_REASONING_EFFORT=high and SGLANG_TIMEOUT_KEEP_ALIVE=900.
• Excluded as InferenceX-only:
  • --enable-metrics: Prometheus scraping for result collection.
  • PYTHONNOUSERSITE and PYTHONUNBUFFERED: container/log plumbing.
  • SGLANG_SIMULATE_ACC_*: synthetic benchmark control, which the cookbook discloses as synthetic acceptance.
  • AIPERF_*, CONC and AGENTIC_WARMUP_GRACE_PERIOD: client and harness settings.
• Pareto coverage: one affected curve, DeepSeek-V4.1-Flash / agentic-coding / h200_sglang / FP4 / run 39318221321. It has 10 valid measured points (TP4+TP8 combined), from the results_bmk artifact, x = E2EL in seconds, y = tput_per_gpu. Counted with infx.workflows.pareto_coverage:
  • P90 E2EL: 5 frontier points (TP8 C2/C4/C8, TP4 C8/C16).
  • P75 E2EL: 5 frontier points (TP8 C2/C4, TP4 C4/C8/C16).

Signed: nvpohanh

[github-actions]

✅✅✅ Verdict: PASS ✅✅✅

Passed and not applicable checks

✅ Check 0 (CODEOWNER): PASS — @nvpohanh is a listed owner of inferencex-e2e/configs/nvidia-master.yaml and of the srt-slurm-recipes/**/h200-*/ recipe path; inferencex-e2e/perf-changelog.yaml falls only under the catch-all owner.

✅ Check 1 (Passing sweep + evals on in-PR commit): PASS — the Run Sweep 39318221321 pull_request run ran on head 5bfd555 (in PR) and concluded success; all 10 agentic / jobs (TP4/TP8 x C1-C16) and the agentic eval / job are success, none skipped (this config is agentic-only, so the fixed-sequence single-node */ and eval / jobs are correctly skipped).

✅ Check 2 (Evals pass): PASS — eval_results_all gives GSM8K n=1319, em_strict 0.9750 ± 0.0043 and flexible 0.9742, logged PASS (>= 0.9 from default) in the agentic eval job; same image lmsysorg/sglang:nightly-dev-cu13-20261013-7224fce1; no SGLANG_SIMULATE_ACC_* in the eval job.

✅ Check 3 (Upstream recipe): PASS — sgl-project/sglang#45311 is MERGED (2026-10-20T06:48:49Z); the DeepSeek-V4.1 cookbook entries dsv41flash-fp4-h200-sglang-agentic-dspark#0/#1 pin this head SHA and image and cover TP4/TP8, every production flag (dsv4 attention, marlin MoE, mem-fraction 0.7, chunked-prefill 4096, prefill-decode-interval 16, SWA replay/tails, DSPARK block 5, cuda-graph bs 64, watchdog, parsers), per-concurrency --max-running-requests (TP4 C2 = 2) and all Engram/thinking/keep-alive env vars; the excluded settings (--enable-metrics, Python env, SGLANG_SIMULATE_ACC_*, AIPerf/harness) are justified as InferenceX-only.

✅ Check 4 (Reuse command): PASS — /use 39318221321 posted by nvpohanh (COLLABORATOR) on its own line, naming the passing in-PR run.

✅ Check 5 (Latest checklist template): PASS — every item of the current PR_REVIEW_CHECKLIST.md template, including the draft-precision, Dynamo metrics and Pareto items, is present and checked.

✅ Check 6 (Upstream image / engine-first): PASS — framework: sglang with upstream lmsysorg/sglang:nightly-dev-cu13-20261013-7224fce1 on H200; SGLang-only change, so the ordering rule does not apply.

✅ Check 7 (No deprecated models/scenarios): PASS — dsv41flash agentic coding (SGLang DSpark) is active in MODELS.md as of 2026-10-20.

✅ Check 8 (No architecture hacks): PASS — no --hf-overrides, json-model-override-args or model-file edits; the rendered launch command contains only serving/tuning flags.

✅ Check 9 (Spec-decode via chat template): PASS — the AgentX replay uses the chat endpoint with SGLANG_DEFAULT_THINKING=1 and SGLANG_DSV41_REASONING_EFFORT=high, so DSpark sees chat-templated traffic.

✅ Check 10 (No engine patches): PASS — the recipe at 5bfd555 has no setup scripts, patch files, heredoc rewrites or wheel installs.

✅ Check 11 (Agentic golden AL): PASS — the rendered TP4 C2 server env has SGLANG_SIMULATE_ACC_LEN=3.51, METHOD=match-expected and TOKEN_MODE=real-draft-token; this equals dsv41flash_dspark.yaml thinking_on with 5 speculative tokens (3.51).

➖ Check 12 (Append-only): N/A — the new changelog entry does not set append-only: true.

✅ Check 13 (Draft precision): PASS — the draft is the DSpark/MTP head bundled in deepseek-ai/DeepSeek-V4.1-Flash, whose checkpoint quantization_config is fp8 block-32 with expert_dtype: fp4 and no draft exclusions. The rendered launch sets no draft quantization, dtype, draft path or KV-dtype override. SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is absent from the recipe and the rendered env, and it defaults to EnvBool(False) in environ.py at the pinned SGLang commit 7224fce1.

✅ Check 14 (Pareto coverage): PASS — one curve (dsv41flash / agentic-coding / h200_sglang / fp4 / run 39318221321 attempt 1, results_bmk, x = E2EL s, y = total_tput_tps per GPU, 10 valid points TP4+TP8). infx.workflows.pareto_coverage counts P90 at 5/5 (TP8 C2/C4/C8, TP4 C8/C16) and P75 at 5/5 (TP8 C2/C4, TP4 C4/C8/C16).

✅ Check 15 (PR description): PASS — the image, TP4 C2 max-running-requests 2 and concurrency 1-16 on TP4/TP8 match the pinned master config and recipe.

✅ Check 16 (NVIDIA overview): PASS — DeepSeek-V4.1-Flash has a populated SGLang cell in the NVIDIA overview Framework Recipes table, which covers the H200 SGLang pair.

➖ Check 17 (Metrics publication): N/A — plain SGLang aggregated serving with no Dynamo TRT-LLM path.

Assessed commit: 5bfd55593e80a10bb9a0e6dd22ec33fd606cf574.