Repository navigation
[NVIDIA][AgentX] GB300 Kimi-K3.1-Pro FP4 Dynamo+SGLang: DFLASH disaggregated and aggregated megakernel points / [NVIDIA][AgentX] GB300 Kimi-K3.1-Pro FP4 Dynamo+SGLang：DFLASH 分离式与聚合式 megakernel 配置点 - #4811
Conversation
…oints Add kimik3.1-fp4-gb300-dynamo-sglang-agentic-dflash-disagg on lmsysorg/sglang:dev-kimi-k31-pro-cu134 with the RadixArk/Kimi-K3.1-Dflash2-Preview draft: TP8/DCP8 prefill with HiCache, feeding one TP16 decode worker (3P1D c300, 2P1D c180, 1P1D c76) or TP8/DCP8 decode workers (1P3D c60 and c36, 1P4D c32). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the contribution!
中文感谢你的贡献！
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| decode: { num-worker: 1, tp: 16, ep: 16, dp-attn: false, dcp-size: 8 } | ||
| - spec-decoding: draft_model | ||
| conc-list: [180] | ||
| srt-recipe: disagg-tp16-variants.yaml:override_2p1d_c180 | ||
| kv-offloading: dram | ||
| kv-offload-backend: { name: hicache } | ||
| prefill: { num-worker: 2, tp: 8, ep: 2, dp-attn: false, dcp-size: 8 } | ||
| decode: { num-worker: 1, tp: 16, ep: 16, dp-attn: false, dcp-size: 8 } | ||
| - spec-decoding: draft_model | ||
| conc-list: [76] | ||
| srt-recipe: disagg-tp16-variants.yaml:override_1p1d_c76 | ||
| kv-offloading: dram | ||
| kv-offload-backend: { name: hicache } | ||
| prefill: { num-worker: 1, tp: 8, ep: 2, dp-attn: false, dcp-size: 8 } | ||
| decode: { num-worker: 1, tp: 16, ep: 16, dp-attn: false, dcp-size: 8 } |
There was a problem hiding this comment.
🟡 (optional) Results for the three TP16-decode rows will be labeled dp-attn: false, but their decode worker actually runs data-parallel attention (attn-dp-size: 2). The master rows at nvidia-master.yaml:7111, 7118 and 7125 do not match the recipe. Fix: set dp-attn: true on the decode entry of all 3 rows, matching the minimax TP16/EP16 decode rows. Same pattern at 3 sites (nvidia-master.yaml:7111, nvidia-master.yaml:7118, nvidia-master.yaml:7125).
Why this was flagged
In disagg-tp16-variants.yaml, decode sets tensor-parallel-size: 16, dcp-size: 8 and attn-dp-size: 2 with enable-dp-lm-head: true, which is DCP8 x attention DP2. The master rows for override_3p1d_c300, override_2p1d_c180 and override_1p1d_c76 declare decode { tp: 16, ep: 16, dp-attn: false, dcp-size: 8 }. AGENTS.md requires the master tp/ep/dp-attn/dcp-size fields to agree with the recipe args. The only dp-attn cross-check, infx/srt_slurm/single_node.py:48, covers single-node recipes and reads enable-dp-attention, not attn-dp-size, so nothing catches the mismatch. The published rows then report these TP16 points as non-DP-attention. The neighbouring minimaxm31flash TP16/EP16 decode rows report dp-attn: true.
Verification: The recipe decode args (disagg-tp16-variants.yaml) set dcp-size: 8 and attn-dp-size: 2, so decode attention runs data-parallel. The master rows at nvidia-master.yaml:7111, 7118 and 7125 still say dp-attn: false. Nothing cross-checks dp-attn against these recipe args. infx/results/agentic/init.py:124 writes decode_dp_attention into the result record.
…block verification Move kimik3.1-fp4-gb300-dynamo-sglang-agentic-dflash-disagg to the multi-arch lmsysorg/sglang:dev-kimi-k31-pro image (digest pinned) and enable speculative-use-block-verification on prefill and decode, matching how the DFLASH golden AL curve was collected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
functionstackx
added
the
full-sweep-fail-fast
|
Will add low-latency points in this PR or in next PR if the current sweep has started for a while |
…2 and c4 Add kimik3.1-fp4-gb300-dynamo-sglang-agentic-dflash-agg-mk: one aggregated TP16 EP1 worker on four GB300 nodes with --enable-nvidia-megakernel, the same dev-kimi-k31-pro image, DFlash2 draft, block verification and verify width 8 (golden AL 3.68) as the disaggregated points. max-running-requests equals the concurrency, so every verify step runs on a megakernel CUDA graph. The megakernel spec needs the trtllm DSA backends, MXFP4 MoE, FP8 KV with 64-token pages, --language-only and an explicit --context-length; TP16 also sets SGLANG_K3_SHARDED_MOE=0. The #4811 changelog entry lists the new key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nvpohanh
changed the title
nvpohanh
removed
the
full-sweep-fail-fast
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=41990069069 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=41990069069 |
|
I am reopening for now but won't trigger sweep |
Port Po-Han Huang's GB300 Dynamo+SGLang DFLASH disaggregated points from #4811: 3P1D c300, 2P1D c180 and 1P1D c76 with a TP16 (DCP8 x attention DP2, EP16 MegaMoE) decode, and 1P3D c60, 1P3D c36 and 1P4D c32 with TP8 DCP8 decodes, served by the Dynamo 1.6.0 the image ships with its KV router and the SGLang chat processor. They replace the SGLang-router disaggregated and wide-EP points; the aggregated c1 point stays a direct SGLang server. Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
|
[by Claude Code] I've started babysitting the pipeline of this PR. I'll keep monitoring it until it passes. If a job fails, I may rerun it when the cause is infrastructure, or push a fix or a merge of the latest |
nvpohanh
added
the
full-sweep-fail-fast
|
nvpohanh
added
non-canary-full-sweep-enabled
|
[by Claude Code] The three failed jobs in run 41990069069 appear to be infrastructure issues, not problems with this PR. All three are the aggregated TP16 megakernel points, and each failed while setting up srt-slurm, before any Slurm job was submitted:
The c4 eval job, which uses the same aggregated recipe on the same runner, got past setup and passed. We will rerun the failed jobs once the jobs still running in this run complete. |
|
/use 41990069069 |
|
@cquil11 staged run 41990069069. This run remains available across future |
|
[by Claude Code] @cquil11 The full sweep run 41990069069 has finished, and every benchmark and eval job passed. The three aggregated TP16 megakernel points (c1, c2, c4) failed during infrastructure setup in the first attempt and passed when rerun in attempt 2. Thanks for pinning it with Performance: https://inferencex.semianalysis.com/inference?unofficialRun=41990069069 |
|
/reuse-sweep-run 41990069069 |
# Conflicts: # inferencex-e2e/perf-changelog.yaml
|
/reuse-sweep-run 41990069069 |
# Conflicts: # inferencex-e2e/perf-changelog.yaml
|
Ingest report — ✅ Results ingested into the Ingest workflow · Merge Ingest run · Benchmark source run
benchmarkFiles 9, rawBenchmarkRows 9, coveredByPerJobFiles 0, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 9233, gpuMetricSamplesNew 0, powerPublicationPoints 9, changelogEntriesWritten 1 Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun. 中文Ingest 报告 — ✅ 结果已写入 上表仅统计本次运行写入的数据，各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。 |
Adds the first Kimi-K3.1-Pro GB300 Dynamo + SGLang AgentX config keys:
kimik3.1-fp4-gb300-dynamo-sglang-agentic-dflash-disagg(disaggregated) andkimik3.1-fp4-gb300-dynamo-sglang-agentic-dflash-agg-mk(aggregated, low latency, with the NVIDIA megakernel). Both runlmsysorg/sglang:dev-kimi-k31-pro(multi-arch digest pinned) with the Dynamo 1.6.0 bundled in the image (dynamo.install: false), and speculate with theRadixArk/Kimi-K3.1-Dflash2-PreviewDFLASH draft (revision pinned).Golden AL. The srt driver injects the committed DFLASH curve from #4798,
kimi-k3.1-pro__dflash__kimi-k3.1-dflash2-preview__default-block.yaml. DFLASH counts the verification width, so curve level N runs--speculative-num-draft-tokens N+1. The c300 point verifies 6 tokens (level 5, golden AL 3.43); every other point verifies 8 (level 7, golden AL 3.68). The curve was collected with block verification, so every role setsspeculative-use-block-verification: trueand the speculative-identity audit matches it with no findings.disagg-tp16-variants.yaml:override_3p1d_c300disagg-tp16-variants.yaml:override_2p1d_c180disagg-tp16-variants.yaml:override_1p1d_c76disagg-tp8-variants.yaml:override_1p3d_c60disagg-tp8-variants.yaml:override_1p3d_c36disagg-tp8-variants.yaml:override_1p4d_c32agg-tp16-mk-variants.yaml:override_tp16_c1agg-tp16-mk-variants.yaml:override_tp16_c2agg-tp16-mk-variants.yaml:override_tp16_c4Node counts include one node shared by the frontend, etcd and the AgentX client.
cutedsl_mladraft attention and an FP8 draft KV cache; prefill and decode use the same verify width. No recipe setsSGLANG_SIMULATE_ACC_*orSGLANG_NVFP4_CKPT_FP8_NEXTN_MOE. The draft is read from the shared Hub cache (HF_HUB_CACHE=/hf_hub_cache), so the pinned revision must be staged there or downloadable from the workers.dsa-*-backend: prims_ts), FP8 KV cache, 16k chunks and shortest-prefill-first scheduling. HiCachewrite_backholds 96 GB per rank, 384 GB per prefill node, inside the 0.80dram-utilizationshare of the gb300-nv node budget. Prefill workers advertise KV routing with a 14400 s session-affinity TTL.enable-linear-replayssm-spec).router-decode-active-request-weightsends each turn to the least-loaded decode worker.--enable-nvidia-megakernel.max-running-requestsequals the concurrency (1, 2 or 4), because the megakernel serves verify steps of 1, 2 or 4 requests and larger batches fall back to the eager path.trtllmDSA prefill and decode backends,flashinfer_mxfp4MoE, an FP8 KV cache with 64-token pages, ReplaySSM speculation,--language-onlyand an explicit--context-length(1048576, the model's maximum).SGLANG_K3_SHARDED_MOE=0: at TP16 the sharded MoE front has no split-K tile for its per-rank width, so graph capture fails without it.MC_TCP_BIND_ADDRESSandWITH_NVIDIA_PEERMEMsettings of the sibling GB300 recipes. One Dynamo frontend uses the KV router and the SGLang chat processor. The AgentX client sends thinking on, temperature 1 and top-p 0.95, with Dynamo session IDs.中文
新增第一批 Kimi-K3.1-Pro GB300 Dynamo + SGLang AgentX config key：
kimik3.1-fp4-gb300-dynamo-sglang-agentic-dflash-disagg（分离式）与kimik3.1-fp4-gb300-dynamo-sglang-agentic-dflash-agg-mk（聚合式低延迟配置，启用 NVIDIA megakernel）。两者均使用lmsysorg/sglang:dev-kimi-k31-pro（固定 multi-arch digest），直接使用镜像自带的 Dynamo 1.6.0（dynamo.install: false），并以RadixArk/Kimi-K3.1-Dflash2-PreviewDFLASH draft（固定 revision）进行推测解码。Golden AL：srt driver 注入 #4798 提交的 DFLASH 曲线
kimi-k3.1-pro__dflash__kimi-k3.1-dflash2-preview__default-block.yaml。DFLASH 计算的是验证宽度，因此曲线的 level N 对应--speculative-num-draft-tokens N+1。c300 配置点验证 6 个 token（level 5，golden AL 3.43）；其余配置点均验证 8 个 token（level 7，golden AL 3.68）。该曲线是在启用 block verification 的情况下采集的，因此每个角色都设置speculative-use-block-verification: true，推测解码身份审计与之完全匹配，没有任何发现项。disagg-tp16-variants.yaml：1 个 TP16 decode worker（DCP8 x attention DP2，EP16 MegaMoE），分别配 3/2/1 个 TP8/DCP8、EP2 prefill worker，并发 300/180/76，节点数 11/9/7。c300 的验证宽度为 6（level 5，golden AL 3.43），其余为 8（level 7，golden AL 3.68）。disagg-tp8-variants.yaml：1 个 TP8/DCP8、EP2 prefill worker，配 3 个 TP8/DCP8、EP2 decode worker（并发 60 与 36，9 个节点）或 4 个（并发 32，11 个节点）。验证宽度均为 8（level 7，golden AL 3.68）。agg-tp16-mk-variants.yaml：1 个聚合式 TP16、EP1 worker，启用 megakernel，并发 1/2/4，5 个节点。验证宽度为 8（level 7，golden AL 3.68）。cutedsl_mla，draft KV cache 为 FP8；prefill 与 decode 的验证宽度一致。recipe 中未设置SGLANG_SIMULATE_ACC_*或SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE。draft 从共享 Hub 缓存（HF_HUB_CACHE=/hf_hub_cache）读取，因此固定的 revision 需要预先放入该缓存，或允许 worker 下载。dsa-*-backend: prims_ts），FP8 KV cache，16k chunk，shortest-prefill-first 调度。HiCachewrite_back每个 rank 96 GB，每个 prefill 节点 384 GB，在 gb300-nv 节点预算的 0.80dram-utilization份额之内。prefill worker 声明 KV 路由及 14400 秒 session affinity TTL。enable-linear-replayssm-spec）。router-decode-active-request-weight将每一轮请求发往负载最低的 decode worker。--enable-nvidia-megakernel。max-running-requests等于并发（1、2 或 4），因为 megakernel 只服务 1、2 或 4 个请求的验证步骤，更大的 batch 会回退到 eager 路径。trtllmDSA prefill 与 decode backend、flashinfer_mxfp4MoE、64 token page 的 FP8 KV cache、ReplaySSM 推测、--language-only，以及显式的--context-length（1048576，即模型最大值）。SGLANG_K3_SHARDED_MOE=0：在 TP16 下，分片 MoE front 的每 rank 宽度没有可用的 split-K tile，不设置该变量会导致 CUDA graph capture 失败。MC_TCP_BIND_ADDRESS与WITH_NVIDIA_PEERMEM设置。一个 Dynamo frontend 使用 KV router 与 SGLang chat processor。AgentX 客户端开启 thinking，temperature 1，top-p 0.95，并携带 Dynamo session ID。AI model disclosure
claude-opus-5-5), via Claude Code.🤖 Generated with Claude Code