[KimiK3][AgentX]: GB200 KimiK3 Day 0 support - #2354
Conversation
Add srt-slurm TP, TEP, and DEP AgentX profiles plus GB200 runner routing and recipe overlay plumbing. 中文：新增 Kimi K3 GB200 首日 AgentX 支持，包括 srt-slurm TP、TEP、DEP 配置，以及 GB200 运行器路由和配方覆盖逻辑。
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
Record the draft pull request URL in the append-only benchmark changelog entry. 中文：在仅追加的基准测试变更日志条目中记录草稿 PR 链接。
cquil11
changed the title
中文：将 origin/main 合并到 GB200 Kimi K3 AgentX 分支。
|
Claude finished @cquil11's task in 8m 24s —— View job Review: Kimi K3 GB200 AgentX Day-0 support
Result: 1 warning, no blocking issues. Left inline on Everything else verified clean:
One non-blocking note: |
Represent the throughput arm as TP1 x DP16 = EP16 in the master configuration, matching the vLLM recipe. 中文：将吞吐量配置修正为 TP1 × DP16 = EP16，与 vLLM 配方保持一致。
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Use the pre-staged /mnt/numa1/models/Kimi-K3 checkpoint on every GB200 compute node while retaining shared-filesystem orchestration and caches. 中文：在每个 GB200 计算节点上使用预置的 /mnt/numa1/models/Kimi-K3 检查点，同时保留共享文件系统上的编排与缓存。
Keep the Day-0 changelog focused on the vLLM recipe performance points and use the model default context length. 中文：精简 Kimi K3 首日性能配置，仅保留与 vLLM 配方一致的性能测试点，并使用模型默认上下文长度。
Move latency and balanced profiles to 16-GPU TP/TEP, use TP2 x DP8 for EP16 throughput, and extend the throughput concurrency sweep through 256. 中文：根据 GB200 HBM 容量调整 Kimi K3 AgentX 扫描。将延迟与均衡配置扩展为 16 GPU TP/TEP，吞吐配置改为 TP2 × DP8（EP16），并将吞吐并发扫描扩展至 256。
cquil11
changed the title
The Dynamo vLLM worker entrypoint does not accept vllm serve tool-call or reasoning parser flags; agentic request parsing belongs to the Dynamo frontend. 中文：移除 Dynamo vLLM worker 不支持的 Kimi K3 工具调用与推理解析参数；AgentX 请求解析由 Dynamo frontend 负责。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30316602813 |
Remove literal shell placeholders from the vLLM worker environment so Gloo and NCCL use the interfaces selected by the srt-slurm launcher. 中文：移除 vLLM worker 环境中的字面量 shell 占位符，由 srt-slurm launcher 为 Gloo 与 NCCL 选择网络接口。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30317459968 |
Patch the pinned agentic srt-slurm checkout to launch one vLLM process per GB200 node, with two local DP ranks and TP2 per rank. Export every per-node metrics endpoint to AIPerf. 中文：修补 AgentX 固定版本的 srt-slurm，使每个 GB200 节点启动一个 vLLM 进程，每个进程承载两个本地 DP rank，并为每个 rank 使用 TP2；同时将所有节点的 metrics endpoint 传给 AIPerf。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30317808862 |
Use TP4 x DP4 for the throughput-oriented EP16 profile after TP2 x DP8 exceeded GB200 HBM during MXFP4 weight loading. 中文：TP2 × DP8 在 MXFP4 权重加载阶段超过 GB200 HBM 容量，因此将吞吐优先的 EP16 配置调整为 TP4 × DP4。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30318430030 |
Pin a tested srt-slurm revision that natively groups TP GPUs within each DP rank and exports every DP metrics endpoint. Remove the vendored patch and reserve enough HBM for one native 1M-context request. 中文：固定经过测试的 srt-slurm 版本，原生支持每个 DP rank 内的 TP GPU 分组，并导出全部 DP metrics endpoint。删除仓库内补丁，同时为原生 1M 上下文请求预留足够 HBM。
| prefill: | ||
| num-worker: 1 | ||
| tp: 16 | ||
| ep: 16 | ||
| dp-attn: false | ||
| additional-settings: | ||
| - "CONFIG_FILE=recipes/vllm/kimi-k3/agentic/agg-gb200-tp16-latency-agentic.yaml" | ||
| decode: | ||
| num-worker: 0 | ||
| tp: 16 | ||
| ep: 16 | ||
| dp-attn: false |
There was a problem hiding this comment.
🟡 WARNING: The TP16 latency profile reports ep: 16, but its recipe (agg-gb200-tp16-latency-agentic.yaml) does not set enable-expert-parallel — experts are TP-sharded, and there is no EP group. vLLM's EP_SIZE = TP × DP rule (cited in commit 86f4e70) only applies when --enable-expert-parallel is on, which is why only the TEP16 and DEP16 recipes set it.
Why it matters: the reported topology metadata is wrong — the frontend will display an EP16 configuration for a run that has no expert parallelism, and it diverges from the repo precedent for pure-TP agentic profiles (dsv4-fp4-gb300-dynamo-vllm-agentic-mtp-agg reports ep: 1 for its TP8/TP4 recipes without the flag, configs/nvidia-master.yaml:7918).
Fix:
| prefill: | |
| num-worker: 1 | |
| tp: 16 | |
| ep: 16 | |
| dp-attn: false | |
| additional-settings: | |
| - "CONFIG_FILE=recipes/vllm/kimi-k3/agentic/agg-gb200-tp16-latency-agentic.yaml" | |
| decode: | |
| num-worker: 0 | |
| tp: 16 | |
| ep: 16 | |
| dp-attn: false | |
| prefill: | |
| num-worker: 1 | |
| tp: 16 | |
| ep: 1 | |
| dp-attn: false | |
| additional-settings: | |
| - "CONFIG_FILE=recipes/vllm/kimi-k3/agentic/agg-gb200-tp16-latency-agentic.yaml" | |
| decode: | |
| num-worker: 0 | |
| tp: 16 | |
| ep: 1 | |
| dp-attn: false |
(Alternatively, if EP16 is intended for the latency profile, add enable-expert-parallel: true to the recipe — but that would no longer match the official multi_node_tp strategy this profile cites.)
| - spec-decoding: none | ||
| conc-list: [32, 64, 96, 128, 192, 256] | ||
| prefill: | ||
| num-worker: 1 | ||
| tp: 4 | ||
| ep: 16 | ||
| dp-attn: true | ||
| additional-settings: | ||
| - "CONFIG_FILE=recipes/vllm/kimi-k3/agentic/agg-gb200-dep16-throughput-agentic.yaml" | ||
| decode: | ||
| num-worker: 0 | ||
| tp: 4 | ||
| ep: 16 | ||
| dp-attn: true |
There was a problem hiding this comment.
🔴 The DEP16 throughput scenario in configs/nvidia-master.yaml (kimik3-fp4-gb200-dynamo-vllm-agentic) sets prefill num-worker: 1, tp: 4 to represent the TP4×DP4=16-GPU recipe (agg-gb200-dep16-throughput-agentic.yaml), but process_agentic_result.py's GPU accounting only multiplies num-worker * tp (no data-parallel factor), so it computes num_gpus=4 instead of 16. This silently inflates every per-GPU throughput/cost metric published for this scenario by 4x. Fix by setting num-worker: 4, tp: 4 (matching the DEP4 disagg precedent) or otherwise ensuring the accounting resolves to 16.
Extended reasoning...
The bug: configs/nvidia-master.yaml lines 8047-8058 add the throughput-oriented search-space entry for kimik3-fp4-gb200-dynamo-vllm-agentic, encoding the DEP16 profile (TP4 × DP4 = 16 GPUs, per agg-gb200-dep16-throughput-agentic.yaml's tensor-parallel-size: 4, data-parallel-size: 4, resources.gpus_per_agg: 16) as prefill num-worker: 1, tp: 4. The GPU-shape accounting in utils/agentic/aggregation/process_agentic_result.py::_gpu_shape() computes num_prefill_gpu = prefill_num_workers * prefill_tp * prefill_pp * prefill_pcp_size — there is no data-parallel-size factor anywhere in that function or in the env vars wired into it. For this entry that evaluates to 1 * 4 * 1 * 1 = 4, not 16.\n\nCode path: .github/workflows/benchmark-multinode-tmpl.yml (lines 226-227) sets PREFILL_NUM_WORKERS and PREFILL_TP verbatim from the master-config's prefill.num-worker/prefill.tp fields for every agentic scenario. _gpu_shape() reads those env vars and multiplies them (plus PP/PCP, both 1 here) to get num_prefill_gpu, which becomes num_gpus for the aggregated (non-disagg) case. That value is then used directly as the denominator in build_agg(): request_nested["throughput"]["per_gpu"] = {"total_tput_tps": request_flat["total_tput_tps"] / num_gpus, ...}. It's also published in the aggregate JSON as the reported GPU/node count for the run.\n\nWhy nothing else catches it: the actual Slurm deployment is driven entirely by the recipe YAML via CONFIG_FILE (srtctl schedules the real 16-GPU topology regardless of these master-config fields), so the benchmark itself runs correctly on 16 GPUs — this is purely an accounting-side defect in result post-processing, invisible at runtime and not something CI's job-success checks would flag. The sibling TP16/TEP16 entries in the exact same PR set tp: 16 and correctly resolve to num_gpus=16, which shows the intended convention and isolates this DEP16 entry as the one exception. The existing DEP4 disagg precedent (dsv4-fp4-gb300-dynamo-vllm-agentic) reaches its 16-GPU total by encoding DP replicas via num-worker: 4, tp: 4, a pattern this new entry doesn't follow.\n\nImpact: every per-GPU throughput and implied per-GPU-cost figure published for the DEP16 throughput profile (concurrency sweep 32-256) will be reported as 4x higher than reality, since the true 16-GPU denominator is replaced with 4. This is exactly the headline metric a 'throughput' scenario exists to report, so the corruption lands in the most visible part of the published benchmark data, not an edge case.\n\nStep-by-step proof:\n1. Recipe agg-gb200-dep16-throughput-agentic.yaml sets resources: {gpus_per_node: 4, agg_nodes: 4, agg_workers: 1, gpus_per_agg: 16} and vllm_config.aggregated: {tensor-parallel-size: 4, data-parallel-size: 4} — 4 nodes × 4 GPUs/node = 16 GPUs actually allocated and used (TP4 × DP4).\n2. configs/nvidia-master.yaml's new throughput search-space entry sets prefill: {num-worker: 1, tp: 4, ...}.\n3. The workflow template exports PREFILL_NUM_WORKERS=1, PREFILL_TP=4 verbatim from those fields.\n4. _gpu_shape() computes num_prefill_gpu = 1 * 4 * 1 * 1 = 4; since decode_num_workers=0, num_gpus = num_prefill_gpu + 0 = 4.\n5. build_agg() divides total_tput_tps (measured across the real 16-GPU deployment) by num_gpus=4 instead of 16 — the published per-GPU throughput is exactly 4x too high, and the aggregate JSON reports 4 GPUs for a run that used 16.\n\nFix: encode the DP replication via num-worker instead of collapsing it, i.e. set prefill num-worker: 4, tp: 4 (matching the DEP4 disagg precedent already in this same file), which yields num_prefill_gpu = 4 * 4 = 16. This changes only the accounting env vars used by result post-processing; the actual deployment remains fully recipe-driven via CONFIG_FILE and is unaffected.
There was a problem hiding this comment.
The config at configs/nvidia-master.yaml:8079-8090 is unchanged from the original report: prefill/decode still set num-worker: 1, tp: 4 for the DEP16 throughput profile. process_agentic_result.py's _gpu_shape() (utils/agentic/aggregation/process_agentic_result.py:148) still computes num_prefill_gpu = prefill_num_workers * prefill_tp with no DP factor, so this resolves to num_gpus=4 for a 16-GPU (TP4xDP4) deployment, inflating per-GPU throughput 4x. No fix has been applied.
| - spec-decoding: none | ||
| conc-list: [1, 2, 4, 8] | ||
| prefill: | ||
| num-worker: 1 | ||
| tp: 16 | ||
| ep: 16 | ||
| dp-attn: false | ||
| additional-settings: | ||
| - "CONFIG_FILE=recipes/vllm/kimi-k3/agentic/agg-gb200-tp16-latency-agentic.yaml" | ||
| decode: | ||
| num-worker: 0 | ||
| tp: 16 | ||
| ep: 16 | ||
| dp-attn: false |
There was a problem hiding this comment.
🟡 The new Kimi K3 latency search-space entry (kimik3-fp4-gb200-dynamo-vllm-agentic, conc-list [1,2,4,8]) sets prefill.ep: 16 / decode.ep: 16, but its recipe (agg-gb200-tp16-latency-agentic.yaml) never sets enable-expert-parallel — it's pure dense TP16 with no MoE expert sharding, unlike its TEP16/DEP16 siblings which do set it. Per repo convention for dense-TP-only agentic configs (see dsv4-fp4-gb300-dynamo-vllm-agentic-mtp-agg), this should be ep: 1; as written, process_agentic_result.py will publish this TP16 profile as an EP16 topology in the aggregated results metadata.
Extended reasoning...
The bug: In configs/nvidia-master.yaml (added by this PR, ~lines 8012-8025), the latency-oriented Kimi K3 GB200 search-space entry sets:
prefill:
num-worker: 1
tp: 16
ep: 16
dp-attn: false
additional-settings:
- "CONFIG_FILE=recipes/vllm/kimi-k3/agentic/agg-gb200-tp16-latency-agentic.yaml"
decode:
num-worker: 0
tp: 16
ep: 16
dp-attn: falseBut the recipe it points at, agg-gb200-tp16-latency-agentic.yaml, only sets tensor-parallel-size: 16 in its vllm_config.aggregated block — it never sets enable-expert-parallel. That makes it a pure dense TP16 deployment with no MoE expert-parallel sharding at all (expert-parallel degree is effectively 1).
Contrast with siblings: The other two new recipes in this PR, agg-gb200-tep16-balanced-agentic.yaml and agg-gb200-dep16-throughput-agentic.yaml, both explicitly set enable-expert-parallel: true (the latter combined with data-parallel-size: 4 for TP4×DP4=EP16). Their master-config entries correctly carry ep: 16. The latency recipe is the odd one out — it deliberately omits expert-parallel (per the official vLLM recipe's multi_node_tp strategy, as opposed to multi_node_tep/multi_node_dep), yet its master-config entry still copies ep: 16 from its siblings.
Repo convention: The established pattern for dense-TP-only agentic aggregate configs is ep: 1. E.g. dsv4-fp4-gb300-dynamo-vllm-agentic-mtp-agg sets tp: 8, ep: 1 and tp: 4, ep: 1 for its two search-space entries, whose recipes (agg-gb300-tp8-mtp-agentic.yaml, agg-gb300-tp4-mtp-agentic.yaml) likewise only set tensor-parallel-size with no enable-expert-parallel — the exact same shape as this Kimi K3 TP16 latency recipe.
Downstream impact: process_agentic_result.py reads PREFILL_EP/DECODE_EP from the master config into the published aggregate result rows as prefill_ep/decode_ep, and derives the top-level ep field as max(prefill_ep, decode_ep). So every published/visualized result point for this TP16 latency profile will be labeled ep: 16, contradicting both the actual serving topology and the recipe's own comment ("TP16 across four GB200 nodes"). This is metadata-only — num_gpus is computed independently from tp × pp × pcp (or num_workers × tp), so GPU accounting, per-GPU throughput, and cost calculations are all unaffected. The only consequence is that anyone reading the published result JSON or a dashboard built on it will see an incorrect ep value for this topology.
Step-by-step proof:
- Master config sets
prefill.ep: 16anddecode.ep: 16for theconc-list: [1, 2, 4, 8]search-space entry pointing atagg-gb200-tp16-latency-agentic.yaml. - That recipe's
vllm_config.aggregatedonly hastensor-parallel-size: 16— noenable-expert-parallel, nodata-parallel-size— so at serve time vLLM runs a plain TP16 dense deployment with EP degree 1. - The launcher (
launch_gb200-nv.sh) passesPREFILL_EP/DECODE_EPfrom the master config (not from the recipe) into the benchmark environment. process_agentic_result.pyreads those env vars intoprefill_ep/decode_epon each result row and computesep = max(prefill_ep, decode_ep) = 16.- The final published/aggregated JSON for this TP16 latency profile therefore reports
ep: 16, even though no expert parallelism is configured — a topology mislabel visible in dashboards and downstream analysis.
Fix: Change prefill.ep and decode.ep to 1 for this latency search-space entry, matching the dsv4-fp4-gb300-dynamo-vllm-agentic-mtp-agg convention for dense-TP-only recipes.
Enable AIPerf burst phase starts only for the Kimi K3 GB200 saturation recipes so warmup and profiling reach the requested concurrency within the benchmark window. Inter-turn replay timing remains unchanged. 中文：仅在 Kimi K3 GB200 饱和度测试配置中启用 AIPerf 阶段起始突发，使预热和正式测试能在基准测试窗口内达到目标并发度，同时保持轮次间回放时序不变。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30324923359 |
Merge the latest main branch while preserving the Kimi K3 changelog entry at the append-only tail. 中文：合并最新 main 分支，并将 Kimi K3 变更日志条目保留在只追加文件的末尾。
Bring in the merged AgentX v1.0 AIPerf default and current benchmark updates while preserving the Kimi K3 changelog entry at the tail.\n\n中文：合并 AgentX v1.0 AIPerf 默认行为及最新基准测试更新，并保留位于末尾的 Kimi K3 变更日志条目。
|
Claude finished @cquil11's task in 2m 44s —— View job Re-review: Kimi K3 GB200 AgentX Day-0 support
Re-review: no new issues, but the two previously flagged blocking issues are still unresolved. Since the last review the only non-merge commits are 64de559 (added per-recipe AIPerf burst phase-start opt-ins) and ff5fb06 (removed them again in favor of the shared default from #2375, which is now merged into this branch) — they net to zero, and the deferral to #2375 is correct. All 8 PR-owned files ( Still open from the previous review (see the existing inline comments — not repeating them):
The non-blocking question about the Dynamo runtime version inside |
Allow the full-context DEP saturation sweep up to 3600 seconds to drain accelerated warmup requests. The grace period exits early when the servers become idle. 中文：将 Kimi K3 DEP 全上下文饱和度测试的加速预热排空上限延长至 3600 秒；服务空闲后会提前结束等待。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30328294420 |
cquil11
changed the title
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30337121602 |
|
/stage-results 30337121602 |
|
@cquil11 staged run 30337121602: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-07-28~r30337121602 This shared staging slot remains available until the next |
中文：将最新 main 分支合并到 Kimi K3 GB200 支持分支。 # Conflicts: # perf-changelog.yaml
Restore the legacy AgentX AIPerf integration with the barrier-aware idle cap, and retain the GB200 Kimi K3 performance changelog entry at the append-only tail. 中文：同步最新 main，恢复带屏障感知空闲上限的旧版 AgentX AIPerf 集成，并将 GB200 Kimi K3 性能变更记录保留在追加式文件末尾。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30399976640 |
Add gb200-nv_3 to the GB200 cluster runner pool so the Kimi K3 sweep can use the capacity provisioned for this bring-up. 中文：将 gb200-nv_3 注册到 GB200 集群运行器池，使 Kimi K3 扫描能够使用本次适配新增的容量。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30405848064 |
cquil11
changed the title
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30406127872 |
|
@cquil11 the AgentX/AIPerf harness has been updated, please merge origin/main into your branch and refresh your submission. Additional tuning may be necessary depending on the config. I apologize for any inconvenience. This is an automated message. |
cquil11
changed the title
1 participant
Summary
/mnt/numa1/models/Kimi-K3checkpoint and keep Kimi K3’s native 1M context.Official recipe: Kimi K3 on GB200.
Shared AgentX phase-start behavior is tracked separately in #2375.
Validation
python -m pytest utils/matrix_logic/ -q— 224 passed on current head.