Kimi K3 vllm GB200 Agentx - #2639
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
1 similar comment
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32092422442 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| - config-keys: | ||
| - kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-dcp16-disagg | ||
| - kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-dcp16-agg | ||
| - kimik3-fp4-gb200-dynamo-vllm-agentic-mooncake-dcp16-agg | ||
| scenario-type: | ||
| - agentic-coding | ||
| description: | ||
| - "Add Kimi-K3 GB200 DCP16 AgentX configurations with Mooncake DRAM offload." | ||
| - "Cover 1P1D, DSpark max-num-seqs 2, and no-spec aggregate profiles using the rebuilt vLLM image." | ||
| pr-link: TBD |
There was a problem hiding this comment.
🔴 perf-changelog.yaml still has pr-link: TBD (line 6088), but this diff is PR #2639, which already exists. Per docs/configuration-procedures.md:254, TBD is only a placeholder used before the PR exists and must be replaced with the real URL immediately after creation; line 272 lists a lingering TBD as a hard stop condition that blocks sweep readiness. Fix: set pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2639.
Extended reasoning...
What's wrong
The new perf-changelog.yaml entry added by this PR (lines 6079-6088) leaves pr-link: TBD. Every sibling entry in the same file — including the three immediately preceding this one (PRs #2621, #2619, #2636) — carries a concrete https://github.com/SemiAnalysisAI/InferenceX/pull/<n> URL. This new entry is the odd one out.
Why it's a bug, not a style nit
docs/configuration-procedures.md:254 spells out the exact rule this repo enforces for this field:
Before the PR exists, the model+hardware playbook permits
pr-link: TBD. Replace it with the real URL immediately after creating the PR.
And docs/configuration-procedures.md:272 lists a stop condition that blocks a config from being sweep-ready:
The changelog change ... still has
TBDwhen the PR is otherwise ready for sweep.
This diff is PR #2639 (per pr-metadata: created 2026-08-18, title "Test Kimi Agentx GB200"). Since the PR already exists at the time this changelog entry is being added, the TBD placeholder condition ("before the PR exists") no longer applies — the entry should have been created with (or promptly updated to) the real URL, not left as TBD.
Why nothing else catches this
YAML parsing/schema validation won't flag pr-link: TBD as invalid — it's a syntactically valid string in a string field, so a naive "does this file parse" check passes. The enforcement is procedural: troubleshooting.md confirms this correction is "enforced by the validator, not by a visually valid YAML parse," i.e. it requires a dedicated readiness check that inspects the value semantically, not the YAML loader itself. Nothing in this diff's three other changed files (the two agg recipes, the disagg recipe, or configs/nvidia-master.yaml) touches or re-derives this field, so it will persist as-is unless someone catches it in review.
Step-by-step proof
docs/configuration-procedures.md:254establishes:TBDis valid only while the PR does not yet exist.pr-metadatafor this diff shows PR Kimi K3 vllm GB200 Agentx #2639 already created at2026-08-18T02:35:20Z.- Therefore, at the time this changelog entry exists in the diff, the PR already exists, and the "before PR exists" exception no longer holds.
- The entry at
perf-changelog.yaml:6088readspr-link: TBD, violating the "replace immediately after creating the PR" instruction. docs/configuration-procedures.md:272explicitly lists "still has TBD when the PR is otherwise ready for sweep" as a stop condition — meaning a downstream readiness validator will block this configuration from being marked sweep-ready until fixed.- Fix: change line 6088 to
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2639, matching the convention used by every adjacent entry (.../pull/2621,.../pull/2619,.../pull/2636).
Fix
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2639| tensor-parallel-size: 16 | ||
| decode-context-parallel-size: 16 | ||
| enable-cumem-allocator: true | ||
| trust-remote-code: true | ||
| no-enable-flashinfer-autotune: true | ||
| max-cudagraph-capture-size: 512 | ||
| # Cap prefill scheduling below the available Mamba cache-block count. | ||
| # The largest configured benchmark point is c48. | ||
| max-num-seqs: 512 | ||
| stream-interval: 10 | ||
| max_num_batched_tokens: 16384 | ||
| language-model-only: true | ||
| attention-backend: "TOKENSPEED_MLA" | ||
| attention-config: '{"mla_prefill_backend": "TRTLLM_RAGGED", "use_prefill_query_quantization": true}' | ||
| speculative-config: '{"model":"Inferact/Kimi-K3-DSpark","attention_backend":"TOKENSPEED_MLA","method":"dspark","num_speculative_tokens":4,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}' | ||
| # Explicit budgets leave room for the TP16/DCP16 all-gather and graph | ||
| # workspace on a 184 GiB GB200 GPU. | ||
| kv-cache-memory: 10737418240 | ||
| decode: | ||
| kv-transfer-config: '{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_load_failure_policy":"recompute","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"enforce_handshake_compat":false,"enable_cross_layers_blocks":false}},{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both","kv_load_failure_policy":"recompute","kv_connector_extra_config":{"load_async":true,"lookup_async":true,"enable_cross_layers_blocks":false,"enable_offload":false}}]}}' | ||
| served-model-name: "moonshotai/Kimi-K3" | ||
| enable-prefix-caching: true | ||
| prefix-match-unit: 128 | ||
| load-format: "safetensors" | ||
| safetensors-load-strategy: "lazy" | ||
| kv-cache-dtype: "fp8" | ||
| tensor-parallel-size: 16 | ||
| decode-context-parallel-size: 16 |
There was a problem hiding this comment.
🟡 The disagg Kimi-K3 DCP16 recipe never sets dcp-comm-backend in its prefill (line 120) or decode (line 146) vllm_config blocks, even though decode-context-parallel-size: 16 is set in both. The two sibling GB200 DCP16 aggregate recipes added in this same PR (and the single-node Kimi-K3 vLLM launcher) always pin dcp-comm-backend: "a2a" alongside DCP16, so this recipe likely runs on a different, unvalidated DCP collective backend than its siblings, undermining benchmark comparability. Recommend adding dcp-comm-backend: "a2a" to both blocks to match the agg profiles.
Extended reasoning...
Both new sibling aggregate recipes in this PR (agg-gb200-dcp16-dspark4-maxseq2-mooncake-agentic.yaml:92 and agg-gb200-dcp16-nospec-mooncake-agentic.yaml:92) pair decode-context-parallel-size: 16 with an explicit dcp-comm-backend: "a2a" line. The disagg recipe added in the same PR sets the identical decode-context-parallel-size: 16 in both its prefill (line 120) and decode (line 146) vllm_config blocks, but never sets dcp-comm-backend in either block. The existing single-node Kimi-K3 launcher (kimik3_fp4_b300_vllm_mtp.sh:47) reinforces that this is a deliberate, non-default pairing for this model: it unconditionally appends --dcp-comm-backend a2a any time DCP is enabled, which would be redundant if vLLM's default DCP backend were already a2a.
The internal-consistency signal is the strongest piece of evidence: the disagg recipe's prefill_environment/decode_environment block (shared via the &kimi_env YAML anchor) sets VLLM_USE_DIRECT_DCP_A2A: "0", VLLM_USE_DIRECT_DCP_Q_GATHER: "0", and VLLM_USE_DIRECT_DCP_KV_GATHER: "0" — the exact tuning knobs that only have meaning once the a2a DCP backend is actually selected. The agg siblings carry the same env vars and explicitly pin dcp-comm-backend: "a2a"; the disagg recipe carries the env vars but omits the backend selector that activates them. That looks like an oversight rather than an intentional divergence, since the author clearly intended to carry over the a2a-specific tuning.
Proof by walkthrough: (1) agg-gb200-dcp16-nospec sets tensor-parallel-size: 16, decode-context-parallel-size: 16, dcp-comm-backend: "a2a" at lines 90-92. (2) The disagg recipe's decode block sets the same tensor-parallel-size: 16 and decode-context-parallel-size: 16 at lines 145-146, but the next config key is enable-cumem-allocator — dcp-comm-backend is simply absent. (3) When vLLM parses this config, it falls back to whatever its internal default DCP comm backend is (not necessarily a2a), so the actual collective implementation exercised on the disagg workers can differ from the agg siblings' validated a2a path, even though every other DCP-adjacent knob (the VLLM_USE_DIRECT_DCP_* env vars) is identical.
The refutation raises a fair point: without confirming vLLM's actual default DCP backend, we can't be certain this causes a functional divergence — if the default already resolves to a2a, the omission is harmless redundancy elsewhere. It's also correct that the DeepSeek-V4 launchers use DCP without an explicit backend, showing the pattern isn't universal across models in this repo. But that only shows DCP-without-backend is acceptable for DeepSeek; it doesn't establish that a2a is the vLLM default, and the Kimi-K3-specific evidence (single-node launcher's unconditional flag, and the a2a-only env knobs mirrored in this exact recipe) points the other way. Because there's no crash and the recipe still executes and returns results either way, this reads as a benchmark-fidelity/consistency gap rather than a functional break — the disagg numbers might not be comparable to the agg numbers the PR is meant to validate against.
Fix: add dcp-comm-backend: "a2a" to both the prefill (near line 120) and decode (near line 146) vllm_config blocks in disagg-gb200-1p1d-dcp16-dcp16-dspark4-mooncake-agentic.yaml, matching the two agg siblings added in this PR.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32094657943 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32097870583 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32097870583 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32144394011 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32155442987 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32155647152 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32166066784 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32180077157 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32180077157 |
|
/stage-results 32180077157 |
|
/reuse-sweep-run 32180077157 |
|
/stage-results 32180077157 |
|
|
/stage-results 32180077157 |
|
@cquil11 how do we move forward with this ? worse case we trigger a re-run, lmk :) |
Add TP16/DCP16 Mooncake configurations for 1P1D and aggregate AgentX coverage using the rebuilt vLLM image. 新增使用重建 vLLM 镜像的 TP16/DCP16 Mooncake 配置，覆盖 1P1D 与聚合 AgentX 场景。 Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
Updated PR link for Kimi-K3 GB200 DCP16 AgentX configurations.
Use the same NVIDIA/srt-slurm v1.0.36 tag as the GB300 AgentX profiles so Mooncake-backed DCP16 recipes use the expected schema. 与 GB300 AgentX 配置统一使用 NVIDIA/srt-slurm v1.0.36，确保 Mooncake DCP16 配置使用兼容的 schema。 Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
Keep max-num-seqs below the Mamba page capacity provided by the explicit 10 GiB KV cache allocation. Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
Drop the 1P1D DCP16/DCP16 disaggregated profile and sweep the two aggregate arms across their intended concurrency ranges: max-num-seqs 512 over 8-48 for the no-spec arm, max-num-seqs 2 over 1-16 for the DSpark K=4 arm. Load weights with fastsafetensors and size the KV cache from gpu-memory-utilization 0.92 (matching agg-gb200-tp16-latency) instead of an explicit 10 GiB kv-cache-memory budget. Mooncake's global_segment_size is per TP rank, so the 96GB setting took 4 x 96 GiB = 412 GB per four-GPU GB200 node while dram-utilization 0.1065 reported only 96 GB. Use 128 GiB per rank (512 GiB per node) with dram-utilization 0.61, the budget already validated for Kimi-K3 on cluster:gb200-nv and consistent with the DeepSeek-V4 GB200 recipes. Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 32144394011 failed all three DCP16 aggregate points with two distinct out-of-memory errors. The DSpark arm died in fastsafetensors get_empty_tensor at shard 5/6: a 2.19 GiB staging buffer failed with 47.58 GiB free, i.e. HBM fragmentation rather than exhaustion. Every other GB200 Kimi-K3 recipe sets PYTORCH_ALLOC_CONF=expandable_segments:True and these two did not, including the two siblings that load a DSpark draft under fastsafetensors without trouble. The no-spec arm loaded, then OOMed in cudagraph capture_end at graph 57/83 with the KV cache at 30.34 GiB. Graph 57 is capture size 608 -- above max-num-seqs 512, and with no drafting capture sizes are sequences, so 32 of the 83 graphs were unreachable. Cap the ladder at max-num-seqs and give the graph pool 3.68 GiB back by derating to 0.90; the DSpark arm's ladder is capped at 64 for the same reason, since 2 seqs x (1+4) tokens can never exceed 10. Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Capture sizes are tokens when drafting and prefill batches run on the captured graphs too, so a 64-wide ladder would have pushed short prefills into eager mode. Match the no-spec arm at 512, which still drops the tail above 512 where cudagraph capture OOMed. Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 32155647152 sat in the health-check loop for over 90 minutes with no worker registering and no crash, so the fastsafetensors path is not worth further iteration on GB200. - Restore load-format safetensors with the lazy strategy on both aggregate arms. Slower, but it does not stage shards on the GPU. - Derate gpu-memory-utilization to 0.85. At 0.92 the KV cache took 30.34 GiB and cudagraph capture OOMed; 0.85 leaves the graph pool, the FlashInfer MoE workspace and fragmentation more room. - Cut the health-check window from 2 hours to 1 (360 x 10s), matching VLLM_ENGINE_READY_TIMEOUT_S, so a wedged worker fails fast instead of holding four GB200 nodes for a second idle hour. Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Roll the two aggregate recipes back to 55d3cd5, keeping only the two intended changes on top: drop the 1P1D DCP16 disaggregated profile and sweep the aggregate arms across concurrency. The tuning attempts layered on since then are all reverted -- the fastsafetensors loader, the removed kv-cache-memory budget, the gpu-memory-utilization derate, the narrowed cudagraph ladder, the expandable-segments allocator config, the 128 GiB Mooncake segment and the shortened health-check window. Neither aggregate arm has completed a run at these settings, so keep the checked-in baseline and change one thing at a time from here. Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
179d897 to
7f31b73
Compare
4 participants
No description provided.