Kimi K3 vllm GB200 Agentx update

[hjjq]

No description provided.

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32396667947
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=32396667947

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32397164224
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=32397164224

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32416586088
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=32416586088

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32424103771
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=32424103771

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32424103771
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=32424103771

[Ankur-singh]

/reuse-sweep-run 32424103771

[Ankur-singh]

As a PR reviewer and CODEOWNER, I have reviewed this and have:

• Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
• Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
• Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. — https://github.com/SemiAnalysisAI/InferenceX/actions/runs/32424103771
• Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. — https://github.com/SemiAnalysisAI/InferenceX/actions/runs/32424103771
• Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
• For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length.
• Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
• Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this.
• If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo.
• If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.).
• Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
  • N/A — this PR adds no single-node recipes. See the additional detail section.
• Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped.
• If this PR uses append-only: true — N/A, the new changelog entry does not use append-only.
• If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.

Additional detail section:

• Image: vllm/vllm-openai:nightly-dev-arm64-cu13.0.1-728d3ad — upstream vLLM Docker Hub org, verified live (digest sha256:815da6c2253bda19195b9ff8f36ecb28640edad445ab051fdd5d5f698144efb0, pushed 2026-08-19 by vllmbot). It runs as shipped: no .patch, git apply, sed -i on engine sources, site-packages edits, or forked engine wheels anywhere in the diff, so no waiver is required.

• Recipe link is N/A: this is a multi-node submission. All five benchmark files live under benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/, and the master-config entry sets multinode: true with framework: dynamo-vllm. The upstream recipe-link requirement covers single-node serve commands only.

• Golden AL source and value: golden_al_distribution/kimik3_dspark_probabilistic_sample_method_block_rejection_sample_method.yaml, kimi-k3.thinking_on.7 = 3.84. The recipes run num_speculative_tokens: 7 with draft_sample_method: probabilistic, so this is the operative curve — the plain kimik3_dspark.yaml reads 3.78 at K=7 and is correctly not the one used. configs/nvidia-master.yaml:8213 and :8235 pin SYNTHETIC_ACCEPTANCE_LENGTH=3.84, an exact match. Verified at runtime rather than by inspection alone: job 96672622385 logs [Synthetic AR] Injecting synthetic acceptance (length=3.84) and the served --speculative-config carries "rejection_sample_method":"synthetic","synthetic_acceptance_length":3.84. Eval jobs correctly retain real target verification ([Synthetic AL] EVAL_ONLY=true: keeping real MTP recipe).

• Sweep and evals: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/32424103771 on head 349aa162879bdae1c39ca71282f5f555c8aa2946 — 13 executed multi-node agentic / jobs and 7 executed multi-node agentic eval / jobs, all success. gsm8k em_strict (n=1319): new TP8xPP2/DCP8 points c16 0.9735, c32 0.9727, c48 0.9727, c72 0.9704, c96 0.9712; re-validated TP16/DCP16 mtp-c16 0.9727 and nospec-c48 0.9621. utils/evals/thresholds.yaml has no kimik3 override, so the default: gsm8k: 0.90 bar applies and all seven clear it.

• Non-DSpark points (c48/c72/c96): these are permitted today. The speculative-decoding A/B retirement table in MODELS.md is explicitly marked "Status: not yet enacted", and this mirrors the already-merged kimik3-fp4-b200-dynamo-vllm-agentic-dspark config, which has the same DSpark-at-low-concurrency / no-spec-at-high-concurrency shape. Flagging it for the core maintainer in case Kimi-K3's day-0 carve-out is intended to override the not-yet-enacted clause.

• Pruned points: kimik3-fp4-gb200-dynamo-vllm-agentic-mooncake-dcp16-agg goes from 8 concurrency points to 3, removing 16, 24, 32, 36 and 44. This is declared in the changelog and the three survivors were re-run green, but it does remove previously published points from that curve.

• srt-slurm bump: v1.0.36 to v1.0.53, scoped to IS_AGENTIC && MODEL_PREFIX == kimik3 on cluster:gb200-nv. Note that kimik3-fp4-gb200-dynamo-vllm-agentic (configs/nvidia-master.yaml:8043) also matches that guard but is neither listed in the changelog config-keys nor re-run in this sweep. It uses the dynamo frontend (dyn-tool-call-parser / dyn-reasoning-parser recipes) and would jump 17 srt-slurm releases on the next main-branch sweep with no validation here. @hjjq — could you either add that key to the changelog or confirm v1.0.53 preserves the dynamo-frontend recipe schema?

• Two follow-ups that do not block this PR: (1) the mooncake backend version reads 0.3.11.post1 on the new entries versus 0.3.12.post1 on the sibling GB200 keys — consistent with the merged B200 entries on the same image, but the two GB200 keys now report different versions. (2) No Kimi-K3 recipe, merged or new, pins thinking mode, yet the golden AL pinned here is a thinking_on value. That is pre-existing and model-wide rather than introduced by this PR, and deserves its own change.

• Merge conflict: perf-changelog.yaml currently conflicts with main (PR AMD AgentX MI355X: GPU sanity dedup, HiCache tuning, node-exclude scoping, and DSv4 disagg image bump #2643 appended after the same PR-Refresh B200 MiniMax M3 TRT AgentX with stream interval 20 #2686 anchor). configs/nvidia-master.yaml is untouched on main and will not conflict. Per .github/AGENT_OPERATIONS.md, resolve by merging origin/main, restoring perf-changelog.yaml byte-for-byte from origin/main, then appending only this PR's entry at the tail — never a 3-way merge. That resolution will move the head SHA, so this sign-off will need re-posting against the new commit; the green sweep above remains valid because 349aa16 stays in the PR's commit list.

Signed: ankur-singh

[Klaud-Cold]

✅✅✅ Verdict: PASS ✅✅✅

✅ Check 0 (CODEOWNER): PASS — @Ankur-singh is a named owner of configs/nvidia-master.yaml (CODEOWNERS line 7); all other changed paths fall only under the * catch-all, which a recognized CODEOWNER satisfies.
✅ Check 1 (passing sweep on in-PR commit): PASS — head 349aa16 carries run 32424103771: 13 executed multi-node agentic / and 7 executed multi-node agentic eval / check-runs, all success (none skipped).
✅ Check 2 (evals pass): PASS — aggregated eval artifact shows gsm8k em_strict 0.9621–0.9735 (n_eff=1319) across all 7 configs, above the default: gsm8k: 0.90 bar (no kimik3 override); new TP8×PP2/DCP8 evals ran on vllm/vllm-openai:nightly-dev-arm64-cu13.0.1-728d3ad, matching this PR's config image (re-validated DCP16 points on …75c2eef, matching their unchanged config).
➖ Check 3 (recipe link): N/A — disaggregated/multi-node submission (multinode: true, framework: dynamo-vllm, all recipes under benchmarks/multi_node/srt-slurm-recipes/**); the recipe-link requirement applies to single-node recipes only.
✅ Check 4 (reuse command): PASS — /reuse-sweep-run 32424103771 posted by Ankur-singh (COLLABORATOR).
✅ Check 5 (latest checklist): PASS — every item of the current docs/PR_REVIEW_CHECKLIST.md template is present and checked in the sign-off.
✅ Check 6 (upstream image + engine-first): PASS — all added/changed entries use vllm/vllm-openai:nightly-dev-arm64-cu13.0.1-728d3ad from the upstream vLLM Docker Hub org; the dynamo-vllm entry runs the upstream vLLM engine itself (vLLM frontend/backend, dynamo.install: false), which is Kimi-K3's mapped first-class engine per MODELS.md, so no additional-framework ordering issue.
✅ Check 7 (deprecated models/scenarios): PASS — kimik3 agentic-coding is active (added 2026-07-27). The non-DSpark points (c48/c72/c96) are permitted today: the MODELS.md spec-decode A/B retirement table is explicitly "Status: not yet enacted… Every non-spec-decode agentic arm above still runs", and this mirrors the merged kimik3-fp4-b200-dynamo-vllm-agentic-dspark shape and the non-spec GB200 config merged in #2639. The "deprecated from day 0" matrix row remains flagged to the core maintainer, as the sign-off did.
✅ Check 8 (no architecture hacks): PASS — no --hf-overrides/model-config edits; language-model-only and VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION are pre-existing in merged Kimi-K3 recipes and do not remove architecture FLOPs.
✅ Check 9 (chat templates): PASS — the agentic replay client drives /v1/chat/completions with --endpoint-type chat (benchmarks/benchmark_lib.sh:2000).
✅ Check 10 (no engine patches): PASS — no .patch/git apply/heredoc rewrites of engine sources or forked wheels; the srt-slurm v1.0.36→v1.0.53 bump is orchestration harness, and the synthetic-acceptance injector rewrites recipe YAML only.
✅ Check 11 (golden AL): PASS — DSpark points (c16, c32) pin SYNTHETIC_ACCEPTANCE=true with SYNTHETIC_ACCEPTANCE_LENGTH=3.84; the vLLM injector sets rejection_sample_method=synthetic + synthetic_acceptance_length; 3.84 exactly matches golden_al_distribution/kimik3_dspark_probabilistic_sample_method_block_rejection_sample_method.yaml kimi-k3.thinking_on.7, the correct curve for draft_sample_method: probabilistic at num_speculative_tokens: 7. No synthetic knobs appear on non-agentic configs.
➖ Check 12 (append-only): N/A — the new perf-changelog.yaml entry does not use append-only: true.

Notes (non-blocking): (1) the sign-off's open question to @hjjq about the srt-slurm v1.0.53 bump also covering un-re-run kimik3-fp4-gb200-dynamo-vllm-agentic is a fair scoping flag but fails no check here. (2) perf-changelog.yaml currently conflicts with main; resolving it will move the head SHA and require a fresh sign-off, though run 32424103771 stays valid while 349aa16 remains in the PR.

[Oseltamivir]

lgtm