[ZJY0516]

Purpose

Kimi-K3 has no end-to-end prefix-cache coverage. FlashKDA exports Mamba prefill checkpoints, a path the Qwen3.5-based test_hybrid_prefix_cache_hit_rate.py never takes, so regressions such as the dropped prompt-end checkpoint fixed by #59146 / #59175 reach users unnoticed. KV offload, P/D and DCP are not tested with K3 at all.

This adds tests/models/kimi_k3/test_prefix_cache.py. It serves riverclouds/Kimi-K3-8L-dummy (the official Kimi-K3 config with 8 layers and no weights) with --load-format dummy, and runs 3 turns where each turn extends the previous prompt. Each turn must:

• hit at least the previous prompt's end checkpoint (cached_tokens), and
• produce the same greedy tokens as a recompute of the same prompt under a fresh cache_salt (check_logprobs_close). This catches restored state that is wrong even though the hit count looks right.

With offloading, GPU prefix caches are reset before each turn, so the hit has to come back from CPU.

Every deployment runs in 4 modes:

• block: Mamba-block granularity.
• partial: --prefix-match-unit 128, i.e. partial hits.
• block-dspark and partial-dspark: the same, plus DSpark speculative decoding as in the recipe. The draft is riverclouds/Kimi-K3-DSpark-for-8L-dummy, which is Inferact/Kimi-K3-DSpark with target_layer_ids remapped to the 8-layer target and no weights. With EAGLE-style block drop, the expected hit is one unit lower.

deployment	GPUs
plain	1
SimpleCPUOffloadConnector (config from the Kimi-K3 recipe)	1
DCP2	2
DCP2 + SimpleCPUOffloadConnector	2
P/D (NIXL) TP1 → TP1	2
P/D TP2 → DEP2 (decode: DP2 + EP, TP1 per rank; prefix caching on P only, since NIXL rejects it on a hybrid decoder with a different TP)	4
P/D + SimpleCPUOffloadConnector (MultiConnector)	2

CI: a new (4xB200) Kimi K3 Prefix Cache Shard %N step runs the file in 2 shards. It installs NIXL with .buildkite/scripts/install-kv-connectors.sh first, because the CI image does not include it. It is triggered by vllm/v1/core/, vllm/v1/spec_decode/, the NIXL and SimpleCPUOffload connectors, vllm/model_executor/layers/mamba/ and vllm/models/kimi_k3/. The existing 1xB200 Kimi K3 step ignores this file.

Known failures (strict xfail)

• dcp2-*-dspark, dcp2-offload-*-dspark: FlashInfer MLA DCP decode crashes on ragged spec-decode batches (causal_seqlens_kv_global must have shape (9,), got (2,)). This is fixed by [Bugfix][Attention] Fix FlashInfer MLA DCP decode on ragged spec-decode batches #59392.

Not covered

P/D + DCP: NIXL with DCP sets cp_kv_cache_interleave_size to the block size (vllm/config/vllm.py), but FlashInfer MLA native DCP requires 1, so the server fails to start. A P TP → D DCP case can be added once that is fixed.

Depends on #59146 or #59175

On current main, every block-granularity case fails at turn 1 with cached=0 expected=17280, which is the bug those PRs fix. Partial cases pass on main. This PR stays a draft until one of them lands.

Not a duplicate

There are no open PRs adding Kimi-K3 prefix-cache, offload, P/D or DCP tests. #59146 and #59175 fix the scheduler bug and add unit tests only.

Test Plan

On GB200 (SM100), with #59146 cherry-picked onto main (379e9a1):

pytest -v -s tests/models/kimi_k3/test_prefix_cache.py
pre-commit run --files tests/models/kimi_k3/test_prefix_cache.py .buildkite/test_areas/models_basic.yaml

Test Result

On main + #59146, one 4xGB200 node. The full run predates two changes: pd-tp1-tp2 was replaced by pd-tp2-dep2, and pd-dcp2 was dropped. It ran 32 cases: 24 passed, 4 failed, 4 xfailed in 5136.53s (1:25:36). The 4 failures are the dcp2*-dspark cases, marked xfail after the run started. The 4 xfails were pd-dcp2. The pd-tp2-dep2 cases were run separately on this branch at c2a0aaa (merged with main): 4 passed in 1138.28s (0:18:58).

deployment	block	partial	block-dspark	partial-dspark
plain, offload	pass	pass	pass	pass
dcp2, dcp2-offload	pass	pass	xfail (#59392)	xfail (#59392)
pd, pd-offload	pass	pass	pass	pass
pd-tp2-dep2	pass	pass	pass	pass

Each case prints its turns. For example, dcp2-offload-partial, with GPU prefix caches reset before every turn:

turn  prompt  cached  expected  matches_recompute
   0    8869       0         0  True
   1    9646    8832      8832  True
   2   10423    9600      9600  True

With DSpark (pd-offload-partial-dspark), the expected hit drops by one 128-token unit:

turn  prompt  cached  expected  matches_recompute
   0   18469       0         0  True
   1   19246   18304     18304  True
   2   20023   19072     19072  True

On main without the fix, plain-block fails at turn 1 (cached=0 expected=17280) and plain-partial passes. Each server start takes about 70–90 s.

With OffloadingConnector instead, DCP2 hits CPU only when the prompt-end checkpoint lands on a multiple of the DCP-scaled MLA chunk. That connector is not in the recipe, so it is not covered here.

No model outputs change (test and CI only), so no eval was run.

AI assistance (Claude Code) was used to write this test.

🤖 Generated with Claude Code

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[ZJY0516]

/ci run --allow-stale

[github-actions]

✅ Triggered Buildkite CI #92145 for commit 947b2bbf64c0.

⚠️ This PR is 89 commits behind upstream main. Running CI at your own risk because --allow-stale was requested; outdated CI configuration may cause failures. Before merging, merge or rebase onto the latest main, then rerun /ci run on the latest PR commit.

[mergify]

Hi @ZJY0516, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

[mergify]

Hi @ZJY0516, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

[ZJY0516]

/ci run --allow-stale

[github-actions]

✅ Triggered Buildkite CI #92163 for commit 9293b44729d0.

⚠️ This PR is 1 commit behind upstream main. Running CI at your own risk because --allow-stale was requested; outdated CI configuration may cause failures. Before merging, merge or rebase onto the latest main, then rerun /ci run on the latest PR commit.

[vllm-agent]

CI selector (shadow): 10 test steps (23 jobs) instead of 10 (25 jobs)

Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today's rules. How it works.

Feedback welcome: reply here if it would skip a step this change needs, or runs something unrelated.

steps (jobs)	Today's rules	Selector	Would skip	Would add
NVIDIA, CPU and others	10 (25)	10 (23)	1 (3)	1 (1)
AMD mirrors	8 (23)	9 (24)	0 (0)	1 (1)

Selector would run (10)

• basic-models-test-other-cpu
• basic-models-tests-extra-initialization ×14
• basic-models-tests-initialization
• basic-models-tests-other
• deepseek-v4-kernel-test-b200
• glm5next-unit-tests
• inkling-unit-tests-b200
• kimi-k3-unit-tests-b200
• qwen4-exp-unit-tests
• qwen4-exp-unit-tests-cpu

Would skip (today's rules run them) (1)

• distributed-model-tests-2-gpus ×3

Would add (today's rules do not run them) (1)

• deepseek-v4-kernel-test-b200 (code map)

AMD mirrors: would skip (0)

none

AMD mirrors: would add (1)

• deepseek-v4-kernel-test-b200 (code map)

_{2 changed files · base b558f160a2 · head 774cf86a7f · Python record: build 92561 at 4056c8ac1f · kernel record: table 4056c8a (build 92561), map 4056c8a · not counted: 10 build steps, 5 A100 steps the generator no longer emits, 0 optional steps the selector would also run}

[AndreasKaratzas]

I know that kimi is fully supported by AMD and that we do have the necessary kernels. And since we are trying to close the parity gap in ci, could you add rocm support on this test?

[ivanium]

sorry for being late. Overall great, one minor comment is how about we do PD TP2->DEP2? which is perhaps closer to the real use case.

[ZJY0516]

I know that kimi is fully supported by AMD and that we do have the necessary kernels. And since we are trying to close the parity gap in ci, could you add rocm support on this test?

You can update it after I merge this

[ZJY0516]

/ci run --allow-stale

[github-actions]

✅ Triggered Buildkite CI #92584 for commit 774cf86a7f91.

⚠️ This PR is 2 commits behind upstream main. Running CI at your own risk because --allow-stale was requested; outdated CI configuration may cause failures. Before merging, merge or rebase onto the latest main, then rerun /ci run on the latest PR commit.