[Hybrid KV Cache][Kimi K3] Retain two committed decode checkpoints

[qianlihuang]

Purpose

This is an extraction from #49574 for the Kimi K3 preserved-thinking case.

With sparse hybrid-cache retention enabled, this PR lets Mamba align groups admit scheduler-aligned states from the committed decode prefix. It keeps at most two additional checkpoints per request in the local prefix-cache hash
map; when a third additional checkpoint is materialized, the oldest local hash is withdrawn without freeing a block that is still referenced.

Periodic checkpoints selected by a positive retention interval remain governed by that interval. Copies already exported to an external KV store remain governed by that store's eviction policy.

Enable the policy with prefix caching and Mamba align mode:

--enable-prefix-caching
--mamba-cache-mode align
VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0
VLLM_PREFIX_CACHE_RETAIN_DECODE_CHECKPOINTS=1

Reuse examples

Kimi K3 preserves and re-renders complete assistant thinking/tool-call history (model card, reference encoding). An exact next-turn request can therefore reuse tokens decoded by the previous request:

• Normal turn: ... assistant(reasoning_content + content) <|end_of_msg|> | user ... can hit through the latest aligned assistant prefix instead of stopping at the earlier prompt boundary.
• Tool turn: ... assistant(reasoning_content + tool_calls) <|end_of_msg|> | tool ... can hit through the latest aligned assistant tool-call prefix before the tool input/result.

The final non-aligned tail is still recomputed. Exact block hashes remain the correctness gate, so modified rendering or dropped thinking safely misses. Rejected speculative tokens are excluded by the existing committed-token cap, and a checkpoint is recorded only when the corresponding Mamba state was actually materialized.

In a P/D deployment, these admitted states are eligible for the existing Decode-side export/offload path and can be matched by Prefill workers using the same backing pool.

Why not exact response-end retention

Keeping exactly one semantic response-end state involves chat renderer/parser and request protocol.

Test plan

.venv/bin/python -m pytest -q tests/v1/core/test_prefix_caching.py
pre-commit run --files \
  vllm/envs.py \
  vllm/v1/core/kv_cache_coordinator.py \
  vllm/v1/core/single_type_kv_cache_manager.py \
  tests/v1/core/test_prefix_caching.py

Test results

• Prefix-caching suite: 95 passed.
• Pre-commit: all hooks passed, including Ruff, formatting, mypy, SPDX,
  and environment-default validation.
• Latest-two comparison: disabled caches the prompt replay point and hits
  (4, 4) tokens; enabled locally retains the latest two decode points and
  hits (16, 20) tokens for an earlier branch and the full assistant replay.
• Added fail-fast coverage for missing retention, Mamba non-align, and
  SWA-only configurations.
• Added speculative-decode coverage for rejected versus committed,
  materialized tokens.
• Model/GPU evaluation: not run. This changes cache admission metadata rather
  than model computation, tokenization, logits, or output accuracy.
  Connector-specific Kimi K3 P/D hit-rate and memory validation remains
  deployment follow-up work.

AI assistance

AI was used to analyze the cache/offload paths, implement the change, and run the checks above.