[Prefix Caching] DeepSeekv4 - Support selective prefix-cache retention for sliding-window KV cache

[wzhao18]

Co-author: @ivanium

Purpose

DeepSeek v4 now exhibits very low effective prefix cache capacity. For example, on TP8 with 8xB300, the reported KV cache capacity is ~14.5x concurrency. However, a microbenchmark that sends 1M-context requests sequentially shows that after the second request is sent, replaying the first request already begins to miss the prefix cache. This means the practical prefix-cache retention capacity is much lower than the expected KV cache capacity.

After investigation, the root cause is that as a new request progresses, its allocation of sliding window cache blocks, despite constantly being freed as window moves, flushes away the existing prefix cache of older requests in the free queue.

This PR addresses the issue by prepending the non-cached blocks to the front of the free queue while appending the cached blocks to the back of the queue. This way, the non-cached blocks are prioritized for reuse, preventing the cached blocks from being flushed away from the transient sliding window block allocation of concurrent requests.

In addition, this PR supports selective cache retention feature to only save SWA cache checkpoints at certain checkpointing intervals, allowing for greater requent-level KV cache capacity. This is enabled through VLLM_PREFIX_CACHE_RETENTION_INTERVAL:

• None (default): preserve the old behavior (all windows of block-size-aligned tokens are cached).
• 0: retain only the latest replayable prompt boundary.
• positive integer: retain checkpoint tails at the configured intervals, plus the latest replayable prompt boundary.

After this change, the same microbenchmark shows >95% prefix-cache hit rate for 14 concurrent 1M-context requests across prompt fractions from 10% to 100%, matching the reported ~14.5x KV cache capacity.

Test Plan

• tests/v1/core/test_prefix_caching.py
• E2E DeepSeek V4 prefix caching eval + performance

Test Result

VLLM_PREFIX_CACHE_RETENTION_INTERVAL=32768 VLLM_ENGINE_READY_TIMEOUT_S=3600 vllm serve deepseek-ai/DeepSeek-V4-Pro \
  --kv-cache-dtype fp8 \
  --trust-remote-code \
  --block-size 256 \
  --enable-prefix-caching \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \
  --attention_config.use_fp4_indexer_cache True \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v4 \
  --max-cudagraph-capture-size 2048 \
  --max-num-batched-tokens 2048 \
  --no-enable-flashinfer-autotune \
  --tensor-parallel-size 8

GSM8K:

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9553|±  |0.0057|
|     |       |strict-match    |     5|exact_match|↑  |0.9560|±  |0.0056|

Trace replay - semianalysisai/cc-traces-weka-no-subagents-051826:

uv run --active aiperf profile \
    --scenario inferencex-agentx-mvp \
    --url localhost:8000 \
    --model deepseek-ai/DeepSeek-V4-Pro \
    --tokenizer deepseek-ai/DeepSeek-V4-Pro \
    --num-dataset-entries 739 \
    --endpoint-type chat \
    --streaming \
    --public-dataset semianalysis_cc_traces_weka_no_subagents \
    --benchmark-duration 1800 \
    --concurrency 16 \
    --use-server-token-count \
    --ui simple

main:

rps=0.2 (avg 0.1) tput_in=11025/s tput_out=44/s done=161 ok=161 err=0
ttft p50=86023ms p75=108751ms p95=127065ms p99=134300ms
itl  p50=155ms  p75=169ms  p95=179ms  p99=190ms
e2e  p50=127751ms p75=173693ms p95=410403ms p99=755707ms
tin  p50=1,072  p75=2,381  p95=6,253  p99=9,344 (tok/s/user)
tout p50=6      p75=7      p95=7      p99=16 (tok/s/user)
seq  isl_avg=123,177    osl_avg=489
tot  in=19,831,512     out=78,775
srv  prefix_cache_hit=0.0% kv_usage=10.3%

branch:

rps=0.4 (avg 0.3) tput_in=37266/s tput_out=196/s done=560 ok=560 err=0
ttft p50=4499ms p75=7092ms p95=12696ms p99=24552ms
itl  p50=54ms   p75=69ms   p95=119ms  p99=156ms
e2e  p50=20679ms p75=39971ms p95=142013ms p99=302469ms
tin  p50=19,381 p75=38,365 p95=139,270 p99=160,284 (tok/s/user)
tout p50=19     p75=26     p95=51     p99=61 (tok/s/user)
seq  isl_avg=120,992    osl_avg=637
tot  in=67,755,382     out=356,899
srv  prefix_cache_hit=74.3% kv_usage=3.8%

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

[gemini-code-assist]

Code Review

This pull request introduces a sparse local retention mechanism for prefix caching, specifically targeting hybrid KV cache models. It adds a new configuration option, prefix_cache_retention_interval, which allows the system to retain sliding-window KV checkpoints at specified token intervals or at the latest prompt boundary. Key changes include updates to the block pool and free queue to support prioritized block reuse, and modifications to the KV cache managers to handle the lifecycle of these retained checkpoint blocks. I have no feedback to provide as there were no review comments.

[ivanium]

Thanks for the PR! This is a valuable direction. I left a few comments. Happy to discuss.

[ivanium]

Maybe also set self.self.cache_config.prefix_cache_retention_interval = None here for safety

[wzhao18]

done

[ivanium]

Just wanted to leave a note that this should also work for Mamba groups

[wzhao18]

I suggest we revisit Mamba in a separate PR to limit the scope of this PR, especially we need to align with the behaviors of the different Mamba modes e.g., "align", "all" etc

[ivanium]

Minor nit: this should be unnecessary as free_blocks() handles it.
On a second thought, I feel here we can prepend blocks without block_hash to the free list, similar to the logic below in the kv cache manager

[wzhao18]

actually the changes here are legacy of my initial changes which performs the similar logic as the sliding window - prepend non-cache and append cached.

I eventually simplified it because I think the non-cached blocks should be already handled by remove_skipped_blocks before free is called. Let me check whether there is still need to do that here as well.

[wzhao18]

I've added the similar logic to free of the SWA KV manager.

[ivanium]

This part is nice, and I think it actually helps the current DSV4 even without this retention mechanism. This is because we have masked out non-256-boundary SWA blocks so they won't have block_hash, but previously we didn't put it into the front of the free list so they may not be re-allocate the first.

[ivanium]

This is what we discussed offline on caching at the end of each turn's prompt, and I personally think this is the most helpful part

[dafeliton]

Hi there,

Does this work with mtp? I'm noticing a 0% cache prefix hit rate with mtp enabled.

[ivanium]

Hi there,

Does this work with mtp? I'm noticing a 0% cache prefix hit rate with mtp enabled.

Thanks for raising the issue. It's a good point. I think this PR doesn't support MTP yet because MTP will additionally drop a block. But this should be fixable.

[wzhao18]

@dafeliton I just added support for MTP. Could you test it again?

[wzhao18]

I've updated the PR with suggestions from @ivanium. It now drops the CLI knob --prefix-cache-retention-interval and instead uses env var VLLM_PREFIX_CACHE_RETENTION_INTERVAL. We will leave the design of a proper interface for the cache retention strategy to a future PR.

For now, a good default value for VLLM_PREFIX_CACHE_RETENTION_INTERVAL may be 0 if only the full prompt will be reused, or 32k if we also want partial prompts to have cache hits.

[njhill]

Thanks @wzhao18 @ivanium! Great work

[ilyaters]

@wzhao18 @ivanium Thank you! Grateful work!