[wangyicong52]

Overview

Follow up on #59857 to enable automatic prefix caching (APC) together with encoder-side SWA bounded replay for DeepSeek-V4.1-Flash under PCP in a colocated model instance. The initial PCP implementation rejected APC; this change enables the PCP + APC + encoder-side replay combination.

Claims

• Enable local APC prefix hits under PCP and rebuild the SWA tail through encoder-side bounded replay while preserving cached compressed KV and indexer keys across ranks.
• Decoder-side tail trimming remains disabled under PCP; scheduled rows execute the normal decoder layers.

Validation

Environment

• Model: DeepSeek-V4.1-Flash
• Hardware: two separate nodes, each with 8x NVIDIA H20 97,871 MiB; driver 535.161.08
• Image: vllm/vllm-openai:nightly-49cb2bbf71cfd36ee5aa33d1e184b1cddff55294
• Baseline: the image + prerequisite [Feature][PCP] Add DeepSeek-V4.1-Flash support #59857
• Candidate: the same image and prerequisite + This PR; APC and bounded replay were enabled.
• Common topology: TP2/PCP4, PP1, DCP1, Humming MoE, eager execution, and synchronous scheduling.

Candidate launch command:

python -m vllm.entrypoints.cli.main serve /models/DeepSeek-V4.1-Flash \
  --host 0.0.0.0 \
  --port 18001 \
  --served-model-name deepseek-v4.1-flash \
  --language-model-only \
  --tokenizer-mode deepseek_v41 \
  --attention-backend FLASHMLA_SPARSE_DSV41 \
  --kv-cache-dtype fp8_ds_mla \
  --block-size 64 \
  --tensor-parallel-size 2 \
  --pipeline-parallel-size 1 \
  --decode-context-parallel-size 1 \
  --prefill-context-parallel-size 4 \
  --enable-expert-parallel \
  --all2all-backend allgather_reducescatter \
  --engram-config '{"cpu_offload":true}' \
  --moe-backend humming \
  --max-model-len auto \
  --max-num-batched-tokens 4096 \
  --max-num-seqs 64 \
  --gpu-memory-utilization 0.9 \
  --enable-prefix-caching \
  --swa-bounded-replay \
  --enforce-eager \
  --no-async-scheduling

Baseline used the same command except:

  --no-enable-prefix-caching \
  --no-swa-bounded-replay

Prefix-repetition performance

Both arms ran concurrently with byte-identical deterministic inputs:

vllm bench serve \
  --backend openai \
  --base-url http://<HOST>:18001 \
  --endpoint /v1/completions \
  --model deepseek-v4.1-flash \
  --tokenizer /tmp/dsv41-tokenizer \
  --tokenizer-mode deepseek_v41 \
  --dataset-name prefix_repetition \
  --prefix-repetition-prefix-len 1024 \
  --prefix-repetition-suffix-len 128 \
  --prefix-repetition-num-prefixes 10 \
  --prefix-repetition-output-len 1 \
  --num-prompts 80 \
  --num-warmups 8 \
  --request-rate inf \
  --max-concurrency 8 \
  --seed 42 \
  --temperature 0 \
  --ignore-eos \
  --percentile-metrics ttft,e2el \
  --disable-tqdm

Metric	Baseline	Candidate	Delta
Successful requests	80/80	80/80	-
Mean TTFT	640.73 ms	386.73 ms	-39.64%
Median TTFT	570.56 ms	346.28 ms	-39.31%
P99 TTFT	907.04 ms	712.10 ms	-21.49%
Effective input throughput	13,837.87 tok/s	22,499.21 tok/s	+62.59%
Request throughput	12.01 req/s	19.53 req/s	+62.59%
Prefix-cache token hit ratio	disabled	80,384/101,376 (79.29%)	-

Every measured request had exactly 1,152 input tokens and one output token. There were no failed requests or Pod restarts.

Full GSM8K

Both arms ran the same full 1,319-sample ModelScope test split:

evalscope eval \
  --eval-type openai_api \
  --api-url http://<HOST>:18001/v1 \
  --api-key EMPTY \
  --model deepseek-v4.1-flash \
  --model-id deepseek-v4.1-flash \
  --datasets gsm8k \
  --dataset-hub modelscope \
  --dataset-args '{"gsm8k":{"few_shot_num":4}}' \
  --seed 42 \
  --eval-batch-size 64 \
  --generation-config '{"temperature":0.0,"top_p":1.0,"max_tokens":8192,"reasoning_effort":"high","seed":42,"timeout":10000}' \
  --timeout 10000

EvalScope version: 1.11.1.

Metric	Baseline	Candidate
Accuracy	1279/1319 (96.97%)	1284/1319 (97.35%)
API errors	0	0

Paired outcomes were 1,277 both correct, 33 both wrong, 7 candidate-only correct, and 2 baseline-only correct. The exact two-sided McNemar p-value was 0.1797, so the observed +0.38 pp accuracy difference is not statistically significant. Candidate GSM8K traffic recorded 674,816/803,206 local prefix-cache hit tokens (84.02%).

Details

Reuse the existing scheduler replay boundary and padding kernel, stage replay starts per local PCP segment, mask each rank's own cacheable slots, and gather those slots back in the same rank-major order as KV publication. All ranks participate even when a peer has no replay rows; SWA and compressor scratch mappings stay unchanged. The additional slot metadata collective applies only to PCP batches with encoder replay enabled. Production changes are confined to DeepSeek-V4.1 files, with the existing non-PCP path and other PCP restrictions retained.

This does not duplicate #61554, which targets cold-prefill decoder trimming with prefix caching disabled. Enable APC and encoder-side replay with --enable-prefix-caching --swa-bounded-replay. The supported scope is NVIDIA CUDA, MRV2, eager execution, synchronous scheduling and PP=DP=DCP=1. KV connectors, speculative decoding, microbatching and MegaMoE remain unsupported with PCP. AI assistance was used for implementation and Codex CLI review; human line-by-line review remains pending, while GPU/model validation is documented above.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

[mergify]

Documentation preview: https://vllm--62183.org.readthedocs.build/en/62183/

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[wangyicong52]

@LucasWilkinson and @ivanium, could you please take a look at this PR when you have time? It brings together the PCP and SWA bounded replay work that you’re leading, which I think is an important step toward improving dsv4.1 prefill performance when PCP is enabled. Thanks a lot!

[LucasWilkinson]

LGTM thanks!

[LucasWilkinson]

/ci run

[github-actions]

✅ @wangyicong52, CI is now available for this PR.

• /ci run starts upstream CI; /amd-ci run starts AMD CI only.
• Run commands test your branch with every commit currently on its upstream target branch. If it is behind, the bot merges the target branch into it first (pull before pushing again). Append --allow-stale to a run command to test an outdated branch as is, at your own risk.
• /ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
• /amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
• /ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

[github-actions]

✅ Triggered Buildkite CI #95781 for commit d95eb1f90527.