[KV Cache] Preserve EAGLE SWA replay blocks for external APC

[wangyicong52]

Purpose

This PR preserves reusable sliding-window replay blocks for EAGLE hybrid KV cache groups after remote KV receive, and aligns Mooncake/NIXL sliding-window transfer clipping with the same EAGLE replay slack so Decode-side local APC can reuse prefixes that were first loaded from external prefix cache.

We found this while benchmarking DeepSeek-V4.1-Flash with PD disaggregation on an 8x NVIDIA H20 setup using the iaas-gpu-cn-beijing.cr.volces.com/serving/vllm:community_deepseekv41-flash-0909 image, EAGLE/DSpark enabled, hybrid KV cache enabled, and long shared-prefix Decode traffic generated by sglang.bench_serving with the generated-shared-prefix dataset. In the 64k shared-prefix Decode case, Prefill-side local APC was hot and Decode-side external prefix cache reported full hits, but Decode-side local APC did not grow at all. A fixed-engine discriminator with Decode switched to a single TP8 engine reproduced the same behavior for serial x+a then x+b requests, ruling out Decode DP cross-engine cache isolation as the primary cause.

The root cause is that sliding-window groups may receive or keep only the local attention tail after external KV loading, while EAGLE local prefix-cache lookup needs one extra lookahead block and can fall back to the previous aligned replay boundary when the prompt tail is partial. In that case the full-attention group can have a reusable prefix, but the SWA group is missing the earlier replay-boundary blocks required for the hybrid fixed-point lookup, so the common local APC hit collapses to zero. This PR fixes that by retaining one cache-hit alignment of replay slack for EAGLE SWA managers and by making Mooncake/NIXL transfer clipping preserve the same replay slack for all SWA groups that share an EAGLE-marked KV cache spec.

Relationship to Existing Work

#50897 is the strongest functional overlap. It introduces successor-aware EAGLE prefix hashes and disables the legacy last-block drop for supporting connectors, including direct Mooncake and NIXL. If that broader protocol change lands and is enabled end to end, it can avoid the same Decode local-APC failure without retaining replay slack. It does not implement the conservative legacy-drop path in this PR, and its hash, publication, and connector lifecycle changes are substantially broader.

#56227 takes a different, DeepSeek-V4.1-specific approach behind an opt-in environment flag: it marks the SWA cache as non-prefix-cacheable and recomputes the trailing window after a hit. That avoids transferring and reusing the SWA prefix rather than preserving replay-boundary blocks, and it does not provide a generic direct Mooncake/NIXL fix for prefix-cacheable SWA groups.

#44082 and #54713 fix colocated SWA lookahead and replay-boundary retention, but they do not cover blocks first materialized by direct remote KV receive. #52287 keeps the EAGLE SWA replay window for the offloading scheduler path, but it does not update direct Mooncake/NIXL transfer clipping. #51295 and #53802 improve hybrid replay/checkpoint boundary handling without preserving replay-boundary SWA blocks received from remote Prefill. #44882 concerns concurrent external-prefix materialization and coalescing; #48189 was closed and explicitly excludes NIXL, hybrid/EAGLE, and full-prompt external-prefix paths; #40268 concerns GPU KV-cache scan pollution.

No existing PR covers the exact path implemented here: preserving legacy EAGLE block-drop semantics while retaining one cache-hit alignment of SWA replay slack consistently in the core manager, direct Mooncake/NIXL transfer clipping, and admission/pool sizing. However, #50897 and #56227 can avoid the same user-visible symptom through alternative designs within their supported scope.

Test Plan

Common Configuration

• Hardware: NVIDIA H20
• Image: iaas-gpu-cn-beijing.cr.volces.com/serving/vllm:community_deepseekv41-flash-0909
• vLLM image version: 0.1.dev20904+g179dd0fa9
• Deployment: Prefill/Decode disaggregation over Mooncake RDMA
• Prefix caching: enabled
• A/B baseline: image-original runtime
• Patched variant: PR head d117baf57c50309bb17affb501b6f4ec791ba3ca

DeepSeek-V4.1-Flash

• Prefill: TP8 on one 8x H20 node
• Decode: TP8 single engine on one 8x H20 node
• Speculative decoding: DSpark, 5 speculative tokens
• Fixed-engine correctness probe: two 64k shared-prefix prompts x+a and x+b, max_tokens=1, sent as a first remote load, serial siblings, and overlapping siblings
• Performance dataset: vLLM prefix_repetition, one 65,504-token prefix, 64-token suffix, one same-prefix warmup
• Performance groups: c1/output1 with 8 formal requests, c8/output1 with 40 formal requests, and c8/output128 with 40 formal requests
• Stress dataset: generated-shared-prefix, 64k system prefix, 64-token question, 1,500-token output, concurrency 64, 320 formal requests

GPT-OSS-20B

• Target: openai/gpt-oss-20b
• Draft: RedHatAI/gpt-oss-20b-speculator.eagle3
• Prefill: TP2 on 2x H20
• Decode: TP2 single engine on 2x H20
• Performance dataset: vLLM prefix_repetition, one 65,504-token prefix, 64-token suffix
• A/B/A order: image baseline A1, PR patch B, image baseline A2
• Performance groups: one warmup plus one formal c1/output1 request, c1/output1 with 8 formal requests, c8/output1 with 40 formal requests, and c8/output128 with 40 formal requests
• Sampling: temperature=0, ignore_eos=true

Representative performance command:

vllm bench serve \
  --backend vllm \
  --base-url http://<router>:<port> \
  --model <local-model-path> \
  --served-model-name <served-model-name> \
  --tokenizer <local-model-path> \
  --dataset-name prefix_repetition \
  --prefix-repetition-num-prefixes 1 \
  --prefix-repetition-prefix-len 65504 \
  --prefix-repetition-suffix-len 64 \
  --prefix-repetition-output-len 128 \
  --num-prompts 40 \
  --num-warmups 8 \
  --max-concurrency 8 \
  --request-rate inf \
  --temperature 0 \
  --ignore-eos

Test Result

DeepSeek-V4.1-Flash Fixed-Engine Correctness

Without the PR, the Decode TP8 single-engine probe recorded zero Decode local APC hits even when the same engine received the serial x+a then x+b requests. The serial and overlap phases each queried 131,210 prefix tokens, produced zero local hits, and reported full external-prefix hits.

With the PR:

• First remote load: Decode external hit 65,551, local hit 0
• Serial x+a then x+b: Decode local hit 65,472, external hit 65,628
• Overlapping sibling requests: Decode local hit 130,944, external hit 156
• Every probe request succeeded and stayed on Decode engine 0

This demonstrates that externally received KV is published into Decode local APC and reused by later same-engine requests.

DeepSeek-V4.1-Flash Prefix-Repetition Performance

Group	Metric	Baseline	PR	Change
c1/output1	Request throughput	0.346 req/s	1.556 req/s	+349.9%
c1/output1	Mean TTFT	2891.2 ms	642.4 ms	-77.8%
c1/output1	P99 TTFT	9605.2 ms	652.3 ms	-93.2%
c8/output1	Request throughput	1.285 req/s	5.657 req/s	+340.1%
c8/output1	Mean TTFT	5524.0 ms	1114.8 ms	-79.8%
c8/output1	P99 TTFT	19013.5 ms	3402.5 ms	-82.1%
c8/output128	Output throughput	315.5 tok/s	755.3 tok/s	+139.4%
c8/output128	Mean TTFT	2665.2 ms	821.9 ms	-69.2%
c8/output128	P99 TTFT	10224.6 ms	1185.7 ms	-88.4%
c8/output128	Mean TPOT	3.81 ms	3.37 ms	-11.4%

All groups completed without failed requests and generated all requested output tokens. Formal TTFT did not include TileLang compilation: compilation completed during service startup, and each formal group ran only after readiness and a same-prefix warmup.

For this random-token prefix-repetition workload, Decode local APC stayed at zero and Decode external prefix cache stayed at 100% in both variants. The measured performance gain was associated primarily with Prefill local APC becoming reusable after one warmup.

DeepSeek-V4.1-Flash 64k/c64 Stress Result

The patched 64k/c64 run completed 320/320 formal requests:

• Request throughput: 2.16 req/s
• Output throughput: 3247.40 tok/s
• Mean / P99 TTFT: 15817.56 / 27784.90 ms
• Mean / P99 TPOT: 8.14 / 9.57 ms
• Prefill local APC across warmup and formal: 99.31%
• Decode external prefix-cache hit: 100.00%
• Decode local APC: 0.31%

The fixed-engine probe still produced 65,472 serial and 130,944 overlapping Decode local-hit tokens after this stress run, but high-concurrency traffic remained dominated by external-prefix materialization. Concurrent load coalescing and capacity churn are not addressed by this PR.

GPT-OSS-20B + EAGLE3 Performance

Baseline values below are the mean of A1 and A2, except output128 where A2 had one transient router-to-Decode connection reset and A1 is used.

Group	Metric	Baseline	PR	Change
pair c1/output1	Request throughput	0.235 req/s	1.797 req/s	+665.50%
pair c1/output1	TTFT	4259.44 ms	556.18 ms	-86.94%
c1/output1	Request throughput	0.498 req/s	1.255 req/s	+152.01%
c1/output1	Mean TTFT	2008.55 ms	796.92 ms	-60.32%
c1/output1	P99 TTFT	4379.61 ms	1330.14 ms	-69.63%
c8/output1	Request throughput	3.551 req/s	4.901 req/s	+38.03%
c8/output1	Mean TTFT	2134.78 ms	1515.96 ms	-28.99%
c8/output1	P99 TTFT	5801.49 ms	2577.43 ms	-55.57%
c8/output128	Output throughput	315.07 tok/s	395.64 tok/s	+25.57%
c8/output128	Mean TTFT	1745.22 ms	1094.30 ms	-37.30%
c8/output128	P99 TTFT	6096.41 ms	2691.77 ms	-55.85%
c8/output128	Mean TPOT	10.56 ms	10.48 ms	-0.81%

All A1 and PR formal requests succeeded and generated the requested output-token counts. The strict pair result was repeatable: baseline A1/A2 TTFT was 4257.04/4261.85 ms, while the PR result was 556.18 ms.

On GPT-OSS, Decode local APC remained zero and Decode external prefix cache remained 100% hit for both baseline and patched variants. The performance gain came from Prefill local APC: the pair changed from 0 to 65,536 local-hit tokens, c1 improved from 55.49% to 86.02%, and c8 improved from 93.65% to 97.55%. GPT-OSS therefore validates a cross-model end-to-end performance benefit but does not independently validate Decode external-to-local APC publication.

AI assistance was used for code analysis, test orchestration, and drafting; the submitter reviewed the changes and end-to-end results.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @wangyicong52.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork