[BugFix] Reuse partial attention pages below their published boundary

[logprobz]

A 2,048-token GLM-5.3 prompt under TP4/DCP4 returned correct answers but reused zero tokens on an identical warm request. The target attention page covers 8,192 tokens and publishes a partial hash at token 2,048. Replay excludes the final prompt token, so the old lookup never examines that published hash.

Allow a matching later partial hash in the same attention page to prove an earlier prefix. Cap the returned length at the requested aligned limit, then preserve the existing EAGLE rewind. This mirrors the full-page fallback. Replicated groups and DCP1 block scaling remain unchanged.

Validation so far:

• Original image source 63bd2b9aa6b9a5ffb4040dd5af17a55f7a64bb43: four focused DCP4 failures, with eight DCP1 and replicated-cache controls passing.
• Fixed lookup: 122 tests passed with the patched manager overlay. All 124 affected CPU tests passed in the complete source-verified image, including divergent-prefix and EAGLE controls.
• Public source head: 74cf920, based on current dev/jovian-judgement.
• Source-verified candidate image: sha256:75af98309b5778c95d89fc197a30823efa62cf615168e07488ed935a2e2f5545.
• Original GPU response: cold cached=0, warm cached=0, appended cached=1792; all three answers were correct.
• Baseline DCP1 retained 4,723,200 tokens in two pressure runs with zero errors/preemptions and at least 4.02 GiB free per GPU.

Fixed-image GPU validation:

• DCP1 control: correct canary answers, shared-prefix reuse, and C1/C8/C16 decode and mixed screens passed; zero runtime errors or preemptions.
• DCP4: all 12 boundary cases and 16 shared-prefix requests passed with per-request token and cache-metric reconciliation. The original 2048-token reproduction now reuses 1792 tokens on warm replay.
• Cold/warm retrieval: exactly 491520 input tokens; warm reuse is 491264 tokens and matches the metric delta.
• All 64 agent turns passed, with individual cached-prefix reuse checked from raw responses.
• Pressure repeats 1 and 2: all three targets passed in each repeat, totaling 288 requests. Retained-prefix lower bound is 6119424 tokens. Minimum free GiB per GPU: [2.9755859375, 2.8193359375, 2.8193359375, 3.4443359375]. Preemptions, OOMs, restarts, and fatal errors were all zero.
• Reported capacity is 14843551 tokens. The 12-sequence cap limits this workload to 6120000 rendered tokens, so the remaining reported capacity is not labeled validated.

Independent source review found no blocking defect. A fresh Astra reviewer verified the changed-file hashes and checked 13680 CPU oracle cases; the old finder disagreed in 1704 cases.

Qualification scope is VRAM-only. The pre-existing per-group connector caller omits DCP/PCP arguments, so this PR makes no live connector-transfer qualification claim.

All required local CPU, GPU correctness, pressure, and independent source-review gates passed on the public head. The exact original LP27 service was restored with its unchanged container ID, image, and configuration fingerprint. LP26 remained stopped and no candidate was promoted. A separate intermittent illegal-access failure during original-image graph startup is preserved and unresolved; this lookup patch does not claim to fix it.

Rechecked current upstream work before readiness, including #644, #654, #677, #678, #482, #519, #533, #524, #670, #674, and #675. PRs #654 and #678 change alignment semantics in the same finder but retain the old search bounded by max_partial_idx. They do not contain this later-tail proof. The public head is unchanged and GitHub reports the PR mergeable.

Summary by CodeRabbit

• Bug Fixes

  • Improved attention cache reuse when matching requests share a cached tail below its published boundary.
  • Corrected cache-hit length calculation across supported attention configurations.

• Tests

  • Added coverage for replaying cached attention tails with matching and divergent request prefixes.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: d1720d5d-00ca-4576-8210-d0504b859b31

📥 Commits

Reviewing files that changed from the base of the PR and between db7a65e and 74cf920.

📒 Files selected for processing (2)

• tests/v1/core/test_partial_attention_rewind.py
• vllm/v1/core/single_type_kv_cache_manager.py

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

FullAttentionManager.find_longest_cache_hit now reuses partial cache entries within the published attention page while enforcing the maximum hit boundary. A parametrized CPU test covers matching and divergent prefixes across supported configurations.

Changes

Partial attention cache rewind

Layer / File(s)	Summary
Cache-hit rewind and regression coverage vllm/v1/core/single_type_kv_cache_manager.py, tests/v1/core/test_partial_attention_rewind.py	The partial-hit search scans the full page, caps the hit length at max_partial_idx, and adds parametrized coverage for matching and divergent prefixes across dcp, replicated, leading_blocks, and drop_eagle settings.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to 74cf9

This change enables reuse of earlier prompt prefixes from partial attention-page cache entries while preserving requested-length limits and existing rewind behavior. Current coverage and validation indicate no remaining merge-blocking risk.

Suggested reviewers: lucaswilkinson, dao007forever

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: reusing partial attention pages below their published boundary.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[voipmonitor]

Independent validation completed on four RTX PRO 6000 Blackwell GPUs at stock memory clocks, using the exact vLLM and B12X source boundary from voipmonitor/vllm:jovian-judgement-community-20260906-r27 plus PR heads #674 and #676.

Tested PR head: 74cf920.

Correctness evidence:

• The unpatched 12-case focused CPU matrix failed all four non-replicated DCP4 cases and passed the eight DCP1 or replicated controls.
• The patched matrix passed all cases, including divergent-suffix and EAGLE-rewind controls.
• TP4/DCP4 DFlash2 K7 with NVFP4 KV cache, a 2,048-token target page, and a 256-token recurrent-state page produced the expected cache boundary grid: 255 -> 0, 256 -> 0, 257 -> 256, 2,047/2,048 -> 1,792, 2,049 -> 2,048, 8,191/8,192 -> 7,936, and 8,193 -> 8,192 cached tokens.
• All boundary requests completed successfully with no preemptions or engine errors.
• The implementation remains conservative: a later chained hash must match the request, the returned hit is capped at the caller's requested aligned boundary, and a divergent suffix misses.

A separate cache-geometry A/B used the same patched code and serving configuration, changing only the recurrent-state page:

• 2,048-token recurrent-state page: 32K cold prefill = 12,759 tok/s.
• 256-token recurrent-state page: 32K cold prefill = 10,200 tok/s.
• Fine recurrent-state pages therefore reduced cold-prefill throughput by 20.1% on this server. Allocated KV capacity was effectively unchanged: 14,819,874 versus approximately 14,803,742 tokens.

This performance difference is not caused by the lookup change; both arms contain this PR. It establishes an important deployment boundary: the 256-token recurrent-state geometry enables fine DFlash replay hits but must not replace the 2,048-token production default without addressing its prefill cost. With the production 2,048/2,048 split geometry, a 2,048-token exact replay has no common hybrid hit and an 8,192-token replay retains 6,144 tokens, because the recurrent group remains the limiting cache group.

Conclusion: the cache lookup fix is correct and mergeable. I recommend merging this PR while keeping fine recurrent-state pages an explicit, separately qualified cache-reuse mode rather than a default launcher change.

[voipmonitor]

This PR is still required; it has not been replaced by #708. The shared R29 image contains its partial-attention-page lookup proof, and #708 consumes those authenticated prefixes when exporting recurrent checkpoints. Lookup and checkpoint export solve different parts of prefix reuse.

I reran all 12 partial-page rewind cases in the composed serving source: they pass, including DCP1, DCP4, replicated groups, speculative rewind and divergent-prefix rejection. Keeping #676 open as the canonical merge target; image inclusion does not mean it has merged into Jovian Judgement.

[voipmonitor]

Included in dev/jovian-judgement through this PR's individual merge. The reviewed head and its contributor commits remain ancestors; the PR is merged and closed.

Source validation: replaying all 32 R35 review heads on the pinned base exactly reproduces the released Docker's vLLM tree; all 6,870 installed tracked files match. JJ additionally preserves Luke's DS4.1 work and #734. The final composition passed 247 focused checkpoint/scheduler, sampler/warmup and native GPU tests. This is combined-source evidence, not a fresh performance or full-model qualification for this individual PR.

Publication-history clarification: the individual merge linked above is in JJ's first-parent history. It replaces the receipt's archived wrapper-merge reference; GitHub's historical merge SHA may still identify that archive. See #731 for component review order and qualification limits.