[BugFix] Preserve aligned hybrid cache reuse alongside endpoint checkpoints - #669
Conversation
Port the qualified hybrid retention, fine-hit, event, exact external-state, and aligned-budget changes onto dev/jovian-judgement. Keep endpoint bundles private and preserve their allocation, reader, and invalidation lifecycle. Retain the behavioral contributions from PRs vllm-project#557, vllm-project#643, vllm-project#645, vllm-project#646, vllm-project#655, vllm-project#656, vllm-project#657, and vllm-project#663. Resolve shared helpers once so fine and coarse replay boundaries are not expanded twice. Validation: 910 cache, scheduler, parser, event, connector, and endpoint regressions pass. The three literal LP26 cache regressions also pass. Co-authored-by: Jason Cook <jasonc@maxlyn.com> Co-authored-by: Martin Vit <martin@voipmonitor.org> Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai> Co-authored-by: haic0 <149741444+haic0@users.noreply.github.com> Co-authored-by: tobymao <toby.mao@gmail.com> Signed-off-by: logprobz <321553542+logprobz@users.noreply.github.com>
|
Warning Review limit reachedNext included review available in 19 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (32)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Signed-off-by: logprobz <321553542+logprobz@users.noreply.github.com>
86acae9
into
local-inference-lab:dev/jovian-judgement
|
Spark qualification of this PR (four-node DGX Spark, GB10 SM121, TP4 over RoCE, GLM-5.3-Flash NVFP4 + DFlash2 k=5, fp8 KV, Cache reuse. Identical replay of page-aligned prompt lengths, which misses entirely on the #646 head (this is the #655 reproducer; it reproduces on Spark for every aligned length I tried), now hits:
Non-aligned lengths were already fine on #646 and are unchanged here (99,303-token prompt: 99,072 reused in 0.44 s; 49,043: 48,896 in 0.31 s; shared-prefix variants identical). Throughput and quality, same image pair, same day. KV pool 4,297,015 tokens (unchanged). Decode c1/c8/c16 at 0k context 47.7 / 131.7 / 229.0 tok/s versus a #646-head band of 42.6–46.9 / 137–151 / 201–229 across paired arms; standalone cold prefill 2840 / 2899 / 2820 tok/s at 8k/32k/128k versus 2849 / 2907 / 2802. Speculative acceptance 2.5–2.8 throughout (that needs #667 on these heads). GSM8K-100 at c16: 92/100 and 96/100 on two runs, against 94–97 on the #646 head. CPU: No regressions observed on Spark; the aligned-length replay fix is a real improvement over #646 on this cluster. |
3 participants
The new endpoint-checkpoint mechanism excludes DFlash2 and external connectors, which still use aligned recurrent checkpoints. That path fails three LP26 cache regressions on the current branch: missing reuse, missing fallback state, and skipped rewind boundaries. This ports the qualified fixes while preserving private endpoint bundles and their ownership guards.
The shared handoff API changes are kept together so core and connector consumers remain compatible. Review the port in these groups:
This carries behavior from #643, #645, #646, #655, #656, #557, #657, and #663. Authorship is preserved in commit trailers. It supersedes logprobz follow-ups #643, #645, #655, #656, #657, and #663; other authors' PRs remain independently owned.
Validation:
The initial serving evidence includes the local GDN variant from now-closed #668 and #653's sampling RNG fix as production prerequisites. BF16 LM-head precision is retained. External connectors have CPU coverage; no live external store was qualified. Concurrent MTP endpoint admission is outside this DFlash2 profile.
Base tested: b7e3d03. Prepared with AI assistance. Independent Astra release review verified source hashes, image provenance, sampling identity, and serving receipts, with bounded candidate approval. A separate local promotion-script preemption check is being corrected before production cutover.
PR #667 already fixes the same GDN defect, so #668 is closed as a duplicate. Our transition cases and #667 metadata/worker tests pass (53 passed, 3 skipped). Final qualification is switching to #667 + this PR + #653; fresh combined-image GPU validation is required before promotion.
MTP qualification update: native2048 recurrent blocks passed full serving qualification. Fine256 MTP exposed a resumed recurrent-state column using target-page units; #672 is the narrow required correction. Fine MTP replay is being requalified with that dependency. Do not interpret cache-hit counts alone as MTP correctness.
Final MTP3 serving qualification
The combined immutable candidate containing #653, #667, #669, #670, #672 and #674 passed full MTP3 serving qualification. All cold/warm/appended cache canaries, 16 shared-prefix requests, 64 agent turns, tools/parser/vision checks and 491,520-token cold/warm retrieval passed, with zero preemptions or runtime errors. The long prompt was fully reused and returned the correct answer in 1.36 seconds warm versus 56.73 seconds cold. Ten decode repeats averaged 133.98 tokens/s.
The installed image passed 1,000 CPU regressions plus three cache contracts, and all 18 scalar restore GPU cases. DFlash2 also passed full serving qualification and two repeated concurrency matrices. Every throughput and latency metric met the 5% LP26 parity gate, with zero preemptions or runtime errors. The qualified image is promoted on port 8100. This is a combined-source qualification, not a standalone performance claim for this PR.