[ziqifan617]

Motivation

Add KVCR as an optional HiCacheStorage L3 backend for local and cross-replica KV reuse over NIXL/UCX.

Together with Dynamo #16117, this enables router-guided peer restore through in-process python -m dynamo.sglang: Dynamo publishes candidate sources and forwards request hints; this PR implements storage access and restores the data through HiCache.

Backup:  HBM -> SGLang HiCache host pages -> KVCR-owned DRAM
Restore: local/peer KVCR DRAM -> locked HiCache host pages -> HBM

This is the HiCacheStorage route, not linker/direct-HBM mode. Remote delivery does not require an intermediate destination-side KVCR cache allocation, and unpinned SGLang host pages are never offered as peer sources. Local L3 works without Dynamo; router-directed cross-replica discovery needs the companion or another compatible hint producer.

Modifications

• Register the optional kvcr backend with page-first host layouts; register physical components once and use indexed RegionDescriptor/MemoryRef APIs for multi-component, multi-pool IO.
• Preserve kv_hints through scheduler prefetch, PD-decode probes/prefetch, deferred tickets, and trailing-pool reads. Reject bare logical-anchor queries without physical ALL_PAGES evidence.
• Intersect legal resume-boundary sets across pools and ranks, rather than reducing sparse checkpoint maxima.
• Add opt-in combined KV/ALL_PAGES-sidecar reads and backups, configurable read batching, replicated-MLA sharing, and prefix-recency alignment. Validate every required pool result and preserve cross-rank ACK sequencing after failures.
• Progress idle peer sources independently and retain completions for their callers. Wait for terminal native results before returning buffers; terminate the worker if native memory access remains uncertain.
• Add adapter/controller regressions, real-NIXL host-byte tests, a backend README, and optional metrics/byte-verification diagnostics. Existing backends retain their default paths.

Accuracy Tests

• Latest source-pinned CPU validation (October 15): 50 cases passed, covering the PD-decode hint and logical-anchor fixes alongside adapter/controller regressions. KVCR transport is mocked in this gate.
• Earlier native validation: 6 real-NIXL host-byte tests passed with adequate container shared memory; this checks host bytes, not model accuracy.
• Earlier end-to-end validation: GLM5.3, two complete TP4 replicas on separate B200 hosts, CP1/PP1, and in-process Dynamo. An 8,192-token restore probe matched source/restored/local-HBM outputs.
• Scoped formatting/lint and git diff --check passed. The October 15 follow-ups have CPU coverage; a new GPU/model run and full upstream CI are not claimed.

Speed Tests and Profiling

Combined IO, read batching, sharing, and recency alignment are opt-in. No inference-speed result is claimed for the latest head; expensive diagnostics remain disabled by default.

Dependencies and Scope

• Install compatible KVCR indexed-region APIs separately; KVCR is not a mandatory SGLang dependency. Optional startup_timeout_seconds additionally requires KVCR #82; transfer deadlines are separate.
• Peer hints are advisory, not source leases; terminal delivery determines usable pages. No framework-page pinning/quarantine or older storage-key compatibility is promised.
• PP>1 and heterogeneous TP head splitting are rejected. Shared MLA additionally requires replicated physical KV/indexer bytes and CP1/PP1; it is not generic sharded-state/SWA support. Multi-node engines and arbitrary hybrid-model accuracy are outside the demonstrated scope.
• Related work: [HiCache] Add KVCR as an L3 storage backend for hint-driven cross-instance KV reuse #36409. Includes Lin Hu's PD-decode hint and logical-anchor query fixes, plus fail-closed regressions.

Review storage/kvcr/{layout,hints,kvcr_store}.py, then controller batching/boundary changes and regressions.

Checklist

• Run scoped formatting/lint checks.
• Add adapter/controller unit regressions and native host-byte tests.
• Document configuration, ownership, and limitations in the backend README.
• Complete broader model-accuracy and inference-speed validation.
• Complete upstream CI and maintainer review.

---

CI States

Latest PR Test (Base): ❌ Missing run-ci label -- add it to run CI tests.
Latest PR Test (Extra): ❌ Blocked -- run-ci is required first.
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.

[ziqifan617]

Superseded by the smaller review slices #45646 → #45647 → #45649. These preserve the HiCache implementation and separate initial enablement from optional performance tuning. Closing this monolithic PR to consolidate review; its branch and benchmark history remain available for reference. The replacement PRs remain drafts with their documented validation and merge gates.