[Fix] HiCache: reclaim redundant host mirrors after storage backup (write_through host-pool starvation)

[phlv]

Motivation

Under --hicache-write-policy write_through with a storage backend (--hicache-storage-backend mooncake etc.), the host (L2) pool deadlocks whenever it is smaller than the device pool, silently starving the storage tier. Three code facts combine into this:

1. Every inserted node immediately pins a host mirror — write_backup() allocates host slots for every node that enters the device radix cache (threshold 1 under write_through).
2. The backup ack never frees the slot — _drain_backup() only calls entry.release_host(), which decrements host_ref_counter; the memory stays attached to node.host_value.
3. The only reclaimer refuses device-resident nodes — evict_host() guards with if not x.evicted: continue, so a mirror can only be reclaimed after the device copy is evicted. The in-band rescue inside write_backup() (evict_host then retry) is a no-op for the same reason.

Under sustained load the device pool stays full, so pinned-mirror demand approaches the full device pool size. With hicache_size below device KV capacity the host pool fills with irreclaimable mirrors, and both staging (hicache_write_rejected_tokens_total{reason="host_alloc_failed"}) and storage prefetch (same reason, stage="storage_prefetch") are rejected wholesale — storage starves in both directions. The default hicache_ratio 2.0 hides the bug by capping mirror demand at half the pool, at the cost of a device-pool-sized slice of host DRAM that is permanently pinned and never reused.

On a production 16-rank deployment (hicache_size ≈ 0.25× device pool, mooncake backend) we measured 329M tokens of write rejections vs 21.6M actually written (94% rejected), hicache_host_reserved_tokens{purpose="write"} pinned at 0, and a storage hit contribution of 0.55%.

Modifications

Make the backup ack meaningful, so the host pool works as a channel into storage instead of a second permanent copy:

• Durable tracking: pages confirmed written to storage are marked durable in a bounded LRU keyed by page hash (an ack is not success — only the completed_tokens prefix is marked; hash-keying also covers nodes split mid-backup).
• Release candidates: a node with device copy resident and all pages durable holds a redundant mirror and becomes releasable. Membership is state-derived; busy nodes (lock_ref/host_ref_counter) are kept, not lost.
• TP-consistent release: candidates are released only under host-pool pressure via a prepare/commit protocol riding the existing storage-drain collective — each rank contributes its pool deficit and last round's release plan (count/tokens/digest as negated pairs, so one MIN all-reduce recovers both min and max), and a plan executes only when every rank reports the identical triple. Mutation never precedes consensus; repeated mismatch fail-closes the feature back to current behavior. Synced ack slots keep MLA-style single-writer deployments (backup_skip) convergent.
• Eviction integration: demoted nodes leave the candidate set; a released-mirror parent whose children were demoted is evicted by pruning its host-only subtree (data is durable in storage or recomputable), so eviction always makes progress; durable marks are revoked when a node leaves the tree entirely.
• Anti-churn: a fully-durable node skips re-staging (age-gated so storage-side eviction can eventually be refreshed).

Opt-in, default off: SGLANG_HICACHE_MIRROR_RELEASE=1 enables the feature (_FREE_FRAC, SGLANG_HICACHE_DURABLE_CAPACITY, _REFRESH_ROUNDS tune it). With the flag unset, one added branch runs per drain round and behavior is unchanged. When enabled, check_hicache_events routes storage draining through drain_storage_control_queues (which carries the protocol fields) instead of the fused ready-counts collective.

Results on the same 16-rank deployment, same load, only the flag flipped: storage write-out 11.3% → 100%, host_alloc_failed write rejections 16.7M tokens → 0, warm-round storage hit 0.53% → 3.17% (12.87% across two engines sharing one mooncake backend, with 2× wall-clock speedup on the warm round).

The same pinning pattern exists in unified_radix_cache.py / hi_mamba_radix_cache.py; this PR scopes to HiRadixCache and the others can follow once the approach is agreed on.

Accuracy Tests

Not applicable — no model-output path is touched. Correctness of the cache state machine is covered by the unit tests below; a released mirror is only ever dropped after the storage backend acked the covering pages, and any uncertainty (partial ack, consensus mismatch, busy nodes) keeps the mirror.

Benchmarking and Profiling

Numbers above; measured with the hicache storage metrics (write_rejected/storage hit source deltas per benchmark arm) on 128-session × 2-round shared-prefix load, 8-way concurrency, cold + warm arms. Feature off shows no measurable drain-round overhead (single boolean branch).

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests: test/registered/unit/mem_cache/test_hicache_mirror_release.py — 26 falsifiable cases (they fail on the unfixed code) covering durable marking with partial acks, split-during-backup, TP consensus commit/mismatch/fail-close, MLA single-writer, released-parent eviction progress, churn guard, and a concurrency regression for the ack-queue snapshot.
• Update documentation / docstrings as needed.
• Provide accuracy and speed benchmark results above.

🤖 Generated with Claude Code

---

CI States

Latest PR Test (Base): ❌ Run #31091637794
Latest PR Test (Extra): ❌ Run #31091637176