[HiCache] L3 storage prefetch lifecycle metrics and cross-tier attribution fixes

[xiezhq-hermann]

Motivation

L3 (storage) observability in HiCache had several gaps that made the exported gauges misleading under real traffic:

• Storage-hit tokens returned by an L3 query were never accounted end to end. There was no way to tell how many were actually reused versus lost to host capacity, transfer shortfall, device capacity, a shrunk splice base, or a dropped host insert.
• The per-request device/host/storage cached-token split credited host/storage for load-backs that were declined or dropped, and lost L3 attribution whenever the L3-loaded tail was evicted from host before admission. In buffer mode, host memory is transport staging, not a resident L2 tier, and was mis-attributed as L2.
• gen_throughput kept re-exporting a stale decode window during long prefill-only stretches, and decode ticks reset the prefill-owned cache_hit_rate gauge to 0.
• Write-through evictions of device leaves that never completed their eager backup were invisible; buffer-mode backup intents swept as stale vanished silently.

Modifications

• Storage prefetch lifecycle accounting in UnifiedRadixCache: _record_storage_prefetch_hit starts accounting at the rank-agreed query result; _resolve_storage_prefetch_tokens / _finish_storage_prefetch retire tokens with a terminal reason (below_threshold, host_capacity, device_capacity, storage_transfer, shrunk, dropped). Cache mode resolves at admission via finish_storage_prefetch_admission; buffer mode resolves at the H2D ack in BufferModePipeline. Two new counters: sglang:storage_prefetch_hit_tokens_total and sglang:storage_prefetch_unfulfilled_tokens_total{reason}.
• prefetched_tokens_total now counts tokens transferred from L3 into host memory, emitted once per completed prefetch.
• Tier attribution: Req gains storage_hit_start, host_loaded_length, and host_hit_is_storage; split_cached_prefix_by_tier takes the absolute L3 span start and the buffer-mode flag; PrefillAdder records the spliced load-back length and accounts admission through a single _account_prefill_cache_admission. The scheduler pops the loaded span via the new pop_prefetch_loaded_span hook.
• hicache_dropped_tokens_total gains reason write_through_unbacked_eviction; UnifiedTreeCore reports unbacked_tokens on EvictDeviceNextNodeResult / EvictDeviceLeafResult. The pool label is now always kv.
• Stale buffer-mode backup intents are counted in hicache_backup_dropped_tokens_total.
• SchedulerMetricsReporter: gen_throughput decays to 0 after 30s without decode stats; decode ticks no longer zero cache_hit_rate.
• cached_tokens_total{cache_source} uses the literal storage label (bounded cardinality).
• Eviction/backup duration histogram defaults trimmed to a coarse, widely supported bucket set; env overrides unchanged.
• Removed derived l3_demand_total_tokens / l3_sum_rate_* counters and the occupancy_ratio snapshot field (redundant with existing exports).

New BasePrefixCache hooks (pop_prefetch_loaded_span, finish_storage_prefetch_admission, discard_storage_prefetch_accounting) have no-op defaults, and StreamingSession forwards them, so other cache implementations are unaffected.

Original commits

• 719b52246f

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging.
• Please feel free to join our Slack channel at https://slack.sglang.io to discuss your PR.

Testing

Ran locally on this branch (CPU):

pytest test/registered/unit/mem_cache/test_unified_radix_cache_unittest.py \
       test/registered/unit/mem_cache/test_hicache_staged_write_back_dispatch.py \
       test/registered/unit/managers/test_prefill_adder.py \
       test/registered/unit/managers/test_schedule_batch_out_of_place.py \
       test/registered/unit/observability/test_stat_loggers_di.py
# 1244 passed, 1257 skipped (GPU/hybrid-only cases), 76 subtests passed

New tests: TestCachedPrefixTierAttribution, test_storage_prefetch_fulfillment_resolves_at_admission, test_retracted_storage_prefetch_accounting_is_omitted, test_short_staged_swa_tail_resolves_device_covered_head, test_write_through_eviction_counts_unbacked_tokens, test_positive_hit_with_lost_anchor_is_reported_as_shrunk, TestHiCacheMetrics (recording-metric collectors). Existing prefetch/buffer-mode tests gained lifecycle-metric assertions.

Also black --check, ruff --select=F401,F821,UP037, and py_compile on all changed files. GPU end-to-end runs were not re-executed from this branch; the change was validated on an internal deployment with L3 storage enabled where the previously stale gen_throughput and zeroed cache_hit_rate gauges now read correctly. Relying on CI for the GPU suites.

---

CI States

Latest PR Test (Base): ❌ Run #33739045495
Latest PR Test (Extra): ❌ Run #33739045300
Latest PR Test (AMD ROCm 7.2): ❌ Run #33739045360

[xiezhq-hermann]

/tag-and-rerun-ci

[xiezhq-hermann]

/tag-and-rerun-ci

[xiezhq-hermann]

/tag-and-rerun-ci

[xiezhq-hermann]

/tag-and-rerun-ci

[xiezhq-hermann]

/tag-and-rerun-ci

[xiezhq-hermann]

/tag-and-rerun-ci