hicache: restore eviction/load-back bandwidth and byte-level metrics

[TianDi101]

Summary

Split out of #31879 (general HiCache L1<->L2 eviction/load-back metrics only, no UMBP-specific or per-tier-hit changes).

Re-adds the HiCache observability that was reverted in feat/umbp-pr (001e9ce) and never carried forward, now adapted to the current async write/load pipeline:

• sglang:evicted_backuped_tokens_total / evicted_regular_tokens_total: split of evicted tokens by whether the node had already been backed up to host (cheap detach) vs dropped with no host copy.
• sglang:gpu_kv_cache_occupancy: alias gauge of token_usage, kept for HiCache dashboards that query it under the old name.
• sglang:eviction_bandwidth_gb_s / load_back_bandwidth_gb_s: real per-batch PCIe bandwidth for the L1<->L2 (device<->host) write-back/load-back path, derived from HiCacheAck.num_tokens + CUDA/HIP event timing.
• sglang:backuped_bytes_total / prefetched_bytes_total: byte-scaled counterparts of the existing backuped_tokens_total/prefetched_tokens_total counters.

This is all L1<->L2 (GPU<->host) traffic inside HiCacheController's write-back/load-back path — it has no dependency on which (if any) L3 storage backend is attached.

Reconciled with the timing_enabled/make_timing_event_pair() guard main already has (for backends without CUDA/HIP event-timing support): extended the write/eviction path in cache_controller.py and hybrid_cache_controller.py to also use make_timing_event_pair() instead of unguarded enable_timing=True (matching the load path's existing handling), and to carry timing_enabled through HiCacheAck so the new bandwidth metrics degrade gracefully where event timing isn't available.

Also restores the corresponding Grafana panels: GPU KV Cache Utilization, HiCache Host Utilization, HiCache Host Used/Total Capacity, L1<->L2 Eviction/Load-back Rate, L1->L2 and L2->L1 PCIe bandwidth, L3 Prefetch/Writeback Rate, and L3 Storage Read/Write Bandwidth (the last two use pre-existing/new general token-and-byte counters, not any UMBP-specific metric). Uses "datasource": {"default": true} rather than a hardcoded UID so it works on any fresh Grafana instance.

Test plan

• Existing hicache/metrics unit tests pass
• Manual: run with --enable-metrics --enable-hierarchical-cache on a backend without CUDA-event-timing support and confirm eviction/load-back still functions (bandwidth metrics simply don't populate, no crash)
• Manual: run on CUDA/HIP and confirm sglang:eviction_bandwidth_gb_s / load_back_bandwidth_gb_s populate with plausible values, and the Grafana panels render

---

CI States

Latest PR Test (Base): ❌ Run #29801027316
Latest PR Test (Extra): ❌ Run #29801027262

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!