fix(hicache): TP/PP write-through & load-back consensus (#28429)

[yorsen]

Motivation

Under TP/PP > 1 with HiCache write_through, the prefill node hangs after a
while and gets killed by the NCCL watchdog (see #28429). Host/device pool
occupancy is per-rank physical state, so a naive per-rank alloc in
write_backup / load_back can succeed on some ranks and fail on others.
That diverges the radix tree across ranks, so the next forward's collectives
mismatch in shape/count and deadlock.

Modifications

• managers/cache_controller.py: split the controller write into a two-phase
  reserve_write (alloc only) / commit_write (enqueue DMA + ack) /
  abort_write (free reservation); add WriteReservation (msgspec.Struct).
• mem_cache/hybrid_cache/hybrid_cache_controller.py: mirror the split for the
  indexer/aux pool_transfers.
• mem_cache/hiradix_cache.py:
  • write_backup: reserve → all_reduce(MIN) consensus → commit on all ranks
    or abort on all ranks (skipping a write-through is harmless).
  • load_back: transactional consensus — load on all ranks or roll back on
    all (free device indices); multi-rank never does the conditional
    evict-and-retry that would re-introduce divergence.
• Tests: CPU unit tests covering the consensus branches and the evict_host
  retry path (registered via register_cpu_ci).

Accuracy Tests

No change to model outputs; the fix only governs when a rank mutates shared
HiCache/radix-tree state. New CPU unit tests pass (17 cases).

Speed Tests and Profiling

Adds one all_reduce(MIN) (CPU/gloo) per node on the write/load path; no main
NCCL traffic. Multi-GPU (TP≥2) soak validation pending on a GPU host.

Checklist

• Format code with pre-commit
• Add unit tests
• Update documentation (N/A)
• Accuracy/speed benchmarks (multi-GPU validation pending)
• Follow SGLang code style

---

CI States

Latest PR Test (Base): ❌ Run #29743947689
Latest PR Test (Extra): ❌ Run #29743947558

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[yorsen]

Hi maintainers, this fixes the HiCache TP/PP write-through hang (#28429).
CI is blocked on the permission gate — could someone authorize a run?
@merrymercy @hzh0425 /tag-and-rerun-ci
Happy to address any review feedback. Thanks!