[P/D disagg] Add HiRadixCache and cache-aware DP routing for decode disaggregation

[huanpengchu]

Summary

Extend decode-side radix cache (#19746) with HiRadixCache (L2 host cache + optional L3 storage cache) integration and cache-aware load balancing for multi-DP decode deployments.

With HiRadixCache enabled, the decode worker uses a multi-level cache hierarchy:

GPU (L1) prefix hits avoid KV transfers entirely, host-memory (L2) hits load back KV via async prefetch, and optional storage (L3, e.g. Mooncake) hits use a two-phase prefetch (L3→L2→L1).

Cache-aware DP routing directs incoming requests to the DP rank with the best prefix match, maximizing cache hit rates across DP workers.

Main Changes

• Decode preallocation queue (decode.py)
  • Add _load_back_and_lock(): match prefix against radix tree, optionally
    load back host-cached KV to GPU, then lock the prefix node.
  • Add _match_prefix() read-only prefix match for budget pre-checks.
  • Integrate HiRadixCache async prefetch: trigger prefetch_and_load_back()
    for L2 host-hit prefixes, wait with timeout before pre-allocation.
  • Two-phase L3 storage prefetch: trigger L3→L2 prefetch for storage-backed
    prefixes, defer pre-allocation until data arrives at host, then L2→L1
    load back to GPU.
  • Add _PlannedPrealloc batched planning: scan all queued requests, plan
    prefix match and allocation budget, then batch-evict before committing.
  • Add _evict_for_prealloc() / _batch_evict_for_prealloc() for eviction
    before pre-allocation with page-aligned delta calculation.
  • Clean up HiRadixCache state on request abort via _cleanup_hicache_aborted_request().
• Cache-aware DP routing (data_parallel_controller.py)
  • Add CACHE_AWARE load balance method for multi-DP decode.
  • Add estimate_prefix_match(): estimate prefix match length against a
    CacheDigest using chained SHA256 hashing on chunk-aligned token boundaries.
  • Route each request to the DP rank with the best prefix match, with
    load-balance override when request imbalance exceeds --dp-routing-max-extra-reqs.
  • Capacity-aware fallback: re-route to another rank when the best-match
    rank has insufficient available KV tokens.
• Cache digest infrastructure (radix_cache.py, hiradix_cache.py)
  • Add RadixCache.build_cache_digest(): traverse the radix tree and produce
    chunk-aligned prefix hashes for cache-aware routing.
  • Track _digest_dirty flag on insert / evict / load-back to avoid
    redundant digest rebuilds.
  • HiRadixCache._skip_node_in_digest(): include evicted-but-backed-up
    nodes in the digest (recoverable via load_back()).
  • Fix evicted-node boundary handling in _match_prefix_device_indices()
    to prevent gaps in device_indices when a descendant was re-filled.
  • Clean up redundant debug logs in HiRadixCache.
• Scheduler / load inquirer (scheduler.py, load_inquirer.py, io_struct.py)
  • Add CacheDigest dataclass to io_struct.py.
  • Build and attach CacheDigest to GetLoadsReqOutput in the load inquirer
    when decode radix cache is enabled.
  • Add SWA/SSM incompatibility guard for decode radix cache in scheduler init.
• CLI / config (server_args.py)
  • Add --disaggregation-decode-enable-hicache, --disaggregation-decode-hicache-ratio,
    --disaggregation-decode-hicache-size, --disaggregation-decode-hicache-write-policy.
  • Add --dp-routing-digest-chunk-size, --dp-routing-max-extra-reqs.
  • Add cache_aware to --load-balance-method choices.
  • Auto-select cache_aware when decode radix cache + DP > 1.

Interface

Enable HiRadixCache on the decode worker with:

--disaggregation-mode decode \
--disaggregation-transfer-backend mooncake \
--disaggregation-decode-enable-radix-cache \
--disaggregation-decode-enable-hicache \
--disaggregation-decode-hicache-ratio 2.0

For multi-DP decode with cache-aware routing (auto-enabled when DP > 1 + radix cache):

--load-balance-method cache_aware

Test Plan

• [✅] Qwen3-32B local PD disagg sanity runs with HiRadixCache enabled
• [✅] Multi-DP decode with cache-aware routing, verify requests route to
  cache-hit ranks
• [✅] GLM5.1 cross-node 1P2D output correctness and metrics validation

---

CI States

Latest PR Test (Base): ❌ Run #26267730577
Latest PR Test (Extra): ❌ Run #26267730531

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[ishandhanani]

Quick thoughts

1. Can you share some performance numbers
2. Can you split out the DP rank routing bits from enabling HiCache. I think they should be 2 separate PRs just for ease of reviewing

[huanpengchu]

快速思考

1. 能否分享一些业绩数据？
2. 能否将 DP 排名路由部分与启用 HiCache 部分分开？我认为为了便于审核，应该分别提交两个 PR。

1. I'll test it, but logically, it's the same latency as enabling radix cache(hit).
2. Ok. I'll submit them separately

[Amaranterre]

i believe cache-aware routing will be a huge leap on throughputs improvements. In my observation, zero cache awareness will result in replicas of kvcache, which hinders throughtput scaling.

And i think decode-side hi-cache is independent with cache-aware DCP routing. Maybe consider submitting in separate PR?

[huanpengchu]

我认为缓存感知路由将极大地提升吞吐量。据我观察，完全不考虑缓存会导致键值缓存（kvcache）出现多个副本，从而阻碍吞吐量的扩展。

我认为解码端的 Hi-Cache 与缓存感知 DCP 路由是相互独立的。或许可以考虑单独提交一个 PR？

Yes, these two functions can be independent of each other. I'm currently testing various throughput and will submit them soon

[huanpengchu]

i believe cache-aware routing will be a huge leap on throughputs improvements. In my observation, zero cache awareness will result in replicas of kvcache, which hinders throughtput scaling.

And i think decode-side hi-cache is independent with cache-aware DCP routing. Maybe consider submitting in separate PR?

I submitted them separately in #26560 and #26561, and included some test results on a single-node 1P4D setup.

[github-actions]

Thanks @huanpengchu. Closing this because it is still a draft and has not been updated in 101 days.

Reopen it if the work is still relevant.

Some directories moved recently, so an older branch may need retargeting:
sgl-kernel/ -> python/sglang/kernels/aot/, python/sglang/jit_kernel/
-> python/sglang/kernels/jit/, docs/ -> docs/docs/ (.mdx),
bench_serving.py -> benchmark/serving.py, test/srt/ -> test/registered/.