Skip to content

Add DeepSeek-V4 decode radix cache with hierarchical reuse - #39055

Open
huangyg-star wants to merge 1 commit into
sgl-project:mainfrom
huangyg-star:dsv4-decode-radix-core
Open

huangyg-star wants to merge 1 commit into
sgl-project:mainfrom
huangyg-star:dsv4-decode-radix-core

Conversation

@huangyg-star

@huangyg-star huangyg-star commented Sep 11, 2026

Copy link
Copy Markdown

Motivation

This PR builds on the SWA-aware decode-side radix cache introduced in
#27770.

PR #27770 enables device-side prefix reuse for sliding-window-attention models,
but it does not support DeepSeek-V4 compressed KV or HiCache.

This PR adds DeepSeek-V4 support to the Decode Radix Cache and integrates it
with HiCache. A Decode worker can therefore reuse prefixes from:

  • L1: device-resident KV cache;
  • L2: host-resident HiCache;
  • L3: storage-backed HiCache.

DeepSeek-V4 requires additional handling to keep its Full KV, active SWA tail,
compressed C4/C128 KV, indexer data, block scales, and request state consistent
across PD transfer and hierarchical cache restore.

The goal is to reduce Prefill-to-Decode KV transfer and avoid repeated
radix-tree traversal over an already matched prefix.

Modifications

Support DeepSeek-V4 in Decode Radix Cache

  • Extend the SWA-aware Decode Radix Cache from [P/D disagg] Decode-side radix cache for SWA hybrid models (unified radix tree) #27770 to DeepSeek-V4.
  • Reuse the Full-attention KV prefix already resident on the Decode worker.
  • Keep the active SWA tail request-local and transfer or allocate it separately.
  • Add PD transfer handling for DeepSeek-V4 compressed KV, indexer data, block
    scales, and request state.
  • Keep the DeepSeek-V4 sidecar data consistent with the Full and SWA cache
    slots throughout allocation, transfer, insertion, and eviction.

Integrate DeepSeek-V4 Decode Radix Cache with HiCache

  • Extend Decode-side prefix matching across device, host, and storage cache
    tiers.

  • Restore L2 host-cache hits to the device before the request starts decoding.

  • Prefetch L3 storage hits into the host cache and include the restorable
    portion in the prefix reported to Prefill.

  • Compute the reusable prefix reported to Prefill as:

    decode_prefix_len = L1 + L2 + L3
    

Reduce repeated radix-tree traversal

Reuse the request's matched and locked device prefix as the anchor for subsequent matching and insertion.
Match and insert only the suffix beyond the admitted prefix instead of walking the same prefix again from the root.
Revalidate the anchor before reuse and fall back to the regular root walk when it is no longer valid.
Preserve the final canonical rematch and lock handoff.
Avoid insertion-driven updates to LRU state, access time, priority, hit count, and write-through policy when the anchor is reused.


CI States

Latest PR Test (Base): ❌ Run #34577348782
Latest PR Test (Extra): ❌ Run #34577347694
Latest PR Test (AMD ROCm 10): ❌ Run #34577348135

Support decode-side prefix reuse for DeepSeek-V4 compressed KV across
device, host, and storage cache tiers.

Add SWA-aware prefix handling, HiCache restore coordination, compressed
sidecar transfer, and trusted insert-anchor reuse without cache-policy
refresh.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hicache Hierarchical Caching for SGLang unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant