Add DeepSeek-V4 decode radix cache with hierarchical reuse - #39055
Open
huangyg-star wants to merge 1 commit into
Open
huangyg-star wants to merge 1 commit into
huangyg-star wants to merge 1 commit into
Conversation
Support decode-side prefix reuse for DeepSeek-V4 compressed KV across device, host, and storage cache tiers. Add SWA-aware prefix handling, HiCache restore coordination, compressed sidecar transfer, and trusted insert-anchor reuse without cache-policy refresh.
huangyg-star
requested review from
ByronHsu,
Duyi-Wang,
HaiShaw,
ShangmingCai,
Ying1123,
alphabetc1,
hanming-lu,
hnyls2002,
huangtingwei9988,
hzh0425,
ispobock,
merrymercy,
sogalin,
xiezhq-hermann and
yizhang2077
as code owners
September 11, 2026 08:03
45 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
This PR builds on the SWA-aware decode-side radix cache introduced in
#27770.
PR #27770 enables device-side prefix reuse for sliding-window-attention models,
but it does not support DeepSeek-V4 compressed KV or HiCache.
This PR adds DeepSeek-V4 support to the Decode Radix Cache and integrates it
with HiCache. A Decode worker can therefore reuse prefixes from:
DeepSeek-V4 requires additional handling to keep its Full KV, active SWA tail,
compressed C4/C128 KV, indexer data, block scales, and request state consistent
across PD transfer and hierarchical cache restore.
The goal is to reduce Prefill-to-Decode KV transfer and avoid repeated
radix-tree traversal over an already matched prefix.
Modifications
Support DeepSeek-V4 in Decode Radix Cache
scales, and request state.
slots throughout allocation, transfer, insertion, and eviction.
Integrate DeepSeek-V4 Decode Radix Cache with HiCache
Extend Decode-side prefix matching across device, host, and storage cache
tiers.
Restore L2 host-cache hits to the device before the request starts decoding.
Prefetch L3 storage hits into the host cache and include the restorable
portion in the prefix reported to Prefill.
Compute the reusable prefix reported to Prefill as:
Reduce repeated radix-tree traversal
Reuse the request's matched and locked device prefix as the anchor for subsequent matching and insertion.
Match and insert only the suffix beyond the admitted prefix instead of walking the same prefix again from the root.
Revalidate the anchor before reuse and fall back to the regular root walk when it is no longer valid.
Preserve the final canonical rematch and lock handoff.
Avoid insertion-driven updates to LRU state, access time, priority, hit count, and write-through policy when the anchor is reused.
CI States
Latest PR Test (Base): ❌ Run #34577348782
Latest PR Test (Extra): ❌ Run #34577347694
Latest PR Test (AMD ROCm 10): ❌ Run #34577348135