[P/D Disagg] DeepSeek-V4 decode radix cache with MTP support by TobyMint · Pull Request #31097 · sgl-project/sglang

TobyMint · 2026-07-14T01:54:10Z

Motivation

PR #27770 (by Ishan Dhanani) added device-only SWA decode radix cache support on the unified radix tree, but explicitly blocked DeepSeek-V4 due to its C128 compression and DSA indexer specifics. This PR removes that guard and adds full DeepSeek-V4 decode radix cache support by porting zhangxiaolei's DS-V4 patch series to the upstream API.

Co-authored-by: zhangxiaolei zhangxiaolei123456@users.noreply.github.com

Accuracy Tests

MMLU (200 examples) on DeepSeek-V4-Flash with decode radix cache enabled:

Score: 0.905
Total latency: 296.983 s
Output throughput: 66.802 token/s

Speed Tests

Shared-prefix benchmark (256 requests, 8 groups, 8192-token system prompts) on 8x NVIDIA H20 (PD disaggregation, NIXL transfer):

Successful requests:         256/256 (100%)
Total token throughput:      7970 tok/s
Mean TTFT:                   1.68s
Mean TPOT:                   130.66ms

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.

CI States

Latest PR Test (Base): ❌ Run #30235651399
Latest PR Test (Extra): ❌ Run #30235651325

gemini-code-assist · 2026-07-14T01:54:13Z

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

Co-authored-by: zhangxiaolei <zhangxiaolei123456@users.noreply.github.com>

…free, full match Co-authored-by: zhangxiaolei <zhangxiaolei123456@users.noreply.github.com>

…le_piecewise_cuda_graph attribute

…urns prompt snapshot

…r import

Match the radix-cache eviction margin: keep enough SWA before the page-aligned insert boundary for the cached key to contain a complete window. `seq_len - 1` is the last committed position. Backport of sgl-project#27770 commit 0bca294.

- Add _release_matched_prefix_lock: unified lock release with swa_prefix_lock_released tracking to avoid double-freeing SWA locks - Add _reclaim_swa_tail_capacity: proactively evict SWA cache entries before allocation to avoid unnecessary admission failures - Call dec_swa_lock_only after prefix match: retain only full-attention lock when decode will transfer fresh SWA tail - Replace inline dec_lock_ref calls with _release_matched_prefix_lock - Recalculate swa_allocatable_tokens via _swa_tail_allocatable_token_budget instead of simple decrement to account for page rounding and eviction - Count swa_evictable_size in SWA admission budget - Hoist uses_swa_tail/swa_tail_len in _pre_alloc and reclaim SWA capacity - Fix resume_retracted_reqs to recalculate SWA budget Backport of sgl-project#27770 commit 6543121.

- test_swa_tail_len_keeps_page_aligned_matchable_window: verify the insertion margin keeps a full SWA window inside the cached key - test_swa_admission_counts_evictable_capacity: SWA budget includes evictable radix-cache entries - test_reclaim_swa_tail_capacity_page_rounds: page-rounding in SWA capacity reclaim triggers correct eviction - test_reclaim_swa_tail_capacity_fails_before_allocation: RuntimeError when SWA eviction cannot satisfy the required capacity - Update test_pop_preallocated_rechecks_budget_after_lock for new dec_swa_lock_only + swa_prefix_lock_released flow - Add swa_available_size mock in test_hisparse_allocator Backport of sgl-project#27770 commits 0bca294 and 6543121.

_uses_dsv4_decode_radix_cache() already guarantees the pool is a DeepSeekV4TokenToKVPool, so compression_ratios always exists. Drop the getattr fallback to fail fast instead of silently degrading to an empty list.

page_size, is_eagle (BasePrefixCache) and cache_protected_len, swa_evicted_seqlen (Req) are always-initialized standard attributes. Drop getattr fallbacks in _maybe_insert_dsv4_decode_radix_prompt so real init bugs surface instead of silently using defaults. force_radix_leaf_creation stays getattr'd: it is a runtime-armed flag that only DS-V4 donation sets, so it is legitimately absent on most reqs.

dongyibo · 2026-08-27T03:24:24Z

@TobyMint hello， can this support Dspark?

…ject/sglang#31097; 2. SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 & SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 & --disaggregation-decode-enable-radix-cache should be enable in D

TobyMint requested review from ByronHsu, HaiShaw, ShangmingCai, Ying1123, alphabetc1, hanming-lu, hnyls2002, hzh0425, ispobock, merrymercy, sogalin, xiezhq-hermann and yizhang2077 as code owners July 14, 2026 01:54

TobyMint force-pushed the dsv4-decode-radix-cache-mtp-support branch 2 times, most recently from 70dbe33 to 3e51bd4 Compare July 15, 2026 03:41

zhangxiaolei123456 mentioned this pull request Jul 16, 2026

[Spec] DSpark support prefill/decode disaggregation #31466

Open

5 tasks

TobyMint force-pushed the dsv4-decode-radix-cache-mtp-support branch from 3e51bd4 to 82c284d Compare July 24, 2026 10:45

ishandhanani and others added 12 commits July 27, 2026 11:51

[P/D disagg] Support device-only SWA decode radix cache

f6b36e8

Enable experimental DSV4 decode radix cache

05272b3

Co-authored-by: zhangxiaolei <zhangxiaolei123456@users.noreply.github.com>

Fix DSV4 decode radix protected prefix length

98cfabf

Co-authored-by: zhangxiaolei <zhangxiaolei123456@users.noreply.github.com>

Instrument SWA allocator over-free

d7e0643

Co-authored-by: zhangxiaolei <zhangxiaolei123456@users.noreply.github.com>

Protect DSV4 decode radix prefix from SWA recovery

0c40d08

Co-authored-by: zhangxiaolei <zhangxiaolei123456@users.noreply.github.com>

Allow DSV4 decode radix EAGLE args

507e955

Co-authored-by: zhangxiaolei <zhangxiaolei123456@users.noreply.github.com>

Log DSV4 decode radix transfer stats

df8500d

Co-authored-by: zhangxiaolei <zhangxiaolei123456@users.noreply.github.com>

Avoid duplicate pending frees in SWA allocator

dbf4323

Co-authored-by: zhangxiaolei <zhangxiaolei123456@users.noreply.github.com>

Avoid DSV4 decode radix reinsertion

c9f16a5

Co-authored-by: zhangxiaolei <zhangxiaolei123456@users.noreply.github.com>

Cache DSV4 decode radix prompt once

0b3aa06

Co-authored-by: zhangxiaolei <zhangxiaolei123456@users.noreply.github.com>

Defer DSV4 prompt radix insert until after prebuilt

1a87224

Co-authored-by: zhangxiaolei <zhangxiaolei123456@users.noreply.github.com>

Support DSV4 decode radix with EAGLE MTP

5688fb1

Co-authored-by: zhangxiaolei <zhangxiaolei123456@users.noreply.github.com>

TobyMint and others added 13 commits July 27, 2026 11:52

Harden DSV4 decode radix speculative args

171cde0

Co-authored-by: zhangxiaolei <zhangxiaolei123456@users.noreply.github.com>

Single-flight DSV4 decode radix prompt inserts

104b461

Co-authored-by: zhangxiaolei <zhangxiaolei123456@users.noreply.github.com>

Insert DSV4 prompt after first decode result

1933c33

Co-authored-by: zhangxiaolei <zhangxiaolei123456@users.noreply.github.com>

Polish DSV4 decode radix prompt donation: full-only leaves, SWA tail …

f8b24c1

…free, full match Co-authored-by: zhangxiaolei <zhangxiaolei123456@users.noreply.github.com>

Fix pre-existing issues: add PureSWATokenToKVPoolAllocator, fix disab…

c116faa

…le_piecewise_cuda_graph attribute

Fix: handle None pre_alloc_size in DecodeReqToTokenPool (default to 0)

17d8080

Fix: use Range immutably (don't set .end); get_fill_ids() already ret…

1cb9419

…urns prompt snapshot

Fix lint: import ordering, line length, test location, missing indexe…

4f9821f

…r import

Keep SWA insertion margin for decode radix cache

4494dfe

Match the radix-cache eviction margin: keep enough SWA before the page-aligned insert boundary for the cached key to contain a complete window. `seq_len - 1` is the last committed position. Backport of sgl-project#27770 commit 0bca294.

Use direct attribute access for compression_ratios in DSV4 helpers

e840ef2

_uses_dsv4_decode_radix_cache() already guarantees the pool is a DeepSeekV4TokenToKVPool, so compression_ratios always exists. Drop the getattr fallback to fail fast instead of silently degrading to an empty list.

TobyMint force-pushed the dsv4-decode-radix-cache-mtp-support branch from 82c284d to 0539913 Compare July 27, 2026 03:53

b8zhong mentioned this pull request Aug 5, 2026

[NVIDIA] DeepSeek V4 Perf Tracking #33636

Open

41 tasks

RyeYuan mentioned this pull request Aug 27, 2026

[FEAT][DSpark] Support the p/d radix cache w/ DSpark HYGON-AI/sglang-das#247

Merged

5 tasks

Navigation Menu

Uh oh!

[P/D Disagg] DeepSeek-V4 decode radix cache with MTP support - #31097

[P/D Disagg] DeepSeek-V4 decode radix cache with MTP support#31097
TobyMint wants to merge 25 commits into
sgl-project:mainfrom
TobyMint:dsv4-decode-radix-cache-mtp-support

TobyMint commented Jul 14, 2026 •

edited by github-actions Bot

Loading

Uh oh!

gemini-code-assist Bot commented Jul 14, 2026

Uh oh!

dongyibo commented Aug 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

TobyMint commented Jul 14, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Accuracy Tests

Speed Tests

Checklist

CI States

Uh oh!

gemini-code-assist Bot commented Jul 14, 2026

Uh oh!

dongyibo commented Aug 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

TobyMint commented Jul 14, 2026 •

edited by github-actions Bot

Loading