Enable unified cache out-of-window slot freeing by default - #34653
Conversation
|
/rerun-test test/registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_hybrid_bitexact.py |
|
Results for 🚀 |
|
/rerun-test test/registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_swa.py |
|
/rerun-test test/registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_full.py |
|
/rerun-test test/registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_mamba.py |
|
Results for 🚀 |
|
Results for 🚀 |
|
Results for 🚀 |
|
/rerun-test test/registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_full.py |
|
Results for 🚀 |
|
/tag-and-rerun-ci |
|
/rerun-test test/registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_full.py |
|
Results for 🚀 |
…EE_OUT_OF_WINDOW_SLOTS on Frees out-of-window SWA slots when a chunked-prefill request is stashed into the unified radix tree (`UnifiedRadixCache.cache_unfinished_req` -> `SWAComponent.free_out_of_window_slots`). We do run UnifiedRadixCache (`impl=UnifiedRadixCache`, components FULL+SWA), so the flag is live for us. Measured no effect on our workload, in either direction. A/B on DeepSeek-V4-Flash-0731 with in=13,513 (2 chunks at chunked_prefill_size=8192) / out=2048 / conc=32, with sgl-project#33805 also applied: on 1138.3 tok/s prefill-phase swa peak 0.07 off 1147.1 tok/s prefill-phase swa peak 0.07 The chunking did happen (`#new-token: 8192` then `1792` per sequence), so this is a real null result, not a missed code path. The reason is that `max_prefill_tokens=16384` admits only one or two sequences per prefill batch, so the out-of-window KV this frees is a per-request transient that never accumulates across the batch -- and with sgl-project#33805 in place the decode side is already reclaiming for every request that has moved on. It should matter with a much wider prefill batch or much longer inputs. Carried anyway to stay close to upstream default. Note the known issue behind upstream sgl-project#34823: under EAGLE the new-leaf path computes `split_pos = swa_evicted_seqlen - result.prefix_len`, mixing raw-token lengths with bigram key units, and split_pos goes negative. That PR gates the option off for EAGLE only and explicitly keeps it for DSpark/DFlash, which is what we run -- but anyone switching this branch to EAGLE should set SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=0 until sgl-project#34823 lands.
1 participant
Motivation
SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTSreleases SWA slots that have already left the sliding window while a chunked prefill is being cached, instead of holding them until the request finishes. It has been opt-in since it landed; this makes it the default.This is a default flip, so it changes behavior for every server running the unified radix cache with an SWA component. The single consumer is the chunked-prefill caching path in
unified_radix_cache.py, gated behindSGLANG_ENABLE_UNIFIED_RADIX_TREE; passingFalserestores the old behavior.The risk worth calling out is not the freeing itself but what it does to prefix reuse: how much SWA data stays resident decides how deep a later request can match, since a hit needs a full window of live SWA below the match point. Freeing earlier could in principle shorten that and silently cost cache hits.
Accuracy
test_unified_radix_cache_kl_hybrid_bitexact.pyis the sharpest check available for this: it asserts prefill and decode score every token identically (kl_divfloor 1e-9, measured exactly 0), and its hicache case additionally asserts the reused prefix length per request. One H200 (SM90, same architecture as the1-gpu-largeCI runner), both fixes in, flag defaulted on:The per-request
cached_tokensassertion in the multi-turn case passes unchanged, so the reusable prefix does not shrink.Two things this does not cover, and I would rather say so than imply otherwise: only the hybrid SWA + mamba configuration was measured, not the SWA-only or mamba-only models in the same directory, and the reason it originally shipped opt-in is not recorded anywhere I could find (the introducing commit carries no rationale). Running the full
unified_radix_tree/suite on this PR would close the first gap.CI States
Latest PR Test (Base): ✅ Run #31661339455
Latest PR Test (Extra): ✅ Run #31661339489