Skip oow slot freeing under eagle - #34823
Merged
Merged
Conversation
ispobock
requested review from
Ying1123,
alphabetc1,
hanming-lu,
hnyls2002,
hzh0425,
merrymercy,
xiezhq-hermann and
yizhang2077
as code owners
August 14, 2026 07:32
Collaborator
Author
|
/rerun-test registered/8-gpu-models/test_gpt_oss_120b.py |
Contributor
|
Results for 🚀 🚀 |
Collaborator
Author
|
/rerun-test registered/8-gpu-models/test_gpt_oss_120b.py |
Contributor
|
Results for 🚀 🚀 |
ehuaa
added a commit
to ehuaa/sglang
that referenced
this pull request
Aug 14, 2026
…EE_OUT_OF_WINDOW_SLOTS on Frees out-of-window SWA slots when a chunked-prefill request is stashed into the unified radix tree (`UnifiedRadixCache.cache_unfinished_req` -> `SWAComponent.free_out_of_window_slots`). We do run UnifiedRadixCache (`impl=UnifiedRadixCache`, components FULL+SWA), so the flag is live for us. Measured no effect on our workload, in either direction. A/B on DeepSeek-V4-Flash-0731 with in=13,513 (2 chunks at chunked_prefill_size=8192) / out=2048 / conc=32, with sgl-project#33805 also applied: on 1138.3 tok/s prefill-phase swa peak 0.07 off 1147.1 tok/s prefill-phase swa peak 0.07 The chunking did happen (`#new-token: 8192` then `1792` per sequence), so this is a real null result, not a missed code path. The reason is that `max_prefill_tokens=16384` admits only one or two sequences per prefill batch, so the out-of-window KV this frees is a per-request transient that never accumulates across the batch -- and with sgl-project#33805 in place the decode side is already reclaiming for every request that has moved on. It should matter with a much wider prefill batch or much longer inputs. Carried anyway to stay close to upstream default. Note the known issue behind upstream sgl-project#34823: under EAGLE the new-leaf path computes `split_pos = swa_evicted_seqlen - result.prefix_len`, mixing raw-token lengths with bigram key units, and split_pos goes negative. That PR gates the option off for EAGLE only and explicitly keeps it for DSpark/DFlash, which is what we run -- but anyone switching this branch to EAGLE should set SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=0 until sgl-project#34823 lands.
ispobock
changed the title
saturn-acc
pushed a commit
to saturn-acc/sglang
that referenced
this pull request
Aug 16, 2026
This was referenced Aug 19, 2026
hanwlax
pushed a commit
to hanwlax/sglang
that referenced
this pull request
Aug 28, 2026
jakki-amd
pushed a commit
to jakki-amd/sglang
that referenced
this pull request
Sep 9, 2026
Atituiset
pushed a commit
to Atituiset/sglang
that referenced
this pull request
Sep 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
#34653 flipped
SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTStoTrue. On a hybrid SWA model with EAGLE speculative decoding this trips the pool invariant checker and kills every TP rank:Root cause: in the new-leaf path,
split_pos = swa_evicted_seqlen - result.prefix_lenmixes units.swa_evicted_seqlenis a raw-token length whileprefix_lenis in bigram key units under EAGLE, sosplit_posgoes negative (observed-64withpage_size 64) and the leaf is neither tombstoned nor split, while its SWA slots were already returned to the allocator. The tree keeps counting them as evictable, so the accounted total exceeds the pool total.Note the blast radius is wider than the flag suggests:
registry.pyroutes every hybrid SWA model toUnifiedRadixCacheunconditionally, withoutSGLANG_ENABLE_UNIFIED_RADIX_TREE.This gates the new default off for the EAGLE family until the unit mismatch is fixed. DSpark, DFlash, non-speculative and hybrid SSM keep the optimization, since
tree_core.is_eagleisparams.is_eagle and MAMBA not in components.Repro
8xH200, gpt-oss-120b,
--page-size 64 --chunked-prefill-size 16384, EAGLE3 draftlmsys/EAGLE3-gpt-oss-120b-bf16,bench_one_batch_server --batch-size 1 8 --input-len 8192 --output-len 512:page_size 64is required to reproduce. Atpage_size 1the page-alignment paths are skipped entirely and the mismatch does not surface.Nightly hit the same signature on B200: https://github.com/sgl-project/sglang/actions/runs/31710357501
CI States
Latest PR Test (Base): ❌ Run #31780346451
Latest PR Test (Extra): 🚫 Run #31781367291