Repository navigation
fix(unified-memory): propagate unified-memory lazy checkpoint policy and handle unallocated session slots - #42072
Merged
ch-wan merged 2 commits intoOct 4, 2026
Conversation
seokwoosong
requested review from
Ying1123,
alphabetc1,
hanming-lu,
hnyls2002,
huangtingwei9988,
hzh0425,
ispobock,
merrymercy,
xiezhq-hermann and
yizhang2077
as code owners
October 1, 2026 14:19
4 of 5 tasks
ch-wan
force-pushed
the
fix/unified-lazy-policy-session
branch
from
October 4, 2026 07:33
b75ee07 to
77a0434
Compare
4 of 5 tasks
…r fixture init_unified_mamba_swa_pools now requires enable_mamba_extra_buffer_lazy, so the shared unified allocator fixture states it explicitly.
ch-wan
force-pushed
the
fix/unified-lazy-policy-session
branch
from
October 4, 2026 09:51
77a0434 to
26cce36
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
With
--mamba-radix-cache-strategy extra_buffer_lazy, a request holds one ping-pong checkpoint slot and the second is allocated only when a track boundary needs it. The unified-memory pool factories never forwardedenable_mamba_extra_buffer_lazy, so unified pools allocated both slots up front while the scheduler ran the lazy policy.Forwarding the flag means unified pools now hold ping-pong buffers whose unallocated entries are
-1.free_mamba_cachealready filters those, but streaming-session cleanup freed the whole buffer and countednumel()slots. The same gap exists on the static pool in lazy mode.Modifications
enable_mamba_extra_buffer_lazythroughinit_unified_mamba_pools,init_unified_mamba_swa_poolsandUnifiedHybridReqToTokenPool, as a required keyword, so a caller that drops it fails with aTypeErrorinstead of silently running eager.StreamingSession._free_slot_mambafrees only the allocated (!= -1) ping-pong entries, andsession_held_mamba_slotscounts only those. This applies to every pool, not just unified ones.python/sglang/test/unified_allocator_fixtures.py) passes the flag explicitly.-1entries on unified and static allocators.Accuracy Tests
Validated on an NVIDIA B200 against main
048b2a07f3, with this PR rebased onto it:test_unified_tri_pool.py: 49 tests and 100 subtests pass, including the two new ones (47 on main). The related unified-memory, Mamba and streaming-session test files give the same results as main.thinkingmachines/Inkling, revisiontest) with--enable-unified-memory --mamba-radix-cache-strategy extra_buffer_lazyand--enable-deterministic-inference, 32 LongBench prompts:pool memory leak detected!, the Mamba pool counting 32 slots more than it holds, one per request;avg_kl_div=0.0.After rebasing onto main
6cec8f98ce(which includes #39982), the tests that build pools through the shared fixture pass:test_unified_pending_event_batches.py,test_unified_dynamic_gate_capacity.pyandtest_unified_float_move_gate.py, along withtest_unified_tri_pool.py(49 tests).Speed Tests and Profiling
No speed impact expected; the session count adds one device read per held slot, and it runs only in the idle leak check.
Checklist
CI States
Latest PR Test (Base): 🚫 Run #37193489703
Latest PR Test (Extra): ❌ Run #37193489469
Latest PR Test (AMD ROCm 10): ❌ Run #37193489678