Skip to content

fix(unified-memory): propagate unified-memory lazy checkpoint policy and handle unallocated session slots - #42072

Merged
ch-wan merged 2 commits into
sgl-project:mainfrom
seokwoosong:fix/unified-lazy-policy-session
Oct 4, 2026
Merged

ch-wan merged 2 commits into
sgl-project:mainfrom
seokwoosong:fix/unified-lazy-policy-session

Conversation

@seokwoosong

@seokwoosong seokwoosong commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

With --mamba-radix-cache-strategy extra_buffer_lazy, a request holds one ping-pong checkpoint slot and the second is allocated only when a track boundary needs it. The unified-memory pool factories never forwarded enable_mamba_extra_buffer_lazy, so unified pools allocated both slots up front while the scheduler ran the lazy policy.

Forwarding the flag means unified pools now hold ping-pong buffers whose unallocated entries are -1. free_mamba_cache already filters those, but streaming-session cleanup freed the whole buffer and counted numel() slots. The same gap exists on the static pool in lazy mode.

Modifications

  • Forward enable_mamba_extra_buffer_lazy through init_unified_mamba_pools, init_unified_mamba_swa_pools and UnifiedHybridReqToTokenPool, as a required keyword, so a caller that drops it fails with a TypeError instead of silently running eager.
  • StreamingSession._free_slot_mamba frees only the allocated (!= -1) ping-pong entries, and session_held_mamba_slots counts only those. This applies to every pool, not just unified ones.
  • The shared unified allocator test fixture (python/sglang/test/unified_allocator_fixtures.py) passes the flag explicitly.
  • Regression tests: the policy reaches both unified factories through the production configurator, and session cleanup and accounting with -1 entries on unified and static allocators.

Accuracy Tests

Validated on an NVIDIA B200 against main 048b2a07f3, with this PR rebased onto it:

  • test_unified_tri_pool.py: 49 tests and 100 subtests pass, including the two new ones (47 on main). The related unified-memory, Mamba and streaming-session test files give the same results as main.
  • Inkling (thinkingmachines/Inkling, revision test) with --enable-unified-memory --mamba-radix-cache-strategy extra_buffer_lazy and --enable-deterministic-inference, 32 LongBench prompts:

After rebasing onto main 6cec8f98ce (which includes #39982), the tests that build pools through the shared fixture pass: test_unified_pending_event_batches.py, test_unified_dynamic_gate_capacity.py and test_unified_float_move_gate.py, along with test_unified_tri_pool.py (49 tests).

Speed Tests and Profiling

No speed impact expected; the session count adds one device read per held slot, and it runs only in the idle leak check.

Checklist


CI States

Latest PR Test (Base): 🚫 Run #37193489703
Latest PR Test (Extra): ❌ Run #37193489469
Latest PR Test (AMD ROCm 10): ❌ Run #37193489678

@ch-wan
ch-wan force-pushed the fix/unified-lazy-policy-session branch from b75ee07 to 77a0434 Compare October 4, 2026 07:33
@ch-wan ch-wan added run-ci CI: run the baseline test suite on this PR bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) parallel-stages CI: stages dispatch together instead of waiting on each other labels Oct 4, 2026
seokwoosong and others added 2 commits October 4, 2026 02:49
…r fixture

init_unified_mamba_swa_pools now requires enable_mamba_extra_buffer_lazy,
so the shared unified allocator fixture states it explicitly.
@ch-wan
ch-wan force-pushed the fix/unified-lazy-policy-session branch from 77a0434 to 26cce36 Compare October 4, 2026 09:51
@ch-wan
ch-wan merged commit bab04cd into sgl-project:main Oct 4, 2026
187 of 251 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) memory-pool parallel-stages CI: stages dispatch together instead of waiting on each other run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants