[mem_cache] Add free_full to release the full side of a tombstoned SWA node - #36637
Merged
Merged
Conversation
hnyls2002
requested review from
Ying1123,
alphabetc1,
hanming-lu,
huangtingwei9988,
hzh0425,
ispobock,
merrymercy,
xiezhq-hermann and
yizhang2077
as code owners
August 27, 2026 06:46
# Conflicts: # python/sglang/srt/mem_cache/allocator/hisparse.py # python/sglang/srt/mem_cache/allocator/swa.py # python/sglang/srt/mem_cache/multi_ended_allocator.py
saturn-acc
pushed a commit
to saturn-acc/sglang
that referenced
this pull request
Aug 31, 2026
nzr-niu
pushed a commit
to nzr-niu/sglang
that referenced
this pull request
Sep 1, 2026
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
free_fullto the allocator family: a plainfree()on the base, full-pool-only on the hybrid SWA allocators, a no-op on the all-SWA oneSWARadixCacheand the unified tree_free_node_valuehelper, so the allocator call and the two eviction counters stay on the same branchBackground
free()releases both halves: it gathersfull_to_swa_index_mapping[idx], drops the entries that read as the padding slot, and returns the rest to the SWA pooldec_swa_lock_only), and that cleared their mapping entries -- sofree()on such a node already resolves to "full side only" today, purely because the sentinel filter absorbs itScope
free_full(x)releases a subset of whatfree(x)releases, and on a tombstoned node the two are already equalPureSWATokenToKVPoolAllocatorgets an explicit no-op override -- itsfull_attn_allocatoris itsswa_attn_allocator, so inheriting the base default would free one pool twicefree_swanow definefree_fullas well; a hybrid subclass silently inheriting the base default is exactly the double free aboveAlso here
Free-group timing.
_recover_tombstone_keeping_locked_fulland theRecoverSWAWithLockedFullhandler reached intoalloc.full_attn_allocator.free(...)directly, which released immediately even inside a free group while every other path deferred to the barrier. They now go throughfree_fulland defer like the rest.Unified needs a distinct action type, not a flag read.
FreeDeviceKVand friends are batched until a barrier, and the same batch'sSWARebuildreassigns the node's full value before the batch drains -- by then the node's tombstone state no longer describes the value being freed.FreeDeviceKVFullOnlyfreezes the decision at emit time instead.Test plan
test_swa_unittest.py:TestFreeFullPartition-- the partition adds up tofree, the SWA peers stay allocated, the mapping is left intact, the release defers inside a free group, and a single-pool allocator falls back tofreetest_unified_radix_cache_unittest.py: theRecoverSWAWithLockedFullcase now assertsfree_fullinstead of the inner allocator, so the free-group defer above is coveredtest/registered/unit/mem_cachepassesCI States
Latest PR Test (Base): 🚫 Run #33136294822
Latest PR Test (Extra): 🚫 Run #33136294751
Latest PR Test (AMD ROCm 7.2): 🚫 Run #33136294714