Skip to content

[mem_cache] Make free_swa sync-free on page_size == 1 - #36723

Merged
hnyls2002 merged 41 commits into
mainfrom
lsyin/swa-drop-peer-mask
Sep 2, 2026
Merged

hnyls2002 merged 41 commits into
mainfrom
lsyin/swa-drop-peer-mask

Conversation

@hnyls2002

@hnyls2002 hnyls2002 commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • free_swa on page_size == 1 gathers the peers and releases them without the swa_indices > 0 filter; the filter's data-dependent output shape read a count back to the host on every SWA release
  • The filter's first job, dropping padding entries, is no longer needed: callers declare intent with free_full for released peers ([mem_cache] Add free_kv_row to release a request's kv row by row range #36721), grouped frees resolve peers at enqueue ([misc] Resolve SWA ownership at enqueue time for grouped free() #36646), and duplicate insert slices are split at the eviction floor ([mem_cache] Split duplicate insert frees at the SWA eviction floor #37481), so the gather never reads a padding entry
  • Its other job, absorbing a caller that frees a range whose peers are already gone, becomes a contract: swa.peer_mapped, an async Invariant (Bucket.FATAL_UNCONTAINABLE) checked at the gather under SGLANG_INVARIANT_CHECK
  • The mirror contract guards free_full: swa.peer_released (mapping[free_index] == 0, Bucket.GUARD) fires when a caller releases only the full side of a slot whose SWA peer is still live, which would otherwise leak that peer; free_group_end frees the pending full ids directly so the check runs once, at enqueue
  • page_size > 1 keeps one filter per group in _release_swa: HiCache LOAD_BACK re-pairs a page-aligned full chunk with an offset SWA one, so a page can hold unmapped slots
free-swa-filter-to-contract

Tests

  • TestSWAPeerMappedContract: the checked predicate is false exactly for a slot whose peer is gone, and free_swa no longer trips torch.cuda.set_sync_debug_mode("error") (CUDA only, gated on the pre-fix form still being flagged)
  • Drop test_free_group_owns_tombstoned_indices: free() on a tombstoned slot is the caller error the contract now reports
  • SWA fixtures that declare an eviction floor now release the peers below it first (free_swa) before calling free_full / inserting with swa_evicted_seqlen, as window eviction does; test_free_full_leaves_the_mapping_intact is dropped because the mapping is always zero at that point under the contract
  • test_self_e2e_perturb_req_to_token runs with SGLANG_INVARIANT_CHECK=0: a perturbed slot reaching free_swa with its peer released is the corruption under test, not a regression

Follows #36646.


CI States

Latest PR Test (Base): 🚫 Run #33678830613
Latest PR Test (Extra): 🚫 Run #33678830328
Latest PR Test (AMD ROCm 7.2): ⏳ Run #33678830257

…/swa-drop-peer-mask

# Conflicts:
#	python/sglang/srt/mem_cache/allocator/swa.py
#	test/registered/unit/mem_cache/test_swa_unittest.py
@hnyls2002
hnyls2002 changed the base branch from main to lsyin/swa-free-group September 1, 2026 21:41
Base automatically changed from lsyin/swa-free-group to main September 2, 2026 06:06
@hnyls2002

Copy link
Copy Markdown
Collaborator Author

Latest PR Test (Base): ❌ Run #33573528101
Latest PR Test (Extra): ✅ Run #33573527976

PR chains passed

@hnyls2002
hnyls2002 merged commit 19c7679 into main Sep 2, 2026
124 of 181 checks passed
@hnyls2002
hnyls2002 deleted the lsyin/swa-drop-peer-mask branch September 2, 2026 21:18
StevenChenSE pushed a commit to StevenChenSE/sglang that referenced this pull request Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants