Skip to content

[mem_cache] Free hybrid SWA pages by one representative per page on page_size > 1 - #38159

Merged
hnyls2002 merged 9 commits into
mainfrom
lsyin/swa-free-reps
Sep 8, 2026
Merged

hnyls2002 merged 9 commits into
mainfrom
lsyin/swa-free-reps

Conversation

@hnyls2002

@hnyls2002 hnyls2002 commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add free_swa_segment(idx, *, start_pos), the SWA-only counterpart of free_segment / free_full_segment, and route every kv-row caller through it; free_swa(idx) stays the set-shaped form (alloc rollback)
  • On page_size > 1 the SWA side now frees by host-derived page ids (mapping[idx[::page_size]] // page_size) instead of expanding pages, filtering > 0 and dedup'ing through torch.unique, so a hybrid SWA free no longer synchronizes the scheduler stream at any page size (follows [mem_cache] Make free_swa sync-free on page_size == 1 #36723, which covered page_size == 1)
  • Add free_page_ids(page_ids) to the paged and token allocators as the page-native primitive (exact pages, no dedup); free_segment is built on it
kv-free-api

Details

  • Inside a free group the composite queues the gathered SWA page ids (swa_page_ids_group) and flushes them with one free_page_ids, so the group flush is fixed-shape too
  • Callers moved to the segment form: window eviction in free_swa_out_of_window_slots, SWARadixCache node frees (free_swa / free / free_full on page-aligned node values), the unified FreeComponentDeviceSlot action; UnifiedSWATokenToKVPoolAllocator splits its existing start_pos branch into free_swa_segment so both hybrid allocators share the API
  • Page-level 1:1 between the full and SWA pools holds on every pairing path (extend / decode alloc drive both pools with one seq_lens, HiCache load-back pairs equal-length page-aligned chunks), so the first token of a page maps into the whole peer page

Tests

  • test_swa_unittest.py: page-set equivalence against torch.unique over every tail length with the mapping cleared per page, set_sync_debug_mode("error") on the segment path in and out of a group (CUDA), the _SWA_PEER_MAPPED predicate on page_size == 4, tree eviction never reaching the inner free, and the group-ownership test parametrized over page_size in (1, 4)

Original commits

  • 80197780d1b0dff8cc3bd556e790de7e7900653c

CI States

Latest PR Test (Base): ✅ Run #34162095517
Latest PR Test (Extra): ✅ Run #34162095388
Latest PR Test (AMD ROCm 7.2): ❌ Run #34162095509

mqhc2020 pushed a commit to mqhc2020/sglang that referenced this pull request Sep 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant