[HiCache] remove large host mem constraint - #28614
Conversation
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
…ller_host # Conflicts: # python/sglang/srt/mem_cache/memory_pool_host.py
This comment was marked as outdated.
This comment was marked as outdated.
|
hey @vladnosiv thank you for your proposal! this PR simply removes the size constraints but leave most mechanism unchanged, and I will have a new PR for buffer-only mode soon which utilizes a different runtime of staging and prefetching, will take a look at your proposal as well. |
| self.evictable_size_ -= len(n.value) | ||
| n.value = None | ||
| self.ongoing_write_through.pop(n.id, None) | ||
| self.evictable_host_leaves.discard(n) |
There was a problem hiding this comment.
should remove n on 'self.evictable_leaves' here?
|
This PR is under stress testing. Will test three scenarios: (1) before applying this PR, L1:L2=1:1, L1:L2=1:3, L1:L2:L3=1:1:2, (2) after applying this PR, L1:L2=1:1, L1:L2:L3=1:1:2, (3) after applying this PR, L1:L2=1:0.5, L1:L2:L3=1:0.5:2.5. If this PR introduces no performance regression in L1+L2+L3, it's perfect. |
|
@stmatengss let me know how does the test go, I also did some style refinement based on your comments thanks |
This PR, tested on Qwen 3 32B with Tp=4, demonstrates no L1 + L2 performance degradation and is ready for merging. L3 testing is scheduled for next week. |
|
cc @YAMY1234 @wenscarl @weireweire for vis |
|
I noticed HiRadixCache.evict_host doesn’t lift the restriction that only evicts host values belonging to evicted nodes. As a result, the L2 entries tied to L1 partitions won’t get evicted. |
…last-leaf fix Upstream sgl-project#29860 (a375e9f, merged). Adapted to our fork's inline _evict_swa in schedule_batch.py (upstream lives in mem_cache/common.py:free_swa_out_of_window_slots). Before: _evict_swa evicted up to pre_len - sliding_window_size - page_size under the assumption that "extra page keeps the frontier below the insert boundary". The env flag SGLANG_OPT_SWA_EVICT_DROP_PAGE_MARGIN toggled whether to apply the extra -page_size subtraction. After: gate on tree_cache.is_chunk_cache() (replicated tree-cache property, uniform across ranks) instead of the env flag: - chunk-cache: no radix tree -> no tombstone-leaf concern; evict up to the window boundary (pre_len - sliding_window_size). - radix: keep max(window, page). The trailing floor page-aligns the frontier, and subtracting at least one page keeps the frontier below the insert boundary (page_floor(seq_len)) so the last leaf is never all-tombstone. This is the case sgl-project#29860 was fixing. The env var SGLANG_OPT_SWA_EVICT_DROP_PAGE_MARGIN is now unused but left in environ.py for backward compatibility (harmless no-op). P2 batch, other PRs skipped this round: - sgl-project#27550 fix(hiradix): wait for extra pool IO - target code path (the completed+pool_transfers_done branch in can_terminate_prefetch) was simplified away by our PR sgl-project#27010 port; the pool_transfers_done invariant is now enforced through the ack-queue ordering instead. - sgl-project#28422 decode-hicache _storage_hit_query - the "pre-query" feature sgl-project#28422 patches does not exist in our fork. - sgl-project#29887 [PP] get kv_buffer_shape - target file (eager_runner.py) does not exist in our fork. - sgl-project#29817 write_back policy refinement - refactor, not a bug fix; our fork's evict() has diverged from the upstream shape and porting cleanly is out of scope for this batch. - sgl-project#28614 remove large host mem constraint - depends on sgl-project#29817's evict refactor for the bulk of its diff, and the standalone piece flips prefetch_capacity_limit from max(0, 0.8*(host-device)) to 0.5*host, a subtle memory-budget semantics change we won't ship silently. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Teng Ma <stmatengss@gmail.com> Co-authored-by: Vladislav Nosivskoy <vladnosiv@gmail.com>
6 participants
Motivation
This PR removes the hardcoded constraint that requires host memory pool to be larger than the device pool, including changes and guards in cache write policy implementations. This will serve as a stepping stone for a more comprehensive buffer-only mode implementation as explored in other PRs: #16909, #20535
Credit to @stmatengss and @vladnosiv as well.
Modifications
Accuracy Tests
Speed Tests and Profiling
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ❌ Run #28221233942
Latest PR Test (Extra): ❌ Run #28221233860