[HiCache] write_back policy refinement - #29817
Merged
Merged
Conversation
xiezhq-hermann
requested review from
Ying1123,
alphabetc1,
hanming-lu,
hnyls2002,
hzh0425,
ispobock,
merrymercy and
yizhang2077
as code owners
July 1, 2026 07:07
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request removes the deprecation warning for the write_back policy in the cache controller, cleans up unused variables, and introduces a staging mechanism for write-back eviction in the hierarchical radix cache. However, a critical race condition was identified in the new eviction path: device memory is freed immediately after queuing the write-back, before the asynchronous copy operation completes. This can lead to data corruption, and it is recommended to defer freeing the device memory until after the write-back synchronization is finished in the flush_staged function.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
hzwzwzw
added a commit
to hzwzwzw/sglang
that referenced
this pull request
Jul 13, 2026
…last-leaf fix Upstream sgl-project#29860 (a375e9f, merged). Adapted to our fork's inline _evict_swa in schedule_batch.py (upstream lives in mem_cache/common.py:free_swa_out_of_window_slots). Before: _evict_swa evicted up to pre_len - sliding_window_size - page_size under the assumption that "extra page keeps the frontier below the insert boundary". The env flag SGLANG_OPT_SWA_EVICT_DROP_PAGE_MARGIN toggled whether to apply the extra -page_size subtraction. After: gate on tree_cache.is_chunk_cache() (replicated tree-cache property, uniform across ranks) instead of the env flag: - chunk-cache: no radix tree -> no tombstone-leaf concern; evict up to the window boundary (pre_len - sliding_window_size). - radix: keep max(window, page). The trailing floor page-aligns the frontier, and subtracting at least one page keeps the frontier below the insert boundary (page_floor(seq_len)) so the last leaf is never all-tombstone. This is the case sgl-project#29860 was fixing. The env var SGLANG_OPT_SWA_EVICT_DROP_PAGE_MARGIN is now unused but left in environ.py for backward compatibility (harmless no-op). P2 batch, other PRs skipped this round: - sgl-project#27550 fix(hiradix): wait for extra pool IO - target code path (the completed+pool_transfers_done branch in can_terminate_prefetch) was simplified away by our PR sgl-project#27010 port; the pool_transfers_done invariant is now enforced through the ack-queue ordering instead. - sgl-project#28422 decode-hicache _storage_hit_query - the "pre-query" feature sgl-project#28422 patches does not exist in our fork. - sgl-project#29887 [PP] get kv_buffer_shape - target file (eager_runner.py) does not exist in our fork. - sgl-project#29817 write_back policy refinement - refactor, not a bug fix; our fork's evict() has diverged from the upstream shape and porting cleanly is out of scope for this batch. - sgl-project#28614 remove large host mem constraint - depends on sgl-project#29817's evict refactor for the bulk of its diff, and the standalone piece flips prefetch_capacity_limit from max(0, 0.8*(host-device)) to 0.5*host, a subtle memory-budget semantics change we won't ship silently. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
62 tasks
Chronostasys
pushed a commit
to MindLab-Research/sglang
that referenced
this pull request
Aug 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Modifications
Accuracy Tests
Speed Tests and Profiling
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ❌ Run #28541126226
Latest PR Test (Extra): ❌ Run #28541125898