[sgl] Window-aware LRU refresh for SWA prefix cache in unified cache by bixue2010 · Pull Request #26615 · sgl-project/sglang

bixue2010 · 2026-05-28T22:20:16Z

Motivation

Summary
Make the SWA (Sliding Window Attention) radix-cache LRU window-aware so that out-of-window SWA nodes can be evicted as they should be. This significantly improves SWA prefix-cache hit rate on long multi-turn workloads, with no behavior change for non-SWA components.

Problem
The radix cache uses LRU to evict SWA cache. On every prefix-cache match (and on every walk-down during insert), all matched ancestors are bumped to MRU. That is correct for the Full KV, but it is wrong for SWA, because for SWA only the last sliding_window_size tokens are actually reachable from any current decode step. As a result:

Very old SWA nodes (outside the active window of every live request) keep getting refreshed to MRU and never get evicted.
Genuinely-hot SWA cache from other branches gets pushed toward the LRU tail and is evicted first.
Under multi-turn pressure, the SWA pool fills up with cache that can no longer contribute to any hit, hit-rate degrades, and eviction churn goes up.

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
Follow the SGLang code style guidance.

Review and Merge Process

Ping Merge Oncalls to start the process. See the PR Merge Process.
Get approvals from CODEOWNERS and other reviewers.
Trigger CI tests with comments or contact authorized users to do so.
- Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ⏳ Run #26739938263
Latest PR Test (Extra): ✅ Run #26739938119

gemini-code-assist · 2026-05-28T22:20:20Z

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

ispobock · 2026-05-29T15:35:30Z

/tag-and-rerun-ci

ispobock · 2026-05-30T15:23:35Z

/rerun-failed-ci

hzh0425 · 2026-05-30T16:35:07Z

        value = node.component_data[self.component_type].value
        return len(value) if value is not None else 0

+    def refresh_lru_on_walkdown(self, node: UnifiedTreeNode) -> None:


Should we avoid adding too many new component interfaces here?
Could we instead merge them into a single interface and use a "phase" parameter to distinguish between different cases?

Additionally, it seems that refresh_lru_on_walkdown hasn't been implemented yet on swa_component

changed to less inferfaces and use phrases.

add comment to indicate why we don't need to do anything in walkdown for swa to avoid confusing.

… bi/swa_lru

bixue2010 · 2026-05-31T05:41:25Z

/rerun-failed-ci

hzh0425

Great work! LGTM

hzh0425 · 2026-05-31T18:01:31Z

/rerun-group radix_cache/unified_radix_tree

github-actions · 2026-05-31T18:01:54Z

Results for /rerun-group radix_cache/unified_radix_tree:

🚀 4-gpu-h100 (2 tests): ❌ View workflow run

cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_dsv4.py
cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_mamba.py

🚀 2-gpu-h100 (2 tests): ❌ View workflow run

cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_full.py
cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_swa.py

🚀 8-gpu-h200 (1 test): ❌ View workflow run

cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_hicache_nightly.py

hzh0425 · 2026-06-01T02:02:06Z

/rerun-group radix_cache/unified_radix_tree

github-actions · 2026-06-01T02:02:30Z

Results for /rerun-group radix_cache/unified_radix_tree:

🚀 4-gpu-h100 (2 tests): ✅ View workflow run

cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_dsv4.py
cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_mamba.py

🚀 2-gpu-h100 (2 tests): ✅ View workflow run

cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_full.py
cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_swa.py

🚀 8-gpu-h200 (1 test): ❌ View workflow run

cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_hicache_nightly.py

bixue2010 · 2026-06-01T06:03:40Z

/rerun-failed-ci

hzh0425 · 2026-06-01T06:55:21Z

/rerun-test test/registered/disaggregation/test_disaggregation_pp.py

github-actions · 2026-06-01T06:55:47Z

Results for /rerun-test test/registered/disaggregation/test_disaggregation_pp.py:

🚀 8-gpu-h20 (1 test): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_pp.py

ispobock · 2026-06-01T08:35:29Z

/rerun-test test_unified_radix_cache_kl_hicache_nightly.py

github-actions · 2026-06-01T08:35:52Z

Results for /rerun-test test_unified_radix_cache_kl_hicache_nightly.py:

🚀 8-gpu-h200 (1 test): ❌ View workflow run

cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_hicache_nightly.py

ispobock · 2026-06-01T11:34:47Z

test_unified_radix_cache_kl_hicache_nightly.py failure seems not related to this PR.

…gl-project#26615)

…26615)

…gl-project#26615)

github-actions Bot and others added 4 commits May 5, 2026 14:14

docs: sync LMSYS SGLang blog cards

0cb5a61

Merge branch 'sgl-project:main' into main

e6f2b88

Merge branch 'sgl-project:main' into main

35c2031

[sgl] Window-aware LRU refresh for SWA prefix cache in unified cache

b5da98d

bixue2010 requested review from Ying1123, hanming-lu, hnyls2002, hzh0425, ispobock, merrymercy, xiezhq-hermann and yizhang2077 as code owners May 28, 2026 22:20

ispobock added the run-ci-extra CI: also run the extra suite (requires run-ci) label May 29, 2026

github-actions Bot added the run-ci CI: run the baseline test suite on this PR label May 29, 2026

ispobock reviewed May 29, 2026

View reviewed changes

Comment thread test/registered/unit/mem_cache/test_unified_radix_cache_unittest.py Outdated

Merge branch 'main' into bi/swa_lru

b3bf499

ispobock assigned ispobock and hzh0425 May 29, 2026

bi and others added 2 commits May 29, 2026 17:24

[sgl] add page_size > 1 test as suggested

37995c3

Merge branch 'main' into bi/swa_lru

763be95

ispobock approved these changes May 30, 2026

View reviewed changes

hzh0425 requested changes May 30, 2026

View reviewed changes

bi added 3 commits May 31, 2026 04:50

[sgl] address feedbacks to reduce inferface and use phrase

e22fcc4

Merge branch 'bi/swa_lru' of https://github.com/bixue2010/sglang into…

5915c94

… bi/swa_lru

fix lint

7a224b1

hzh0425 approved these changes May 31, 2026

View reviewed changes

sgl-project deleted a comment from github-actions Bot May 31, 2026

remove override as it only available >py 3.12

0cb6b75

Merge branch 'main' into bi/swa_lru

273558e

ispobock approved these changes Jun 1, 2026

View reviewed changes

ispobock merged commit 6965fe0 into sgl-project:main Jun 1, 2026
119 of 147 checks passed

amd-bot mentioned this pull request Jun 2, 2026

[CI Monitor] Daily Report - 2026-06-02 bingxche/sglang-ci-bot#91

Open

hzh0425 mentioned this pull request Jun 2, 2026

[SWA] Improve SWA prefix-cache retention under memory pressure #27009

Closed

mqhc2020 pushed a commit to mqhc2020/sglang that referenced this pull request Jun 2, 2026

[sgl] Window-aware LRU refresh for SWA prefix cache in unified cache (s…

2f42e02

…gl-project#26615)

hanming-lu pushed a commit that referenced this pull request Jun 3, 2026

[sgl] Window-aware LRU refresh for SWA prefix cache in unified cache (#…

9c835bf

…26615)

HanHan009527 mentioned this pull request Jun 11, 2026

[HiCache] DeepSeek-V4 host-mem L2 + NVMe/disk L3 (file backend) bytedance-iaas/sglang#564

Merged

hzwzwzw pushed a commit to hzwzwzw/sglang that referenced this pull request Jun 15, 2026

[sgl] Window-aware LRU refresh for SWA prefix cache in unified cache (s…

c7edd9a

…gl-project#26615)

luoroger37 mentioned this pull request Jul 23, 2026

[HiCache][DSV4] Complete SWA pressure and host-pool fixes bytedance-iaas/sglang#659

Merged

ispobock mentioned this pull request Aug 6, 2026

[Roadmap] Unified Hybrid Radix Cache Refactor #20415

Open

56 of 66 tasks

Chronostasys pushed a commit to MindLab-Research/sglang that referenced this pull request Aug 24, 2026

[sgl] Window-aware LRU refresh for SWA prefix cache in unified cache (s…

39cb63f

…gl-project#26615)

Conversation

bixue2010 commented May 28, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

Review and Merge Process

CI States

Uh oh!

gemini-code-assist Bot commented May 28, 2026

Uh oh!

ispobock commented May 29, 2026

Uh oh!

Uh oh!

ispobock commented May 30, 2026

Uh oh!

hzh0425 May 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bixue2010 May 31, 2026

Choose a reason for hiding this comment

Uh oh!

bixue2010 commented May 31, 2026

Uh oh!

hzh0425 left a comment

Choose a reason for hiding this comment

Uh oh!

hzh0425 commented May 31, 2026

Uh oh!

github-actions Bot commented May 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hzh0425 commented Jun 1, 2026

Uh oh!

github-actions Bot commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bixue2010 commented Jun 1, 2026

Uh oh!

hzh0425 commented Jun 1, 2026

Uh oh!

github-actions Bot commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ispobock commented Jun 1, 2026

Uh oh!

github-actions Bot commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ispobock commented Jun 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bixue2010 commented May 28, 2026 •

edited by github-actions Bot

Loading

hzh0425 May 30, 2026 •

edited

Loading

github-actions Bot commented May 31, 2026 •

edited

Loading

github-actions Bot commented Jun 1, 2026 •

edited

Loading

github-actions Bot commented Jun 1, 2026 •

edited

Loading

github-actions Bot commented Jun 1, 2026 •

edited

Loading