Repository navigation
[Perf][Core] Skip the null head of sliding-window block tables - #61724
Merged
Merged
Conversation
A sliding-window block table keeps one null block per out-of-window block, so for long requests it is almost all null. Find that head by binary search and skip it when collecting unhashed block ids and when freeing a request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Juntian Liu <Juntianl777@gmail.com>
Juntian777
requested review from
ApostaC,
WoosukKwon,
ZJY0516,
alexm-redhat,
heheda12345,
ivanium,
njhill,
orozery,
robertgshaw2-redhat and
ywang96
as code owners
October 15, 2026 01:00
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
njhill
reviewed
Oct 15, 2026
njhill
left a comment
njhill
left a comment
Member
There was a problem hiding this comment.
Thanks @Juntian777!
Do you know whether this change by itself made noticeable difference e2e?
Address review: move the null-head trim from free() into pop_blocks_for_free() (new trim_null_head(), no copy when there is no null head) so the deferred-free path benefits too, and trim the residency row HiSparseResidentManager returns. touch() skips the null block like free_blocks() does, which drops its now-redundant is_null check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Juntian Liu <Juntianl777@gmail.com>
Contributor
Author
|
@njhill Thanks for the review comments! I've addressed them and added the E2E improvement numbers to the summary. This happens with DSV41F wideEP because it's GPU computation too fast, and larger DEP is more likely to have CPU stragglers at high concurrency. |
njhill
reviewed
Oct 15, 2026
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Member
|
/ci run |
|
✅ Triggered Buildkite CI #95212 for commit |
|
✅ @Juntian777, CI is now available for this PR.
|
Contributor
|
Hi @Juntian777, the pre-commit checks have failed. Please run: uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-filesThen, commit the changes and push to your branch. For future commits, |
Member
|
/ci run |
|
✅ Triggered Buildkite CI #95240 for commit |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
A sliding-window KV group keeps one null block per out-of-window block, so a long request's block table is almost all null (8,183 of 8,192 entries for a 128K-token prompt with a 128-token window and 16-token blocks).
KVCacheBlocks.get_unhashed_block_ids_all_groupsandSingleTypeKVCacheManager.freewalked every entry in Python. This PR skips that null head.null_head_len()finds the leading null run by binary search and checks it with onelist.countof the shared null block. If the null blocks are not one contiguous head, it falls back to a scan.get_unhashed_block_ids_all_groupsandpop_blocks_for_free(so bothfreeand the deferred-free path) start after the head.free_blocksskips any other null blocks; the null block'sref_cntis not maintained (seeBlockPool.__init__).Not a duplicate: I searched open PRs for
null block sliding window free,null_block free_blocks,sliding window null blocksandget_unhashed_block_ids. #42276 and #55818 change SWA eviction and allocation, not these walks.Where it matters
The cost is paid per sliding-window group, on the scheduler's critical path, each time a request is admitted with async KV loads or freed. It shows up when GPU steps are short and requests turn over quickly, as in DeepSeek V4.1 Flash, WideEP DEP16 decode (GB300, 64 requests/rank, 169.5K-token prompts):
End to end, output throughput went from 8,892 to 9,025 tok/s/GPU (+1.5%).
Microbenchmark
One request on one sliding-window group (window 128, block size 16):
KVCacheBlocks.get_unhashed_block_ids_all_groups(collect) andSlidingWindowManager.free(free). Median of 300 calls on a Grace CPU, main (8530aed2b7) vs this PR, consistent across two runs.Test Plan
New test
test_null_head_len: empty, no-null, all-null, contiguous-head and non-contiguous tables, plusget_unhashed_block_ids_all_groupson two groups.Test Result
312 passed. Pre-commit (ruff, format, typos, mypy) passes.
AI assistance (Claude Code) was used to write and benchmark this change.
🤖 Generated with Claude Code