Repository navigation
[Core][HiSparse] Keep unpinned blocks last in the free queue - #59283
Draft
MatthewBonanni wants to merge 1 commit into
Draft
MatthewBonanni wants to merge 1 commit into
MatthewBonanni wants to merge 1 commit into
Conversation
BlockPool.unpin_blocks documents its blocks as last-resort eviction candidates, but blocks freed afterwards were appended behind them, so an allocation could evict HiSparse's GPU prefix copies before blocks nobody would reuse. FreeKVCacheBlockQueue now keeps a last-resort segment at its tail that later appends stay in front of, and unpin_blocks uses it. Only HiSparse unpins blocks; without them the queue behaves as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
MatthewBonanni
force-pushed
the
perf/hisparse-last-resort-copies
branch
from
September 30, 2026 19:43
94d4d36 to
2591660
Compare
26 of 53 tasks
6 tasks done
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Optional performance improvement, stacked on #59282. #59282 fixes the crash on its own; this PR only affects how many HiSparse GPU prefix copies survive memory pressure, and so how many prefix pages are read from host instead.
BlockPool.unpin_blocksdocuments unpinned blocks as "last-resort eviction candidates", but it appends them to the free queue like any other block, so blocks freed afterwards are queued behind them. In #59282's scenario, a finished request's cached indexer block is freed after its MLA copies were unpinned, so the resumed request's allocation evicts a copy it was about to adopt while that indexer block survives.Changes
FreeKVCacheBlockQueuekeeps a last-resort segment at its tail:append/append_ninsert in front of it, the newappend_last_resort_nappends behind it, andpopleft/popleft_n/removeadvance its head when they take it.BlockPool.unpin_blocksusesappend_last_resort_n, which makes its documented behavior true.Only the HiSparse coordinator unpins blocks. Without unpinned blocks the segment is empty and the queue orders blocks exactly as before, at the cost of one pointer comparison per popped or removed block.
This does change HiSparse's eviction policy: unpinned blocks, meaning finished requests' copies and pages a running request released but still reads, now outlast every ordinary cached block rather than aging in LRU order. Evicting either kind forces reads from host.
Copies kept in #59282's scenario:
free_blocksTest Plan
test_hisparse_prefix_hit_under_pressure_adopts_surviving_copies[10]now expects all three copies; it keeps two without the last-resort segment.test_free_kv_cache_block_queue_last_resort_stays_behind: later appends stay ahead of the segment after its first block is removed or popped.tests/v1/core/test_prefix_caching.pyandtests/v1/core/test_kv_cache_utils.py, withCpuPlatformselected.Test Result
fixture 'caplog_vllm' not found, from running outside the top-level conftest, and are identical on main.AI assistance (Claude Code) was used for the investigation, implementation, tests, and this description.
🤖 Generated with Claude Code