feat(kv-cache): support multiple sliding window groups in HybridKVCac…

[DZADSL72-00558]

…heCoordinator with tests

Changes to be committed:
new file: tests/v1/core/test_hybrid_kv_cache_coordinator.py
modified: vllm/v1/core/kv_cache_coordinator.py

Purpose

dd support for multiple sliding window groups in HybridKVCacheCoordinator.

Previously, the coordinator only supported a single sliding window configuration. This PR extends it to handle multiple sliding window groups.

Test Plan

pytest tests/v1/core/test_hybrid_kv_cache_coordinator.py -v

Test Result

collected 3 items

tests/v1/core/test_hybrid_kv_cache_coordinator.py::TestHybridKVCacheCoordinatorMultipleSlidingWindows::test_verify_and_sort_multiple_sliding_windows PASSED [ 33%]
tests/v1/core/test_hybrid_kv_cache_coordinator.py::TestHybridKVCacheCoordinatorMultipleSlidingWindows::test_cache_hit_multiple_sliding_windows PASSED [ 66%]
tests/v1/core/test_hybrid_kv_cache_coordinator.py::TestHybridKVCacheCoordinatorMultipleSlidingWindows::test_partial_cache_hit_different_sliding_windows PASSED [100%]

=========================================================== 3 passed in 0.58s ============================================================
(b

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request introduces support for multiple sliding window groups in the HybridKVCacheCoordinator, which is a necessary feature for models like Amazon NOVA. The changes are well-implemented, refactoring the existing logic and adding a new path to handle multiple sliding window configurations. The implementation correctly identifies the common cache hit prefix across full attention and all sliding window groups by iterating through them and finding the minimum common hit length. The sliding window groups are sorted by window size, which is a good optimization for the cache hit search. The pull request also includes a comprehensive new test suite that validates the sorting logic and various cache hit scenarios, including partial hits and eviction. The code is clean, and the logic appears to be sound. Overall, this is a solid contribution that extends the functionality of the KV cache coordinator.

[LucasWilkinson]

cc @heheda12345

[heheda12345]

It's not correct to take the min of all sliding window groups. For example, we may have request ABCDE with cache hit (assume block_size=1):
block_table for sw=2: [miss, miss, miss, D, E] -> hit length = 5
block_table for sw=3: [miss, B, C, D, miss] -> hit length = 4
the min length is 4, sw=2 needs the kv of token CD but we don't cache the kv of token C for length 4 of sw=2.

[heheda12345]

One possible solution is to write a new coordinator for the most general case. The main idea is to gradually reduce the length until converged:

def find_longest_cache_hit():
    length = max_cache_hit_length
    while True:
         prev_length = length
         for  manager_cls  in self.different_type_managers:
              length = min(length, manager_cls.find_longest_cache_hit(...))
         if length == prev_length: break

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @DZADSL72-00558.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[DarkLight1337]

Closing as superseded by #31707