[Perf][KV Offload] Avoid quadratic ARC batch eviction

[mindungil]

Purpose

ARC batch eviction repeatedly scans its internal cache lists from the beginning for each block selected. As a result, evicting many blocks can require quadratic work.

Fix

Keep monotonic iterators over those lists while collecting candidates, so each entry is visited at most once. Cache mutations remain deferred until all requested candidates are found, preserving atomic eviction.

This also preserves eviction order and protected/pinned block handling.

Performance

Manager-level microbenchmark with a full ARC cache, replacing half of its blocks (median of 3 runs, AMD EPYC 9355, Python 3.12.3):

Cache / evicted blocks	Before	After	Speedup
1,000 / 500	7.162 ms	0.435 ms	16.5x
5,000 / 2,500	179.178 ms	2.205 ms	81.3x
10,000 / 5,000	696.385 ms	4.904 ms	142.0x

This measures CPU eviction bookkeeping, not end-to-end serving throughput.

Related work

ARC was introduced in #27039, and #37874 later made eviction atomic by collecting all candidates before modifying cache state. This PR preserves that behavior while avoiding repeated scans during batch eviction.

#45757 skips eviction when failure can be determined in advance; this PR optimizes successful ARC eviction after that check passes.

No open PR was found for the same repeated-scan issue. #50422 changes the eviction interface but not ARC candidate selection.

Test Plan

.venv/bin/python -m pytest -q tests/v1/kv_offload/cpu/test_manager.py
.venv/bin/python -m pytest -q tests/v1/kv_offload/
.venv/bin/pre-commit run --files \
  vllm/v1/kv_offload/cpu/policies/arc.py \
  tests/v1/kv_offload/cpu/test_manager.py

Test Result

tests/v1/kv_offload/cpu/test_manager.py: 27 passed
tests/v1/kv_offload/: 492 passed, 1 skipped
pre-commit: all applicable hooks passed
mypy (Python 3.10-3.13): passed

The regression tests verify single-pass scanning, repeated T2 fallback after the T1 iterator is exhausted while its virtual size remains above the target, and atomic failure.

Model evaluation: N/A. This changes CPU cache bookkeeping only and does not affect model output.

AI assistance

This PR includes AI-assisted code and analysis from OpenAI Codex. I reviewed the changes and take responsibility for the contribution.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR is described.
• The test plan is provided.
• Test and performance results are included.
• Documentation update is not required because user-facing behavior is unchanged.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[orozery]

Thanks @mindungil !
Can you please add a test to cover for this:

Missing test coverage: the case where virtual_t1_size >= target_t1_size for the entire batch but T1's eligible entries run out mid-way (all remaining are protected/pinned), forcing repeated fallback to T2 with an exhausted T1 iterator. That's the path where the old re-scan and the new exhausted-iterator-returns-None differ most in mechanism.

[mindungil]

Thanks for pointing this out. Added test_batch_eviction_falls_back_after_t1_iterator_exhausted in 5cd9fb5.

The test keeps virtual_t1_size above the target, leaves only one eligible T1 entry, and makes every remaining T1 entry protected or pinned. After selecting that entry, two subsequent selections retry the exhausted T1 iterator and fall back to T2. It also asserts that T1 and T2 are each scanned only once; the previous implementation visits T1/T2 13/3 times for this setup, while the updated implementation visits them 6/2 times.

Validation:

• targeted regression test: passed
• CPU manager tests: 27 passed
• full tests/v1/kv_offload/: 492 passed, 1 skipped
• pre-commit and mypy for Python 3.10-3.13: passed

[orozery]

Thanks @mindungil !