[Bugfix][KV Offload] Size CPU cache slots for one KV group

[zupengwang]

Fixes #56871.

An offload key identifies one KV cache group, but CPU slots can reserve the combined footprint of all groups. For eight equally sized, disjoint groups, a budget that should hold 16 group keys holds only two.

This change carries each selected group's byte size into the offloading config and sizes the shared slot pool for the largest group. Worker views start at the beginning of each slot for each group, preserving GPU strides, aliases, unpadded payload sizes, and partial chunks. Packed GPU layouts use group-specific transfer regions instead of copying the whole multi-group block. The layout applies to CPU offloading and the CPU primary tier of tiered offloading.

Slots remain fixed-size for secondary-tier transfers. Unequal groups can still leave unused space inside a slot. A storage-format identifier including row size separates compact rows from previously persisted rows; existing files remain on disk but are not reused. Single-group and external backends retain their existing layout by default.

Duplicate-work check

The issue is claimed at #56871 (comment). Issue discussion and open/closed PR searches were rechecked on 2026-09-15. Related work covers group eligibility (#54743), node-local sizing (#53474), capacity metrics (#53902, #56867), and worker interfaces (#56076); it does not implement this per-group slot layout.

Validation

Base: cd10ed6f9f6b37a8ace9cf380007e66fe12ec0c3, after the v0.29.0 release. Linux, Python 3.12.13, PyTorch 2.13.0+cu130, RTX 3090.

• Capacity reproducer: all four cases fail on the unmodified base and pass with the fix. Eight equal groups hold 16 keys instead of two; unequal groups and blocks_per_chunk=1,3 are covered.
• Configuration, factory, and shared-region suites: 142 passed.
• CPU manager/policies, tier factories, tiered offloading, real filesystem I/O, mocked object storage, file identity, and connector scheduler suites: 422 passed.
• GPU worker, canonical layout, and connector registration suites: 67 passed, 1 skipped (ROCm-only test). Includes packed and aliased GPU pages, unequal groups, an empty group in a transfer, sparse chunk tails, and slot reuse across groups.
• Qwen3.5-0.8B, four cache groups, four prompts: cold generation followed by successful GPU prefix-cache reset and CPU reload. Both base and candidate load 32,907,264 bytes. Generated token IDs and their logprobs match exactly between base and candidate; cold/reloaded token IDs also match in each run. Each reloaded prompt reports 1,088 cached tokens. Cold/reloaded logprob differences also occur identically on the base.
• All applicable pre-commit hooks pass, including mypy for Python 3.10; the manual Python 3.12 mypy hook also passes.

The GPU tests used reused native extensions with the checked-out Python sources. DeepSeek-V4.1 TP=8 production validation and throughput measurement remain outside this evidence.

The model comparison used the same standalone model_roundtrip.py harness from each checkout, invoked with PYTHONPATH="$PWD" RUN_NAME=baseline or RUN_NAME=candidate, using .venv/bin/python /ch_data/wzp/oss-ai-infra/vllm-offload-groups-20260915-evidence/model_roundtrip.py. The harness and JSON results are included in the review materials.

Commands

.venv/bin/python -m pytest -q \
  tests/v1/kv_connector/unit/offloading_connector/test_config.py \
  tests/v1/kv_offload/test_factory.py \
  tests/v1/kv_offload/cpu/test_shared_offload_region.py

.venv/bin/python -m pytest -q \
  tests/v1/kv_offload/cpu/test_manager.py \
  tests/v1/kv_offload/cpu/policies \
  tests/v1/kv_offload/tiering/test_factory.py \
  tests/v1/kv_offload/tiering/test_tiering_offloading.py \
  tests/v1/kv_offload/tiering/test_fs_tier.py \
  tests/v1/kv_offload/tiering/test_obj_tier.py \
  tests/v1/kv_offload/test_file_mapper.py \
  tests/v1/kv_connector/unit/offloading_connector/test_scheduler.py

.venv/bin/python -m pytest -q \
  tests/v1/kv_offload/cpu/test_gpu_worker.py \
  tests/v1/kv_offload/cpu/test_canonical_layout.py \
  tests/v1/kv_connector/unit/offloading_connector/test_worker.py

.venv/bin/python -m pre_commit run --files $(git diff --name-only HEAD^ HEAD)
.venv/bin/python -m pre_commit run mypy-3.12 --hook-stage manual \
  --files $(git diff --name-only HEAD^ HEAD)

AI assistance: Codex assisted with implementation, testing, and this description.

[mergify]

Documentation preview: https://vllm--56953.org.readthedocs.build/en/56953/

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[zupengwang]

@ApostaC @orozery could you review the shared-slot layout and persistent-cache compatibility in this PR? It fixes #56871 by sizing each CPU slot for the largest selected KV group and placing each group's data at the slot origin, while retaining fixed-size rows for secondary tiers.

Remote validation at 8480dace6: 631 tests passed, 1 ROCm-only test skipped; Qwen3.5-0.8B generation tokens and logprobs match the base after GPU prefix-cache reset and CPU reload on an RTX 3090 (reused native extensions). The DeepSeek-V4.1 TP=8 production setup remains untested. Thanks!