Hybrid KV offload: planner, MultiConnector, and mamba alignment for hybrid models#38261

Open

malaiwah wants to merge 35 commits into

vllm-project:mainfrom

malaiwah:codex/hybrid-kv-offload

Commits on Mar 28, 2026

Add hybrid offload planning scaffolding

Michel Belleau
authored and
malaiwah
committed
Add partial GPU block transfer metadata

Michel Belleau
authored and
malaiwah
committed
Add per-group offload hashing

Michel Belleau
authored and
malaiwah
committed
Add per-group hybrid offload accounting

Michel Belleau
authored and
malaiwah
committed
Relax hybrid planner hash-size assumptions

Michel Belleau
authored and
malaiwah
committed
Emit hybrid partial GPU transfer specs

Michel Belleau
authored and
malaiwah
committed
Validate hybrid chunk sizing inputs

Michel Belleau
authored and
malaiwah
committed
Log offloading worker job completion

Michel Belleau
authored and
malaiwah
committed
Tolerate compatible offload metadata on preemption

Michel Belleau
authored and
malaiwah
committed
Allow hybrid planner fallback for unsplittable groups

Michel Belleau
authored and
malaiwah
committed
Offset hybrid hash/store chunks for late full blocks

Michel Belleau
authored and
malaiwah
committed
Load first hybrid chunk from group start

Michel Belleau
authored and
malaiwah
committed
Add hybrid offload validation and handle_preemptions compatibility

authored and
malaiwah
committed
Add SupportsHMA to LMCacheConnectorV1 for hybrid model support

authored and
malaiwah
committed
Fix hybrid offload multi-hash block ordering bug

M. Belleau
authored and
malaiwah
committed
Add MultiConnector HMA support, load backpressure, and metrics safety

M. Belleau
authored and
malaiwah
committed
Add weighted load selection to MultiConnector

M. Belleau
authored and
malaiwah
committed
Fix MultiConnector handle_preemptions for stock vLLM signature

M. Belleau
authored and
malaiwah
committed
Add per-connector Prometheus metrics to MultiConnector

M. Belleau
authored and
malaiwah
committed
Add per-connector Prometheus metrics to MultiConnector via stats pipeline

M. Belleau
authored and
malaiwah
committed
Cache computed planner properties in HybridOffloadPlanner.__post_init__

M. Belleau
authored and
malaiwah
committed
Materialise block hashes once in update_state_after_alloc

authored and
malaiwah
committed
Cache HybridChunkBlockHashList per request in OffloadingConnectorScheduler

authored and
malaiwah
committed
Cache combined chunk hashes in HybridChunkBlockHashList

authored and
malaiwah
committed
Validate hybrid block/unit divisibility once at init

authored and
malaiwah
committed
Add test for HybridChunkBlockHashList chunk hash caching

authored and
malaiwah
committed
Remove unverified-external-KV assertion in mamba block alignment

authored and
malaiwah
committed
offloading: remove dead _ensure_transfer_supported, downgrade hot-path logs to DEBUG

M. Belleau
authored and
malaiwah
committed
Fix MultiConnector deferred resolution, load fallback, and mamba alignment

malaiwah
and
claude
committed
Handle disagreeing KV group prefix lengths gracefully

malaiwah
and
claude
committed
Fix crash on group-disagree: zero external tokens when falling back

malaiwah
and
claude
committed
Add load timeout for external KV cache transfers

malaiwah
and
claude
committed
Fix pre-commit failures: typos, line length, mypy type errors

malaiwah
and
claude
committed

Commits on Mar 29, 2026