[faresobeid]

Router replay now works with inference.kv_cache_offload on vLLM 0.31, both native (OffloadingConnector) and Mooncake (MooncakeStoreConnector), with or without P/D. Per-role P/D offload (decode-side Mooncake with save_decode_cache) is in #3832; decode writes are not clamped, and prefill loads of decode-saved blocks go through the same clamp.

Problem on 0.31. vLLM 0.31 keeps routing rows in a local store keyed by block hash. No connector moves rows with the KV, so a block filled by an offload load has no rows on this instance. A request that needs them fails the engine step with BlockObjectStoreError. Upstream allows offload connectors and fails closed.

1. Load clamp. External loads stop at block_floor(routed_experts_prompt_start). vLLM never returns rows before that position, so the request never reads the loaded blocks' rows.

• Native: wrap OffloadingConnector._max_loadable_tokens.
• Mooncake: wrap MooncakeStoreConnector.get_num_new_matched_tokens and shrink load_specs[rid].kvpool_cached_tokens (and clear tail_key_boundaries), so update_state_after_alloc's assert holds.
• Requests resumed after preemption (num_output_tokens > 0) and decode-side P/D requests are not limited: they emit rows only from their first forward, which always recomputes that token, so they keep their offload hits.

Why clamp instead of restoring rows. Restoring rows with offloaded blocks would need each connector to carry an extra per-block object, and the row store lives in the worker (TP rank 0) while load decisions are made in the scheduler. Neither connector has a hook for that. The clamp is a scheduler-side bound and keeps the hits that matter (multi-turn reloads).

2. Loaded-block cut (still needed on 0.31). Loaded blocks enter the GPU prefix cache. A later request with a lower routed_experts_prompt_start that hits them locally (for example a first turn hitting a prefix reloaded by another rollout's later turn) needs their rows and would crash the engine. The scheduler marks blocks covered by num_external_computed_tokens (unmarked when reallocated), and _get_local_prefix_cache_hit cuts such hits at the first loaded block the request needs rows from, then recomputes. Requests with output and decode-side P/D requests are skipped. It also covers non-P/D requests on a decode instance that hit NIXL-loaded blocks.

Hits kept: multi-turn reloads (verifiers sets routed_experts_prompt_start = path_len - 1). Hits lost: offloaded prefixes past it (first turns, shared prompts), which are recomputed.

validate_router_replay_without_kv_offload is deleted; docs updated. vLLM 0.31 allows offload connectors with routed-experts capture, and the P/D routed-experts bypass on main covers NIXL inside MultiConnector.

Checks run

• pytest -m "not gpu" tests/unit/orchestrator tests/unit/test_configs.py tests/unit/inference on this head: 316 passed. Ruff: clean.
• Tests: the parametrized load-limit test, and the loaded-block tracking on vLLM's real KVCacheManager.

GPU validation (2026-10-06, vLLM 0.31.0, 1x 8xH200, Qwen3-30B-A3B TP1, BI, native offload 40 GB CPU, 2000 GPU blocks; compared with a no-offload reference)

check	result
A = P+200 with start=len(P) after P was evicted (external load)	3/3 rows identical; external hits 2992/4096/2592 = block_floor(start)
B = P+50 with start=0 right after A (local hit on A's loaded blocks)	3/3 identical; hit cut (0 local, 0 external)
C = P+70 with start=0 after eviction	3/3 identical (0 external)
negative control: same head without the cut	engine dies at the first B: BlockObjectStoreError ("auxiliary output object does not exist")
native offload, 6 two-turn conversations, turn 2 vs reference	6/6 identical (tokens, rows, start, masks)
Mooncake (RDMA) on prefill + NIXL 1P1D, same conversations	6/6 identical; prefill external hits 16512; no errors
decode-saved KV (#3832: decode Mooncake cpu.num_bytes = 0 + save_decode_cache), same conversations, re-run on this stack (job 2900)	6/6 identical (tokens, 264 rows, start, masks); prefill external hits 16880 = sum block_floor(start), vs 16512 (the prompt-only maximum, the row above) → 23 completion blocks loaded from decode-saved KV; no errors

Tokens were identical everywhere.

---

Note

Medium Risk
Changes vLLM scheduler and KV offload connector behavior for routed-expert capture; incorrect bounds could cause BlockObjectStoreError or silent prefix-cache misses, though unit tests cover the core logic.

Overview
Router replay can now run together with inference.kv_cache_offload (native and Mooncake). The RL config no longer rejects that combination, and the inference docs describe the new behavior instead of calling the mix unsupported.

Because offloaded KV blocks do not carry routed-expert rows in vLLM’s local store, prime-rl adds two vLLM scheduler patches: clamp external loads so they stop at the block before routed_experts_prompt_start (with exceptions for P/D decode and resumed requests), and cut local prefix-cache hits at externally loaded blocks when a request would need missing routing rows. Multi-turn reloads keep safe offload hits; shared or first-turn prefixes may be recomputed.

Unit tests cover _routed_experts_load_limit, loaded-block tracking, and prefix-hit cutting on vLLM’s KVCacheManager.

^{Reviewed by Cursor Bugbot for commit b7ef693. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1dc6dc3. Configure here.}