[Bugfix][V1] SimpleCPUOffload: defer sliding-window mid-flight block free (load-WAR alternative to #47291)

[Saddss]

Purpose

Alternative, host-side fix for the SimpleCPUOffload load-path write-after-read
(WAR) race in #47282 — the residual gap left after the store-path fix #46278
(#45704). The worker-side CUDA-event fix is already up in #47291; this PR
implements the block-lifetime deferral approach @hclsys suggested, so the two
can be compared.

Root cause

Under async scheduling the scheduler runs ahead of the device. A running
sliding-window request (chunked-local / R-SWA behave the same) frees KV blocks
mid-flight when its attention window advances:

remove_skipped_blocks -> _remove_blocks_in_range -> block_pool.free_blocks

The scheduler treats the block as free, but the in-flight forward dispatched in
the previous step (processed+1) is still reading it on the compute stream. With
SimpleCPUOffload the freed block is immediately reused as a load destination and
the CPU->GPU copy (separate stream) overwrites it mid-read -> garbled output.

This is the mid-flight free path. #45357's deferral only covers
request-completion frees, so it does not cover this case.

Fix

Extend #45357's fence machinery to the mid-flight skipped-block free:

• block_pool.free_blocks_maybe_deferred() holds skipped blocks under the
  current schedule-step fence instead of returning them to the pool;
  drain_skipped_frees() releases them once processed_step_seq reaches that
  fence (the in-flight reader step has completed).
• single_type_kv_cache_manager._remove_blocks_in_range() frees through
  free_blocks_maybe_deferred().
• scheduler enables it only when defer_block_free is active (async + kv
  consumer), sets the fence at the top of schedule(), drains after
  processed_step_seq advances.

Disabled -> byte-for-byte the old immediate free.

Scope

Only mid-flight-freeing attention types are affected: sliding-window,
chunked-local, R-SWA. Full / sink / cross attention free only at request
completion (already covered by #45357).

Mamba / linear-attention layers are not affected — confirmed empirically
(Falcon-H1-0.5B, both align and all mamba cache modes, same repro; 0 garble
over 1128 valid completions while mid-flight frees fired 1.5k-2.5k times and
external-hit loads ran ~25%). A freed Mamba state snapshot has no in-flight
reader: the recurrent forward reads only the current state block.

Correctness + performance (measured)

Gemma-4-31B-NVFP4 + fp8 KV, VLLM_USE_SIMPLE_KV_OFFLOAD=1,
--kv-offloading-size 100 --max-num-seqs 64 --gpu-memory-utilization 0.95
(GPU KV ~30.8k tokens = 1.88x concurrency, i.e. heavy offload). 300 prompts x
3 rounds x concurrency 64 = 900 completions/variant, max_tokens 256,
temperature 0.7. #45357 active in all variants. All three variants overlaid on
the same vllm/vllm-openai:nightly (files verified byte-identical to the fork
baseline), so only the changed files differ.

variant	load guard	hard garble	garble rate	wall clock (900 req)
NOFIX	none	44 / 900	4.89%	728 s
EVENT (#47291)	compute-done event	0 / 900	0.00%	718 s
DEFER (this PR)	host-side defer	0 / 900	0.00%	704 s

Steady-state engine metrics (mean over 72x 10s samples):

variant	gen tok/s	KV util	waiting	external hit	preemptions
NOFIX	292.7	83.0%	49.5	26.4%	0
EVENT	294.5	80.5%	48.8	27.5%	0
DEFER	299.7	84.6%	49.1	27.8%	0

Throughput / KV util / queue depth / external-hit / preemptions all line up;
wall clock within 3% noise (defer marginally fastest). DEFER instrumentation:
863,525 blocks deferred over the run, but each is held ~1 step only (max
deferred-queue depth 30, drained every processed step), so the in-flight held
set is tiny and swamped by offload cost itself. Free-pool idle blocks dipped to
106 at the tightest point without triggering preemption — a tighter
memory / higher-concurrency setup could make defer bite, but this run did not
measure it.

Relationship to #47291 (event fix)

Same race, two places to cut it:

• Event ([Bugfix][V1] SimpleCPUOffload: order CPU->GPU loads after compute (WAR fix) #47291): load copy waits on a compute-done CUDA event on the
  worker — one spot, covers both load-WAR and store-RAW.
• Defer (this PR): hold the freed block host-side until the reader step is
  done — covers load-WAR; store-RAW still relies on the existing store
  event (the DEFER variant above keeps it; dropping it would re-break the
  store path, which is a pure cross-stream RAW that host-side lifetime
  management cannot reach).

Defer is a correct, equally-fast load-side alternative, but it is a partial
fix (still needs the event for store) and more intrusive (per-manager, SWA-only
today; chunked-local / R-SWA / any future mid-flight-freeing manager must be
wired individually), whereas the event is one path-agnostic spot in
worker.get_finished.

Test Plan

• Unit: tests/v1/core/test_block_pool_defer.py (fence/drain semantics, 3 passed).
• E2E: the 3-variant garble + perf comparison above; ruff + pre-commit clean.

Draft — opening to compare defer vs the event fix (#47291) before either lands.