[Core] Preempt hybrid requests whose sync KV load failed

[ivanium]

Purpose

Follow-up to #50388, which stopped EngineCore from crashing on a KV load failure with a hybrid KV cache by recomputing the affected request in full. For a request in the WAITING_FOR_REMOTE_KVS state that works: its blocks are freed once the transfer finishes and it is re-admitted. For a running request (a synchronous load under kv_load_failure_policy="recompute") the recompute is done in place, rewinding num_computed_tokens to 0 while the request keeps its blocks and stays in running. That is not a valid restart:

• With overlapping batches (async scheduling, pipeline parallelism) the next step was scheduled before the failure was reported, so its allocate_slots already ran remove_skipped_blocks on the loaded prefix: the sliding-window and Mamba groups now point at the shared null block for the positions they no longer needed. The recompute from 0 writes those positions' KV into the null block and the attention right after them reads it back.
• The in-flight step's output for that request was computed on the failed load, but nothing marks it stale, so it is delivered as a real token.
• Any per-request worker state derived at admission stays as it was. The DeepSeek-V4.1 SWA bounded replay in [Feat][Model] Support encoder-side SWA-bounded replay for DeepSeek-V4.1-Flash #56227 has such state (the replay boundary), which is what surfaced this.

This PR preempts such requests instead of rewinding them in place. _preempt_request already does everything a from-scratch restart needs: it frees the blocks, moves the request to the head of the waiting queue, reports it in preempted_req_ids so the worker drops its state, and with drop_stale_output=True drops the in-flight output. Before preempting, the request's blocks are evicted from the prefix cache; a sync load is cached at allocation, so the retry would otherwise hit the garbage. Single-group models keep the block-precise in-place rewind, and the async path is unchanged.

Retry semantics: the re-admitted request goes through normal admission, so the connector is asked for a match again (as it is today for the request-level _handle_failed_recving path). A connector that keeps offering a load it cannot deliver would retry each admission.

Not a duplicate: #56135, #50742, #45497, #48216 and #54733 address the crash that #50388 fixed or work out a precise per-group truncation point; #53298 keeps the in-place rewind and repairs Model Runner V2 state around it for single-group models. None of them changes how a running hybrid request is restarted after #50388.

Test Plan

pytest tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py tests/v1/core/test_scheduler.py tests/v1/core/test_async_scheduler.py

The new test_hybrid_sync_load_failure_preempts_request covers a running hybrid request whose sync load fails: it is preempted with no output appended, its blocks are released and no longer cached, and the next schedule() re-admits it from scratch with preempted_req_ids set.

Test Result

207 passed. Pre-commit (ruff, mypy) clean. Not run end to end against a KV connector that injects load failures; the unit test exercises the scheduler path only.

🤖 Developed with Claude Code; all changes reviewed and tested by the author.