[fangyuchu]

Purpose

Add a resume_requests_after_recovery option to FaultToleranceConfig (--fault-tolerance-config='{"resume_requests_after_recovery": true}', default False) that lets in-flight requests continue generating after the engine recovers from a fault, instead of being aborted — with no duplicated or lost tokens visible to the client.

Today, when the busy loop raises under the fault-tolerance framework, EngineCoreSentinel.on_fault() aborts every running request (finish_requests(..., FINISHED_ABORTED) + _send_abort_outputs), and the client receives an abort notification — even though the engine is about to recover via retry and could have continued serving them.

With resume_requests_after_recovery: true:

Engine side (EngineCoreSentinel.on_fault):

1. Contaminated KV eviction (_clear_contaminated_blocks): the faulted step may have partially written KV for in-flight tokens. The blocks covering [num_computed_tokens - num_in_flight_tokens, num_computed_tokens - 1] of each running request are evicted from the prefix cache, so the post-recovery re-prefill never reuses KV written by the failed step.
2. Preempt instead of abort: all running requests are moved back to the waiting queue via the scheduler's existing _preempt_request (frees blocks, resets num_computed_tokens = 0). num_in_flight_tokens / num_stale_output_tokens are then zeroed because the faulted step's output frames will never return — there is nothing left to drain. prev_step_scheduled_req_ids is cleared.
3. After retry completes and the busy loop restarts, the scheduler re-schedules the preempted requests automatically. The re-prefill covers the prompt plus already-generated tokens and hits the retained prefix cache, so recovery is cheap; decoding then continues and only new tokens are streamed. The client observes a stall, not a failure — no abort, no duplicated or lost tokens.

Worker side (WorkerSentinel._clean_worker_state):

• Model runner V2: unchanged — all per-request state is dropped via _remove_request, and resumed requests are rebuilt from scratch by add_requests after recovery. No resume-specific handling needed.
• MRV1: the persistent batch is still cleared, but CachedRequestState (model_runner.requests, num_prompt_logprobs) is retained, because the resumed-request path (scheduled_cached_reqs.resumed_req_ids) indexes model_runner.requests[req_id] directly.

Default is False — existing behavior (abort on fault) is unchanged unless explicitly opted in.

Test Plan

E2E (tests/v1/fault_tolerance/test_fault_tolerance_e2e.py, requires 2 GPUs + nixl_ep):

DP_SIZE=2 .venv/bin/python -m pytest tests/v1/fault_tolerance/test_fault_tolerance_e2e.py -v -s

test_injected_fault_retry_recovers_all_ranks is extended to cover the resume path: a long streaming completion (max_tokens = fault_step + 50) runs on rank 0 with resume_requests_after_recovery: true, while a fault is injected into the DP-sync allreduce (sync_cudagraph_and_dp_padding) on rank 1 at fault_step. The test asserts that:

• at fault time the stream stalls — still alive, no exception;
• after retry on both engines, the stream resumes and completes with exactly max_tokens chunks (no loss, no duplication) and finish_reason == "length".

Manual E2E (external LB mode, DP+EP, --enable-fault-tolerance, resume_requests_after_recovery: true):

1. Launch the server; a client sends a long streaming generation.
2. Inject a fault right after the DP allreduce while the request is in flight — the stream stalls but receives no abort/error.
3. Send retry via POST /fault_tolerance/apply.
4. After recovery the stalled stream resumes generating and completes normally.

Test Result

• E2E test file above passes on a 2-GPU nixl_ep setup (2 passed).
• Manual E2E: after injecting the post-allreduce fault mid-generation and issuing retry, the in-flight streaming request resumed and generated to completion; the client received no abort notification and no duplicate tokens, and both engines returned to healthy and kept serving.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @fangyuchu.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.