[Frontend][Core] Re-add shutdown timeout - allowing in-flight requests to finish

[markmc]

Re-apply #34730 and #36270 which were reverted by #36628 because of test_external_lb_dp.py failures in Distributed Tests (4 GPUs)

---

Relates to #24885 and supersedes a subset of #32420

🤖 Co-authored with Claude Code
Co-authored-by: @njhill

Overview

This PR adds support for a shutdown timeout in the API server and engine core, allowing controlled handling of a SIGTERM signal - either immediate termination by default (--shutdown-timeout=0) or wait for in-flight requests to finish (--shutdown-timeout=SECONDS).

Background

Previously, SIGTERM to the API server or multi-API server launcher would cause immediate process termination without coordinating request handling. This PR addresses that by:

1. Rejecting new requests when shutdown is initiated (not queuing them like pause does)
2. Handling in-flight requests according to --shutdown-timeout:
   • Default, no timeout - abort all in-flight requests immediately
   • With timeout - Wait for in-flight requests to complete (up to --shutdown-timeout seconds)
3. Using a separate shutdown state machine in the engine core to manage the shutdown lifecycle

Scope and Constraints

This PR is intentionally scoped to the API server use case with SIGTERM handling. The following are explicitly out of scope:

• KV transfers: unlike [Frontend] Enable drain shutdown mode for non-DP deployments #32420, prefill instances are not (yet) waiting for KV transfers to complete before shutting down
• Library use case: No changes to direct LLM or AsyncLLM usage (though shutdown() methods are added for future use)
• ScalingMiddleware reuse: Request rejection happens at engine core level, not middleware
• Signal handling redesign: No new shutdown pipe mechanism; minimal changes to existing handlers
• Ctrl-C behavior: Process groups and Ctrl-C handling remain unchanged; only SIGTERM is handled
• /health endpoint changes: No new /live endpoint or changes to health check behavior
• External/hybrid load balancer validation: Assumes parallel SIGTERM to multiple API servers works (untested, similar to pause behavior)

Implementation Approach

Core Mechanism: EngineShutdownState State Machine

The implementation uses a separate EngineShutdownState enum to manage shutdown lifecycle independently from pause state:

class EngineShutdownState(IntEnum):
    RUNNING = 0
    REQUESTED = 1
    SHUTTING_DOWN = 2

Why a separate state machine:

• Shutdown has different semantics than pause (permanent vs temporary)
• Shutdown needs request rejection, pause needs request queuing
• Different lifecycle: pause can be resumed, shutdown cannot
• Cleaner separation of concerns

How it works:

1. SIGTERM signal handler sets engine_core.shutdown_state = EngineShutdownState.REQUESTED
2. Engine core's _handle_shutdown() method checks state each iteration:
   • RUNNING: Continue normal operation
   • REQUESTED: Abort/wait based on config, transition to SHUTTING_DOWN, reject new requests
   • SHUTTING_DOWN: Exit when no work remaining
3. Engine core's _handle_client_request() calls _reject_add_in_shutdown() and _reject_utility_in_shutdown() which check shutdown state and reject new work
4. Engine core's run_busy_loop() exits when _handle_shutdown() returns False

Idle Engine Wake-up

When the engine is idle and blocked on input_queue.get(), it cannot detect shutdown state changes. To deal with this, we:

1. Add WAKEUP sentinel to EngineCoreRequestType for waking idle engine
2. Add a new signal-context-safe SignalCallback to allow triggering a callback from a signal handler
3. Add a signal callback that pushes WAKEUP to input_queue
4. Trigger the signal callback from the signal handler
5. Main loop wakes from blocking get(), checks shutdown_state, proceeds

Shutdown Flow

Single API Server:

SIGTERM → signal_handler() sets shutdown_event
       → _handle_shutdown() coroutine wakes up
       → Retrieves timeout from vllm_config.shutdown_config.wait_timeout
       → Calls engine_client.shutdown(timeout=timeout)
       → Shutdown propagates through client hierarchy:
          - AsyncLLM.shutdown() → EngineCoreClient.shutdown()
          - MPClient detaches finalizer, calls engine_manager.shutdown()
          - CoreEngineProcManager.shutdown() terminates/waits for processes
       → Then cancels server tasks and stops SSL refresher

Multi-API Server Launcher:

SIGTERM → signal_handler() sets shutdown_requested flag, raises SystemExit
       → finally block executes
       → Computes shared deadline from configured wait_timeout
       → Calls shutdown(deadline=deadline) on all managers:
          - api_server_manager.shutdown(deadline=deadline)
          - local_engine_manager.shutdown(deadline=deadline) if present
          - coordinator.shutdown(deadline=deadline) if present
       → Each manager terminates/waits for processes within deadline

Headless Mode:

SIGTERM → signal_handler() sets shutdown_requested flag, raises SystemExit
       → finally block executes
       → Retrieves timeout from vllm_config.shutdown_config.wait_timeout
       → Calls engine_manager.shutdown(timeout=timeout)
       → Process manager terminates/waits for processes with timeout

[markmc]

I expect Distributed Tests (4 GPUs) to fail - triggering tests just to re-establish that as a baseline

[gemini-code-assist]

Code Review

This pull request introduces a shutdown timeout feature to vLLM, allowing for a grace period to complete in-flight requests before shutting down. The changes include adding a shutdown_timeout parameter to the VllmConfig and EngineArgs, modifying the signal handlers to initiate a graceful shutdown, and updating the process management to respect the timeout. The close() methods were renamed to shutdown().

[njhill]

@markmc I wanted to note a couple of follow-on things I'd noticed, probably not directly related to the 4-GPU test breakages but thought I would record here anyhow:

• I realized that in the timeout case, we don't abort requests when the timeout has expired. It would be nice if we did that consistently whether it's 0 or some larger number, and in all cases allow additional small time for those abort responses to propagate
• Related, I'm not sure even in the existing abort case, whether we probably allow for those abort responses to propagate before things are shut down. I need to look more closely but in particular was wondering whether we wait until the output processing thread has finished processing work in the output queue before continuing to exit the engine core proc.

Don't necessarily need to address these as part of the re-added PR of course.

[wojciech-wais]

@markmc @njhill
I'd like to help with this relanding effort. I was the co-author on the original #34730 and have been working on the related shutdown improvements in the RFC #24885 space (#36258).

On the 4-GPU test failure (#36624):

From the bisection in #36624, the failure is a race condition in run_engine_core() _ with 4 API servers per DP rank, EngineCore_DP1's worker hits BrokenPipeError because the parent exits before
initialization completes. I suspect the SignalCallback thread + shutdown_state enum replacement of the simple signal handler changes process lifecycle timing enough to expose this race under heavier
initialization load. The [1] variant (single API server) passes because initialization is lighter.

A few things I could investigate:

1. Narrow the culprit in run_engine_core() _ the original PR changed both the signal handler mechanism (SignalCallback thread) and added os.setpgrp(). We already confirmed that os.setpgrp()
   alone is risky - can orphan engine cores if the parent crashes. I can test whether keeping the simpler signal handler while still adding the shutdown state machine avoids the race.

2. Address @njhill's follow-on points _ abort requests when timeout expires + allow abort responses to propagate before engine core exits. I can look into whether the output processing thread drains
   its queue before run_busy_loop() exits.

3. Our /live probe PR ([Feature] Add /live liveness probe, LLM.shutdown(), and k8s shutdown docs #36258) depends on this landing _ it adds app.state.draining = True in the signal handler and a /live endpoint that stays 200 during drain. Happy to rebase it on top of this
   once the test failure is resolved.

I don't have a 4-GPU setup to reproduce test_external_lb_dp[4] locally, but I can do code analysis and prepare fixes for review.
How do you think about addressing the (1) and (2)?

[njhill]

Thanks @wojciech-wais for looking at this... I will defer to @markmc because I don't know of his plan or whether he's already started on this, but would be great if we can confirm the issue and incorporate a clean fix in this PR. I am interested and will try to have a closer look today too.

The follow-on points could also be fixed here or handled in a follow-on PR...

[njhill]

@markmc fyi I separately found/fixed this issue #36950, thought there is a chance it could be implicated in the problem here. I didn't test that fix with this PR yet though.

[markmc]

@markmc fyi I separately found/fixed this issue #36950, thought there is a chance it could be implicated in the problem here.

Thanks, let's find out!

[mergify]

Hi @markmc, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[wojciech-wais]

@njhill
Following up on your two points I've opened #36964 which addresses both of them:

1. Abort requests on shutdown
   When the engine core receives SIGTERM/SIGINT, all in-flight requests are now explicitly aborted via scheduler.finish_requests(None, FINISHED_ABORTED) and abort outputs
   are sent to clients (reusing the existing _send_abort_outputs() from the pause path).

2. Drain output thread before exit
   After sending abort outputs, the ENGINE_CORE_DEAD sentinel is pushed and the output thread is joined with a 5s timeout, ensuring all pending messages (including
   abort responses) are flushed through ZMQ before the process terminates.

This is a standalone PR against main. It will work independently of the shutdown timeout relanding here, but will compose cleanly with it. Once #36666 will be merged, the same mechanism will apply when the timeout
expires (abort + drain).

[markmc]

@markmc fyi I separately found/fixed this issue #36950, thought there is a chance it could be implicated in the problem here.

Thanks, let's find out!

Distributed Tests (4 GPUs) is indeed passing now

I'd prefer merge this now as is, so we can get back to the baseline of what was previously working and merged - I'll note Nick's suggested improvements in #24885