[Feature] Abort in-flight requests and drain outputs on shutdown#36964
[Feature] Abort in-flight requests and drain outputs on shutdown#36964wojciech-wais wants to merge 1 commit intovllm-project:mainfrom
Conversation
When the engine core receives SIGTERM/SIGINT, abort all in-flight requests via the scheduler and send abort outputs to clients before exiting. Then signal the output thread to drain all pending messages (including abort responses) by pushing ENGINE_CORE_DEAD sentinel and joining the thread with a 5s timeout. Previously, the engine core would exit immediately on signal without notifying clients that their requests were aborted. This left clients hanging with no response. Now clients receive proper abort responses before the process terminates. Addresses follow-up points from njhill on PR vllm-project#36666: 1. Abort requests when shutdown timeout expires 2. Wait for output thread to drain before exiting Signed-off-by: Wojciech Wais <wojciech.wais@gmail.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a graceful shutdown mechanism for the engine core. When a SIGTERM or SIGINT is received, it now aborts in-flight requests, sends abort notifications to clients, and waits for the output queue to drain before exiting. This is a valuable improvement that prevents clients from being left in a hanging state. The implementation is clean and includes corresponding unit tests. My main feedback is to make the shutdown timeout configurable to handle various operational environments, which I've detailed in a specific comment.
| # Signal the output thread to exit and wait for it to flush | ||
| # all pending messages (including the abort outputs above). | ||
| self.output_queue.put_nowait(EngineCoreProc.ENGINE_CORE_DEAD) | ||
| self.output_thread.join(timeout=5.0) |
There was a problem hiding this comment.
The 5-second timeout for draining the output thread is hardcoded. In scenarios with high load or a slow network, this might not be enough time to send all pending messages, including the abort responses. This could result in some clients not receiving an abort notification, which is the problem this PR aims to solve. This same hardcoded value is also used in _send_engine_dead.
To improve robustness and maintainability, this timeout should be extracted into a constant. A future improvement could be to make this value configurable, for example, via an environment variable.
1 participant
When the engine core receives SIGTERM/SIGINT, abort all in-flight requests via the scheduler and send abort outputs to clients before exiting. Then signal the output thread to drain all pending messages (including abort responses) by pushing ENGINE_CORE_DEAD sentinel and joining the thread with a 5s timeout.
Previously, the engine core would exit immediately on signal without notifying clients that their requests were aborted. This left clients hanging with no response. Now clients receive proper abort responses before the process terminates.
Addresses follow-up points from njhill on PR #36666:
Purpose
Test Plan
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.