Skip to content

[Observability] log sub process killing in process manager force kill - #52314

Merged
DarkLight1337 merged 6 commits into
vllm-project:mainfrom
andyxning:detailed_log_in_processes_shutdown
Sep 30, 2026
Merged

DarkLight1337 merged 6 commits into
vllm-project:mainfrom
andyxning:detailed_log_in_processes_shutdown

Conversation

@andyxning

@andyxning andyxning commented Aug 14, 2026 •

Copy link
Copy Markdown
Member

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

This is a follow up of #52299.

Add sub process killing log in process manager force kill by kill_process_tree. For now, the kill process tree logic is not logged.

Test Plan

  1. setting export VLLM_ENGINE_READY_TIMEOUT_S=120
  2. run vllm serve Qwen/Qwen3.6-27B --max-num-seqs 512 --renderer-num-workers 1 --tensor-parallel-size 2 --pipeline_parallel_size 2 --data-parallel-size 1

Test Result

a detailed log will be recorded about sub process force kill.

RuntimeError: Frontend process failed during engine core initialization. See root cause above. Failed frontend proc(s): {'ApiServer_1': None}
DEBUG 08-14 08:25:07 [v1/utils.py:605] [shutdown] Process manager: start process_count=2 timeout=5.0s names=ApiServer_0,ApiServer_1
INFO 08-14 08:25:07 [v1/utils.py:615] [shutdown] Process manager: send sigterm to process ApiServer_0
DEBUG 08-14 08:25:08 [v1/utils.py:647] [shutdown] Process manager: complete
DEBUG 08-14 08:25:08 [v1/utils.py:605] [shutdown] Process manager: start process_count=1 timeout=5.0s names=EngineCore
INFO 08-14 08:25:08 [v1/utils.py:615] [shutdown] Process manager: send sigterm to process EngineCore
(EngineCore pid=2441) INFO 08-14 08:25:08 [v1/engine/core.py:1317] [shutdown] EngineCore: trigger received signal=SIGTERM
WARNING 08-14 08:25:13 [v1/utils.py:635] [shutdown] Process manager: force killing remaining processes count=1
WARNING 08-14 08:25:13 [v1/utils.py:640] [shutdown] Process manager: force killing remaining process EngineCore pid 2441
WARNING 08-14 09:47:25 [utils/system_utils.py:274] [shutdown] force killing sub process VLLM::Worker_PP0_TP0 pid 6909
WARNING 08-14 09:47:25 [utils/system_utils.py:274] [shutdown] force killing sub process VLLM::Worker_PP0_TP1 pid 6910
WARNING 08-14 09:47:25 [utils/system_utils.py:274] [shutdown] force killing sub process VLLM::Worker_PP1_TP0 pid 6911
WARNING 08-14 09:47:25 [utils/system_utils.py:274] [shutdown] force killing sub process VLLM::Worker_PP1_TP1 pid 6912

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)

Signed-off-by: Andy Xie <andy.xning@gmail.com>
@andyxning
andyxning marked this pull request as ready for review August 27, 2026 06:17

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@DarkLight1337
DarkLight1337 enabled auto-merge (squash) September 30, 2026 04:27
@DarkLight1337

Copy link
Copy Markdown
Member

/ci run

@github-actions github-actions Bot added the ready ONLY add when PR is ready to merge/full CI is needed label Sep 30, 2026
@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #92038 for commit 287fa57e52d2.

@DarkLight1337
DarkLight1337 merged commit 29773c7 into vllm-project:main Sep 30, 2026
138 of 139 checks passed
@andyxning
andyxning deleted the detailed_log_in_processes_shutdown branch September 30, 2026 06:27
liulanze added a commit to liulanze/vllm that referenced this pull request Oct 9, 2026
vllm-project#52314 started logging each child's name() before SIGKILLing it, inside
suppress(ProcessLookupError). psutil raises NoSuchProcess, not
ProcessLookupError, when the child was reaped after children() listed
it. That race is likely on the force-kill path, where the target is
still tearing down and reaping its own children. The exception escapes
kill_process_tree, so the remaining children and the target itself are
never killed, and callers such as shutdown() abort their loop over the
remaining processes.

Suppress NoSuchProcess as well: a child that is already gone needs no
SIGKILL, and skipping it avoids signalling a recycled pid.

Signed-off-by: Lanze Liu <lanzetech@gmail.com>
liulanze added a commit to liulanze/vllm that referenced this pull request Oct 12, 2026
Per review, keep this PR to the vllm-project#52314 regression: move children() back
out of the try and drop the target-exit test.

In the remaining test, end the child that exits mid-walk with SIGTERM
instead of SIGKILL and assert it, so it is clearly told apart from the
processes that kill_process_tree SIGKILLs.

Signed-off-by: Lanze Liu <lanzetech@gmail.com>
michael-gpu pushed a commit to practical-inference/vllm-ampere-extended that referenced this pull request Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants