[Bugfix] Escalate diffusion worker shutdown and retain survivors

[hsliuustc0106]

Purpose

Fixes #7124. Part of #7107.

After one diffusion rank dies, another worker can ignore SIGTERM and remain alive after executor shutdown. The cleaner previously stopped after a shared graceful-exit deadline and a shared SIGTERM deadline, then the executor discarded its process references.

Add a bounded SIGKILL/join phase, continue peer cleanup after signal/join OS errors, and report any final survivors with their names and PIDs. Retain survivors for another explicit executor shutdown attempt. A nonblocking cleaner lock prevents the worker monitor and an explicit shutdown retry from performing process operations concurrently. Ordinary result-pump and pending-future cleanup remains in finally.

The issue contains the motivation, goals, non-goals, use cases, and detailed design. This PR does not change HWR behavior or provide automatic service restart. A process still alive after the final deadline is logged and retained; no unconditional GPU-memory recovery guarantee is made.

Test Plan

Real CPU children reproduce the failure: one child is already dead, another acknowledges that it ignores SIGTERM, and the shutdown queue raises an error. The regression fails on main because the second child remains alive; with the fix it exits via SIGKILL and is reaped. The cooperative case exits normally. Focused tests cover shared deadlines, signal/join errors, survivors and repeat shutdown, and concurrent cleanup.

CUDA_VISIBLE_DEVICES='' VLLM_TARGET_DEVICE=cpu \
  /tmp/codex-pr6607-vllm028/bin/python -m pytest -n 0 -q \
  tests/diffusion/test_multiproc_engine_concurrency.py \
  tests/diffusion/test_result_pump.py

vLLM Version: 0.28.0; Python 3.12.13; torch 2.13.0+cu129; transformers 5.14.1.

vLLM-Omni Commit: based on 06541d1f523dc25f13c4258732dcd0ed4cb248ff.

Test Result

• CPU suites: 97 passed. Warnings include existing dependency/development-version warnings and Python's warning about forking from a multithreaded test process.
• Base regression: 1 failed, specifically assert not worker.is_alive(); task-owned children were cleaned in finally and the implementation restored afterward.
• Local pre-commit: all applicable hooks passed except mypy-3.10, which reports the same 10 diagnostics on base and this branch (6 production, 4 tests; no new diagnostics). Hook pins/configuration were preserved in an external config except that markdownlint used installed Node v22.23.1 after the automatic LTS download stalled. No repository hook changes or applicable hook skips.
• Full precheck completed: no new correctness/scope blockers; existing mypy debt and CPU-only validation disclosed. git diff --check passed.
• No GPU workload was run. These tests establish process-cleanup behavior, not full-model serving recovery or device-memory correctness.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/diffusion/continuous_batching.md.

Module owners: @Isotr0py @princepride

@hsliuustc0106, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit fd071ae08139 produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.