Skip to content

[Core] Integrate HWR resilience changes for lifecycle validation - #7164

Closed
baonudesifeizhai wants to merge 10 commits into
vllm-project:mainfrom
baonudesifeizhai:hwr-resilience-integration-20260906
Closed

baonudesifeizhai wants to merge 10 commits into
vllm-project:mainfrom
baonudesifeizhai:hwr-resilience-integration-20260906

Conversation

@baonudesifeizhai

Copy link
Copy Markdown
Contributor

Purpose

Integrate the HWR follow-ups to #7107 into one draft for lifecycle fault-injection validation. Original commit authorship is preserved.

Source PR Change
#7126 Escalate worker shutdown to SIGKILL and retain survivors for retry
#7128 Bound domain-initialization lock waits
#7131 Skip store-size scans when no total limit is configured
#7133 Add explicit quarantine cleanup
#7140 Add opt-in serialized build admission
#7144 Expose full-checksum validation in startup configuration
#7145 Add the retention diagnostic, including the tensor-size follow-up
#7147 Add registration bypass and progress logs

Resolved overlapping configuration/documentation changes and added coverage for combined validation and registration policies.

Defaults remain metadata validation and concurrent admission. Quarantine cleanup is explicit; serialized admission requires a compatible store policy and does not provide strict storage reservations. Registration bypass avoids the risky path; it does not establish or fix a driver root cause. The retention PR is diagnostic only.

Test Plan

vLLM Version: 0.28.0; Python 3.13.12, Torch 2.13.0+cu130, safetensors 0.8.0.

vLLM-Omni Commit: integration HEAD 688446d23af1dda5e1c5d40940b0ccca162252a7, based on ff6e906a25de1ca3122c0d0d5250fde5a0ddbc7b.

With project/test dependencies installed (GPU hidden for this CPU pass):

CUDA_VISIBLE_DEVICES='' PYTHONPATH="$PWD" python -m pytest \
  -o 'addopts=--strict-markers --strict-config' -q \
  tests/host_weight_runtime \
  tests/diffusion/test_multiproc_engine_concurrency.py \
  tests/diffusion/test_result_pump.py \
  tests/diffusion/model_loader/test_diffusers_loader.py \
  tests/config/test_omni_config.py \
  tests/entrypoints/test_async_omni_diffusion_config.py \
  tests/diffusion/offloader/test_distributed_layerwise_backend.py \
  tests/diffusion/offloader/test_cuda_host_registration.py

The addopts override removes the unavailable xdist option from this local environment; strict checks remain enabled.

Test Result

  • 615 passed, 2 skipped. Two additional Helios tests failed with No CUDA GPUs are available; both fail identically on the base commit with GPUs hidden.
  • 144/144 local lifecycle fault cases passed: 12 phase boundaries × 3 repeats × ext4/tmpfs × concurrent/serialized admission. Real non-root processes, BF16 payloads, files/flock/mmap, SIGSTOP/SIGKILL; four concurrent readers and four explicit recovery attempts per case. Verified tensor contents, producer count, active-reader protection and released FD/mmap/locks. These are CPU store tests, not multi-GPU inference or automatic service recovery.
  • Retention diagnostic still shows ~30.67 MiB private-memory growth after one million get_tensor() calls at 64 elements; cached reuse grows ~0.02 MiB in the same environment.
  • Applicable pre-commit hooks pass except mypy: 45 errors versus 51 on the base, with no new diagnostics after normalizing line numbers.

Real multi-GPU rank-failure/resource-release validation on this combined branch remains pending. Keep this PR in draft while completing that lifecycle campaign.

hsliuustc0106 and others added 10 commits September 6, 2026 16:45
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
(cherry picked from commit c79c841)
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
(cherry picked from commit d455411)
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
(cherry picked from commit 00d6984)
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
(cherry picked from commit c6a0c7f)
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
(cherry picked from commit 90815cc)

Signed-off-by: roG0d <baonudesifeizhai@gmail.com>
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
(cherry picked from commit d989aa6)
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
(cherry picked from commit 86166fd)
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
(cherry picked from commit 743953b)
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
(cherry picked from commit 729ccd7)

Signed-off-by: roG0d <baonudesifeizhai@gmail.com>
Signed-off-by: roG0d <baonudesifeizhai@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants