fix(update): terminate stray venv holder processes on unattended Windows update - #67229
fix(update): terminate stray venv holder processes on unattended Windows update#67229Dolverin wants to merge 1 commit into
Conversation
…ows update Unattended updates (bootstrap one-click passes --yes) dead-ended whenever a transient process held the install venv: cron job runs spawn pythonw from venv\Scripts at all hours, the venv-holder guard exited 2, and the retry on the next click simply collided with the next job. The update flow pauses the gateways — and with them the in-gateway cron ticker — before this guard, so no replacements can spawn; terminate the remaining strays (graceful, then force), re-scan, and only refuse when holders persist or reappear (e.g. a supervised desktop backend). Interactive runs keep the plain refusal, now with a hint that --yes clears strays automatically. Also makes two venv-health tests host-aware (they built a POSIX-layout fake venv but never patched _is_windows, so they only passed off-Windows).
teknium1
left a comment
There was a problem hiding this comment.
Thanks for isolating the post-pause venv-holder path. Current main does retain the reported refusal at hermes_cli/main.py:9847-9851.
Problems
hermes_cli/main.py:9916terminates every detector match, but the detector includes the Desktopservebackend (hermes_cli/main.py:9262-9276). Current main deliberately refuses that backend because it is supervised and can respawn (hermes_cli/main.py:9836-9845). The sole immediate re-scan athermes_cli/main.py:9917can be clean before a later respawn, reopening the.pyd-lock race the guard protects against. The new test attests/hermes_cli/test_update_venv_health.py:376-381covers only a holder still present at that immediate scan.- This makes
--yesprocess-destructive, while the unchanged parser help (hermes_cli/subcommands/update.py:47-51) and updating guide (website/docs/getting-started/updating.md:105) describe it as prompt automation and instruct users to close holders or use--force-venv.
Suggested changes
- Preserve refusal for Desktop-supervised/external holders, or add a tested stable-quiescence lifecycle before update mutation.
- Document any approved destructive
--yesbehavior.
Automated hermes-sweeper review.
| # then re-scan. Anything that reappears (a supervised desktop | ||
| # backend) still trips the refusal below. | ||
| print("→ Clearing stray Hermes venv process(es) before the dependency sync…") | ||
| _terminate_venv_python_holders(_venv_holders) |
There was a problem hiding this comment.
_detect_venv_python_processes() includes the Desktop serve backend, while current main explicitly treats killing that supervised process as a race because it can respawn. A single immediate re-scan can be clean before the supervisor restarts it. Exclude supervised/external holders or add a bounded, tested quiescence check before allowing dependency mutation.
|
Closing this PR because the current implementation broadens --yes into terminating every detected venv holder, including the Desktop-supervised serve backend. The immediate re-scan does not prove stable quiescence before mutation, so it can reopen the lock race the existing fail-closed guard prevents. The safer contract remains: refuse supervised/external holders, use the external Desktop handoff, and require the explicit --force-venv override when the operator accepts that risk. Any future automatic-quiescence work should be a separate, lifecycle-proven design. |
What does this PR do?
Fixes a permanent dead-end of unattended Windows updates at the venv-process guard.
Today the update flow pauses gateways and then refuses (exit 2) when any other process runs from the install venv. On a busy install that refusal is effectively unconditional: cron job runs spawn
pythonwfromvenv\Scriptsat all hours, and abandoned CLI sessions linger. The desktop one-click handoff exits cleanly and launches the updater, the updater dies at the guard, and the next retry simply collides with the next job — the update can never converge without manual process hunting.Key observation: once
_pause_windows_gateways_for_update()has run, the in-gateway cron ticker is stopped too, so nothing spawns replacements. The remaining holders are strays that nothing supervises — killing them is safe, and that is exactly what an unattended update should do instead of dead-ending.This PR teaches
hermes updateto converge on unattended runs:--yesis set (the desktop bootstrap updater always passes--yes), terminate them gracefully (psutilterminate(), wait up to 20 s, thenkill()), logging each PID + cmdline, then re-scan.--yes) keep the plain refusal; the message now hints that--yesclears strays automatically.--force-venvsemantics are untouched, and the new code path is Windows-gated (_is_windows()), no-op without psutil.Scope note / related PRs: #61515 reaps gateway-spawned venv children inside the gateway pause (complementary — this PR covers everything the pause legitimately can't: cron job runs, stray CLIs). #62445 kills external holders from the desktop handoff (desktop path only; this PR fixes it in the update flow itself, so plain
hermes updateCLI runs converge too). #63318 surfaces blockers in the UI (orthogonal).Related Issue
No tracking issue; reproduction and a live run are included below.
Type of Change
Changes Made
hermes_cli/main.py_terminate_venv_python_holders()(graceful → force, bounded waits, never raises)--yesruns terminate strays and re-scan before refusing--yeshint;_detect_venv_python_processesdocstring updated to match the new contracttests/hermes_cli/test_update_venv_health.py--yes+ strays cleared → past guard; holders persist → still exit 2; interactive → never terminates; terminate → force-kill only survivors_is_windows, so they could only pass off-Windows)How to Test
venv\Scripts\pythonw.exe -c "import time; time.sleep(600)"hermes update --yes --force --branch main→ Terminating venv holder PID …, the holder is gone, and the update proceeds instead of exiting 2.Unit:
pytest tests/hermes_cli/test_update_venv_health.py -q→ 22 passed.Checklist
Code
fix(scope):,feat(scope):, etc.)pytest tests/hermes_cli/test_update_venv_health.py -q— 22 passed; full suite not run locally)Documentation & Housekeeping
docs/, docstrings) — or N/A — docstrings updated; no user-facing docs affectedcli-config.yaml.exampleif I added/changed config keys — N/ACONTRIBUTING.mdorAGENTS.mdif I changed architecture or workflows — N/AScreenshots / Logs
Live unattended update on an install that had been dead-ended for two days (stray holder planted deliberately):