fix(update): reap orphaned Desktop backends + stop managed runtime child trees (salvages #81327) by teknium1 · Pull Request #82179 · NousResearch/hermes-agent

teknium1 · 2026-08-09T02:38:37Z

What does this PR do?

Fixes the Windows GUI-updater dead-end where an update aborts with "Hermes is still running. Close all Hermes windows and try the update again." even though the user has zero windows open. Salvages #81327 intact underneath (authorship preserved) and adds the missing half on top.

Symptom (real logs, 2026-08-09, two consecutive failed updates)

[update] waiting for Hermes to exit…
✗ Other Hermes processes are running from this install's venv:
  PID 12580  python.exe  ...venv\Scripts\python.exe -m hermes_cli.main serve --host 127.0.0.1 --por  ← Hermes Desktop backend (close the desktop app)
update stage stage=update state=Failed error=Some("Hermes is still running. Close all Hermes windows and try the update again.")

…followed by the fallback bootstrap dying on the same lock:

stage=venv state=Failed error="Cannot remove item ...venv\Scripts\python.exe: Access to the path 'python.exe' is denied."

Root cause

The Desktop→hermes-setup handoff can lose the teardown race: Electron SIGTERMs its backend and quits, but the python.exe -m hermes_cli.main serve backend (and its managed .hermes-runtime interpreter child) survives. At that point the Desktop is gone — nothing will ever respawn that backend — but hermes update's venv-holder guard treats every serve backend as "the Desktop supervises this, killing it is futile" and refuses. That assumption is only true while the Desktop is alive.

Fix

Two commits:

Salvaged from fix(install): stop managed runtime child trees on Windows #81327 (@helix4u, commit intact): scripts/install.ps1's venv sweep stops each verified venv process with its full child tree (taskkill /T /F) instead of Stop-Process on the parent only, so a managed-runtime child can't keep holding the install (force_kill_other_hermes() only kills hermes.exe, misses pythonw.exe backend → venv rebuild fails with "Access denied" on Windows #70026). Includes the behavioral regression test.
New — hermes_cli/update_cmd.py: the venv-holder guard now classifies remaining holders with _orphaned_desktop_backend_pids():
- A serve/dashboard backend whose supervising parent is provably dead (parent PID gone, or recycled — parent created after the child) is an orphaned straggler: reap its tree via the new _stop_process_trees() (taskkill /T /F, mirroring the Desktop's forceKillProcessTree), re-scan, and proceed.
- A backend whose Desktop is still alive, any non-backend holder (operator REPL, stray script), or an unprovable case (no psutil) → refusal preserved exactly as before. --force still doesn't bypass; --force-venv semantics unchanged.

Related Issue

Related to #70026 (managed-runtime child holding the install) and the #39828 family (Desktop handoff race).

Type of Change

🐛 Bug fix (non-breaking change that fixes an issue)

How to Test

pytest tests/hermes_cli/test_update_orphan_backend_reap.py tests/hermes_cli/test_update_venv_health.py tests/hermes_cli/test_scan_venv_blockers.py -o "addopts=--timeout-method=thread" — 41 passed.
On Windows: scripts/run_tests.sh tests/test_install_ps1_venv_process_tree.py (from fix(install): stop managed runtime child trees on Windows #81327).

E2E verification (real Windows 11 box)

Spawned a detached, orphaned python with a backend-shaped argv (... -m hermes_cli.main serve --host 127.0.0.1): classifier returned [pid], _stop_process_trees killed it — verified dead via psutil.
Same setup with a non-backend argv: classifier returned None (refusal preserved).
The live Desktop backend on the box (parent alive): classifier returned None (refusal preserved).

Checklist

Read the Contributing Guide; conventional commits; no unrelated changes
Tests added (unit + guard integration + live-parent/orphan/PID-reuse edge cases)
Tested on my platform (Windows 11, real process trees)
Cross-platform: guard is Windows-only by existing _is_windows() gating; no POSIX behavior change
Contributor credit: fix(install): stop managed runtime child trees on Windows #81327's commit cherry-picked/rebased intact (author: Gille / @helix4u, noreply email — no AUTHOR_MAP entry needed)

…e venv-holder guard The GUI-updater handoff race: the Desktop fires SIGTERM + app.quit() and spawns hermes-setup, but its Python backend (`python.exe -m hermes_cli.main serve`) can survive the teardown. The Desktop is gone -- nothing will respawn that backend -- yet the venv-holder guard refused on it and the update dead-ended with "Hermes is still running" while the user had zero windows open (observed twice on 2026-08-09, 01:59 and 02:17, bootstrap-installer.log). New `_orphaned_desktop_backend_pids()` classifies remaining holders: a serve/dashboard backend whose supervising parent is provably dead (PID gone, or recycled -- parent created after the child) is a straggler safe to reap. Any live-parent backend, non-backend holder, or unprovable case keeps the refusal exactly as before. Reaping uses the new `_stop_process_trees()` (taskkill /T /F), mirroring the Desktop's forceKillProcessTree and install.ps1's venv sweep so the managed .hermes-runtime interpreter child dies with its launcher (#70026). Builds on #81327 (salvaged intact underneath): that fixed the same parent-only-kill gap in install.ps1's venv sweep; this closes the remaining dead-end in the `hermes update` guard itself. E2E on a real Windows box: spawned a detached orphan with a backend-shaped argv -> classifier returned its PID and the tree reap killed it; a non-backend orphan and the live Desktop backend (parent alive) both returned None (refusal preserved).

github-actions · 2026-08-09T02:39:51Z

૮ >ﻌ< ა ci review

_{ran on bfc2a55}

⚠️ Warnings

OSV vulnerability scan · View job

21 known vulnerabilities found in pinned dependencies.

CVE-2026-59873 in package-lock.json
CVE-2026-24842 in package-lock.json
CVE-2025-71329 in website/package-lock.json
CVE-2026-26960 in package-lock.json
CVE-2026-23745 in package-lock.json
CVE-2026-59874 in package-lock.json
CVE-2026-70608 in package-lock.json
CVE-2026-31802 in package-lock.json
CVE-2026-69247 in uv.lock
CVE-2026-59875 in package-lock.json
CVE-2026-69249 in uv.lock
CVE-2026-69248 in uv.lock
CVE-2026-29786 in package-lock.json
GHSA-r292-9mhp-454m in package-lock.json
CVE-2026-23950 in package-lock.json
CVE-2026-53655 in package-lock.json
CVE-2025-71330 in website/package-lock.json
CVE-2026-59871 in package-lock.json
CVE-2026-69247 in uv.lock
CVE-2026-69249 in uv.lock

How to fix:

Review the findings in the Security tab. Update the affected dependencies if a patched version is available.

debug info

CI timings

CI timings · View report · View job

Wall time 6m1s vs 6m36s (-8.8%). 9 job(s) slower, 14 faster, 1 unchanged.

Python tests / Run tests slice 5/12: +40.0s
Python tests / Run tests slice 7/12: +31.0s
Python tests / Run tests slice 2/12: -26.0s
Python tests / Run tests slice 8/12: -22.0s
Python tests / Run tests slice 11/12: -20.0s

helix4u · 2026-08-09T02:46:29Z

Thanks for consolidating this and preserving the #81327 commit/authorship. The installer-side tree kill and the orphan recovery both make sense.

I think we should also bring over the Desktop teardown portion of #77436 before merging this. releaseBackendLock() still sends SIGTERM to the primary backend before forceKillProcessTree(). If that launcher exits before taskkill /T runs, Windows can no longer enumerate its descendants. This PR repairs one aftermath of that race, but the owning Electron path can still create the orphan in the first place.

Could we also add coverage for a holder set containing both an orphaned serve root and one of its managed-runtime descendants? _orphaned_desktop_backend_pids() currently refuses the whole set when any holder has a live parent. If the scanner returns both processes, the descendant still has the orphaned backend as its live parent, so the fallback may refuse even though the complete tree is safe to reap.

My preference would be:

port the backend-child.ts, main.ts, and behavioral TS test portion of fix(desktop): drain Windows update process trees #77436 onto current main;
keep this CLI orphan reaper as downstream defense;
make the orphan classification tree-aware, or at least cover the mixed root/descendant shape;
leave fix(desktop): drain Windows update process trees #77436’s older scanner changes out, since that area has moved through fix(update): venv-blocker scan truncated cmdlines, breaking the gateway exemption and dead-ending Desktop updates #82158.

With that, this should cleanly supersede both #81327 and the relevant part of #77436.

teknium1 · 2026-08-09T02:53:43Z

Good catch — all three points addressed in #82191 (raised as a follow-up since this merged while your comment landed):

Desktop teardown from fix(desktop): drain Windows update process trees #77436 ported (stopBackendTreesForUpdate() — tree-kill the live root first, no SIGTERM pre-signal; @4adwentures credited as co-author). Its scanner half left out per your note, since fix(update): venv-blocker scan truncated cmdlines, breaking the gateway exemption and dead-ending Desktop updates #82158 moved that code.
Orphan classification is now tree-aware: holders inside an accepted orphan root's tree fold into the root (only roots returned; taskkill /T reaps descendants). Your mixed root/descendant case is covered by tests — and it turned out even more real than the review suggested: in live testing the uv-managed interpreter worker re-execs with the same serve argv and a live parent (the orphan root), so the pre-fix classifier refused a genuinely safe set. The mocks missed it; the real process tree didn't.
Coverage added for root+managed-runtime child, grandchild depth, stray-outside-tree (still refuses), and mid-classify exits.

E2E on a real Windows box: mixed orphan tree (root + 3 descendants) classified to [root] and fully reaped; the live Desktop backend still refuses.

@4adwentures

…te trees without pre-signalling Follow-up to #82179 addressing helix4u's review comment (#82179 issuecomment-5229441571). Three parts: 1. Desktop teardown (salvaged from #77436, @4adwentures): the update hand-off's releaseBackendLock() sent SIGTERM to the primary backend BEFORE taskkill /T. If the launcher exits first, Windows can no longer enumerate its descendants and they survive holding the venv — the Electron path that creates the orphan #82179 then has to repair. New stopBackendTreesForUpdate() tree-kills the live root first, with the behavioral vitest from #77436. The scanner half of #77436 is deliberately NOT taken (superseded by #82158's full-cmdline scan). 2. Tree-aware orphan classification: _orphaned_desktop_backend_pids() previously refused the whole holder set when any holder had a live parent. But the scanner legitimately returns an orphaned serve root AND its descendants (the venv trampoline's uv-managed interpreter worker — which carries the same backend argv — plus .hermes-runtime children). Those have a live parent: the orphan root itself. Now holders inside an accepted orphan root's tree fold into that root (only roots are returned; taskkill /T reaps descendants), and live-parent backends defer to the ancestry check instead of refusing outright. Anything outside an orphan tree still refuses. 3. Tests for the mixed shapes: root+managed-runtime child, grandchild depth, non-descendant stray alongside an orphan root (still refuses), descendant exited mid-classify. E2E on a real Windows box: spawned a detached backend-shaped orphan that itself spawned children (3 python descendants); the scanner-shaped mixed holder set classified to [root], taskkill /T reaped root and all descendants. The live Desktop backend on the box still classified None (refusal preserved). The first E2E attempt caught exactly the trampoline/worker case the mocks missed — the live worker re-execs with the same backend argv and a live parent — which is what part 2 fixes. Co-Authored-By: 4adwentures <296413879+4adwentures@users.noreply.github.com>

@4adwentures

…te trees without pre-signalling Follow-up to NousResearch#82179 addressing helix4u's review comment (NousResearch#82179 issuecomment-5229441571). Three parts: 1. Desktop teardown (salvaged from NousResearch#77436, @4adwentures): the update hand-off's releaseBackendLock() sent SIGTERM to the primary backend BEFORE taskkill /T. If the launcher exits first, Windows can no longer enumerate its descendants and they survive holding the venv — the Electron path that creates the orphan NousResearch#82179 then has to repair. New stopBackendTreesForUpdate() tree-kills the live root first, with the behavioral vitest from NousResearch#77436. The scanner half of NousResearch#77436 is deliberately NOT taken (superseded by NousResearch#82158's full-cmdline scan). 2. Tree-aware orphan classification: _orphaned_desktop_backend_pids() previously refused the whole holder set when any holder had a live parent. But the scanner legitimately returns an orphaned serve root AND its descendants (the venv trampoline's uv-managed interpreter worker — which carries the same backend argv — plus .hermes-runtime children). Those have a live parent: the orphan root itself. Now holders inside an accepted orphan root's tree fold into that root (only roots are returned; taskkill /T reaps descendants), and live-parent backends defer to the ancestry check instead of refusing outright. Anything outside an orphan tree still refuses. 3. Tests for the mixed shapes: root+managed-runtime child, grandchild depth, non-descendant stray alongside an orphan root (still refuses), descendant exited mid-classify. E2E on a real Windows box: spawned a detached backend-shaped orphan that itself spawned children (3 python descendants); the scanner-shaped mixed holder set classified to [root], taskkill /T reaped root and all descendants. The live Desktop backend on the box still classified None (refusal preserved). The first E2E attempt caught exactly the trampoline/worker case the mocks missed — the live worker re-execs with the same backend argv and a live parent — which is what part 2 fixes. Co-Authored-By: 4adwentures <296413879+4adwentures@users.noreply.github.com>

@4adwentures

…te trees without pre-signalling Follow-up to NousResearch#82179 addressing helix4u's review comment (NousResearch#82179 issuecomment-5229441571). Three parts: 1. Desktop teardown (salvaged from NousResearch#77436, @4adwentures): the update hand-off's releaseBackendLock() sent SIGTERM to the primary backend BEFORE taskkill /T. If the launcher exits first, Windows can no longer enumerate its descendants and they survive holding the venv — the Electron path that creates the orphan NousResearch#82179 then has to repair. New stopBackendTreesForUpdate() tree-kills the live root first, with the behavioral vitest from NousResearch#77436. The scanner half of NousResearch#77436 is deliberately NOT taken (superseded by NousResearch#82158's full-cmdline scan). 2. Tree-aware orphan classification: _orphaned_desktop_backend_pids() previously refused the whole holder set when any holder had a live parent. But the scanner legitimately returns an orphaned serve root AND its descendants (the venv trampoline's uv-managed interpreter worker — which carries the same backend argv — plus .hermes-runtime children). Those have a live parent: the orphan root itself. Now holders inside an accepted orphan root's tree fold into that root (only roots are returned; taskkill /T reaps descendants), and live-parent backends defer to the ancestry check instead of refusing outright. Anything outside an orphan tree still refuses. 3. Tests for the mixed shapes: root+managed-runtime child, grandchild depth, non-descendant stray alongside an orphan root (still refuses), descendant exited mid-classify. E2E on a real Windows box: spawned a detached backend-shaped orphan that itself spawned children (3 python descendants); the scanner-shaped mixed holder set classified to [root], taskkill /T reaped root and all descendants. The live Desktop backend on the box still classified None (refusal preserved). The first E2E attempt caught exactly the trampoline/worker case the mocks missed — the live worker re-execs with the same backend argv and a live parent — which is what part 2 fixes. Co-Authored-By: 4adwentures <296413879+4adwentures@users.noreply.github.com>

@4adwentures

…te trees without pre-signalling Follow-up to NousResearch#82179 addressing helix4u's review comment (NousResearch#82179 issuecomment-5229441571). Three parts: 1. Desktop teardown (salvaged from NousResearch#77436, @4adwentures): the update hand-off's releaseBackendLock() sent SIGTERM to the primary backend BEFORE taskkill /T. If the launcher exits first, Windows can no longer enumerate its descendants and they survive holding the venv — the Electron path that creates the orphan NousResearch#82179 then has to repair. New stopBackendTreesForUpdate() tree-kills the live root first, with the behavioral vitest from NousResearch#77436. The scanner half of NousResearch#77436 is deliberately NOT taken (superseded by NousResearch#82158's full-cmdline scan). 2. Tree-aware orphan classification: _orphaned_desktop_backend_pids() previously refused the whole holder set when any holder had a live parent. But the scanner legitimately returns an orphaned serve root AND its descendants (the venv trampoline's uv-managed interpreter worker — which carries the same backend argv — plus .hermes-runtime children). Those have a live parent: the orphan root itself. Now holders inside an accepted orphan root's tree fold into that root (only roots are returned; taskkill /T reaps descendants), and live-parent backends defer to the ancestry check instead of refusing outright. Anything outside an orphan tree still refuses. 3. Tests for the mixed shapes: root+managed-runtime child, grandchild depth, non-descendant stray alongside an orphan root (still refuses), descendant exited mid-classify. E2E on a real Windows box: spawned a detached backend-shaped orphan that itself spawned children (3 python descendants); the scanner-shaped mixed holder set classified to [root], taskkill /T reaped root and all descendants. The live Desktop backend on the box still classified None (refusal preserved). The first E2E attempt caught exactly the trampoline/worker case the mocks missed — the live worker re-execs with the same backend argv and a live parent — which is what part 2 fixes. Co-Authored-By: 4adwentures <296413879+4adwentures@users.noreply.github.com>

@4adwentures

…te trees without pre-signalling Follow-up to NousResearch#82179 addressing helix4u's review comment (NousResearch#82179 issuecomment-5229441571). Three parts: 1. Desktop teardown (salvaged from NousResearch#77436, @4adwentures): the update hand-off's releaseBackendLock() sent SIGTERM to the primary backend BEFORE taskkill /T. If the launcher exits first, Windows can no longer enumerate its descendants and they survive holding the venv — the Electron path that creates the orphan NousResearch#82179 then has to repair. New stopBackendTreesForUpdate() tree-kills the live root first, with the behavioral vitest from NousResearch#77436. The scanner half of NousResearch#77436 is deliberately NOT taken (superseded by NousResearch#82158's full-cmdline scan). 2. Tree-aware orphan classification: _orphaned_desktop_backend_pids() previously refused the whole holder set when any holder had a live parent. But the scanner legitimately returns an orphaned serve root AND its descendants (the venv trampoline's uv-managed interpreter worker — which carries the same backend argv — plus .hermes-runtime children). Those have a live parent: the orphan root itself. Now holders inside an accepted orphan root's tree fold into that root (only roots are returned; taskkill /T reaps descendants), and live-parent backends defer to the ancestry check instead of refusing outright. Anything outside an orphan tree still refuses. 3. Tests for the mixed shapes: root+managed-runtime child, grandchild depth, non-descendant stray alongside an orphan root (still refuses), descendant exited mid-classify. E2E on a real Windows box: spawned a detached backend-shaped orphan that itself spawned children (3 python descendants); the scanner-shaped mixed holder set classified to [root], taskkill /T reaped root and all descendants. The live Desktop backend on the box still classified None (refusal preserved). The first E2E attempt caught exactly the trampoline/worker case the mocks missed — the live worker re-execs with the same backend argv and a live parent — which is what part 2 fixes. Co-Authored-By: 4adwentures <296413879+4adwentures@users.noreply.github.com>

@4adwentures

…te trees without pre-signalling Follow-up to NousResearch#82179 addressing helix4u's review comment (NousResearch#82179 issuecomment-5229441571). Three parts: 1. Desktop teardown (salvaged from NousResearch#77436, @4adwentures): the update hand-off's releaseBackendLock() sent SIGTERM to the primary backend BEFORE taskkill /T. If the launcher exits first, Windows can no longer enumerate its descendants and they survive holding the venv — the Electron path that creates the orphan NousResearch#82179 then has to repair. New stopBackendTreesForUpdate() tree-kills the live root first, with the behavioral vitest from NousResearch#77436. The scanner half of NousResearch#77436 is deliberately NOT taken (superseded by NousResearch#82158's full-cmdline scan). 2. Tree-aware orphan classification: _orphaned_desktop_backend_pids() previously refused the whole holder set when any holder had a live parent. But the scanner legitimately returns an orphaned serve root AND its descendants (the venv trampoline's uv-managed interpreter worker — which carries the same backend argv — plus .hermes-runtime children). Those have a live parent: the orphan root itself. Now holders inside an accepted orphan root's tree fold into that root (only roots are returned; taskkill /T reaps descendants), and live-parent backends defer to the ancestry check instead of refusing outright. Anything outside an orphan tree still refuses. 3. Tests for the mixed shapes: root+managed-runtime child, grandchild depth, non-descendant stray alongside an orphan root (still refuses), descendant exited mid-classify. E2E on a real Windows box: spawned a detached backend-shaped orphan that itself spawned children (3 python descendants); the scanner-shaped mixed holder set classified to [root], taskkill /T reaped root and all descendants. The live Desktop backend on the box still classified None (refusal preserved). The first E2E attempt caught exactly the trampoline/worker case the mocks missed — the live worker re-execs with the same backend argv and a live parent — which is what part 2 fixes. Co-Authored-By: 4adwentures <296413879+4adwentures@users.noreply.github.com>

helix4u and others added 2 commits August 8, 2026 19:29

fix(install): stop managed runtime child trees on Windows

2300541

teknium1 mentioned this pull request Aug 9, 2026

fix(install): stop managed runtime child trees on Windows #81327

Closed

19 tasks

teknium1 merged commit 826bf9b into main Aug 9, 2026
46 checks passed

teknium1 deleted the fix/windows-update-backend-reap branch August 9, 2026 02:46

teknium1 mentioned this pull request Aug 9, 2026

fix(update): tree-aware orphan reap + drain Desktop update trees without pre-signalling (follow-up to #82179) #82191

Merged

5 tasks

This was referenced Aug 9, 2026

fix(update): refresh installer bootstrap-cache scripts on every update (stale hermes-setup mitigation) #82229

Merged

feat(desktop): repo-owned Windows update hand-off script — stop depending on the frozen hermes-setup binary #82328

Merged

Navigation Menu

Uh oh!

fix(update): reap orphaned Desktop backends + stop managed runtime child trees (salvages #81327) - #82179

fix(update): reap orphaned Desktop backends + stop managed runtime child trees (salvages #81327)#82179
teknium1 merged 2 commits into
mainfrom
fix/windows-update-backend-reap

teknium1 commented Aug 9, 2026

Uh oh!

github-actions Bot commented Aug 9, 2026 •

edited

Loading

CI timings · View report · View job

Uh oh!

helix4u commented Aug 9, 2026

Uh oh!

Uh oh!

teknium1 commented Aug 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

teknium1 commented Aug 9, 2026

What does this PR do?

Symptom (real logs, 2026-08-09, two consecutive failed updates)

Root cause

Fix

Related Issue

Type of Change

How to Test

E2E verification (real Windows 11 box)

Checklist

Uh oh!

github-actions Bot commented Aug 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

૮ >ﻌ< ა ci review

⚠️ Warnings

OSV vulnerability scan · View job

debug info

CI timings · View report · View job

Uh oh!

helix4u commented Aug 9, 2026

Uh oh!

Uh oh!

teknium1 commented Aug 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Aug 9, 2026 •

edited

Loading