Skip to content

fix(update): reap gateway venv children so the Windows venv guard self-clears - #61515

Closed
lEWFkRAD wants to merge 2 commits into
NousResearch:mainfrom
lEWFkRAD:fix/windows-update-venv-child-reap
Closed

fix(update): reap gateway venv children so the Windows venv guard self-clears#61515
lEWFkRAD wants to merge 2 commits into
NousResearch:mainfrom
lEWFkRAD:fix/windows-update-venv-child-reap

Conversation

@lEWFkRAD

@lEWFkRAD lEWFkRAD commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes the Windows hermes update permanent dead-end at the venv-process guard.

_pause_windows_gateways_for_update() stops gateway PIDs before the venv guard runs, but on Windows a stopped gateway does not reap the helpers it spawned from the install venv's python (stdio MCP servers, the perfmon probe). Those orphans keep native .pyd files mapped, so _detect_venv_python_processes() refuses the update — and since nothing ever cleans the orphans up, every retry fails identically. The guard can never self-clear.

This PR makes the pause step responsible for the gateway's own venv children:

  1. Snapshot before stop_snapshot_gateway_venv_children() walks each running gateway's descendant tree before the gateway is stopped (once the parent exits, the children's ppid dangles and a later walk misses them), keeping only processes that hold this install's venv (same three signals the guard has always used, now factored into a shared predicate _process_holds_install_venv()).
  2. Reap after stop_terminate_gateway_venv_children() force-terminates the captured helpers (terminate_pid(force=True)taskkill /T /F) and waits (bounded) for them to exit, so the guard that runs next sees a clean slate. The helpers respawn when the gateway restarts via the existing resume path — no state is lost.
  3. Desktop backend excluded — the Electron-supervised hermes_cli.main serve backend is deliberately never reaped (the app respawns it within seconds; the guard's "close the desktop app" refusal remains the right answer there).
  4. Stale marker sweep_sweep_stale_update_marker() in the update preflight removes a HERMES_HOME\.hermes-update-in-progress marker whose owning PID is dead (left behind when a wedged hermes-setup.exe is killed). A live-owner marker — including our own parent setup process when the desktop drives the update — is never touched.

Everything is Windows-guarded (_is_windows()), best-effort, and never raises; POSIX behavior is unchanged. If a reap fails (access denied, already gone), the venv guard still runs and still refuses — the guard remains the backstop, this just clears the self-inflicted case.

Related Issue

Fixes #61514

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

  • hermes_cli/main.py
    • New _venv_lock_prefixes() / _process_holds_install_venv() — venv-holder predicate extracted from _detect_venv_python_processes() so the guard and the reaper classify holders identically (pure refactor of the guard; behavior unchanged, existing tests pass).
    • New _snapshot_gateway_venv_children() / _terminate_gateway_venv_children() — capture-before-stop + reap-after-stop, wired into _pause_windows_gateways_for_update().
    • New _process_is_desktop_backend() — safety exclusion for the supervised desktop backend.
    • New _pid_is_alive() / _sweep_stale_update_marker() — dead-owner marker sweep, wired into _cmd_update_impl() preflight (mirrors the staleness self-heal the Electron launch gate already does in apps/desktop/electron/update-marker.ts).
  • tests/hermes_cli/test_update_venv_child_reap.py — 20 new tests: predicate (venv exe / trampoline / unrelated / empty), snapshot (captures MCP+perfmon, excludes desktop backend + unrelated python, dedups across gateways, dead-gateway and no-psutil safety, off-Windows no-op), reap (force-kills each child, best-effort on per-PID failure, off-Windows/empty no-ops), _pid_is_alive (psutil path + conservative no-psutil default), marker sweep (dead-owner removed, live-owner preserved, malformed removed, absent no-op, POSIX untouched).

How to Test

  1. pytest tests/hermes_cli/test_update_venv_child_reap.py tests/hermes_cli/test_update_venv_health.py -q — new suite plus the existing guard suite (the refactored _detect_venv_python_processes behavior).
  2. Repro (Windows): with a gateway running stdio MCP servers off the install venv, run hermes update. Before this PR: after "Stopping Windows gateway process(es)…" the venv guard fires on the orphaned MCP/perfmon children and exits 2 on every retry. After: the pause step prints → Reaped N venv helper process(es) the gateway left running (MCP servers, perfmon) and the update proceeds; the helpers are back after the gateway resumes.
  3. Stale marker: create %LOCALAPPDATA%\hermes\.hermes-update-in-progress containing a dead PID + timestamp, run hermes update — it prints → Cleared a stale update-in-progress marker … and continues. With a live PID in the marker it is left untouched.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass — targeted update suites; 4 pre-existing failures reproduce identically on an unmodified checkout on this host (2 × test_update_venv_health.py build a POSIX venv/bin/python without patching _is_windows, so they fail on any real Windows host; 2 × test_cmd_update.py fire the real venv guard when actual venv processes are running on the test machine)
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: Windows 11 Pro (10.0.26200)

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — docstrings on all new helpers; N/A otherwise
  • I've updated cli-config.yaml.example if I added/changed config keys — N/A (no config changes)
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — all new behavior is _is_windows()-gated; POSIX paths are no-ops with explicit tests
  • I've updated tool descriptions/schemas if I changed tool behavior — N/A

Screenshots / Logs

Guard output from the wedged state this fixes (2026-07-09, live install):

✗ Other Hermes processes are running from this install's venv:
  PID 5040  python.exe  ...venv\Scripts\python.exe ...client_lookup_mcp.py
  PID 7956  python.exe  ...venv\Scripts\python.exe ...client_lookup_mcp.py
  ...
  On Windows these keep native extension files (.pyd) locked, so the
  dependency update would fail partway and leave a broken install.

The listed PIDs are children the just-paused gateway left behind — closing every Hermes window does not clear them, so the update could never proceed without manual taskkill or --force-venv.

🤖 Generated with Claude Code

…f-clears

On Windows, `hermes update` pauses the gateway before its venv-process guard,
but pausing a gateway does not reap the helper processes it spawned from the
install venv's python — stdio MCP servers and the perfmon probe. Those children
survive as orphans holding native `.pyd` files mapped, so the venv-process guard
(`_detect_venv_python_processes`) sees them and refuses the update indefinitely.
Every blind retry hits the same wall; a killed `hermes-setup.exe` also leaves a
stale `.hermes-update-in-progress` marker behind (observed July 2026 with
`client_lookup_mcp` / `hermes-perfmon`).

Fix (all Windows-guarded; POSIX behavior unchanged):

- `_pause_windows_gateways_for_update` snapshots each gateway's venv-resident
  descendants BEFORE stopping it (a dead parent's dangling ppid defeats an
  after-the-fact walk), then reaps them once the gateways are down. They are
  re-spawned when the gateway restarts (resume path), so no state is lost. The
  supervised desktop backend (`serve`/`dashboard`) is a child of the Electron
  app, not a gateway, and is explicitly excluded from the reap.
- `_cmd_update_impl` sweeps a stale `.hermes-update-in-progress` marker whose
  owning PID is dead before the guard runs, so the CLI recovery path also clears
  the desktop's launch gate. A live-owner marker (a genuine concurrent update,
  incl. our own parent setup) is never touched.
- Extract the venv-holder predicate (`_venv_lock_prefixes` /
  `_process_holds_install_venv`) so the guard and the reaper classify holders
  identically.

Adds tests for the predicate, snapshot (incl. desktop-backend exclusion, dedup,
dead-gateway/no-psutil safety), reap (force-kill each, best-effort on error),
`_pid_is_alive`, and the stale-marker sweep (dead/live/malformed/absent/POSIX).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@alt-glitch alt-glitch added type/bug Something isn't working comp/cli CLI entry point, hermes_cli/, setup wizard platform/windows Native Windows-specific behavior or breakage P2 Medium — degraded but workaround exists sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades labels Jul 9, 2026

@tonydwb tonydwb left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Summary

Verdict: Comment

Overview

  • 2 files changed, +581/-30 lines
  • Windows venv update: reap gateway child processes so the guard self-cleans
  • Uses psutil for cross-platform process tree traversal

Looks Good

  • Clean fix for a real Windows issue
  • psutil is already a dependency in this project
  • Proper process cleanup on Windows

Note

  • +581 lines includes substantial process-reaping logic — worth verifying on a Windows machine

Reviewed by Hermes Agent

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing a real update-recovery gap. Current main pauses gateway PIDs but has no descendant cleanup (hermes_cli/main.py:9083-9110), then immediately applies the venv-holder guard (hermes_cli/main.py:9465-9489).

Problems

  • hermes_cli/main.py:9079 in this PR identifies a desktop backend with "serve" in cmdline_low or "dashboard" in cmdline_low. That also matches ordinary helper commands such as the documented stdio MCP command mcp-server-time (skills/autonomous-ai-agents/hermes-agent/references/native-mcp.md:38). Such a gateway child is excluded at PR line 9147, so it can remain locked and reproduce the guard failure.

Suggested changes

  • Match the actual hermes_cli.main serve / dashboard invocation with token-aware parsing, not arbitrary substrings.
  • Add a snapshot regression test for mcp-server-time or server.py being captured, alongside the existing exact desktop-backend exclusion test (tests/hermes_cli/test_update_venv_child_reap.py:121-137).

Automated hermes-sweeper review.

Comment thread hermes_cli/main.py Outdated
flow (the venv guard refuses and tells the user to close the app instead).
Used as a safety exclusion when reaping gateway children.
"""
return "serve" in cmdline_low or "dashboard" in cmdline_low

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This substring check excludes any helper whose command merely contains serve, including the documented stdio MCP command mcp-server-time. Since the snapshot then skips it, that child can retain the venv lock and defeat the purpose of this fix. Please match the tokenized hermes_cli.main serve / dashboard invocation instead, and add a mcp-server-time regression case.

@teknium1 teknium1 added the sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform label Jul 11, 2026
@lEWFkRAD

Copy link
Copy Markdown
Contributor Author

Addressed in 9489f4f. Desktop backend detection now tokenizes the command line and requires the exact hermes_cli.main serve or hermes_cli.main dashboard module/subcommand sequence, so helpers such as mcp-server-time and server.py remain eligible for cleanup. Added regression coverage for both names; all 21 focused updater child-reap tests pass.

@OutThisLife

Copy link
Copy Markdown
Collaborator

Closing in favor of #74436, which fixes the cause this works around.

Your diff treats the symptom correctly — a venv holder blocks the updater — but the holder in these reports is the dashboard's own detached hermes update, spawned unregistered while the desktop updater was already running. The in-progress marker existed but nothing enforced it as a lock, so two updaters could mutate one checkout at once. #74436 makes that marker a real cross-process lock claimed by every update entrypoint, so a second updater is refused instead of racing.

The venv-holder detection you were working around is untouched and still correct; it should just rarely fire now.

Thanks for digging into this — the logs and repro in here were genuinely useful in tracing the orchestration bug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/install-update Installer, updater, packaging, wheels, doctor comp/cli CLI entry point, hermes_cli/, setup wizard P2 Medium — degraded but workaround exists platform/windows Native Windows-specific behavior or breakage sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Windows: hermes update dead-ends permanently at the venv-process guard — pausing the gateway orphans its venv-resident MCP/perfmon children

5 participants