Skip to content

fix(serve): exit desktop backend when its parent process dies - #61364

Closed
0xDevNinja wants to merge 1 commit into
NousResearch:mainfrom
0xDevNinja:fix/61349-serve-orphan-watchdog
Closed

fix(serve): exit desktop backend when its parent process dies#61364
0xDevNinja wants to merge 1 commit into
NousResearch:mainfrom
0xDevNinja:fix/61349-serve-orphan-watchdog

Conversation

@0xDevNinja

Copy link
Copy Markdown
Contributor

What does this PR do?

The Electron desktop spawns each serve/dashboard backend (the primary window backend plus every per-profile pool backend) as a plain child. On macOS/Linux the before-quit handler only sends a fire-and-forget SIGTERM, and a force-quit or crash never runs that handler at all. Any backend that doesn't die promptly reparents to launchd (PPID=1) and lingers as a ~330 MB orphan still holding a LISTEN socket — the report saw 16 accumulate (~5.3 GB) over a day of normal use.

tui_gateway.slash_worker already self-terminates when its spawning gateway disappears; the serve backend never got the same guard, so this mirrors that watchdog in the serve/dashboard startup path: poll the original PPID (with a psutil create_time PID-reuse guard, since Linux reparents to a subreaper rather than PID 1) and os._exit(0) once the parent is gone.

This is a surface-agnostic fix that also catches the force-quit / crash cases that before-quit (Option A/B in the issue) can't, and needs no changes to the Electron shutdown path.

It's gated on HERMES_DESKTOP=1 — already set on both desktop spawn paths (main.ts:6126, main.ts:6384) — so a standalone hermes serve & / nohup / systemd launch, where the parent legitimately exits, is unaffected.

Related Issue

Fixes #61349

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

  • hermes_cli/main.py: added _arm_desktop_orphan_watchdog() (+ _backend_is_orphaned() helper) and call it in cmd_dashboard() after the named-profile re-exec routing and before the server boots.
  • HERMES_DISABLE_ORPHAN_WATCHDOG=1 opts out; HERMES_BACKEND_WATCHDOG_POLL_S tunes the poll interval (default 2s).
  • tests/hermes_cli/test_backend_orphan_watchdog.py: orphan-detection truth table (ppid change, create_time mismatch, healthy parent) + arm/no-op gating on the env flags.

How to Test

  1. Launch the desktop app, open a couple of profile chats, then quit via the Dock (or force-quit).
  2. ps -eo pid,ppid,command | grep "hermes_cli.main.*serve" | grep -v grep — no PPID=1 orphans remain (they exit within one poll interval of the app dying).
  3. Standalone hermes serve & then close the shell — backend keeps running (watchdog only arms under HERMES_DESKTOP=1).
  4. pytest tests/hermes_cli/test_backend_orphan_watchdog.py -q

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix
  • I've run the tests and they pass
  • I've added tests for my changes
  • I've tested on my platform: macOS 15 (ARM64)

Documentation & Housekeeping

  • I've updated relevant documentation (docstrings) — or N/A
  • N/A — no config keys added/changed
  • N/A — no architecture/workflow change
  • I've considered cross-platform impact — the guard uses getppid() + psutil create_time (portable); the standalone-launch gate keeps non-desktop use unchanged on every platform

@alt-glitch alt-glitch added type/bug Something isn't working comp/desktop Electron desktop app (apps/desktop/*) comp/cli CLI entry point, hermes_cli/, setup wizard P3 Low — cosmetic, nice to have labels Jul 9, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for targeting a verified desktop lifecycle gap. The premise still holds on current upstream main: apps/desktop/electron/main.ts:6323-6333 sends only SIGTERM on non-Windows, and before-quit calls that path at apps/desktop/electron/main.ts:9067-9096. Current hermes_cli/main.py has no equivalent backend watchdog.

Problems

  • hermes_cli/main.py:11962-11965 adds HERMES_DISABLE_ORPHAN_WATCHDOG and HERMES_BACKEND_WATCHDOG_POLL_S. AGENTS.md:62-64 requires behavioral settings to use config.yaml/setup rather than raw HERMES_* variables. These appear to be test/debug controls rather than required user configuration.
  • tests/hermes_cli/test_backend_orphan_watchdog.py:52-69 replaces the thread with a no-op _DummyThread, so it verifies construction but never executes the target that calls os._exit(0) in hermes_cli/main.py:11980-11983.

Suggested changes

  • Remove the new environment knobs, or route any genuinely user-facing setting through config.yaml. Inject or mock sleep for test speed instead.
  • Execute a captured thread target with _backend_is_orphaned forced true and os._exit replaced by a sentinel, then assert the exit branch is reached.

Automated hermes-sweeper review.

Comment thread hermes_cli/main.py Outdated
if os.environ.get("HERMES_DISABLE_ORPHAN_WATCHDOG") == "1":
return

poll_s = max(0.05, _watchdog_env_float("HERMES_BACKEND_WATCHDOG_POLL_S", 2.0))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This introduces a non-secret behavioral HERMES_* setting solely for watchdog tuning, alongside the opt-out above. AGENTS.md requires behavioral configuration to use config.yaml rather than raw env vars; please remove these test/debug knobs or wire a genuine user-facing setting through the config surface.

return _DummyThread()

monkeypatch.setattr(hermes_main.threading, "Thread", _fake_thread)
hermes_main._arm_desktop_orphan_watchdog()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fake thread records only kwargs and its start() never invokes the target, so the test does not exercise the branch that calls os._exit(0). Capture and invoke the target with _backend_is_orphaned forced true and a mocked exit function.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 11, 2026
@0xDevNinja
0xDevNinja force-pushed the fix/61349-serve-orphan-watchdog branch from 461dd37 to 72b17fe Compare July 15, 2026 11:35
@0xDevNinja

Copy link
Copy Markdown
Contributor Author

Both fair — fixed:

  • Dropped the env knobs. HERMES_DISABLE_ORPHAN_WATCHDOG and HERMES_BACKEND_WATCHDOG_POLL_S are gone (along with the _watchdog_env_float helper). They were only ever test-timing controls, so per AGENTS.md they had no business being raw HERMES_* vars. The poll interval is now a plain _arm_desktop_orphan_watchdog(poll_s=_BACKEND_WATCHDOG_POLL_S) default that tests inject directly. The only env read left is the HERMES_DESKTOP=1 gate, which is an existing signal the desktop already sets on both spawn paths — not new configuration.
  • The test now executes the loop. Replaced the no-op _DummyThread with a _CapturingThread that captures the target, then test_watchdog_loop_exits_process_once_orphaned drives it with _backend_is_orphaned forced true on the third poll and os._exit swapped for a sentinel — asserting the exit branch is reached with code 0 and that it polled (and didn't exit) while the parent was still alive.

6 tests, 0.13s.

The Electron desktop spawns each serve/dashboard backend (the primary
window backend plus every per-profile pool backend) as a plain child. On
macOS/Linux the `before-quit` handler only SIGTERMs them fire-and-forget,
and a force-quit or crash never runs that handler at all. Any backend that
doesn't die promptly reparents to launchd (PPID=1) and lingers as a
~330 MB orphan still holding a LISTEN socket — 16 accumulated over a day
of normal use in the report (issue NousResearch#61349).

tui_gateway.slash_worker already self-terminates when its spawning gateway
disappears; the serve backend never got the same guard. Mirror that
watchdog: poll the original PPID (with a psutil create_time PID-reuse
guard, since Linux reparents to a subreaper rather than PID 1) and
os._exit(0) once the parent is gone.

Gated on HERMES_DESKTOP=1 (already set on both desktop spawn paths) so a
standalone `hermes serve &` / nohup / systemd launch — where the parent
legitimately exits — is unaffected. The poll interval is a plain function
default injected by tests, not a user-facing env knob.

Fixes NousResearch#61349
@0xDevNinja

Copy link
Copy Markdown
Contributor Author

Rebased onto current main — the branch had gone conflicting.

Re-applied both hunks on the current cmd_dashboard shape: upstream added an SSH-session-token read right after the named-profile re-exec block, so the watchdog now arms between the re-exec and that read — same intent as before (after the execvpe that desktop skips, before the long-lived server boot). _backend_is_orphaned / _arm_desktop_orphan_watchdog are otherwise unchanged, and the threading / time as _time imports they use are still in place upstream.

6 pass in tests/hermes_cli/test_backend_orphan_watchdog.py; 151 across every dashboard/serve test file, ruff clean.

@teknium1

Copy link
Copy Markdown
Contributor

Resolved on main by PR #83406 (rebase-merged), which carries the parent-death watchdog + group-kill (from #73066) and the Desktop-boot reap of already-orphaned serve backends. Special credit here: this PR was the EARLIEST submission of the serve orphan watchdog idea (July 9), predating the implementation that ultimately landed — the design direction was yours first. Thank you!

@teknium1 teknium1 closed this Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard comp/desktop Electron desktop app (apps/desktop/*) P3 Low — cosmetic, nice to have sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Desktop app leaves orphaned processes after quit (macOS, ~330MB each)

3 participants