Skip to content

fix(serve): self-reap the desktop backend when its parent dies - #79112

Closed
Rmohid wants to merge 1 commit into
NousResearch:mainfrom
Rmohid:agent/claude/serve-orphan-watchdog-3d88
Closed

fix(serve): self-reap the desktop backend when its parent dies#79112
Rmohid wants to merge 1 commit into
NousResearch:mainfrom
Rmohid:agent/claude/serve-orphan-watchdog-3d88

Conversation

@Rmohid

@Rmohid Rmohid commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Makes the headless hermes serve backend reap itself when its parent dies, closing the one orphan path that no parent-side fix can reach.

The desktop app spawns its backend as hermes serve --host 127.0.0.1 --port 0 and tears it down in before-quit. That teardown is correct, and #76245 / #76244 make it more reliable still. But all three live on the parent path, and there is a case where the parent's code never runs at all: force-quit, SIGKILL, or a fatal GPU abort. main.ts already acknowledges this in a comment — "FATAL GPU aborts skip before-quit".

When that happens on macOS there is no PR_SET_PDEATHSIG (this repo documents the constraint verbatim in tools/mcp_stdio_watchdog.py), so the kernel reparents the backend to pid 1 instead of reaping it. It keeps its 127.0.0.1 LISTEN socket and its resident memory forever. That is the PPID=1 accumulation in #61349.

Since the parent's code is exactly what did not run, the fix has to be child-side.

This is not a new mechanism for this repo. tui_gateway/slash_worker.py already solves precisely this with a parent-death watchdog, and tests/test_slash_worker_watchdog.py pins it. #61349 reports orphans from both serve and slash_workerslash_worker is already covered by this idiom; serve never was. This PR extends the proven pattern to serve, deliberately mirroring the existing code and tests rather than inventing anything.

Relationship to the other open orphan issues

This is a complement, not a replacement:

Issue Path it fixes Covers force-quit / SIGKILL?
#76245 — before-quit doesn't wait for SIGTERM parent No
#76244 — uvicorn graceful shutdown unbounded child, but only once SIGTERM arrives No
this PR — backend self-reaps on parent death child, needs no signal at all Yes

Related Issue

Addresses the force-quit / abort path of #61349, and reduces the pile-up behind #78872 and #78821.

To be precise about scope: #61349 likely has more than one cause, and the graceful-quit paths belong to #76245 / #76244. This PR does not supersede those — it covers the case where no teardown code runs at all. I've deliberately written "addresses" rather than "fixes" for that reason.

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

  • hermes_cli/main.py (+72, purely additive — no existing lines changed):
    • _serve_is_orphaned(original_ppid, getppid=os.getppid) — same predicate/signature as slash_worker._is_orphaned.
    • _should_start_serve_watchdog(headless_backend, env, os_name) — the gate, as a pure function so it is directly testable.
    • _start_serve_parent_death_watchdog(original_ppid) — daemon thread; os._exit(0) once orphaned.
    • _env_float(...) + _SERVE_WATCHDOG_POLL_S (HERMES_SERVE_WATCHDOG_POLL_S, default 2.0s), mirroring the slash_worker helper so a typo'd env var can't raise at import.
    • One call site in cmd_dashboard.
  • tests/hermes_cli/test_serve_parent_death_watchdog.py (new, 9 tests).

The gate

cmd_dashboard backs both dashboard and serve, so the watchdog is gated on all three of:

  1. _headless_backendserve only. A human's foreground hermes dashboard must never self-reap.
  2. HERMES_DESKTOP=1 — the desktop's own backend only. A deliberate nohup hermes serve & legitimately reparents to pid 1 when its shell exits; killing that would be a regression, not a fix. This env var is already load-bearing here — cmd_dashboard branches on it a few lines below.
  3. POSIX — same gate tools/mcp_tool.py uses.

Why the call sits after the re-exec

The named-profile re-exec is os.execvpe on POSIX, which preserves pid/ppid, so a ppid recorded after it stays valid. On Windows that same branch is subprocess.Popen, where it would not — hence both the placement and the POSIX gate. A test asserts the ordering so a future refactor can't silently move it above the re-exec.

How to Test

Both arms, against a real backend (not a mock). Substitute any Python with the repo importable:

1. Treatment — desktop-shaped launch self-reaps:

cat > /tmp/parent.py <<'PY'
import subprocess, sys, os
p = subprocess.Popen([sys.executable, "-m", "hermes_cli.main",
                      "serve", "--host", "127.0.0.1", "--port", "0"])
open(os.environ["PIDFILE"], "w").write(str(p.pid)); sys.exit(p.wait())
PY
PIDFILE=/tmp/child.pid HERMES_HOME=/tmp/isolated-home HERMES_DESKTOP=1 \
  HERMES_SERVE_WATCHDOG_POLL_S=0.2 python /tmp/parent.py &
sleep 25
CHILD=$(cat /tmp/child.pid); PARENT=$(ps -p $CHILD -o ppid= | tr -d ' ')
lsof -nP -p $CHILD | grep LISTEN     # backend is up and listening
kill -9 $PARENT                       # force-quit: no teardown code runs
sleep 3; ps -p $CHILD || echo "self-reaped"

2. Control — a standalone daemon is left alone. Same script with HERMES_DESKTOP unset: the child survives at PPID=1. This both reproduces the original bug and demonstrates the gate protects a deliberate nohup hermes serve &.

3. Unit + integration tests:

bash scripts/run_tests.sh --files "tests/hermes_cli/test_serve_parent_death_watchdog.py:tests/hermes_cli/test_serve_command.py:tests/test_slash_worker_watchdog.py"

Results on my machine (macOS 15 / Darwin 25.5.0, Python 3.11.14)

  • Treatment: backend self-reaped 1s after the parent's SIGKILL; LISTEN socket released.
  • Control: backend survived at PPID=1 — the bug reproduced.
  • No spurious firing: under a live parent the backend ran 25s+ without self-reaping.
  • Mutation check: with the watchdog loop neutered, the integration test fails with the real "orphan survived" message — confirming the test actually discriminates rather than passing vacuously.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix (single commit, no unrelated changes)
  • I've run pytest tests/ -q and all tests pass — partially; see note below
  • I've added tests for my changes
  • I've tested on my platform: macOS 15 (Darwin 25.5.0, ARM64), Python 3.11.14

Note on the full-suite checkbox, stated plainly rather than ticked: I ran the change's blast radius — test_serve_parent_death_watchdog.py, test_serve_command.py, test_slash_worker_watchdog.py, test_dashboard_param_clamps.py, test_apply_profile_override.py, test_web_server_cron_profiles.py, test_argparse_flag_propagation.py, tests/dashboard/46 tests, 0 failures under scripts/run_tests.sh. I did not complete a clean full-suite run: running scripts/run_tests.sh tests/hermes_cli rebuilt my checkout's .venv and removed pytest from it, breaking the runner on the next invocation. That looks like the hazard tests/hermes_cli/test_managed_uv.py:45-46 mocks (the SQLite-repair path probing the real checkout's venv) being reached unmocked somewhere else in that directory. It is unrelated to this change — it reproduced before it — and I'm happy to file it separately if it isn't already known.

Documentation & Housekeeping

  • I've updated relevant documentation — N/A (behavior is internal; the rationale is in the code comments and commit message)
  • I've updated cli-config.yaml.example — N/A (no config keys; the one env knob is test-only tuning with a safe default)
  • I've updated CONTRIBUTING.md / AGENTS.md — N/A
  • I've considered cross-platform impact — yes, explicitly: the watchdog is POSIX-gated because the Windows re-exec branch invalidates the recorded ppid. Windows behavior is unchanged.
  • I've updated tool descriptions/schemas — N/A

🤖 Generated with Claude Code

Three `hermes_cli.main serve --host 127.0.0.1 --port 0` processes were found
orphaned at ppid 1 from a single ~3-minute restart burst, each holding a live
127.0.0.1 LISTEN socket with zero connections and 53-138 MB resident.

The spawner is the desktop Electron app (apps/desktop/electron/
backend-command.ts serveBackendArgs emits exactly that argv). Its teardown is
correct -- before-quit does SIGTERM then forceKillProcessTree over the primary
backend and the pool -- but structurally cannot cover this: on force-quit /
SIGKILL / fatal GPU abort that handler never runs, and main.ts says so itself
("FATAL GPU aborts skip before-quit"). No macOS crash report exists for the
window, consistent with SIGKILL. macOS has no PR_SET_PDEATHSIG (documented in
tools/mcp_stdio_watchdog.py), so the kernel reparents the backend to pid 1
instead of reaping it.

No parent-side fix can close this, because the parent's code is exactly what
did not run. So the child reaps itself, using the same watchdog idiom as
tui_gateway/slash_worker.py: record ppid at startup, exit once it changes.

Gated to the one launch shape that can leak -- headless `serve` only (never the
interactive `dashboard`, which shares cmd_dashboard), HERMES_DESKTOP=1 only (a
deliberate `nohup hermes serve &` legitimately reparents to pid 1), and POSIX
only (the profile re-exec is os.execvpe here, preserving pid/ppid; on Windows
it is subprocess.Popen and the recorded ppid would be stale). Started after the
re-exec for that same reason.

Verified end to end against a real serve backend on Darwin 25.5.0, both arms:
with HERMES_DESKTOP=1 the backend self-reaped 1s after its parent was SIGKILLed
and released its LISTEN socket; without it the backend survived at ppid 1,
reproducing the leak and confirming standalone daemons stay protected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@teknium1

Copy link
Copy Markdown
Contributor

Resolved on main by PR #83406 (rebase-merged), which carries the parent-death watchdog + group-kill (from #73066) and the Desktop-boot reap of already-orphaned serve backends. Special credit here: this PR was the EARLIEST submission of the serve orphan watchdog idea (July 9), predating the implementation that ultimately landed — the design direction was yours first. Thank you!

@teknium1 teknium1 closed this Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants