fix: reap terminal shells on shutdown

[Michaelyklam]

Thinking Path

• Issue [bug] api/terminal.py: Node child processes become zombies when Popen handle is not reaped (same class as #1912) #2577 reports zombie child processes around the embedded terminal surface.
• api/terminal.py owns PTY shell lifecycle, not arbitrary grandchildren, so the safest WebUI slice is to harden the shell lifecycle it directly controls.
• The existing close path already sends SIGHUP and waits, but graceful process shutdown did not explicitly close all tracked terminals, and timeout escalation did not wait again after SIGKILL.
• This PR adds a Linux parent-death signal for the spawned PTY shell, a graceful shutdown reaper, and regression coverage for those contracts.

What Changed

• Added _terminal_shell_preexec_fn() and passed it to the PTY shell Popen(...) so Linux sends SIGTERM to the shell when the WebUI parent dies.
• Registered close_all_terminals() via atexit to best-effort close and reap active embedded shells during graceful WebUI shutdown.
• Updated close_terminal() to wait again after SIGKILL escalation so timed-out shells are reaped.
• Added focused tests for the preexec wiring, SIGKILL wait path, all-terminal close snapshot, and shutdown registration.
• Added an Unreleased changelog entry.

Why It Matters

This reduces the chance that embedded workspace-terminal shells survive WebUI shutdown/restart paths or remain unreaped after timeout cleanup. It also keeps the fix scoped to the process that api/terminal.py actually spawns and controls.

Refs #2577

Verification

• env -u HERMES_CONFIG_PATH -u HERMES_WEBUI_HOST /home/michael/.hermes/hermes-agent/venv/bin/python -m pytest tests/test_terminal_process_cleanup.py -q → 4 passed
• python3 -m py_compile api/terminal.py
• git diff --check

Risks / Follow-ups

• PR_SET_PDEATHSIG is Linux-specific and best-effort; non-Linux or restricted runtimes skip it and keep existing behavior.
• This does not claim to solve Docker PID-1 orphan reaping for grandchildren if a deployment runs WebUI as PID 1 without an init shim; that may still need a separate Docker/runtime fix.
• SIGKILL still cannot run Python cleanup hooks; the pdeathsig path is defense-in-depth for the spawned shell, while atexit covers graceful exits.

Model Used

AI-assisted change with repository inspection, targeted editing, and shell-based test verification.

[nesquena-hermes]

Summary

Reading the diff at api/terminal.py and the new tests/test_terminal_process_cleanup.py: this implements exactly the two-pronged mitigation I recommended on #2577 (pdeathsig defense-in-depth + graceful-shutdown atexit hook), and the test coverage matches the contracts. The CI green-light on 3.11/3.12/3.13 plus the focused tests give me reasonable confidence this lands the WebUI-side hardening cleanly. One small portability nit and one open question about the issue's original symptom worth flagging before merge.

Code reference

api/terminal.py:73-86 — the pdeathsig preexec:

def _terminal_shell_preexec_fn() -> None:
    """Ask Linux to terminate the PTY shell when the WebUI parent dies."""
    try:
        import ctypes
        libc = ctypes.CDLL(None)
        libc.prctl(1, signal.SIGTERM)  # PR_SET_PDEATHSIG=1, SIGTERM=15
    except Exception:
        pass

api/terminal.py:259-279 — the SIGKILL wait + atexit reap path:

try:
    os.killpg(term.proc.pid, signal.SIGKILL)
except ProcessLookupError:
    pass
try:
    term.proc.wait(timeout=1.0)
except (subprocess.TimeoutExpired, ProcessLookupError):
    pass

The atexit.register(close_all_terminals) at module load plus the snapshot-then-iterate pattern in close_all_terminals (so we don't mutate _TERMINALS while iterating) are both correct.

Diagnosis

Lands the right scope. The fix is scoped to what api/terminal.py actually controls (its own PTY shell), and explicitly does not over-claim to fix the Docker PID-1 grandchild reaping case. That's the right call — that's a separate runtime concern that needs an init shim (tini / docker run --init), not a Python patch.

Tests match contracts. test_terminal_shell_uses_parent_death_signal_preexec exercises the Popen kwargs, test_close_terminal_waits_again_after_sigkill proves the second wait(timeout=1.0) is actually called after escalation (the previously-missing reap), and test_close_all_terminals_closes_snapshot confirms snapshot semantics. The string-match-on-source test at the bottom is a belt-and-suspenders sanity check on the wiring.

One minor portability concern. libc = ctypes.CDLL(None) resolves to the calling process's primary libc on Linux/glibc, which is correct, but on musl-based Alpine images prctl is still present under the same name, so the path works there too. On macOS, CDLL(None) succeeds and prctl doesn't exist — that raises AttributeError, which the broad except Exception catches. Good. The api/providers.py:90-105 pattern uses ctypes.CDLL("libc.so.6") explicitly; either approach works, just noting the two terminal.py and providers.py paths are now slightly stylistically divergent. Not a blocker.

Open question on the original report. The reporter on #2577 observed Node zombies as direct children of the WebUI PID. This patch fixes the case where the shell itself becomes a zombie, but Node-as-direct-child still implies either (a) WebUI running as PID 1 without an init shim, or (b) something setting PR_SET_CHILD_SUBREAPER upstream. The PR description correctly notes this in "Risks / Follow-ups." I'd suggest the reporter confirm their deployment topology when they re-test — if they're seeing the zombies inside Docker, this patch alone may not be sufficient and the actionable follow-up is a Dockerfile --init / tini change.

Verification

Tests in CI cover the Python side. The Linux pdeathsig behavior can't be unit-tested cleanly without spawning a real process and killing the parent, so the string-presence test is a reasonable proxy. For a manual reality check post-merge: start a terminal session, sleep 1000 inside the PTY, kill -9 the WebUI PID, then ps -ef --forest after ~2s — the shell PID should be gone. Skip on macOS.

LGTM modulo the open question above. Nice clean slice.