fix(gateway): add hard-exit watchdog to prevent drain hang on stop/restart

[shengting]

Problem

When the gateway receives SIGUSR1 (restart) or is stopped while agent
sessions are active, _stop_impl() may hang indefinitely if any session
thread cannot be interrupted ("zombie" sessions — e.g. a blocking
subprocess in terminal(), or a synchronous network call that ignores
asyncio cancellation).

On systems managed by launchd (macOS) or systemd (Linux), the
process is then killed with SIGKILL after TimeoutStopSec, which bypasses
log flushing and DB close.

Reproducing scenario:

1. Start a gateway session running a long terminal() call
2. Send SIGUSR1 to restart the gateway (hermes restart)
3. _drain_active_agents() times out; _interrupt_running_agents() fires
   but the blocking thread doesn't respond
4. _stop_impl() blocks in the 5s interrupt wait loop — or until the
   service manager's SIGKILL

Fix

Two layered watchdogs in _stop_impl(), both using daemon threads:

Layer 1 — Unconditional stop watchdog

Added at the very start of the shutdown sequence (before drain), fires
after drain_timeout + 20s regardless of where the sequence gets stuck:

_watchdog_delay = self._restart_drain_timeout + 20.0
import threading as _threading_wd
def _unconditional_exit_watchdog(delay: float) -> None:
    import time as _time, os as _os
    _time.sleep(delay)
    _os._exit(0)  # noqa: SIM115
_wd_thread = _threading_wd.Thread(
    target=_unconditional_exit_watchdog,
    args=(_watchdog_delay,),
    daemon=True,
    name="gateway-stop-watchdog",
)
_wd_thread.start()

This covers the blind spot of Layer 2: when _running_agents is cleared
before blocking threads actually finish, the per-zombie check below sees
an empty dict and doesn't fire — but those threads may still be holding
resources. Layer 1 catches this case.

Layer 2 — Per-zombie watchdog

Triggered only when zombie sessions remain after the interrupt + 5s wait.
Fires after 10s, allowing _stop_impl() to continue (log flush, DB
close, adapter disconnect) while guaranteeing exit:

if self._running_agents:
    _hard_exit_delay = 10.0
    import threading as _threading
    def _hard_exit_watchdog(delay: float) -> None:
        import time as _time, os as _os
        _time.sleep(delay)
        _os._exit(0)  # noqa: SIM115
    _t = _threading.Thread(
        target=_hard_exit_watchdog,
        args=(_hard_exit_delay,),
        daemon=True,
    )
    _t.start()

Why os._exit(0) not sys.exit()

sys.exit() raises SystemExit which asyncio and Python's atexit
machinery may catch or defer — the process might still not exit.
os._exit(0) bypasses all cleanup handlers and is the correct
last-resort in a daemon/service process. The # noqa: SIM115 silences
the linter's preference for sys.exit(), which is inappropriate here.

Behavior

Normal shutdown (no active sessions):
Both watchdog threads start but never fire — they are daemon threads and
exit automatically when the main process terminates cleanly.

Shutdown with interruptible sessions:
Sessions are cancelled and drained within drain_timeout. Process exits
cleanly before either watchdog fires.

Shutdown with zombie sessions (threads blocked, cannot be interrupted):
Layer 2 watchdog fires after 10s — process hard-exits via os._exit(0).
Layer 1 was already running but superseded.

Catastrophic hang (drain loop itself stuck, _running_agents cleared early):
Layer 1 watchdog fires unconditionally at drain_timeout + 20s.

Testing

Reproduced hang by starting a session running sleep 120 in terminal()
tool, then issuing hermes restart (SIGUSR1).

• Without this patch: process hangs until SIGKILL from launchd/systemd.
• With this patch: process exits within drain_timeout + 20s, log and DB
  close complete normally in the non-zombie path.

[tonydwb]

Code Review Summary

Verdict: Approved

Overview

Two-layer hard-exit watchdog to prevent gateway _stop_impl() from hanging indefinitely when zombie sessions can't be interrupted. A belt-and-suspenders approach covering both the per-zombie blind spot and the catastrophic hang case.

Looks Good

• Layer 1: unconditional exit after drain_timeout + 20s at start of shutdown
• Layer 2: per-zombie guard fires 10s after active sessions remain post-interrupt
• Correct use of os._exit(0) for last-resort (bypasses cleanup handlers intentionally)
• Daemon threads that auto-exit on clean shutdown
• Well-documented rationale and behavior matrix
• # noqa: SIM115 annotated with explanation

---

Reviewed by Hermes Agent

[shengting]

Hi @tonydwb, the CI workflow runs for this PR are awaiting approval (fork PR security policy). Could you please approve them when you get a chance? The PR has already been approved by you — just need the CI checks to pass before merge. Thanks!

[shengting]

Hi @tonydwb, gentle ping — the CI workflow runs for this PR are still awaiting approval (fork PR security policy). It's been about two weeks since your approval. Could you approve the CI runs when you get a chance? Thanks!

[shengting]

Hi @tonydwb, another gentle ping — this PR was approved on 5/31 but the CI workflow runs are still awaiting approval (fork PR security policy). Could you approve the CI runs when you get a chance so we can get this merged? It's been nearly 3 weeks now. Thanks!

[shengting]

Hi @tonydwb, another ping — it's been over 4 weeks since approval on 5/31 and the CI runs still haven't been triggered. The fork PR security policy requires maintainer approval for CI. Could you please approve the CI workflow runs so this can be merged? Thanks!

[terry197913]

Hi @tonydwb 👋 — following up on the four previous pings from @shengting (6/9, 6/13, 6/20, 6/29).

This PR was approved on 5/31 but the CI workflow runs are still awaiting maintainer approval due to the fork PR security policy. Could you please approve the CI runs so this can be merged?

I also wanted to note why this fix remains necessary even after the recent upstream drain work:

• feat(gateway+dashboard-auth): external drain coordination (safe-shutdown Phase 2) — salvage #50519 #52937 (external drain coordination) prevents new turns from being accepted during a planned shutdown and waits for active_agents == 0 — but it relies on the gateway's stop path completing cleanly.
• feat(gateway): suppress home-channel shutdown broadcast on flagged drains #54824 suppresses noisy home-channel broadcasts during flagged drains.
• This PR (fix(gateway): add hard-exit watchdog to prevent drain hang on stop/restart #35676) addresses a different failure mode: if a session thread is stuck in a blocking subprocess (terminal()) or a synchronous network call that ignores asyncio cancellation, _stop_impl() hangs indefinitely — and the external drain coordinator will wait forever too. The hard-exit watchdog is a last-resort safety net that sits underneath the drain coordination layer.

The two approaches are complementary, not redundant. Thanks for your time!

[shengting]

@tonydwb — thanks for taking a look at this PR. As @terry197913 beautifully articulated, the hard-exit watchdog (#35676) sits underneath the drain coordination layer (#52937) and addresses a complementary failure mode: stuck blocking calls that make _stop_impl() hang indefinitely.

The PR is approved and the analysis from an independent contributor confirms the design rationale. Could you approve the CI workflow runs so we can get this merged? It has been 6 weeks since initial approval.

[teknium1]

Thanks for the shutdown-safety work. An automated hermes-sweeper review found that current main already provides the requested hard-exit behavior through a later, broader implementation.

• gateway/run.py:5658 bounds active-agent draining; gateway/run.py:8224-8230 interrupts timed-out work and waits only five seconds.
• gateway/run.py:5997-6029 moves potentially wedged agent cleanup off the event loop under a 30-second bound.
• gateway/run.py:20964-21050 routes normal and SystemExit gateway shutdown paths through os._exit, preventing interpreter finalization from joining stuck non-daemon workers.
• tests/gateway/test_gateway_process_exit.py:61-110 covers the os._exit termination guarantee.
• This shipped in cde3ca4ebf59e42a422c2239d917b5124a59ec5e (v2026.7.1).