fix(gateway): hard-exit after graceful shutdown so a wedged worker can't strand the gateway

[Kyzcreig]

Symptom

Restarting the gateway occasionally left it "down but not exited" for ~2.5 minutes (measured: a 149s blackout on one restart), during which launchd/systemd could not relaunch it — the old process was still alive, so the supervisor wouldn't spawn a new one. Most restarts are sub-second; this is intermittent.

Root cause (measured, not theorized)

From gateway-exit-diag.log for the slow restart:

• asyncio.run.returned at T+0 — async teardown done in ~6s (adapters disconnected, SessionDB closed, sessions persisted)
• atexit.hook at T+149s — the interpreter hung for 2.5 min after the loop returned

The gateway runs each agent turn on a ThreadPoolExecutor (gateway/run.py). In CPython, those workers are daemon=False, and concurrent.futures registers an atexit hook that joins every worker thread before the interpreter can exit. If an agent turn was still executing in a worker at SIGTERM, the process blocked at interpreter finalization until that in-flight turn completed. Verified empirically on 3.11.15: ThreadPoolExecutor worker daemon=False, concurrent.futures.thread._python_exit atexit-join present.

Across 185 recorded shutdowns the hang fired once (149s); every other exit was sub-second — it needs a worker mid-blocking-call at the SIGTERM instant.

gateway/run.py::main() already force-exits for exactly this reason (_exit_after_graceful_shutdown), but production runs via hermes_cli/gateway.py::run_gateway(), which lacked the backstop.

Fix

Apply the same graceful-teardown-then-hard-exit pattern to the production path, plus diagnostics.

• hermes_cli/gateway.py: run_gateway() force-exits via a new _force_process_exit() at both the clean (0) and failure (1) tails, after start_gateway()'s graceful teardown. Nothing of value is lost — the graceful work already ran; we hard-exit the husk. The helper flushes stdio and logs any lingering non-daemon threads (greppable force-exit: lingering non-daemon ...) so the next stall is diagnosable at a glance. Wrapped in a helper so tests neutralize os._exit.
• gateway/run.py: _shutdown_executor() now does a bounded drain (_executor_drain_timeout(), default 8s, HERMES_GATEWAY_EXECUTOR_DRAIN_TIMEOUT override) and logs in-flight worker count + drain time (Executor drain: N worker(s) in-flight ... drained in Xs / clean in Xs). A wedged worker no longer blocks here; the CLI backstop finalizes it.

Tests (RED→GREEN verified)

• test_run_gateway_force_exits_zero_after_clean_shutdown / ..._exits_nonzero_when_start_gateway_reports_failure — force-exit fires with the right code (fail on old fall-through/sys.exit).
• test_force_process_exit_calls_os_exit_and_flushes — flushes stdio, calls os._exit(code).
• test_shutdown_executor_bounded_by_drain_timeout — _shutdown_executor returns within budget even when a worker blocks 5s (RED-proven: unbounded variant blocks 4.97s and fails).

Existing run_gateway() end-to-end tests updated to neutralize the new force-exit (shared _install_fake_gateway_run helper stubs _force_process_exit), mirroring the existing tests/gateway/test_gateway_process_exit.py pattern for main().

[github-actions]

🔎 Lint report: fix/gateway-shutdown-force-exit vs origin/main

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 12271 on HEAD, 12271 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 6430 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

[greptile-apps]

Greptile Summary

This PR adds a hard os._exit backstop to the production gateway entrypoint (hermes_cli/gateway.py::run_gateway) so that a wedged non-daemon ThreadPoolExecutor worker — which concurrent.futures would otherwise join via atexit, blocking interpreter finalization — cannot strand the gateway "down but not exited" after graceful teardown. A matching bounded-drain loop is added to _shutdown_executor so clean exits remain clean, with detailed logging for the wedged case.

• hermes_cli/gateway.py: Introduces _force_process_exit(code) (flushes stdio, logs lingering non-daemon threads, calls os._exit) and replaces sys.exit(1) on the failure tail while adding _force_process_exit(0) at the clean tail; the KeyboardInterrupt console path is intentionally left as a plain return.
• gateway/run.py: Replaces the unconditional wait=False shutdown with a bounded polling drain (default 8 s, HERMES_GATEWAY_EXECUTOR_DRAIN_TIMEOUT override) and emits structured log lines for both the clean and stalled cases.
• Tests: New targeted tests for the force-exit paths and the bounded drain; existing tests updated to neutralize _force_process_exit via a shared _install_fake_gateway_run helper.

Confidence Score: 4/5

Safe to merge; the graceful-teardown logic is unchanged and the hard exit fires only after it completes, so no in-flight work is dropped.

The fix is well-targeted and backed by three new tests covering the exact failure modes. The bounded drain correctly uses Python's while/else idiom, the thread-name filter matches the executor's thread_name_prefix, and the os._exit path is cleanly isolated in a monkeypatchable helper. Two minor observations remain: a missing blank line (PEP 8 E302) after _executor_drain_timeout, and the KeyboardInterrupt console path still returning normally — meaning a wedged worker at Ctrl+C time could produce the same hang in interactive use, though the production daemon path is fully covered.

gateway/run.py — missing blank line after _executor_drain_timeout; hermes_cli/gateway.py — KeyboardInterrupt exit path returns without _force_process_exit.

Important Files Changed

Filename	Overview
gateway/run.py	Adds _executor_drain_timeout() helper and bounded polling drain to _shutdown_executor; a missing blank line between the new function and the following module-level regex constant is a PEP 8 E302 violation.
hermes_cli/gateway.py	Adds _force_process_exit helper and wires it into both exit tails of run_gateway; the KeyboardInterrupt console path deliberately returns normally and is the only remaining path susceptible to a wedged-worker hang.
tests/gateway/test_session_env.py	Adds test_shutdown_executor_bounded_by_drain_timeout which proves the drain returns within budget even when a worker blocks 5 s — correctly tests the core correctness of the bounded wait.
tests/hermes_cli/test_gateway.py	Updates _install_fake_gateway_run to neutralize _force_process_exit, adds three new targeted tests for the force-exit paths; all existing tests correctly updated to the new contract.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Supervisor as launchd/systemd
    participant Process as gateway process
    participant Executor as ThreadPoolExecutor
    participant Worker as worker thread

    Supervisor->>Process: SIGTERM
    Process->>Executor: _shutdown_executor()
    Executor->>Worker: "shutdown(wait=False, cancel_futures=True)"
    Note over Executor,Worker: bounded poll up to 8 s
    alt worker finishes within budget
        Worker-->>Executor: thread exits cleanly
    else worker still running at deadline
        Executor-->>Process: "timed_out=True logged"
    end
    Process->>Process: _exit_diag(gateway.exit_clean)
    Process->>Process: _force_process_exit(0)
    Note over Process: flush stdio, log lingering threads
    Process->>Process: os._exit(0) skips atexit join
    Process-->>Supervisor: process exited immediately
    Supervisor->>Process: respawn

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Supervisor as launchd/systemd
    participant Process as gateway process
    participant Executor as ThreadPoolExecutor
    participant Worker as worker thread

    Supervisor->>Process: SIGTERM
    Process->>Executor: _shutdown_executor()
    Executor->>Worker: "shutdown(wait=False, cancel_futures=True)"
    Note over Executor,Worker: bounded poll up to 8 s
    alt worker finishes within budget
        Worker-->>Executor: thread exits cleanly
    else worker still running at deadline
        Executor-->>Process: "timed_out=True logged"
    end
    Process->>Process: _exit_diag(gateway.exit_clean)
    Process->>Process: _force_process_exit(0)
    Note over Process: flush stdio, log lingering threads
    Process->>Process: os._exit(0) skips atexit join
    Process-->>Supervisor: process exited immediately
    Supervisor->>Process: respawn

Loading

Comments Outside Diff (1)

1. hermes_cli/gateway.py, line 4558-4566 (link)

   KeyboardInterrupt path left unguarded against wedged-worker hang

   The KeyboardInterrupt handler returns normally without calling _force_process_exit, so a non-daemon executor worker still running at Ctrl+C time will block interpreter finalization via concurrent.futures's atexit join — the same hang this PR fixes for the daemon path. In an interactive console session the user can force-kill manually, but it is the one remaining path where the 149 s stall could recur (e.g., a developer stopping a local gateway while a slow agent turn is in flight).

_{Reviews (1): Last reviewed commit: "fix(gateway): hard-exit after graceful s..." | Re-trigger Greptile}