fix(gateway): default restart_drain_timeout to 0 to kill systemd crash loop

[teknium1]

Summary

A gateway restart now interrupts in-flight agents immediately instead of holding the process open for a grace window, which kills the systemd crash-loop reported in #31981.

Root cause: restart_drain_timeout (the gateway's own drain timer) and systemd's TimeoutStopSec (the kill timer) were two independently-set values. On a stale unit where TimeoutStopSec < drain, systemd SIGKILLed the gateway mid-cleanup, before it could remove its PID/lock file. The next startup then saw the stale lock, exited with "another gateway instance is already running", and under Restart=on-failure looped forever (one reporter's diag log hit 63 MB over 13 days).

Setting the drain default to 0 makes the mismatch structurally impossible: with drain 0 the generated unit gets TimeoutStopSec=90 against a near-instant drain, so systemd never kills mid-cleanup. The contract is now explicit — restart the gateway, in-flight work stops. A grace window large enough to "save" a long agent turn would have to outlast an unbounded task, which is impossible.

Changes

• hermes_cli/config.py: restart_drain_timeout default 180 → 0; comment rewritten to state the contract and the timer-coupling rationale.
• cli-config.yaml.example: doc comment updated to match (60 → 0).
• gateway/run.py: stale-unit warning suggested hermes gateway service install --replace — that subcommand does not exist. Corrected to hermes gateway install --force.
• tests/gateway/test_gateway_shutdown.py: the one test that depended on a positive default drain now sets an explicit 5.0 to cover the opt-in grace-window path.

Notes:

• DEFAULT_GATEWAY_RESTART_DRAIN_TIMEOUT derives from DEFAULT_CONFIG, so it tracks the new default automatically — single source of truth.
• The unit-generation tests are invariant-based (max(60, DEFAULT) + 30), so they carried the change with no edits and now assert TimeoutStopSec=90.
• parse_restart_drain_timeout("0") returns 0.0 cleanly (no coercion-to-default edge with the new 0 default).

Validation

	Before	After
Default drain	180s grace window on every restart	0 — interrupt immediately
systemd kill-timer vs drain	could be SIGKILLed mid-cleanup (stale unit)	90s ≫ near-instant drain; race impossible
Stale-unit warning command	hermes gateway service install --replace (invalid)	hermes gateway install --force
Crash loop on stale unit	infinite	cannot form

Targeted suites green: test_gateway_service (175), test_restart_drain (21), test_gateway_shutdown, test_shutdown_forensics — 196 tests.

Closes #31981.

Infographic

[tonydwb]

Code Review Summary

Verdict: Approved

Clean config change that defaults restart_drain_timeout to 0 to prevent systemd crash loops. The old 180s default was causing SIGKILL-mid-cleanup races when TimeoutStopSec was shorter than the drain window. The new default (0 = interrupt immediately) sidesteps both the unbounded-task problem and the SIGKILL race. Tests are updated to explicitly opt into a grace window when needed. Well-documented with clear rationale in the code comments.

---

Reviewed by Hermes Agent