fix(gateway): stop stale systemd timeout mismatch loop

[konsisumer]

What does this PR do?

When the gateway detects a stale systemd unit whose TimeoutStopSec is shorter than the configured drain timeout headroom, it now fails startup loudly instead of only logging a warning. The startup error uses the real repair command for the installed unit scope, writes startup_failed runtime status, and best-effort stops the current unit so a misconfigured service does not immediately re-enter the same crash loop.

Related Issue

Fixes #31981

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• gateway/run.py: turn a stale systemd drain-timeout mismatch into a startup-blocking error, print the remediation message to stderr, persist startup_failed, and best-effort stop the misaligned unit.
• gateway/shutdown_forensics.py: return the detected systemd scope plus the correct repair command, and add a helper to stop the current systemd unit without shelling through unsafe string interpolation.
• tests/gateway/test_runner_startup_failures.py: cover the new startup refusal path and verify the unit-stop attempt plus runtime-status update.
• tests/gateway/test_shutdown_forensics.py: cover scope-aware repair command generation and the new unit-stop helper.

How to Test

1. Run pytest tests/gateway/test_shutdown_forensics.py tests/gateway/test_runner_startup_failures.py tests/gateway/test_restart_drain.py tests/gateway/test_runner_fatal_adapter.py -q.
2. Run python scripts/check-windows-footguns.py gateway/run.py gateway/shutdown_forensics.py tests/gateway/test_runner_startup_failures.py tests/gateway/test_shutdown_forensics.py.
3. Run ruff check gateway/run.py gateway/shutdown_forensics.py tests/gateway/test_runner_startup_failures.py tests/gateway/test_shutdown_forensics.py and git diff --check.
4. Attempt the broad local suite with /opt/homebrew/bin/timeout -k 30 480 sh -c 'pytest tests/ -q -x --timeout=60 "$@"' sh.
5. Note that step 4 currently stops during unrelated collection at tests/hermes_cli/test_dashboard_auth_401_reauth.py with ModuleNotFoundError: No module named 'fastapi' in this environment.

What platforms tested on

• macOS on Darwin 24.6.0 arm64 (local)

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run pytest tests/ -q and all tests pass
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: macOS on Darwin 24.6.0 arm64

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) — or N/A
• I've updated cli-config.yaml.example if I added/changed config keys — or N/A
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
• I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Screenshots / Logs

• Broad local suite attempt stopped during unrelated collection because fastapi is unavailable in this environment: tests/hermes_cli/test_dashboard_auth_401_reauth.py -> ModuleNotFoundError: No module named 'fastapi'.

[liuhao1024]

✅ Verified — stale systemd timeout mismatch now prevents SIGKILL mid-drain

Reviewed the diff for the gateway startup refusal logic and stop_systemd_unit helper.

• Scope tracking: check_systemd_timing_alignment now correctly captures "user" vs "system" scope via the flag loop variable (shutdown_forensics.py:367-388). The scope field is passed through to run.py and used by stop_systemd_unit.
• Auto-stop: stop_systemd_unit uses the correct --user flag based on scope (shutdown_forensics.py:404-418). Timeout of 3.0s is appropriate for a best-effort stop.
• Clean exit path: run.py:4338-4372 — on mismatch, the runner logs the error, attempts stop_systemd_unit, writes startup_failed runtime status, and calls _request_clean_exit. The return True signals "handled gracefully" (not "started successfully").
• Test coverage: test_runner_refuses_start_for_stale_systemd_timeout_mismatch covers the happy path with both scope values and verifies the exit reason contains the repair command.

The fix correctly converts a warning into a hard gate, preventing systemd's SIGKILL from hitting mid-drain. No issues found.

[teknium1]

Thanks for this — your diagnosis was spot-on and matched what we found.

We took a different fix direction in #54066: rather than make systemd's TimeoutStopSec chase the drain timeout, we set the restart_drain_timeout default to 0. A restart now interrupts in-flight work immediately and exits fast, so the kill-timer (90s) always far exceeds a near-instant drain and the mismatch that caused the crash loop is structurally impossible — no refuse-to-start or auto-stop logic needed. We also corrected the stale-unit warning's bad command (hermes gateway service install --replace → hermes gateway install --force), which your PR description correctly flagged.

The "refuse to start + auto-stop the misaligned unit" behavior here is a meaningful design change with real downside risk (a detection false-positive turns a cosmetic warning into a hard outage), so we opted not to take it. Closing in favor of #54066, which resolves the crash loop at the root. Appreciate the thorough write-up and tests.