fix(gateway): revive gateway on /restart under Restart=on-failure units

[randomuser2026x]

Problem

The in-chat /restart command leaves the gateway dead on systemd deployments whose unit file uses Restart=on-failure (the most common setup recommended in deployment guides and used by hermes setup on Ubuntu). The gateway drains, exits cleanly, and never comes back — the only recovery is a host reboot.

Confirmed reproduction: operator-managed unit file at `/etc/systemd/system/hermes-gateway.service` with `User=ubunutu`, `Restart=on-failure`, `RestartForceExitStatus=75`. After `/restart` from Feishu the gateway stays down indefinitely until someone runs `sudo systemctl restart hermes-gateway` or reboots the host.

Root cause

Three independent bugs in series, any one of which breaks `/restart` on affected deployments:

1. Wrong exit-code assumption (`gateway/run.py` `_stop_impl`): the Linux/systemd branch returned exit code `0` under the assumption that all systemd units use `Restart=always` and would relaunch on a clean exit. Units with `Restart=on-failure` never see `0` as a trigger, so systemd does nothing.

2. Hardcoded `--user` scope (`gateway/run.py` `_launch_systemd_restart_shortcut`): the planned-restart helper assumed the unit was registered as a user service. On system-level deployments (the common case) `systemctl --user show hermes-gateway` returns an empty `MainPID`, the PID-equality check fails, and the helper silently returns without launching anything.

3. Polkit denial on `systemd-run --system` (deployment reality): non-root gateway units (e.g. `User=ubunutu`) invoke `systemd-run --system` to create the transient restart helper, but Polkit rejects it with `Interactive authentication required`. The `--user` fallback requires a D-Bus user session that is absent on headless servers. So even after the scope bug above is fixed, the helper still cannot start in real deployments.

Fix

Always exit `75` (EX_TEMPFAIL) on service-managed restarts. Combined with the existing `RestartForceExitStatus=75` directive in the unit file, systemd treats the planned restart as a controlled failure and revives the gateway via `Restart=on-failure`, with `RestartSec` as the only delay.

```diff

• self._exit_code = (

• GATEWAY_SERVICE_RESTART_EXIT_CODE
  

• if sys.platform == "darwin" or not os.environ.get("INVOCATION_ID")
  

• else 0
  

• )

• self._exit_code = GATEWAY_SERVICE_RESTART_EXIT_CODE
  ```

The planned-restart helper is still attempted (for setups that want sub-`RestartSec` restarts), but it is no longer load-bearing. If `systemd-run` is denied by Polkit or the user bus is missing, the helper silently fails and systemd's own `Restart=` machinery takes over.

A second commit also fixes the scope detection in `_launch_systemd_restart_shortcut` so the helper (when it does work) targets the correct scope:

```python
system_pid = _query_pid([]) # systemctl show ...
user_pid = _query_pid(["--user"])

if current_pid == system_pid: scope = [] # /etc/systemd/system
elif current_pid == user_pid: scope = ["--user"] # systemctl --user
else: return # not systemd
```

Behavior matrix after fix

Unit file	Helper outcome	Gateway revived by
`Restart=always`, root gateway	✓ sub-second	helper or systemd
`Restart=on-failure`, root gateway	✓ sub-second	helper or systemd (exit 75)
`Restart=on-failure`, non-root gateway	✗ Polkit denied	systemd (exit 75)
Headless without D-Bus user session	✗ no user bus	systemd (exit 75)
Not under systemd	n/a	existing non-systemd paths

Safety

• `StartLimitBurst` / `StartLimitIntervalSec` in the unit file still bound accidental restart loops.
• macOS launchd path is unchanged (`KeepAlive.SuccessfulExit=false`).
• No impact on crash recovery: crashes still produce the same exit codes they always did.

Verification

End-to-end on Ubuntu 24.04 with hermes-gateway as a `/etc/systemd/system/` service under `User=ubunutu`, using `Restart=on-failure`, `RestartSec=30`, `RestartForceExitStatus=75`, `StartLimitIntervalSec=600`, `StartLimitBurst=5`.

• `/restart` from Feishu: gateway drains in <1s, exits 75, systemd revives it ~30s later. No manual intervention.
• Repeated 5× in quick succession: works correctly, then `start-limit-hit` kicks in as expected.
• After `systemctl reset-failed`: `/restart` works again.

Tests

`tests/gateway/test_gateway_shutdown.py`:

• Renamed `test_gateway_stop_systemd_service_restart_exits_cleanly` → `test_gateway_stop_systemd_service_restart_uses_tempfail`.
• Asserts `_exit_code == GATEWAY_SERVICE_RESTART_EXIT_CODE` (75) instead of `0`.

```
$ pytest tests/gateway/test_gateway_shutdown.py
14 passed in 16.58s
```

[benschewel]

Independent root-cause analysis on a separate systemd deployment reached the same conclusion, and we verified this fix end-to-end.

Evidence from our incident (system unit with Restart=on-failure):

• The regression is commit b14e15c48e ("clean service restart notifications"), which switched the planned-restart exit from 75 to 0 on the assumption that units use Restart=always. Our unit is Restart=on-failure, so the clean exit-0 was treated as success and the gateway stayed down (~48 min until a manual systemctl restart).
• The exit-diag log confirmed the incident was the first-ever Linux SystemExit: 0; every prior restart exited 75 and relaunched fine.
• _launch_systemd_restart_shortcut() queries the --user scope, which no-ops on a system unit, so it couldn't compensate — exactly what the _query_pid(scope_flags) change here addresses.

We applied the same core change (self._exit_code = GATEWAY_SERVICE_RESTART_EXIT_CODE) as a hotfix and verified on a real box: SIGUSR1 → process exits 75 → Restart=on-failure relaunches within RestartSec → Telegram reconnects, NRestarts=1. Works. +1 to merging.

One note for anyone mirroring the unit change: SuccessExitStatus=75 would be the wrong directive under Restart=on-failure — it reclassifies 75 as success and would skip the relaunch. RestartForceExitStatus=75 as used here is correct.

[teknium1]

Merged via #56362 (commit 2f167a2) — cherry-picked onto current main with your authorship preserved in git log. Thanks for the fix!

The salvage probes both systemctl show and systemctl --user show to pick the right scope (system-unit deployments were silently no-op'ing on the hardcoded --user), and always exits 75 so RestartForceExitStatus=75 revives the unit under Restart=on-failure too, not just Restart=always. A small follow-up commit corrected a comment (Hermes-generated units set StartLimitIntervalSec=0, so StartLimitBurst doesn't bound loops) and added the AUTHOR_MAP entry.