fix(gateway): revive gateway on /restart under system-scope + Restart=on-failure units - #56362
Merged
Merged
Conversation
teknium1
force-pushed
the
hermes/hermes-d04d58a5
branch
from
July 1, 2026 12:01
cb610a2 to
a7189f4
Compare
alt-glitch
added
type/bug
The in-chat /restart command was leaving the gateway dead on systemd
deployments using Restart=on-failure (the default for many
operator-managed and tutorial-style unit files). The gateway drained,
exited cleanly (code 0), and was never revived — the only recovery was
a host reboot.
Root cause was a multi-layer assumption mismatch:
1. gateway/run.py:_stop_impl assumed all systemd units use
Restart=always, so the Linux/systemd branch returned exit code 0
and relied on a `systemd-run` transient helper to restart the unit
immediately. Units with Restart=on-failure never see a clean exit
as a trigger, so nothing revived the process.
2. gateway/run.py:_launch_systemd_restart_shortcut hardcoded
`--user` scope, so it could not even locate the unit PID on
system-level deployments (the common case for
/etc/systemd/system/hermes-gateway.service). It silently returned
without launching the helper.
3. Even after the scope detection was fixed, the helper could not
actually start: non-root gateway units (User=ubunutu) hit a Polkit
denial on `systemd-run --system` ("Interactive authentication
required"), and `--user` requires a D-Bus user session that is
typically absent on headless servers.
The fix is two-fold:
* `_stop_impl` now always exits with GATEWAY_SERVICE_RESTART_EXIT_CODE
(75 / EX_TEMPFAIL) on service-managed restarts, regardless of
platform. Combined with RestartForceExitStatus=75 in the unit file,
systemd treats the planned restart as a controlled failure and
revives the gateway via Restart=on-failure, with RestartSec as the
only delay. The planned-restart helper is still attempted (for
RestartSec=0 setups that want sub-second restarts) but is no longer
load-bearing.
* `_launch_systemd_restart_shortcut` now probes both system and user
scopes via MainPID equality and uses whichever scope actually owns
the gateway process. It bails out safely if neither matches.
StartLimitBurst in the unit file still bounds accidental restart
loops, and the macOS launchd path is unchanged.
Verified end-to-end on Ubuntu 24.04 with hermes-gateway as a
/etc/systemd/system/... service running under User=ubunutu. The
unit uses Restart=on-failure, RestartSec=30, RestartForceExitStatus=75,
StartLimitIntervalSec=600, StartLimitBurst=5. /restart from Feishu now
drains cleanly, exits 75, and the gateway is back online ~30s later
without manual intervention.
Tests: tests/gateway/test_gateway_shutdown.py renamed the affected
case to test_gateway_stop_systemd_service_restart_uses_tempfail and
now asserts exit_code == GATEWAY_SERVICE_RESTART_EXIT_CODE.
14/14 tests in this module pass.
- Correct the exit-75 comment: Hermes-generated units set StartLimitIntervalSec=0 (rate limiting disabled), so StartLimitBurst does not bound loops. The real bound is that genuine crashes exit non-zero-but-not-75, and RestartForceExitStatus=75 only whitelists the planned code. - Add randomuser2026x AUTHOR_MAP entry (CI blocks unmapped emails).
teknium1
force-pushed
the
hermes/hermes-d04d58a5
branch
from
July 1, 2026 12:12
a7189f4 to
507b7a4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/restartnow revives the gateway on systemd deployments it previously left dead: system-scope units and units usingRestart=on-failurerather thanRestart=always.Salvage of #50204 by @randomuser2026x, cherry-picked onto current
mainwith authorship preserved, plus a follow-up comment fix + AUTHOR_MAP entry.Root cause
Two independent gaps in the planned-restart path:
--user._launch_systemd_restart_shortcut()ransystemctl --user show ... MainPIDto confirm it owned the unit. On a system-unit deployment (/etc/systemd/system/hermes-gateway.service,User=/Group=) that returns an empty MainPID, the PID-equality check fails, and the helper silently no-ops — no relaunch is scheduled.Restart=always. The primary teardown path exited 0 on Linux systemd (conditional onINVOCATION_ID), which relies onRestart=alwaysto respawn. Operator-managed units usingRestart=on-failuretreat exit 0 as success and never restart — gateway stays dead until a reboot. (The fallback path at the bottom of the same function already exited 75 unconditionally, somainwas internally inconsistent.)Changes
gateway/run.py: probe bothsystemctl showandsystemctl --user show, pick the scope whose MainPID matches this PID (bail if neither does, rather than restart the wrong unit); thread the resolved scope throughsystemd-runand the reset-failed/restart commands.gateway/run.py: always set the service-restart exit code to 75.RestartForceExitStatus=75in the generated unit whitelists the planned code, so it revives under bothRestart=alwaysandRestart=on-failure; genuine crashes still exit non-zero-but-not-75.scripts/release.py: AUTHOR_MAP entry for the contributor (CI gate).Validation
/restart--userMainPID) → dead/restart--userunitRestart=on-failureunittests/gateway/test_gateway_shutdown.py: 20/20 pass (assertion updated to expect exit 75).systemctlproving all three scope branches (system picked, user picked, mismatch bails).Infographic
Nous Research