Skip to content

Fix/systemd restart on failure - #35899

Closed
wildekek wants to merge 3 commits into
NousResearch:mainfrom
wildekek:fix/systemd-restart-on-failure
Closed

Fix/systemd restart on failure#35899
wildekek wants to merge 3 commits into
NousResearch:mainfrom
wildekek:fix/systemd-restart-on-failure

Conversation

@wildekek

Copy link
Copy Markdown

What does this PR do?

Changes Restart=always to Restart=on-failure in the systemd unit templates generated by generate_systemd_unit().

The problem: Restart=always causes systemd to respawn the gateway service immediately after any exit, including the clean SIGTERM exit during system reboot/shutdown. When multiple gateway services are installed (e.g., per-profile gateways), they restart in a loop as fast as systemd can kill them, blocking the shutdown sequence and preventing the system from rebooting.

Why this approach: Restart=on-failure preserves all existing restart behavior for actual failures — crashes (non-zero exit), signal termination, and the existing RestartForceExitStatus=75 mechanism for deliberate graceful restarts via SIGUSR1 — but won't respawn after a clean exit (exit code 0 from SIGTERM during shutdown), so the system can reboot without fighting the service manager.

Related Issue

Fixes # — no existing issue filed. Discovered and fixed while debugging a Debian system with 3 gateway services that couldn't reboot.

Changes Made

  • hermes_cli/gateway.py (line 2220): Restart=always → Restart=on-failure in the system-level unit template
  • hermes_cli/gateway.py (line 2255): Restart=always → Restart=on-failure in the user-level unit template
  • hermes_cli/gateway.py (lines 210, 3239): Updated docstrings/comments referencing Restart=always to match

How to Test

  1. Run hermes gateway install — confirm generated unit has Restart=on-failure
  2. pytest tests/hermes_cli/test_gateway_service.py -k "systemd" -q — all 40 tests pass
  3. pytest tests/hermes_cli/test_gateway.py -q — all 33 tests pass
  4. (Manual) On a system with running gateway services: systemctl reboot should complete cleanly

teknium1 and others added 3 commits May 29, 2026 09:29
The v0.15.0 PyPI wheel shipped every plugin's Python code but none of its
plugin.yaml manifests, so plugin discovery (hermes_cli/plugins.py) found zero
plugins and ALL gateway platforms failed with "No adapter available for
<platform>" (discord, slack, mattermost, ...). Same gap also dropped the
web-search provider manifests (NousResearch#28149).

Declare manifest coverage in both packaging channels:
- wheel: [tool.setuptools.package-data] plugins += **/plugin.yaml, **/plugin.yml
- sdist: MANIFEST.in recursive-include plugins plugin.yaml plugin.yml
  (Homebrew and other downstream packagers build from the sdist)

Verified by building the wheel before/after: plugin.yaml count went 0 -> 69,
discord's manifest now ships. Adds a regression test asserting both channels
cover manifests.

Fixes NousResearch#34034

Co-authored-by: outsourc-e <201563152+outsourc-e@users.noreply.github.com>
Co-authored-by: Dhruvil Parikh <41384593+dparikh79@users.noreply.github.com>
Co-authored-by: ousiaresearch <261687298+ousiaresearch@users.noreply.github.com>
Co-authored-by: libre-7 <6366424+libre-7@users.noreply.github.com>
Restart=always causes systemd to respawn the gateway service
immediately after any exit, including during system reboot/shutdown
when systemd sends SIGTERM. This races with the shutdown sequence.

Restart=on-failure still restarts on crashes (non-zero exit,
signal termination, timeout) and keeps the RestartForceExitStatus=75
mechanism for deliberate graceful restarts via SIGUSR1. But it
won't respawn after a clean shutdown, so the system can reboot
without fighting the service manager.
@wildekek
wildekek requested a review from a team May 31, 2026 14:32
@alt-glitch alt-glitch added type/bug Something isn't working comp/cli CLI entry point, hermes_cli/, setup wizard comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists labels May 31, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

The Restart=on-failure change in this PR is already covered by #35102 (which also removes --replace from ExecStart and fixes the NixOS module) and #32989. The version bump, MANIFEST.in, and pyproject.toml packaging changes are unrelated to the systemd fix and should be a separate PR. Consider closing in favor of #35102.

@wildekek wildekek closed this May 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants