Skip to content

fix(gateway): refresh systemd units before update restart - #34464

Open
leprincep35700 wants to merge 1 commit into
NousResearch:mainfrom
leprincep35700:fix/update-gateway-restart-docker-safe
Open

fix(gateway): refresh systemd units before update restart#34464
leprincep35700 wants to merge 1 commit into
NousResearch:mainfrom
leprincep35700:fix/update-gateway-restart-docker-safe

Conversation

@leprincep35700

@leprincep35700 leprincep35700 commented May 29, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Mark the Gateway systemd drain exit code 75 as a successful service exit status.
  • Before hermes update restarts Gateway services, refresh every discovered hermes-gateway*.service unit rather than only the active profile.
  • Read each installed unit’s pinned HERMES_HOME, regenerate that unit in its own profile scope, and restore the caller environment afterwards.
  • Keep refresh best-effort for permission-limited system services.
  • Reject explicit service names that could escape the systemd unit directory.

Why

An update can pull the fix for a stale unit but still restart through that stale unit. With multiple profiles, refreshing only the caller’s unit leaves hermes-gateway-coder, hermes-gateway-assistant, and other discovered services stale before they are signaled. This change refreshes each exact unit before restarting it.

Fixes #31048.

Review resolution

The multi-profile review is addressed in ff7b1f2fd501: the regression test installs default and coder units with different pinned homes, refreshes both by service name, and asserts that each generated unit retains its own home. A separate test covers traversal-style explicit service names. The review thread has been replied to and resolved.

Validation

Focused update/systemd suite: 38 passed
Ruff: passed
py_compile: passed
git diff --check: passed
Independent re-review: PASS
GitHub checks: 23 success, 7 skipped, 1 neutral, no failure/pending

Rebased head: ff7b1f2fd501; main base: c44de9985.

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/cli CLI entry point, hermes_cli/, setup wizard comp/gateway Gateway runner, session dispatch, delivery labels May 29, 2026
@leprincep35700
leprincep35700 force-pushed the fix/update-gateway-restart-docker-safe branch from 689e235 to 09dfa14 Compare May 29, 2026 10:41
@leprincep35700

Copy link
Copy Markdown
Contributor Author

Post-current-main reassessment:

I rechecked this after the recent gateway/service restart work landed, especially #36188, #35679, #34805, and related gateway fixes.

Conclusion: this PR still has a useful remaining scope, but the framing should be updated.

What upstream now covers:

  • fix(gateway): clean service restart flow #36188 cleaned up the service-managed planned restart flow and makes the systemd path exit cleanly with 0 in the normal managed case.
  • That makes SuccessExitStatus=75 less central than when this PR was opened.

What is still not covered on current main:

Recommendation:

  • Keep this PR, but rebase/update it against current main.
  • Present the primary value as “refresh systemd units before update restart so unit-template fixes are applied before restart”.
  • Treat SuccessExitStatus=75 as compatibility/harmless cleanup rather than the main fix after fix(gateway): clean service restart flow #36188.

@leprincep35700
leprincep35700 force-pushed the fix/update-gateway-restart-docker-safe branch 2 times, most recently from a54f456 to 3899412 Compare June 1, 2026 09:13
@leprincep35700

Copy link
Copy Markdown
Contributor Author

Rebased onto current main and updated the wording/scope around the newer restart semantics.

  • New head: 38994129c
  • The refreshed branch keeps the surviving value: best-effort systemd unit refresh before update-triggered gateway restarts, so unit-template fixes are applied before the service is signaled.
  • Local validation:
    • /opt/venv/bin/python -m pytest tests/hermes_cli/test_cmd_update.py::TestCmdUpdateSystemdRestartPrep -q -o addopts='' --tb=short2 passed
    • /opt/venv/bin/python -m py_compile hermes_cli/main.py hermes_cli/gateway.py tests/hermes_cli/test_cmd_update.py tests/hermes_cli/test_gateway_service.py → OK
    • git diff --check → OK
  • Note: I also tried the broader tests/hermes_cli/test_gateway_service.py; in this Docker/root environment several pre-existing systemd/DBus/root-service tests fail because systemctl --user cannot reach a user D-Bus session / root system-service install is refused. The new scoped regression tests above pass.
  • Attribution was refreshed to the mapped GitHub noreply author email.

@leprincep35700
leprincep35700 force-pushed the fix/update-gateway-restart-docker-safe branch 2 times, most recently from bf1503a to 22b04ec Compare June 9, 2026 09:51

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for preserving a real update/restart failure mode: current main still enumerates active hermes-gateway* systemd services and restarts them without a pre-restart unit refresh (hermes_cli/main.py:10560-10604). That matters because a stale WorkingDirectory can fail at systemd's CHDIR step before gateway boot-time refresh code runs (hermes_cli/gateway.py:2667-2671).

Problems

  • The proposed helper refreshes only the active profile's two scope paths (hermes_cli/main.py:7831-7834 in PR head). refresh_systemd_unit_if_needed() derives a single service path from the current HERMES_HOME (hermes_cli/gateway.py:1778-1795 on main), while the update path restarts every discovered hermes-gateway* unit. Additional profile services can therefore still restart through stale units.

Suggested changes

  • Refresh each discovered service/profile before its update-triggered signal/restart, rather than refreshing only the caller's active profile.
  • Add a multi-profile regression test proving each selected unit refreshes before restart.

Automated hermes-sweeper review.

Comment thread hermes_cli/main.py Outdated
@teknium1 teknium1 added sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 13, 2026
@leprincep35700
leprincep35700 force-pushed the fix/update-gateway-restart-docker-safe branch from 22b04ec to ff7b1f2 Compare July 13, 2026 20:54
@leprincep35700

Copy link
Copy Markdown
Contributor Author

Rebased onto current main (c44de9985) and addressed the multi-profile systemd review. hermes update now refreshes every discovered named unit before restart, reads each unit's pinned HERMES_HOME, regenerates it in that profile scope, restores the caller environment, and rejects service names that could escape the systemd unit directory. The central test now asserts that default and coder units each retain their own home.

Validation:

  • focused update/systemd suite: 38 passed
  • Ruff, py_compile, and git diff --check: passed
  • independent re-review: PASS

Updated head: ff7b1f2fd501.

@teknium1 teknium1 added the area/install-update Installer, updater, packaging, wheels, doctor label Jul 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/install-update Installer, updater, packaging, wheels, doctor comp/cli CLI entry point, hermes_cli/, setup wizard comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hermes update fails to restart gateway on Linux systemd (exit code 75 handoff broken)

3 participants