Skip to content

fix(gateway): make intentional systemd stops exit cleanly - #24351

Open
heathley wants to merge 1 commit into
NousResearch:mainfrom
heathley:fix/systemd-gateway-stop-exit-status-24344
Open

heathley wants to merge 1 commit into
NousResearch:mainfrom
heathley:fix/systemd-gateway-stop-exit-status-24344

Conversation

@heathley

Copy link
Copy Markdown

Summary

Fix intentional systemd gateway stops being recorded as failures.

Today there are two separate failure paths behind this:

  1. hermes gateway stop --system writes the planned-stop marker from a sudo/root context, so the marker can end up unreadable to the actual gateway service user.
  2. Direct systemctl stop never writes a planned-stop marker at all, so the gateway cannot distinguish an intentional stop from an unexpected SIGTERM and exits with status 1 by design.

This change moves the correctness path into the systemd unit itself by adding a non-recursive ExecStop helper that writes the planned-stop marker as the service user before systemd delivers SIGTERM.

Closes #24344.

What changed

  • Added an internal hermes gateway _write-planned-stop --pid <pid> helper
  • Added ExecStop=... gateway _write-planned-stop --pid $MAINPID to generated systemd units
  • Updated the existing CLI stop prewrite path to reuse the same helper
  • Relaxed planned-stop marker read permissions after writing so cross-user stop paths remain readable by the gateway service process
  • Added regression coverage for:
    • generated systemd unit includes the ExecStop helper
    • planned-stop helper writes a readable marker
    • systemd_stop() marks the stop as planned before stopping
    • internal helper command success/failure dispatch

Why this approach

This preserves Hermes' existing non-zero exit behavior for genuinely unexpected SIGTERM, which is still needed for Restart=on-failure recovery.

It intentionally does not use SuccessExitStatus=1, since that would mask real unexpected shutdown failures.

Verification

Tested with targeted regression coverage:

  • tests/hermes_cli/test_gateway_service.py -k "execstop_helper or planned_stop_only or systemd_stop"
  • tests/hermes_cli/test_gateway.py -k "write_planned_stop_internal_command"

@heathley
heathley force-pushed the fix/systemd-gateway-stop-exit-status-24344 branch from 7a6f82c to a0b94c7 Compare May 12, 2026 12:32
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery comp/cli CLI entry point, hermes_cli/, setup wizard labels May 12, 2026
@teknium1

Copy link
Copy Markdown
Collaborator

Thanks for tracing both the root-owned marker path and direct systemctl stop path. The premise remains valid on current main: gateway/run.py:20490-20539 treats an unmarked SIGTERM as unexpected, and gateway/run.py:20799-20804 exits 1; generated units at hermes_cli/gateway.py:2753-2801 have no ExecStop marker writer.

Problems

  • Current main deliberately omits ExecStop in both templates (tests/hermes_cli/test_gateway_service.py:426-449, :537-561) after commit 002c45998, which removed a recursive hermes gateway stop ExecStop. The proposed helper is non-recursive, but this requires a deliberate salvage that preserves that distinction rather than a clean application of the stale patch.
  • The added generation coverage checks only the system unit even though the diff adds ExecStop to both user and system templates.

Suggested changes

  • Port the marker-only helper to the current templates and replace both existing no-ExecStop regressions with assertions that it cannot re-enter service management.
  • Cover both unit scopes and the hidden command dispatch.

Automated hermes-sweeper review.

@teknium1 teknium1 added sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 13, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Systemd gateway service exits with status 1 on intentional stop

3 participants