Skip to content

fix(gateway): use via_service for macOS restart to avoid KeepAlive race - #37063

Open
drzeast-png wants to merge 3 commits into
NousResearch:mainfrom
drzeast-png:pr/macos-via-service
Open

fix(gateway): use via_service for macOS restart to avoid KeepAlive race#37063
drzeast-png wants to merge 3 commits into
NousResearch:mainfrom
drzeast-png:pr/macos-via-service

Conversation

@drzeast-png

Copy link
Copy Markdown

Problem

On macOS launchd, KeepAlive=true restarts the gateway on any exit (same semantics as systemd Restart=always). However, the in-chat /restart path was hardcoded to use detached=True, which spawns a shell watcher that:

  1. Polls the old PID
  2. After the old gateway exits, runs hermes gateway restart

This races with launchd's KeepAlive:

  • Watcher starts → old gateway is dying
  • KeepAlive (which is faster) launches a new gateway first
  • Watcher sees old PID gone, thinks it's safe to run hermes gateway restart
  • New gateway gets SIGTERM → dies
  • KeepAlive restarts again → another watcher spawns
  • kill loop that eventually unloads the launchd job

Solution

Make Darwin match systemd by using via_service=True so the gateway exits cleanly and launchd restarts it natively. No watcher, no hermes gateway restart CLI, no racing.

Diff (1 line, in gateway/run.py)

- if _under_service or _in_container:
+ if _under_service or _in_container or sys.platform == "darwin":
      self.request_restart(detached=False, via_service=True)

Behavior comparison

Scenario Before (detached=True) After (via_service=True)
Linux systemd restart ✅ via_service (correct) ✅ via_service
Linux Docker restart ✅ via_service (correct) ✅ via_service
macOS launchd restart ❌ detached watcher → race ✅ via_service
Manual restart (no supervisor) ✅ detached watcher (correct) ✅ detached watcher

Test plan

  • Trigger /restart from WeChat on macOS — gateway restarts cleanly, no kill loop
  • launchd launchctl list | grep hermes shows new PID after restart
  • No 'detached helper' log lines after restart
  • Companion PR (drop _HERMES_GATEWAY guard) unblocks the restart path itself

@liuhao1024

Copy link
Copy Markdown
Contributor

I found an issue that looks worth fixing before merge.

gateway/run.py:10627

The change if _under_service or _in_container or sys.platform == "darwin" unconditionally treats every macOS process as service-managed. When via_service=True, the gateway exits with code 75 (SystemExit(75) at line 19276) — this only works when a service manager (launchd's KeepAlive) is watching. A bare macOS Terminal process (hermes gateway) would exit 75 with nothing to restart it.

Why it matters: Any macOS user running the gateway from Terminal (not under a launchd plist) would lose their gateway on /restart — it exits 75 and stays dead.

Suggested fix: Detect actual launchd management instead of blanket-darwin. PR #37094 (same issue, same file) does this correctly by checking XPC_SERVICE_NAME:

_under_launchd = sys.platform == "darwin" and os.environ.get(
    "XPC_SERVICE_NAME", "0"
) not in ("", "0")

launchd sets XPC_SERVICE_NAME to the job label (e.g. ai.hermes.gateway-alex) for managed jobs and "0" for shell-launched processes. This is the same signal used by the systemd INVOCATION_ID check two lines above.

@liuhao1024

Copy link
Copy Markdown
Contributor

I found an issue with the scope of this fix.

gateway/run.py:10627 — the condition sys.platform == "darwin" applies via_service=True (exit 0) to all macOS processes, including bare shell-launched ones that are NOT managed by launchd. For a shell-launched gateway, exiting 0 means nothing relaunches it — the gateway stays dead until manual recovery.

The comment says "Launchd on macOS has KeepAlive=true which restarts the gateway on any exit" — but this is only true for launchd-managed jobs. A bare python gateway/run.py process on macOS has no KeepAlive; the via_service=True path would silently kill it.

There's a more precise fix in PR #37094 that checks XPC_SERVICE_NAME (which launchd sets to the job label for managed processes, and "0" for shell-launched ones):

_under_launchd = sys.platform == "darwin" and os.environ.get(
    "XPC_SERVICE_NAME", "0"
) not in ("", "0")

This correctly detects launchd-managed processes without affecting bare shell-launched ones. I'd recommend closing this in favor of #37094.

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery labels Jun 2, 2026
@drzeast-png
drzeast-png force-pushed the pr/macos-via-service branch 5 times, most recently from 8e336b2 to 334bd43 Compare June 4, 2026 11:37
@drzeast-png
drzeast-png requested a review from a team June 4, 2026 11:37
@drzeast-png
drzeast-png force-pushed the pr/macos-via-service branch 11 times, most recently from eb19434 to 6a91e1c Compare June 5, 2026 05:30
张东 and others added 3 commits June 5, 2026 17:18
…estarts

Allow restart/stop from inside the gateway (e.g. via WeChat/Telegram)
so the user can manage the gateway remotely. Trust launchd KeepAlive
as the only respawn policy; the upstream blanket guard (NousResearch#35679) blocks
legitimate manual restarts. Supersedes the still-open PR NousResearch#35815
(loop-detector variant).
iLink can return ret=-3 (not just ret=-2/errcode=-14) to indicate a
stale context_token that needs to be refreshed.  Without this fix the
adapter retries 5 times with a dead token, then gives up — causing
intermittent send failures for cron deliveries and other bot-initiated
messages.

Also adds debug logging to _api_post for sendmessage responses so
future session issues are easier to diagnose.
On macOS launchd, KeepAlive=true restarts the gateway on any exit (same
semantics as systemd Restart=always).  Previously the in-chat /restart
path used detached=True, spawning a shell watcher that polls the old PID
and runs 'hermes gateway restart' after exit. This races with launchd's
KeepAlive — the watcher starts after KeepAlive has already launched a
new gateway, then sends SIGTERM to the new instance, causing a kill loop
and eventually unloading the launchd job.

Fix: make Darwin match systemd by using via_service=True so the gateway
exits with code 75 and launchd natively restarts it. No watcher, no
'hermes gateway restart' CLI, no racing.

@austinpickett austinpickett left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Approved

Three complementary fixes in one coherent PR from the same author:

  1. via_service=True on macOS (gateway/run.py): launchd KeepAlive restarts the gateway on ANY exit (same semantics as Restart=always). Adding sys.platform == "darwin" to the existing _under_service or _in_container gate routes Darwin through request_restart(via_service=True) — clean exit, no detached watcher shell racing with KeepAlive. Correct fix.

  2. Remove _HERMES_GATEWAY blanket guard (hermes_cli/gateway.py): With via_service now used on macOS, the guard is no longer needed there. On Linux/systemd the INVOCATION_ID path already handles it. The guard was incorrectly blocking legitimate in-chat hermes gateway restart from messaging platforms.

  3. WeChat _is_stale_session_ret hardening: ret=-3 is exclusive to stale-session (never rate-limiting), so removing the errmsg gate for it is correct. Explanatory docstring added.

Note: See my comment on #37057 (companion PR) — recommend closing that in favor of this one which is the complete fix.

@teknium1

Copy link
Copy Markdown
Contributor

Thanks for the macOS restart investigation. The title fix is now implemented on current main through a more precise launchd probe.

Problems

  • The PR’s blanket Darwin condition would route interactive macOS gateways through the service path. Current main checks XPC_SERVICE_NAME at gateway/slash_commands.py:1302-1309; tests/gateway/test_restart_service_detection.py:57-64 verifies that XPC_SERVICE_NAME="0" stays on detached restart.
  • Commit abc3662bf6076045e4d4dc1e14a74cb35d69b86e (/restart bricks a launchd-managed gateway on macOS — exits 0, KeepAlive.SuccessfulExit=false won't revive it #43475) already delivers the launchd behavior and coverage, after this handler moved out of gateway/run.py.
  • The Weixin and _HERMES_GATEWAY guard changes are separate concerns and are not covered by the superseding macOS fix.

Suggested changes

  • Re-scope any salvage to independently justified Weixin behavior with focused regression tests; do not retain the superseded blanket-Darwin hunk.
  • Evaluate the guard removal separately against current main’s restart-loop protections.

Automated hermes-sweeper review.

@teknium1 teknium1 added sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants