Skip to content

fix(gateway): replace blanket internal-restart guard with loop detector - #35815

Closed
drzeast-png wants to merge 1 commit into
NousResearch:mainfrom
drzeast-png:fix/restart-loop-detector
Closed

fix(gateway): replace blanket internal-restart guard with loop detector#35815
drzeast-png wants to merge 1 commit into
NousResearch:mainfrom
drzeast-png:fix/restart-loop-detector

Conversation

@drzeast-png

Copy link
Copy Markdown

Problem

The current _HERMES_GATEWAY guard (introduced in PR #35679 to fix #30719) blanket-refuses all stop/restart commands from inside the gateway process. This prevents legitimate manual restarts (e.g. from WeChat, Telegram) while KeepAlive is active, forcing users to SSH in or use an external terminal.

Additionally, this guard does not fully prevent the original cron + KeepAlive respawn loop — cron jobs run as subprocesses that inherit _HERMES_GATEWAY=1, but the guard triggers on the first restart attempt, not on the loop pattern itself.

Solution

Replace the blanket guard with a loop detector that tracks restart/stop timestamps in a persistent file (.gateway-restart-tracker under HERMES_HOME). Since the file lives on disk, it survives process death — critical for catching KeepAlive respawn cycles.

How it works

  • Each stop/restart call from inside the gateway (_HERMES_GATEWAY==1) checks the tracker
  • If ≥3 attempts occur in a 60-second sliding window → loop detected, operation refused
  • Single manual restarts always succeed (can't hit 3 in 60s by hand)
  • The tracker file persists across KeepAlive restarts, so cron-triggered restarts accumulate

Behavior comparison

Scenario Old guard New loop detector
Manual restart from WeChat/Telegram ❌ Blocked ✅ Allowed
cron fires restart once an hour ❌ Blocked (if inside gateway) ✅ Allowed
cron fires restart every 10s + KeepAlive (original #30719 loop) ❌ Blocks 1st attempt (breaks the loop but also breaks valid use) ✅ Blocks on 3rd attempt (breaks the loop, allows valid use)

Recovery

When the loop detector triggers, the error message includes recovery instructions:
launchctl unload ~/Library/LaunchAgents/ai.hermes.gateway.plist

Check ~/.hermes/.gateway-restart-tracker to inspect recent timestamps.

The current _HERMES_GATEWAY guard (PR NousResearch#35679) blanket-refuses all stop/
restart commands from inside the gateway process. This prevents legitimate
manual restarts (e.g. from WeChat, Telegram) while KeepAlive is active,
forcing users to SSH in or use an external terminal.

Replace with a smarter loop detector that tracks restart timestamps in a
persistent file. If ≥3 stop/restart attempts occur in a 60s sliding
window, a KeepAlive respawn cycle is detected and the operation is
refused. A single manual restart always succeeds.

This is defense-in-depth against the original issue NousResearch#30719 (cron +
KeepAlive respawn loop) without the false-positive rejection of valid
manual restarts. The tracker file survives process death, so cron-triggered
restarts are counted across respawns.
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery comp/cron Cron scheduler and job management labels May 31, 2026
@drzeast-png

Copy link
Copy Markdown
Author

Superseded — the loop-detector variant has been replaced with a more direct approach. See the new PR which simply drops the blanket guard entirely (Trust launchd KeepAlive as the only respawn policy).

Rationale for switching from loop-detector → drop-guard:

  1. The loop-detector tracks timestamps in a file, but the underlying issue is the guard itself preventing legitimate user restarts
  2. KeepAlive unconditional (PR fix(gateway): keep launchd KeepAlive unconditional for macOS resilience #35668) is already in flight, making the guard redundant
  3. macOS restart race fix (in flight separately) prevents the original kill-loop pattern at its source
  4. Simpler is better — fewer moving parts, no tracker file to inspect

Closing this in favor of the simpler approach. Thanks to anyone who took a look.

@drzeast-png drzeast-png closed this Jun 1, 2026
drzeast-png pushed a commit to drzeast-png/hermes-agent that referenced this pull request Jun 4, 2026
…estarts

Allow restart/stop from inside the gateway (e.g. via WeChat/Telegram)
so the user can manage the gateway remotely. Trust launchd KeepAlive
as the only respawn policy; the upstream blanket guard (NousResearch#35679) blocks
legitimate manual restarts. Supersedes the still-open PR NousResearch#35815
(loop-detector variant).
drzeast-png pushed a commit to drzeast-png/hermes-agent that referenced this pull request Jun 4, 2026
…estarts

Allow restart/stop from inside the gateway (e.g. via WeChat/Telegram)
so the user can manage the gateway remotely. Trust launchd KeepAlive
as the only respawn policy; the upstream blanket guard (NousResearch#35679) blocks
legitimate manual restarts. Supersedes the still-open PR NousResearch#35815
(loop-detector variant).
drzeast-png pushed a commit to drzeast-png/hermes-agent that referenced this pull request Jun 5, 2026
…estarts

Allow restart/stop from inside the gateway (e.g. via WeChat/Telegram)
so the user can manage the gateway remotely. Trust launchd KeepAlive
as the only respawn policy; the upstream blanket guard (NousResearch#35679) blocks
legitimate manual restarts. Supersedes the still-open PR NousResearch#35815
(loop-detector variant).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Agent can schedule gateway-restart cron job that kills its own runtime, creating respawn loop with launchctl/systemd KeepAlive

2 participants