Skip to content

fix(gateway,cron): prevent agent gateway-restart loops (salvage #30728) - #35679

Merged
teknium1 merged 2 commits into
mainfrom
hermes/hermes-e976ff2c
May 31, 2026
Merged

fix(gateway,cron): prevent agent gateway-restart loops (salvage #30728)#35679
teknium1 merged 2 commits into
mainfrom
hermes/hermes-e976ff2c

Conversation

@teknium1

Copy link
Copy Markdown
Contributor

Summary

Stops an agent from killing its own gateway in a SIGTERM-respawn loop. Two defenses: hermes gateway stop/restart refuse to run from inside the gateway process, and hermes cron create rejects prompts/scripts containing gateway-lifecycle commands.

Salvages #30728 by @SimoKiihamaki (fixes #30719) onto current main, with two follow-up fixes.

Root cause

Agent schedules a cron job that restarts the gateway (launchctl kickstart / hermes gateway restart) → gateway gets SIGTERM → launchd/systemd KeepAlive revives it → auto-resume re-runs the same session → loop every ~10s until manual kill. Real community incident (2026-05-22 and again 2026-05-25).

Changes

  • hermes_cli/gateway.py — guard stop/restart subcommands when running inside the gateway
  • hermes_cli/cron.py — regex pre-flight rejecting gateway-lifecycle commands in cron payloads
  • gateway/run.py — (contributor commit) original env marker; superseded below
  • tests/hermes_cli/test_gateway_restart_loop.py — 33 tests

Follow-up fixes on top of the salvage

  1. Reuse the existing marker. Gateway already exports _HERMES_GATEWAY=1 at startup (gateway/run.py:740) and cli.py already keys off it for the same "am I inside the gateway?" check. Dropped the redundant new HERMES_IN_GATEWAY var; guards now use _HERMES_GATEWAY. One marker for one fact.
  2. Tighten the cron regex. Removed the greedy \bgateway.*restart alternation — it false-positived on legitimate prompts that merely mention an unrelated gateway + a restart ("summarize the API gateway logs and report restart events", "check if the payment gateway needs a restart"). The specific hermes gateway (restart|stop|start) pattern already covers the real command.
  3. Reworked the two negative guard tests to sentinel the first downstream call (they previously drove real signal delivery and tripped the live-system test guard), and added false-positive regression cases.

Validation

Result
Targeted suite 33/33 passed
E2E cron filter blocks real launchctl/systemctl/hermes gateway restart payloads; allows legit gateway-mentioning prompts
E2E CLI guard hermes gateway stop/restart exit 1 with "Refusing…" when _HERMES_GATEWAY=1

Not in scope

Issue #30719 also proposed a loop detector (#3) and "don't auto-resume turns whose last tool calls touched gateway/cron" (#4). #4 is the narrowest belt-and-suspenders net — worth a follow-up, not needed here once the known scheduling paths are blocked.

Original PR: #30728. Commit authorship preserved via rebase merge.

Infographic

gateway-restart-loop-defense

…eway commands (#30719)

Three defenses against SIGTERM-respawn loops when agent schedules its
own gateway restart under launchd/systemd KeepAlive:

1. HERMES_IN_GATEWAY env var: gateway sets it at startup; stop/restart
   subcommands refuse to run when set (exit 1 with clear message).

2. Cron create payload filter: regex pre-flight rejects prompts/scripts
   containing hermes gateway restart/stop, launchctl kickstart/unload,
   systemctl restart/stop, and pkill patterns.

3. 30 new tests: pattern matching (14), cron block (5), gateway guard (4),
   safe command negatives (7).
@github-actions

github-actions Bot commented May 31, 2026

Copy link
Copy Markdown
Contributor

🔎 Lint report: hermes/hermes-e976ff2c vs origin/main

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 9526 on HEAD, 9525 on base (🆕 +1)

🆕 New issues (1):

Rule Count
unresolved-import 1
First entries
tests/hermes_cli/test_gateway_restart_loop.py:12: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`

✅ Fixed issues: none

Unchanged: 4945 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

@teknium1
teknium1 force-pushed the hermes/hermes-e976ff2c branch from b00becb to 014de2b Compare May 31, 2026 04:06
…n regex

Follow-up to the salvaged #30728:
- Gateway already exports _HERMES_GATEWAY=1 at startup (gateway/run.py) and
  cli.py already keys off it. Drop the redundant new HERMES_IN_GATEWAY var;
  guard stop/restart on _HERMES_GATEWAY instead. One marker for one fact.
- Drop the greedy \bgateway.*restart alternation from the cron lifecycle
  filter — it false-positived on legit prompts that merely mention an
  unrelated gateway + a restart (API/payment gateway monitoring). The
  specific 'hermes gateway (restart|stop|start)' pattern already covers the
  real command.
- Rework the two negative guard tests to sentinel the first downstream call
  so they don't drive real signal delivery (tripped the live-system guard).
- Add false-positive regression cases to test_safe_commands.
@teknium1
teknium1 force-pushed the hermes/hermes-e976ff2c branch from 014de2b to 0a09ed3 Compare May 31, 2026 04:12
@alt-glitch alt-glitch added type/bug Something isn't working comp/gateway Gateway runner, session dispatch, delivery comp/cron Cron scheduler and job management comp/cli CLI entry point, hermes_cli/, setup wizard P1 High — major feature broken, no workaround labels May 31, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Duplicate of #33395 — both salvage #30728 to fix #30719 (agent gateway-restart SIGTERM-respawn loop). This PR reuses _HERMES_GATEWAY env marker and tightens the cron regex; #33395 is the earlier submission with the same scope.

@teknium1
teknium1 merged commit bd72d33 into main May 31, 2026
23 checks passed
@teknium1
teknium1 deleted the hermes/hermes-e976ff2c branch May 31, 2026 06:05
drzeast-png pushed a commit to drzeast-png/hermes-agent that referenced this pull request Jun 4, 2026
…estarts

Allow restart/stop from inside the gateway (e.g. via WeChat/Telegram)
so the user can manage the gateway remotely. Trust launchd KeepAlive
as the only respawn policy; the upstream blanket guard (NousResearch#35679) blocks
legitimate manual restarts. Supersedes the still-open PR NousResearch#35815
(loop-detector variant).
drzeast-png pushed a commit to drzeast-png/hermes-agent that referenced this pull request Jun 4, 2026
…estarts

Allow restart/stop from inside the gateway (e.g. via WeChat/Telegram)
so the user can manage the gateway remotely. Trust launchd KeepAlive
as the only respawn policy; the upstream blanket guard (NousResearch#35679) blocks
legitimate manual restarts. Supersedes the still-open PR NousResearch#35815
(loop-detector variant).
drzeast-png pushed a commit to drzeast-png/hermes-agent that referenced this pull request Jun 5, 2026
…estarts

Allow restart/stop from inside the gateway (e.g. via WeChat/Telegram)
so the user can manage the gateway remotely. Trust launchd KeepAlive
as the only respawn policy; the upstream blanket guard (NousResearch#35679) blocks
legitimate manual restarts. Supersedes the still-open PR NousResearch#35815
(loop-detector variant).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard comp/cron Cron scheduler and job management comp/gateway Gateway runner, session dispatch, delivery P1 High — major feature broken, no workaround type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Agent can schedule gateway-restart cron job that kills its own runtime, creating respawn loop with launchctl/systemd KeepAlive

3 participants