Skip to content

fix(gateway,cron): guard cron model-tool path + auto-resume loop breaker (#30719) - #56240

Merged
teknium1 merged 1 commit into
mainfrom
hermes/hermes-d60c9f0b
Jul 1, 2026
Merged

fix(gateway,cron): guard cron model-tool path + auto-resume loop breaker (#30719)#56240
teknium1 merged 1 commit into
mainfrom
hermes/hermes-d60c9f0b

Conversation

@teknium1

@teknium1 teknium1 commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Summary

Completes the #30719 gateway restart-loop defenses. Defenses 1–2 already landed on main under different names; this closes the two remaining gaps and hardens the filter.

The foot-gun: an agent schedules hermes gateway restart (or launchctl kickstart ai.hermes.gateway), the cron fires → SIGTERM → supervisor KeepAlive revives the gateway → auto-resume replays the offending session → the turn re-runs the same logic. A ~10s SIGTERM-respawn loop until broken by hand (real macOS repro, 2026-05-22).

What was already on main

  • Defense 1_HERMES_GATEWAY=1 guard: hermes gateway stop|restart refuse from inside the gateway, and terminal_tool hard-blocks lifecycle commands at execution time.
  • Defense 2 (CLI layer)hermes_cli.cron.cron_create rejected lifecycle prompts/scripts.

What this PR adds

  • Defense 2 → chokepoint. The agent's cronjob model tool calls cron.jobs.create_job directly, bypassing the CLI filter, so model-tool-scheduled lifecycle jobs were only caught at execution. Moved the filter into a shared cron/lifecycle_guard.py enforced at create_job — the single path every caller (CLI + model tool) hits. _contains_gateway_lifecycle_command is re-exported from hermes_cli.cron so terminal_tool's import is unchanged. Relative script paths resolve under HERMES_HOME/scripts exactly as the scheduler runs them, so a bare script name is scanned as the file that actually executes.
  • Defense 3 (new): auto-resume loop breaker. gateway/restart_loop_guard.py counts restart-interrupted boots in a rolling window (gateway.restart_loop_guard, default 3 boots / 60s) and skips auto-resume for that boot once tripped. The gateway still comes up and serves real inbound messages — it just stops replaying the session that keeps killing it, putting a human back in the loop. Catches every SIGTERM source defenses 1–2 don't (e.g. a raw terminal("launchctl kickstart …")). Fails open on any state error.
  • Regex tightened over main's version: dropped benign hermes gateway start; require the gateway identifier on the launchctl/systemctl branches (launchctl unload ai.hermes.update-checker.plist and systemctl restart hermes-meta.service no longer false-positive); added the inverse pkill gateway … hermes token order; fixed the binary-script bypass (decode errors="replace" instead of swallowing UnicodeDecodeError).

Changes

  • cron/lifecycle_guard.py (new): shared checker + check_gateway_lifecycle raising GatewayLifecycleBlocked.
  • cron/jobs.py: enforce the guard in create_job.
  • hermes_cli/cron.py: re-export the checker; drop the now-redundant CLI-layer guard.
  • gateway/restart_loop_guard.py (new) + gateway/run.py: loop breaker + wiring in _schedule_resume_pending_sessions.
  • hermes_cli/config.py: gateway.restart_loop_guard defaults.
  • tests: tests/hermes_cli/test_gateway_restart_loop.py extended.

Validation

Result
Targeted suite (restart-loop + gateway service + cronjob tools) 318 passed, 0 failed
Regression sweep (config + terminal_tool + cron/) 753 passed, 0 failed
E2E: agent cronjob create (prompt + script abuse) blocked with #30719 hint
E2E: loop breaker (burst trips / spread ignored / disable) correct
ruff clean

Credit

Design and much of defense-2 originate from PR #33395 (@kshitijk4poor), which salvaged #30728 (@SimoKiihamaki). Rebuilt against current main because defenses 1–2 had already landed under different names (_HERMES_GATEWAY), so cherry-picking #33395 wholesale would have re-added a duplicate HERMES_IN_GATEWAY env flag. Both authors preserved as Co-authored-by.

Closes #30719.

Infographic

restart-loop breaker

…breaker (#30719)

Completes the #30719 restart-loop defenses. Defenses 1-2 (the
_HERMES_GATEWAY guard on `hermes gateway stop|restart` + terminal_tool,
and the cron-creation lifecycle filter) already landed on main, but two
gaps remained:

- The agent's `cronjob` model tool calls cron.jobs.create_job directly,
  bypassing the hermes_cli.cron.cron_create CLI filter, so lifecycle
  commands scheduled via the model tool were only blocked at execution
  time (terminal_tool), not at creation. Moved the filter to a shared
  cron/lifecycle_guard.py enforced at create_job — the single chokepoint
  every job-creation path hits (CLI + model tool). Re-exported
  _contains_gateway_lifecycle_command from hermes_cli.cron so
  terminal_tool's import keeps working.
- No breaker for the auto-resume loop itself. Defenses 1-2 cover the
  cron/CLI/terminal paths, but any other SIGTERM source (e.g. a raw
  terminal("launchctl kickstart ai.hermes.gateway")) still triggers the
  boot->auto-resume->re-run cycle. Added gateway/restart_loop_guard.py:
  counts restart-interrupted boots in a rolling window (config
  gateway.restart_loop_guard, default 3 boots / 60s) and skips
  auto-resume for that boot once tripped. The gateway still comes up and
  serves real inbound messages; it just stops replaying the session that
  keeps killing it, putting a human back in the loop.

Also tightened the lifecycle regex over main's version: dropped
`hermes gateway start` (benign), required the gateway identifier on the
launchctl/systemctl branches (so `launchctl unload
ai.hermes.update-checker.plist` and `systemctl restart
hermes-meta.service` no longer false-positive), added the inverse
pkill token order, and fixed the binary-script bypass (decode with
errors='replace' instead of swallowing UnicodeDecodeError). The
create_job guard resolves relative script paths under HERMES_HOME/scripts
the same way the scheduler does, so a bare script name is scanned as the
file that actually runs.

Design and much of defense-2 originate from PR #33395 (@kshitijk4poor),
which itself salvaged #30728 (@SimoKiihamaki). Rebuilt against current
main since defenses 1-2 had already landed under different names.

Closes #30719.

Co-authored-by: SimoKiihamaki <simo.kiihamaki@gmail.com>
Co-authored-by: kshitijk4poor <82637225+kshitijk4poor@users.noreply.github.com>
@alt-glitch alt-glitch added type/bug Something isn't working comp/gateway Gateway runner, session dispatch, delivery comp/cron Cron scheduler and job management sweeper:risk-automation Sweeper risk: may affect CI, automerge, label sync, or maintainer automation P2 Medium — degraded but workaround exists labels Jul 1, 2026
@teknium1
teknium1 merged commit b48cacb into main Jul 1, 2026
31 checks passed
@teknium1
teknium1 deleted the hermes/hermes-d60c9f0b branch July 1, 2026 09:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:risk-automation Sweeper risk: may affect CI, automerge, label sync, or maintainer automation type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Agent can schedule gateway-restart cron job that kills its own runtime, creating respawn loop with launchctl/systemd KeepAlive

2 participants