fix(gateway,cron): guard cron model-tool path + auto-resume loop breaker (#30719) - #56240
Merged
Conversation
…breaker (#30719) Completes the #30719 restart-loop defenses. Defenses 1-2 (the _HERMES_GATEWAY guard on `hermes gateway stop|restart` + terminal_tool, and the cron-creation lifecycle filter) already landed on main, but two gaps remained: - The agent's `cronjob` model tool calls cron.jobs.create_job directly, bypassing the hermes_cli.cron.cron_create CLI filter, so lifecycle commands scheduled via the model tool were only blocked at execution time (terminal_tool), not at creation. Moved the filter to a shared cron/lifecycle_guard.py enforced at create_job — the single chokepoint every job-creation path hits (CLI + model tool). Re-exported _contains_gateway_lifecycle_command from hermes_cli.cron so terminal_tool's import keeps working. - No breaker for the auto-resume loop itself. Defenses 1-2 cover the cron/CLI/terminal paths, but any other SIGTERM source (e.g. a raw terminal("launchctl kickstart ai.hermes.gateway")) still triggers the boot->auto-resume->re-run cycle. Added gateway/restart_loop_guard.py: counts restart-interrupted boots in a rolling window (config gateway.restart_loop_guard, default 3 boots / 60s) and skips auto-resume for that boot once tripped. The gateway still comes up and serves real inbound messages; it just stops replaying the session that keeps killing it, putting a human back in the loop. Also tightened the lifecycle regex over main's version: dropped `hermes gateway start` (benign), required the gateway identifier on the launchctl/systemctl branches (so `launchctl unload ai.hermes.update-checker.plist` and `systemctl restart hermes-meta.service` no longer false-positive), added the inverse pkill token order, and fixed the binary-script bypass (decode with errors='replace' instead of swallowing UnicodeDecodeError). The create_job guard resolves relative script paths under HERMES_HOME/scripts the same way the scheduler does, so a bare script name is scanned as the file that actually runs. Design and much of defense-2 originate from PR #33395 (@kshitijk4poor), which itself salvaged #30728 (@SimoKiihamaki). Rebuilt against current main since defenses 1-2 had already landed under different names. Closes #30719. Co-authored-by: SimoKiihamaki <simo.kiihamaki@gmail.com> Co-authored-by: kshitijk4poor <82637225+kshitijk4poor@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Completes the #30719 gateway restart-loop defenses. Defenses 1–2 already landed on
mainunder different names; this closes the two remaining gaps and hardens the filter.The foot-gun: an agent schedules
hermes gateway restart(orlaunchctl kickstart ai.hermes.gateway), the cron fires → SIGTERM → supervisor KeepAlive revives the gateway → auto-resume replays the offending session → the turn re-runs the same logic. A ~10s SIGTERM-respawn loop until broken by hand (real macOS repro, 2026-05-22).What was already on main
_HERMES_GATEWAY=1guard:hermes gateway stop|restartrefuse from inside the gateway, andterminal_toolhard-blocks lifecycle commands at execution time.hermes_cli.cron.cron_createrejected lifecycle prompts/scripts.What this PR adds
cronjobmodel tool callscron.jobs.create_jobdirectly, bypassing the CLI filter, so model-tool-scheduled lifecycle jobs were only caught at execution. Moved the filter into a sharedcron/lifecycle_guard.pyenforced atcreate_job— the single path every caller (CLI + model tool) hits._contains_gateway_lifecycle_commandis re-exported fromhermes_cli.cronsoterminal_tool's import is unchanged. Relative script paths resolve underHERMES_HOME/scriptsexactly as the scheduler runs them, so a bare script name is scanned as the file that actually executes.gateway/restart_loop_guard.pycounts restart-interrupted boots in a rolling window (gateway.restart_loop_guard, default 3 boots / 60s) and skips auto-resume for that boot once tripped. The gateway still comes up and serves real inbound messages — it just stops replaying the session that keeps killing it, putting a human back in the loop. Catches every SIGTERM source defenses 1–2 don't (e.g. a rawterminal("launchctl kickstart …")). Fails open on any state error.hermes gateway start; require the gateway identifier on the launchctl/systemctl branches (launchctl unload ai.hermes.update-checker.plistandsystemctl restart hermes-meta.serviceno longer false-positive); added the inversepkill gateway … hermestoken order; fixed the binary-script bypass (decodeerrors="replace"instead of swallowingUnicodeDecodeError).Changes
cron/lifecycle_guard.py(new): shared checker +check_gateway_lifecycleraisingGatewayLifecycleBlocked.cron/jobs.py: enforce the guard increate_job.hermes_cli/cron.py: re-export the checker; drop the now-redundant CLI-layer guard.gateway/restart_loop_guard.py(new) +gateway/run.py: loop breaker + wiring in_schedule_resume_pending_sessions.hermes_cli/config.py:gateway.restart_loop_guarddefaults.tests/hermes_cli/test_gateway_restart_loop.pyextended.Validation
cronjobcreate (prompt + script abuse)Credit
Design and much of defense-2 originate from PR #33395 (@kshitijk4poor), which salvaged #30728 (@SimoKiihamaki). Rebuilt against current
mainbecause defenses 1–2 had already landed under different names (_HERMES_GATEWAY), so cherry-picking #33395 wholesale would have re-added a duplicateHERMES_IN_GATEWAYenv flag. Both authors preserved asCo-authored-by.Closes #30719.
Infographic