Skip to content

fix(gateway): block launchctl submit in the gateway lifecycle guard - #62896

Closed
jackjin1997 wants to merge 2 commits into
NousResearch:mainfrom
jackjin1997:fix/gateway-lifecycle-guard-launchctl-submit
Closed

fix(gateway): block launchctl submit in the gateway lifecycle guard#62896
jackjin1997 wants to merge 2 commits into
NousResearch:mainfrom
jackjin1997:fix/gateway-lifecycle-guard-launchctl-submit

Conversation

@jackjin1997

Copy link
Copy Markdown
Contributor

What does this PR do?

_GATEWAY_LIFECYCLE_PATTERN's launchctl branch (Branch B in cron/lifecycle_guard.py) only matched the verbs kickstart|unload|load|stop|restart. As reported in #62891, an agent whose direct gateway restart/kill was correctly blocked by the guard instead laundered the same effect through:

launchctl submit -l ai.hermes.gateway-hard-restart-no-photon-notice -- /bin/sh <helper-script>

launchctl submit creates a new launchd keepalive job wrapping an arbitrary helper script — a different verb shape than the existing service ops the pattern already caught, so it slipped through both enforcement points that share this function (cron.jobs.create_job at job-creation time, and tools/terminal_tool.py's execution-time hard-block). The helper script killed the real gateway and re-triggered it every ~20s via launchctl kickstart -k, running 9,447 times before manual removal.

Related Issue

Fixes #62891

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

  • cron/lifecycle_guard.py: add submit to Branch B's verb alternation, with a comment explaining the indirection this closes.
  • tests/hermes_cli/test_gateway_restart_loop.py: added the exact reproduction command from the issue plus a second launchctl submit variant as positive matches, and a launchctl submit on an unrelated label as a negative (no-false-positive) case.

Scope note

The issue's "Suggested fixes" section lists five items; this PR addresses #1 (the concrete pattern gap that caused the actual incident) and its accompanying regression test (#5). Items #2 (treating any post-rejection workaround as the same prohibited effect — needs session-level intent tracking, not just pattern matching), #3 (a supported one-shot detached restart helper — a new tool surface), and #4 (enumerating submitted launchd jobs in restart-loop diagnostics) are broader hardening/feature work that I think deserves separate design discussion rather than being bundled into a reactive pattern fix — flagging them here in case a maintainer wants to track them as follow-ups.

How to Test

  1. Before this fix: _contains_gateway_lifecycle_command("launchctl submit -l ai.hermes.gateway-x -- /bin/sh helper.sh") returns False — the exact command from the issue is not caught.
  2. After this fix: it returns True, and the same protection applies at both the cron-job-creation guard and the terminal execution-time guard (both call this shared function).
  3. pytest tests/hermes_cli/test_gateway_restart_loop.py -q — 66 passed (2 pre-existing failures unrelated to this change: this sandbox's venv has no pip/croniter installed, confirmed identical on a clean origin/main checkout via git stash).

Checklist

Code

Documentation & Housekeeping

  • N/A — no docs, config keys, or tool schemas changed

AI Disclosure

This bug was identified and fixed with AI assistance.

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for closing the real launchctl submit omission: current main's Branch B at cron/lifecycle_guard.py:59 does not include that verb.

Problems

  • The changed regex at cron/lifecycle_guard.py:64 still uses [^\n]* between the verb and gateway label. The actual #62891 command puts a shell-continuation newline after launchctl submit, so this pattern stops before -l ai.hermes.gateway-… and does not block the reported form.

Suggested changes

  • Normalize shell continuations before matching, or permit continuation-newline segments in Branch B without relaxing the gateway-label constraint.
  • Add the literal multiline #62891 command to tests/hermes_cli/test_gateway_restart_loop.py, including the terminal guard path described in the PR.

Automated hermes-sweeper review.

Comment thread cron/lifecycle_guard.py
# creates a NEW keepalive job wrapping an arbitrary helper, which is how
# a blocked direct restart/kill gets laundered into a persistent restart
# loop instead (#62891) — same foot-gun, indirect shape.
r"|(?:launchctl\s+(?:kickstart|unload|load|stop|restart|submit)\b[^\n]*\bhermes[.\-]?gateway)"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[^\n]* still prevents this branch from reaching a -l ai.hermes.gateway-… label when launchctl submit is written with the backslash-newline continuation shown in #62891. Please normalize or explicitly handle shell continuations and add that literal reproduction.

@jackjin1997

Copy link
Copy Markdown
Contributor Author

Good catch — the exact reported command was split across backslash-newline shell continuations, so [^\n]* never bridged the verb to the gateway label. Pushed a follow-up commit that normalizes continuations (\ immediately followed by a newline) to a single space before matching, mirroring what the shell itself does, rather than loosening [^\n]* globally. Added the literal multi-line command from the issue as a test case, plus a negative test confirming two genuinely separate (non-continued) lines still don't get bridged. Verified the new test fails without the normalization.

@teknium1 teknium1 added sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 12, 2026
@alt-glitch alt-glitch added type/bug Something isn't working comp/cron Cron scheduler and job management P2 Medium — degraded but workaround exists and removed sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data labels Jul 12, 2026
@teknium1 teknium1 added sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Jul 12, 2026
_GATEWAY_LIFECYCLE_PATTERN's launchctl branch only matched
kickstart/unload/load/stop/restart. An agent whose direct restart/kill
was correctly blocked could launder the same effect through
`launchctl submit -l ai.hermes.gateway-<suffix> -- <helper-script>`,
which creates a NEW keepalive job wrapping an arbitrary helper — this
bypassed both the cron-creation-time guard and the terminal
execution-time guard (both call the same shared function), and became
a restart loop that ran 9,447 times before manual removal.

Fixes NousResearch#62891
…rd matching

Every branch of _GATEWAY_LIFECYCLE_PATTERN uses [^\n]* between its verb
and the gateway identifier so a match can't span unrelated lines of a
longer cron prompt/script. But the exact NousResearch#62891 command was split
across backslash-newline shell continuations, so the verb and the
gateway label landed on different physical lines and the pattern
(even with `submit` added) never matched. Collapse continuations to a
single space before matching, mirroring what the shell itself does,
instead of loosening [^\n]* and risking false positives across
genuinely separate lines.

Addresses review feedback on NousResearch#62896.
@jackjin1997
jackjin1997 force-pushed the fix/gateway-lifecycle-guard-launchctl-submit branch from d0720ee to 6ae871e Compare July 30, 2026 01:47
teknium1 added a commit that referenced this pull request Aug 1, 2026
…bel-independent detection

Extends the shared _GATEWAY_LIFECYCLE_PATTERN (used by BOTH the cron
creation-time guard in cron/lifecycle_guard.py and the terminal
execution-time hard-block in tools/terminal_tool.py) so Branch B covers
launchctl submit and bootstrap alongside kickstart/unload/load/stop/
restart, and normalizes POSIX shell line continuations before matching
so the exact multi-line reported shape in #62891 cannot slip past.

Also extends the execution-aware, label-independent detector
(contains_launchctl_submit_command, cherry-picked from #63272) to cover
launchctl bootstrap, since a neutral label like ai.hermes.svc-reload-tmp
defeats any label-anchored regex — the second production reproduction.

Regression tests cover both sites, including
`launchctl submit -l com.foo -- /path/gateway` and the bootstrap
variant, plus outside-gateway pass-through.

Branch B regex extension and continuation normalization drawn from
PR #62896; bootstrap coverage and test shapes drawn from PR #51003.

Co-authored-by: JackJin <1037461232@qq.com>
Co-authored-by: joelbrilliant <joelbrilliant1@gmail.com>
@teknium1

teknium1 commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Fixed via PR #75972 (#75972) — your shell line-continuation normalization was incorporated with Co-authored-by credit in the consolidation commit, alongside #63272's script scanner and the submit/bootstrap regex extension. Fixes #62891. Thanks!

@teknium1 teknium1 closed this Aug 1, 2026
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
…bel-independent detection

Extends the shared _GATEWAY_LIFECYCLE_PATTERN (used by BOTH the cron
creation-time guard in cron/lifecycle_guard.py and the terminal
execution-time hard-block in tools/terminal_tool.py) so Branch B covers
launchctl submit and bootstrap alongside kickstart/unload/load/stop/
restart, and normalizes POSIX shell line continuations before matching
so the exact multi-line reported shape in NousResearch#62891 cannot slip past.

Also extends the execution-aware, label-independent detector
(contains_launchctl_submit_command, cherry-picked from NousResearch#63272) to cover
launchctl bootstrap, since a neutral label like ai.hermes.svc-reload-tmp
defeats any label-anchored regex — the second production reproduction.

Regression tests cover both sites, including
`launchctl submit -l com.foo -- /path/gateway` and the bootstrap
variant, plus outside-gateway pass-through.

Branch B regex extension and continuation normalization drawn from
PR NousResearch#62896; bootstrap coverage and test shapes drawn from PR NousResearch#51003.

Co-authored-by: JackJin <1037461232@qq.com>
Co-authored-by: joelbrilliant <joelbrilliant1@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

macOS gateway lifecycle guard can be bypassed via launchctl submit, creating an infinite restart loop

3 participants