Skip to content

fix: close gateway-lifecycle guard gaps in cron guard and execute_code - #68289

Closed
arcimun wants to merge 1 commit into
NousResearch:mainfrom
arcimun:fix/gateway-lifecycle-guard-gaps
Closed

fix: close gateway-lifecycle guard gaps in cron guard and execute_code#68289
arcimun wants to merge 1 commit into
NousResearch:mainfrom
arcimun:fix/gateway-lifecycle-guard-gaps

Conversation

@arcimun

@arcimun arcimun commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

What

Closes two gaps in the gateway lifecycle guard that allowed the agent to kill its own gateway process — causing real outages (26h on 2026-07-13, another on 2026-07-19).

Changes

1. cron/lifecycle_guard.py

  • Python argv-list bypass: regex used \s+ which doesn't match commas/quotes in subprocess.run([...]) form
  • Missing launchctl verbs: bootout, bootstrap, disable, remove
  • pkill regex: \bhermes\b doesn't match hermes_cli.main
  • gateway run/start --replace not blocked

2. tools/code_execution_tool.py

  • execute_code had NO gateway-lifecycle guard — os.system() / subprocess.run() sailed through
  • Several bundled skills document bootout recipes, giving the model a standing invitation

Testing

All variants tested 2026-07-19: bootout, kickstart -k, pkill -f hermes_cli.main, gateway run --replace, subprocess.run([...]) list form

Two real incidents (2026-07-13 and 2026-07-19) exposed gaps where the
gateway could be killed from inside itself:

1. cron/lifecycle_guard.py:
   - Python argv-list form (subprocess.run([...])) was not matched
     because the regex used \s+ which doesn't match commas/quotes
   - launchctl bootout/bootstrap/disable/remove were missing — bootout
     unloads the job so KeepAlive can't revive it (26h outage)
   - pkill regex used \bhermes\b which fails on hermes_cli.main
   - gateway run/start --replace was not blocked (SIGTERMs live gateway)

2. tools/code_execution_tool.py:
   - execute_code had NO gateway-lifecycle guard at all — os.system()
     and subprocess.run() inside Python scripts sailed through
   - Several bundled skills document bootout recipes, giving the
     model a standing invitation to run them
@alt-glitch alt-glitch added type/bug Something isn't working comp/cron Cron scheduler and job management tool/code-exec execute_code sandbox P2 Medium — degraded but workaround exists sweeper:risk-automation Sweeper risk: may affect CI, automerge, label sync, or maintainer automation sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data labels Jul 21, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Related: #63272, #62896, #51980, and merged #56240 address distinct gateway-lifecycle guard bypasses. This PR extends coverage to argv-list, launchctl, process-name, --replace, and execute_code paths.

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for extending a real gateway self-termination defense. Current main still has the reported lifecycle and execute_code gaps.

Problems

  • cron/lifecycle_guard.py:73 only makes the Hermes and launchctl branches argv-list aware. The next systemctl branch (:75) still requires whitespace, so subprocess.run(["systemctl", "restart", "hermes-gateway"]) remains unmatched.
  • cron/lifecycle_guard.py:95 accepts any gateway run --replace substring. That conflicts with the existing command-shaped guard contract at cron/lifecycle_guard.py:45-47 and can reject safe prose.
  • The PR adds no regression tests. Existing coverage in tests/hermes_cli/test_gateway_restart_loop.py:28-89,313-381 does not exercise execute_code or the new variants.

Suggested changes

  • Make systemctl and --replace argv-list aware, preserving a concrete Hermes/module command anchor for --replace.
  • Add direct execute_code gateway-marker tests plus positive and safe-prose pattern cases.

Automated hermes-sweeper review.

Comment thread cron/lifecycle_guard.py
# through this gap, booted the job out, and the Mac gateway stayed dead for
# 26h until a manual restart. `disable`/`remove` are blocked for the same
# reason (they leave the label un-startable).
r"|(?:launchctl[\"',\s]+(?:kickstart|unload|load|stop|restart|bootout|bootstrap|disable|remove)\b[^\n]*\bhermes[.\-]?gateway)"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes the Hermes and launchctl branches argv-list aware, but the systemctl branch immediately below still begins with systemctl\s+. subprocess.run(["systemctl", "restart", "hermes-gateway"]) therefore remains unblocked; apply the same delimiter handling there and add a regression case.

Comment thread cron/lifecycle_guard.py
# victim logged `parent_pid=1` (its own launchd parent), which reads like
# launchd killed it and hides the actual caller. Covers both the CLI shape
# and the module shape (`python -m hermes_cli.main gateway run --replace`).
r"|(?:gateway\s+(?:run|start)\b[^\n]*--replace)"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This matches arbitrary prose containing gateway run --replace, although the existing guard deliberately requires command-shaped identifiers to avoid false positives. Require a Hermes CLI or hermes_cli.main invocation prefix, and add a safe-prose regression case.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 30, 2026
@MongLong0214

Copy link
Copy Markdown

Current-main follow-up: I reproduced a remaining execute_code path that the source-text check at this head does not cover. A Python child can assemble "launch" + "ctl" and "sub" + "mit" dynamically, so the lifecycle operation only exists in the resolved process argv; the source scanner returns false while check_execute_code_guard() approves under approvals.mode=off/yolo. I filed #83630 with a no-side-effect reproduction and runtime-resolution acceptance criteria. I am keeping the scope separate from this PR’s missing-verb work and will credit the prior incidents and analysis here.

@Baophan00

Copy link
Copy Markdown
Contributor

Nice work closing the source-text gaps. I found one remaining bypass while reproducing this locally: the new guard in code_execution_tool.py only scans source text, but once the script is approved it can still assemble dangerous argv dynamically at runtime, e.g. program = "launch" + "ctl" then subprocess.run([program, verb, ...]). That path never touches terminal_tool.py, so the hard-block is skipped.

I opened #83630 with a no-side-effect reproduction and what I think the acceptance criteria should be: inspect argv at child runtime, cover dynamic assembly and os.system/shell=True/os.exec*, and keep yolo/off from waiving it.

If useful I can prepare a follow-up PR.

@alt-glitch alt-glitch removed sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades labels Aug 11, 2026
teknium1 pushed a commit that referenced this pull request Aug 24, 2026
… lifecycle guard (#68289)

execute_code lacked the lifecycle guard entirely, and Python argv-list
forms (subprocess.run([...])) separated command words with brackets and
commas the shell-shaped pattern could not see. Mirror the terminal_tool
guard in execute_code (ownership-gated per #92560) and strip argv-list
punctuation in the token-join re-scan. Salvaged from PR #68289 by
@arcimun, adapted to the ownership gate and current guard structure.
teknium1 pushed a commit that referenced this pull request Aug 24, 2026
… lifecycle guard (#68289)

execute_code lacked the lifecycle guard entirely, and Python argv-list
forms (subprocess.run([...])) separated command words with brackets and
commas the shell-shaped pattern could not see. Mirror the terminal_tool
guard in execute_code (ownership-gated per #92560) and strip argv-list
punctuation in the token-join re-scan. Salvaged from PR #68289 by
@arcimun, adapted to the ownership gate and current guard structure.
@teknium1

Copy link
Copy Markdown
Contributor

Merged via PR #93336 (c94ee2e) — the execute_code guard and the argv-list detection were cherry-picked with your authorship preserved in git log, adapted to the PID-ownership gate (#92560) and the current guard structure (argv punctuation stripped in the token-join re-scan rather than widening the raw pattern, keeping prose safe). Your Branch B verb additions (bootout/bootstrap/disable/remove) had already landed via #93297, and the --replace Branch E shape is noted. Thanks — the 2026-07-19 verification table in this PR made live repro straightforward.

@teknium1 teknium1 closed this Aug 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-automation Sweeper risk: may affect CI, automerge, label sync, or maintainer automation tool/code-exec execute_code sandbox type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants