Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 33 additions & 3 deletions cron/lifecycle_guard.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,18 +51,48 @@ class GatewayLifecycleBlocked(ValueError):
# `start` is intentionally excluded: starting a gateway from inside a
# gateway is benign (a no-op or "already running" error), and a
# legitimate cron job might start a sibling profile's gateway.
r"(?:hermes\s+gateway\s+(?:restart|stop))"
# The separator class is ["',\s]+ rather than \s+ throughout: the same
# command reaches this guard both as a shell string ("launchctl bootout X")
# and as a Python argv list from execute_code
# (subprocess.run(["launchctl", "bootout", "X"])), where the tokens are
# separated by quotes and commas instead of spaces. Verified 2026-07-19:
# with \s+ the list form was the one variant that slipped past every other
# branch.
r"(?:hermes[\"',\s]+gateway[\"',\s]+(?:restart|stop))"
# Branch B: launchctl ops on a hermes-gateway label. macOS launchd
# labels look like `ai.hermes.gateway` / `hermes-gateway`. Requiring the
# gateway identifier prevents blocking unrelated hermes services (e.g.
# `launchctl unload ai.hermes.update-checker.plist`).
r"|(?:launchctl\s+(?:kickstart|unload|load|stop|restart)\b[^\n]*\bhermes[.\-]?gateway)"
# `bootout`/`bootstrap` (the modern launchctl verbs) are the worst of the
# set and were missing: `bootout` *unloads* the job, so KeepAlive can no
# longer revive it, and the `bootstrap` that would re-register it dies with
# the gateway it just killed. Real incident 2026-07-13: the agent fell
# through this gap, booted the job out, and the Mac gateway stayed dead for
# 26h until a manual restart. `disable`/`remove` are blocked for the same
# reason (they leave the label un-startable).
r"|(?:launchctl[\"',\s]+(?:kickstart|unload|load|stop|restart|bootout|bootstrap|disable|remove)\b[^\n]*\bhermes[.\-]?gateway)"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes the Hermes and launchctl branches argv-list aware, but the systemctl branch immediately below still begins with systemctl\s+. subprocess.run(["systemctl", "restart", "hermes-gateway"]) therefore remains unblocked; apply the same delimiter handling there and add a regression case.

# Branch C: systemctl ops on a hermes-gateway unit.
r"|(?:systemctl\s+(?:-\S+\s+)*(?:restart|stop|start)\b[^\n]*\bhermes[.\-]?gateway)"
# Branch D: pkill / kill targeting the hermes gateway process. Both
# token orders because real reproductions show both.
r"|(?:p?kill\b[^\n]*\bhermes\b[^\n]*\bgateway)"
# No right-hand \b after `hermes`: the process as it actually appears in
# `ps` is `hermes_cli.main gateway run`, and `\bhermes\b` does NOT match
# `hermes_cli` because `_` is a word character — so the single most likely
# kill command (`pkill -f "hermes_cli.main gateway"`) sailed straight
# through this guard until 2026-07-19. Matching `hermes` as a prefix also
# covers `hermes-gateway`, `hermesd`, etc. `gateway` must still appear.
r"|(?:p?kill\b[^\n]*\bhermes[^\n]*\bgateway)"
r"|(?:p?kill\b[^\n]*\bgateway\b[^\n]*\bhermes)"
# Branch E: starting a SECOND gateway with `--replace`. Branch A excludes
# bare `start`/`run` on the theory that starting a gateway from inside one
# is benign ("already running"), and that holds — but only WITHOUT this
# flag. `--replace` means "SIGTERM whoever holds the slot, then take it",
# i.e. it is a restart wearing a start's clothes. Real incident 2026-07-19:
# a second `gateway run --replace` SIGTERM'd the live gateway mid-task; the
# victim logged `parent_pid=1` (its own launchd parent), which reads like
# launchd killed it and hides the actual caller. Covers both the CLI shape
# and the module shape (`python -m hermes_cli.main gateway run --replace`).
r"|(?:gateway\s+(?:run|start)\b[^\n]*--replace)"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This matches arbitrary prose containing gateway run --replace, although the existing guard deliberately requires command-shaped identifiers to avoid false positives. Require a Hermes CLI or hermes_cli.main invocation prefix, and add a safe-prose regression case.

)


Expand Down
28 changes: 28 additions & 0 deletions tools/code_execution_tool.py
Original file line number Diff line number Diff line change
Expand Up @@ -1195,6 +1195,34 @@ def execute_code(
if not code or not code.strip():
return tool_error("No code provided.")

# Hard-block gateway-lifecycle commands, mirroring the guard in
# terminal_tool. Without this, execute_code is a straight bypass: the
# terminal() path refuses `launchctl bootout ai.hermes.gateway`, but the
# identical command inside `os.system(...)` / `subprocess.run([...])` here
# sailed through and SIGTERM'd the gateway mid-task — verified 2026-07-19,
# every variant (bootout / kickstart -k / hermes gateway restart) returned
# approved=True. That matters because several bundled skills still document
# a bootout recipe in their troubleshooting sections (grsai-provider, which
# the agent reads while generating images, among them), so the model has a
# standing invitation to run exactly this. Killing the gateway from inside
# it also kills this very subprocess, so the restart may never complete.
if os.environ.get("_HERMES_GATEWAY") == "1":
from hermes_cli.cron import _contains_gateway_lifecycle_command
if _contains_gateway_lifecycle_command(code):
return json.dumps({
"status": "error",
"error": (
"Blocked: cannot restart or stop the gateway from inside the "
"gateway process. The gateway would kill this script before it "
"could complete (SIGTERM propagates to child processes). Run "
"`hermes gateway restart` from a shell outside the gateway, or "
"let the watchdog (~/.hermes/scripts/gateway-watchdog.sh) "
"handle recovery."
),
"tool_calls_made": 0,
"duration_seconds": 0,
}, ensure_ascii=False)

# Dispatch: remote backends use file-based RPC, local uses UDS
from tools.terminal_tool import _get_env_config, _docker_has_host_access
_env_config = _get_env_config()
Expand Down