Skip to content

fix(cron): scope lifecycle guard to real shell scripts; per-job cron flag via ContextVar - #76797

Open
samson-mak wants to merge 3 commits into
NousResearch:mainfrom
samson-mak:fix/approval-gate-cron-leak
Open

fix(cron): scope lifecycle guard to real shell scripts; per-job cron flag via ContextVar#76797
samson-mak wants to merge 3 commits into
NousResearch:mainfrom
samson-mak:fix/approval-gate-cron-leak

Conversation

@samson-mak

@samson-mak samson-mak commented Aug 2, 2026

Copy link
Copy Markdown

Summary

Fixes three approval-gate defects that appear after a gateway restart:

1. cron/lifecycle_guard.py — false-positive "cannot restart or stop the gateway" blocks

_iter_referenced_shell_scripts treated any full-path executable (/Users/.../venv/bin/python3, ~/.hermes/bin/claude, …) as a shell script to read and scan. Real binaries were decoded as UTF-8 with errors="replace" — which embeds NUL bytes — and the resolver then crashed with ValueError: embedded null byte (only OSError was caught). In the common case it returned unsafe=True, hard-blocking legitimate dispatch commands with a fake gateway-lifecycle verdict.

Fix: only treat files as shell scripts when they end in .sh/.bash/.zsh or their first 512 bytes carry a POSIX-shell shebang (_is_shell_script_file). Also fixed dot-source (Path(".").name is "" so . script.sh was never scanned) and added (OSError, ValueError) guards on all os.open/resolve paths.

2. cron/scheduler.py — process-wide HERMES_CRON_SESSION leak

run_job set os.environ["HERMES_CRON_SESSION"] = "1" process-wide. After a gateway restart, the long-lived scheduler process kept the flag set, so every live session (Slack/Discord/…) was misclassified as a cron job and execute_code was hard-denied via approvals.cron_mode: deny.

Fix: per-job ContextVar (HERMES_CRON_SESSION_CONTEXTVAR) set inside the lock-guarded try: and reset in finally: (with a None token guard), so an exception can't leak the terminal-cwd lock.

3. tools/approval.py_is_cron_session()

All four approval-gating sites now check the ContextVar first with an env fallback (backward compatible for standalone cron processes).

Tests

New tests/cron/test_lifecycle_guard_regressions.py (12 tests):

  • real full-path binary (venv/bin/python3 -c ...) → not flagged
  • . /path/evil.sh and source /path/evil.sh → flagged
  • bash /path/evil.sh → flagged
  • large (>4 KB) extensionless #!/bin/sh script with a restart command → flagged
  • real cron.scheduler.run_job exception path → ContextVar reset + writer lock released (no leak)

tests/tools/test_execute_code_approval_cluster.py: 21 passed — the 3 one-shot/smart-approval tests were failing on machines whose config.yaml allowlists execute_code (the permanent allowlist short-circuited the gateway approval branch). The gw_session fixture now snapshots/restores _permanent_approved (second commit), plus a new test_guard_permanent_allowlist_is_isolated pins the allowlist shortcut.

Suite results: test_lifecycle_guard_regressions.py 12 passed · test_terminal_cwd_lock.py 4 passed · test_cron_approval_mode.py 26 passed · test_execute_code_approval_cluster.py 21 passed.

Notes

  • Reviewed by gpt-5.6-sol in 3 rounds (final verdict ACCEPT; no remaining findings).
  • Reproduction & fix validated on a live gateway: after restart, HERMES_CRON_SESSION no longer set, full-path python3 no longer blocked.

…flag via ContextVar

Three approval-gate defects after gateway restart:
1. lifecycle_guard treated any full-path executable as a shell script —
   venv python3 / claude binaries were classified, decoded as UTF-8 (with
   embedded NUL bytes) and could crash the resolver (ValueError: embedded
   null byte) or block legit dispatches with a false 'cannot restart or
   stop the gateway' verdict.
2. _is_shell_script_file had a 4096-byte total-size cap, so large
   extensionless shell scripts (#!/bin/sh + gateway restart) bypassed the
   guard; it now reads only the first 512 bytes for shebang classification.
3. dot-source (`.` / `source`) never scanned scripts because
   Path(".").name is ''.
4. scheduler set HERMES_CRON_SESSION=1 process-wide, so after a gateway
   restart every live session inherited cron mode and execute_code was
   hard-denied; replaced with a per-job ContextVar set inside the lock-guarded
   try and reset in finally (lock-leak-safe).

approval: _is_cron_session() reads the ContextVar first, falls back to env.
gateway/session_context: new HERMES_CRON_SESSION_CONTEXTVAR (default False).

Tests: tests/cron/test_lifecycle_guard_regressions.py — 12 tests covering
real full-path binary, dot-source, bash, large extensionless scripts, and the
real cron.scheduler.run_job cleanup path (ContextVar reset + writer lock
release). Reviewed by gpt-5.6-sol (3 rounds; final ACCEPT).
@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/gateway Gateway runner, session dispatch, delivery comp/cron Cron scheduler and job management area/sessions Session lifecycle, resume, persistence, history P2 Medium — degraded but workaround exists needs-decision Awaiting maintainer decision before any implementation sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades labels Aug 2, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Related to #76773 and the cron ContextVar cluster (#56796, #58663). The lifecycle-guard patches differ on whether an extensionless text script without a shell shebang remains scanable, while this PR also contains the separate cron-context repair; please choose the intended guard policy before consolidating.

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for isolating the cron approval marker; the underlying leak is present on current main at cron/scheduler.py:3011-3014, and the existing worker handoff copies ContextVars at cron/scheduler.py:3564-3569.

Problems

  • cron/lifecycle_guard.py:222-226 narrows referenced-script scanning to extensions or recognized shell shebangs. That drops directly executed extensionless text scripts without a shebang from the lifecycle scan, weakening the intentional defense-in-depth guard. The current implementation deliberately scans slash-containing paths (cron/lifecycle_guard.py:222-223), and the member note correctly flags this policy decision.
  • tests/cron/test_lifecycle_guard_regressions.py:111-161 forces an exception before the agent worker submission, so it does not verify the actual scheduler worker propagation or concurrent gateway isolation.

Suggested changes

  • Preserve extensionless text-script scanning while skipping clearly binary headers (the related #76773 approach uses a NUL-byte header distinction).
  • Add a real worker-handoff/concurrent-context regression around cron/scheduler.py:3568-3569.

Automated hermes-sweeper review.

Comment thread cron/lifecycle_guard.py
if "/" in executable or executable.endswith((".sh", ".bash", ".zsh")):
if executable.endswith((".sh", ".bash", ".zsh")):
yield _resolve_terminal_script_path(executable, cwd)
elif "/" in executable:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This excludes a directly executed extensionless text script with no shebang from scanning, so ./restart-helper can contain hermes gateway restart and evade the lifecycle guard. Please preserve scanning for text candidates and skip only clearly binary files (for example, a bounded NUL-byte header check).

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved in 7deebd1a0 — _is_shell_script_file now uses a bounded NUL-byte header check: text candidates (no NUL in first 512 bytes) are scanned, clearly binary files are skipped. ./restart-helper (extensionless, no shebang) is covered by the new test_extensionless_text_script_without_shebang_is_scanned.

@teknium1 teknium1 added the sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform label Aug 2, 2026
@samson-mak

Copy link
Copy Markdown
Author

Guard policy decision: bounded NUL-byte header check — scan text candidates (no NUL in first 512 bytes), skip clearly binary files. Landed in 7deebd1a0, aligned with the #76773 approach; ready to consolidate.

@samson-mak

Copy link
Copy Markdown
Author

@teknium1 both findings addressed in 7deebd1a0. @alt-glitch guard policy decided.

1. lifecycle_guard — NUL-byte header check (_is_shell_script_file)
Read first 512 bytes; NUL present → binary → skip; no NUL → text candidate → scan. Preserves extensionless shebang-less text-script scanning (./restart-helper evasion case, new regression test) while keeping the binary NUL-crash fixes from f03e785. Also updated test_full_path_non_shell_binary_is_not_scanned: the old 2 MiB b"x" payload has no NUL byte, so under NUL detection it classifies as text and would trip the 1 MiB read cap (unsafe → block); payload is now NUL-padded.

2. Real worker-handoff regressiontest_run_job_handoff_propagates_cron_context_and_isolates_concurrent_thread drives the actual cron.scheduler.run_job through the scheduler.py:3568-3569 copy_context submission: worker thread sees HERMES_CRON_SESSION_CONTEXTVAR True, a concurrent plain thread (live gateway session) sees False, scheduler thread resets to False after return.

Validation (Hermes venv, worktree at 7deebd1a0):

  • test_lifecycle_guard_regressions.py + test_terminal_cwd_lock.py + test_cron_approval_mode.py44 passed
  • test_execute_code_approval_cluster.py → 17 passed / 3 failed — identical at HEAD (pre-existing, unrelated)

Guard policy (for consolidation with #76773): bounded NUL-byte header check — scan text candidates, skip binary.

The gw_session fixture now snapshots/restores tools.approval._permanent_approved
so tests don't depend on developer config.yaml command_allowlist entries
(this dev machine allowlists execute_code, which short-circuited the gateway
approval branch and broke 3 one-shot/smart-approval tests). Adds
test_guard_permanent_allowlist_is_isolated to pin the allowlist shortcut.
@samson-mak
samson-mak force-pushed the fix/approval-gate-cron-leak branch from 7deebd1 to f39ef75 Compare August 2, 2026 15:10
… regression test

Addresses teknium1/hermes-sweeper review on NousResearch#76797:
- _is_shell_script_file now uses a bounded NUL-byte header check: scan
  text candidates (POSIX shells execute extensionless shebang-less text
  scripts via ENOEXEC fallback), skip clearly binary files. Fixes the
  ./restart-helper bypass class while keeping binary NUL-crash fixes.
- test_full_path_non_shell_binary: payload now NUL-padded (2MiB no-NUL
  'x' payload would be classified as text and trip the 1MiB cap).
- New: extensionless no-shebang text script is scanned.
- New: real run_job worker-handoff regression — ContextVar propagates
  into the worker thread via copy_context (scheduler.py:3568-3569),
  concurrent non-cron thread stays False, scheduler thread resets after.

(restored via cherry-pick of 7deebd1a0 onto PR head f39ef75)
@mohitagrawal-marvis

Copy link
Copy Markdown

+1 — I hit this same bug in the wild and opened #77233 (now closed in favor of this PR), so I can confirm this fix is needed and correct.

Independent reproduction: a terminal command referencing a venv interpreter by path (e.g. .venv/bin/python -m src.main) crashed the guard with ValueError: embedded null byte — the referenced file was decoded as UTF-8, tokenized as shell text, and Path.resolve() blew up on the embedded NUL bytes. This broke legitimate cron/script dispatch immediately after the referenced-script scanning landed.

Validating the policy here: my own testing confirmed the exact behavior this PR implements — a NUL-byte-laden binary executable passes through cleanly (no crash, no false block), while real shell scripts (.sh/shebang) are still scanned and a lifecycle command embedded in them is still blocked. The scan-selection approach here (only treat .sh/.bash/.zsh or shebang-bearing files as scripts) is strictly better than my NUL-stripping approach, which still scanned binaries and could false-positive on a binary that merely contains the command string.

Also appreciated: the dot-source . script.sh fix and the HERMES_CRON_SESSION ContextVar leak fix riding along — both real.

@alt-glitch alt-glitch added sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data and removed sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data labels Aug 3, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Related: #56796 and #58663 address the cron-session ContextVar family. This PR also repairs lifecycle-guard false positives and carries distinct cleanup coverage, so it is not a same-scope duplicate; maintainer comparison is needed.

@alt-glitch alt-glitch removed needs-decision Awaiting maintainer decision before any implementation area/sessions Session lifecycle, resume, persistence, history labels Aug 3, 2026
@GottZ

GottZ commented Aug 3, 2026

Copy link
Copy Markdown

This was generated by AI during triage.

Graph note (no action implied — a maintainer has already reviewed this thread).

Our triage graph places this PR in a complex with 1 related pull request (#77233). They were checked against each other at the diff level and no consolidation is indicated — they address distinct causes.

Full neighbourhood: https://hermes-triage.gottz.de/?node=76797

This note exists so the relationship stays discoverable from the thread itself.

@alt-glitch alt-glitch removed the comp/gateway Gateway runner, session dispatch, delivery label Aug 3, 2026
@alt-glitch alt-glitch added needs-decision Awaiting maintainer decision before any implementation comp/gateway Gateway runner, session dispatch, delivery tool/terminal Terminal execution and process management and removed sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data labels Aug 3, 2026
@alt-glitch alt-glitch added the sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data label Aug 3, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Related: #77383 is the focused lifecycle-guard repair and #56796 is the focused cron ContextVar repair. This conflicting branch bundles both; please rebase and decide whether to split or consolidate.

@alt-glitch alt-glitch removed the sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades label Aug 3, 2026
@saved-j

saved-j commented Aug 3, 2026

Copy link
Copy Markdown

Hey! I ran into the same wall and made a complementary fix — there are a couple of holes left even with your scoping. Verified on a live machine (macOS, gateway under launchd with no HOME in the env):

  1. RuntimeError: Could not determine home directoryexpanduser() in _resolve_terminal_script_path() / _resolve_script_path() raises this (not ValueError) when a token starts with ~ and the env has no HOME. You catch (OSError, ValueError) — the RuntimeError slips through and kills the guard exactly like the NUL byte did.

  2. tools/terminal_tool.py _read_script_in_env() — decodes any file up to 1 MB as UTF-8 with errors="replace", binaries included. Even with your iterator scoping: when the recursive scan walks a script that execs venv/bin/python inside, the binary gets decoded into garbage text containing gateway/restart — and the command gets falsely blocked with "cannot restart or stop the gateway". Your PR doesn't touch this file.

My PR (#78056): reject NUL bytes before touching pathlib/files + catch (OSError, ValueError, RuntimeError) at all four sites in lifecycle_guard.py, plus a NUL guard in _read_script_in_env (local and remote branches) matching the existing #76762 check.

On the live box: full-path python -c, ~ with no HOME, NUL-bearing tokens — all pass; real hermes gateway restart still blocked.

I think both PRs should land together — they fix adjacent layers of the same guard. Happy to rebase if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cron Cron scheduler and job management comp/gateway Gateway runner, session dispatch, delivery needs-decision Awaiting maintainer decision before any implementation P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state tool/terminal Terminal execution and process management type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants