Skip to content

fix(cron): make the lifecycle guard total — sanitize at ingestion, not per-syscall - #80258

Merged
kshitijk4poor merged 3 commits into
NousResearch:mainfrom
kshitijk4poor:fix/lifecycle-guard-total
Aug 6, 2026
Merged

fix(cron): make the lifecycle guard total — sanitize at ingestion, not per-syscall#80258
kshitijk4poor merged 3 commits into
NousResearch:mainfrom
kshitijk4poor:fix/lifecycle-guard-total

Conversation

@kshitijk4poor

Copy link
Copy Markdown
Collaborator

Summary

The gateway lifecycle guard is now a total function — every input maps to a verdict, no input can crash it or stall it — by sanitizing untrusted bytes at three boundaries instead of catching exceptions at whichever syscall crashed that week.

Root cause of the class: the guard feeds untrusted byte streams (shlex-tokenized binaries, remote cat output) into OS-path and shell-text operations with no ingestion boundary. Each incident (#76762, #77703, #77780, #78256, #77729) got a per-callsite except; @tilllt's regression suite on #79454 (comment) showed 4 members of the class still open on merged main, and ~30 open community PRs each hot-fix another fragment.

Changes

  • cron/lifecycle_guard.py_expand_candidate_path(): single ingestion chokepoint for path candidates. Rejects NUL/empty tokens before any Path OS call and tolerates ValueError/RuntimeError/OSError from expanduser() (T1/T2, plus the HOME-unset launchd crash from fix(guard): never crash or false-positive on binary/executable paths #78056). Both _resolve_terminal_script_path and _resolve_script_path route through it.
  • cron/lifecycle_guard.py_sanitize_remote_script_text(): applies the local-read contract (NUL = binary = nothing to scan; >1 MiB = fail closed) to whatever any read_remote_script callback returns, at the recursion boundary. The guard stops trusting its callbacks — fix(terminal): skip binary content on referenced-script remote-read fallback (#77703) #79454 hardened one callback; this covers every current and future one (T3/T4).
  • cron/lifecycle_guard.pycontains_gateway_lifecycle_command_or_referenced_script() is total by construction: direct regex scans (pure string ops) run first; the best-effort filesystem walk is wrapped so an unexpected failure logs a warning and falls back to the direct-scan verdict instead of breaking every terminal command until gateway restart.
  • tools/terminal_tool.py — remote fallback reads head -c 1048577 instead of cat, so an oversized file never crosses the wire. One byte over budget is enough for the sanitizer to fail closed.
  • tests/hermes_cli/test_gateway_restart_loop.py — tilllt's T1–T4 adopted as regression tests, plus an adversarial never-raises sweep (NUL paths, unset HOME, over-long paths) and a walk-crash fallback test.

Validation

tilllt's T1–T4 harness against merged main (49d8a155c) vs this branch:

Test main @ 49d8a15 this PR
T1 ~\x00 candidate via terminal walk crash (ValueError: embedded null byte) pass (False)
T2 ~\x00 candidate via cron script path crash (ValueError: embedded null byte) pass (no raise)
T3 NUL binary from remote callback false-positive block pass (False)
T4 >1 MiB remote text scanned unbounded fail-closed (True)

Live probes (real imports, no guard mocks): all positive detections preserved — direct commands, launchctl kickstart/submit, nested wrapper scripts, remote-callback detection of a real lifecycle script; a 170 MB callback payload goes from a 30+ minute superlinear-shlex stall (#79838's field report) to a 0.02 s fail-closed verdict.

Tests: test_gateway_restart_loop.py 91 passed; test_terminal_tool.py + tests/cron/ 422 passed.

Credit

…t per-syscall

The guard feeds untrusted byte streams (tokenized binaries, remote cat
output) into OS-path and shell-text operations; every incident so far
(NousResearch#76762, NousResearch#77703, NousResearch#77780, NousResearch#78256, NousResearch#77729) was hot-fixed with an except at
whichever frame crashed that week. tilllt's regression suite on NousResearch#79454
showed 4 members of the class still open on merged main. Close the class
at three boundaries instead:

- _expand_candidate_path(): single ingestion chokepoint for path
  candidates — reject NUL/empty tokens before any Path OS call and
  tolerate ValueError/RuntimeError/OSError from expanduser (T1/T2, plus
  the HOME-unset launchd crash). Both _resolve_terminal_script_path and
  _resolve_script_path now go through it.
- _sanitize_remote_script_text(): apply the local-read contract (NUL =
  binary = nothing to scan; >1MiB = fail closed) to whatever any
  read_remote_script callback returns, at the recursion boundary — the
  guard stops trusting its callbacks (T3/T4).
- contains_gateway_lifecycle_command_or_referenced_script() is now total
  by construction: direct regex scans (pure string ops) run first; the
  best-effort filesystem walk is wrapped so an unexpected failure logs a
  warning and falls back to the direct-scan verdict instead of killing
  every terminal command until gateway restart.

terminal_tool's remote fallback also bounds the read at the source
(head -c 1MiB+1 instead of cat), so a 166MB ELF never crosses the wire —
the superlinear-shlex 30-minute stall from NousResearch#79838's field report drops
to a 0.02s fail-closed verdict.

Regression tests: tilllt's T1-T4 adopted verbatim, plus an adversarial
never-raises sweep (NUL paths, unset HOME, over-long paths) and a
walk-crash fallback test.
- _sanitize_remote_script_text: compare re-encoded BYTES against the cap,
  not characters — a >1MiB multibyte file truncated at the head -c byte
  bound decodes to fewer chars than bytes and would have scanned the
  truncated text instead of failing closed (the exact local/remote
  divergence this PR closes).
- terminal_tool: replace the three hardcoded 1MiB literals with
  lifecycle_guard._MAX_REFERENCED_SCRIPT_BYTES so the budget cannot
  drift; use the redirect-safe 'head -c N < path' form from
  tools/image_source.py so leading-dash paths stay out of argv.
- Public guard wrapper: drop the duplicate depth-0 direct scan — the
  walk already runs it; the except-path now falls back to the pure
  string scans, preserving the direct verdict when the walk crashes.
@alt-glitch alt-glitch added type/bug Something isn't working comp/cron Cron scheduler and job management tool/terminal Terminal execution and process management P2 Medium — degraded but workaround exists sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data labels Aug 6, 2026

@monerostar monerostar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ubuntu 26.04 on linux-5800x.

Ran the PR's tests/hermes_cli/test_gateway_restart_loop.py against three trees:

tree result
origin/main 7 failed, 84 passed (ValueError embedded null on tilde+NUL candidates; remote/oversized/adversarial cases)
sibling #80241 6 failed, 85 passed (still misses tilde-NUL walk, oversized remote fail-closed, total-function cases)
this PR 91 passed

Sanitizing at ingestion (_expand_candidate_path / remote text contract) is the right whole-class shape vs per-syscall catches. Prefer this over the narrower #80241 patch.

Looks good from Linux here.

…esolvable totality

3-reviewer simplify pass (reuse/quality/efficiency) findings:

- cron/scheduler.py _run_job_script: the ORIGINAL that
  lifecycle_guard._resolve_script_path documents mirroring had the exact
  same unguarded expanduser() — a NUL-bearing script value survives
  creation (the guard treats it as nothing-to-scan) and crashed the
  scheduler at fire time with ValueError instead of a clean job failure.
  Same ingestion contract applied; regression test added.
- lifecycle_guard._resolve_script_path: get_hermes_home() -> Path.home()
  raises RuntimeError when neither HERMES_HOME nor HOME resolves
  (arbitrary-UID containers); the cron entry point called it bare.
  Caught -> None; totality test added.
- terminal_tool: stale 'cat ...' docstring updated to the bounded
  head -c form.
- lifecycle_guard: dead 'script_text and' condition dropped (guarded by
  'if not script_text: continue' directly above).

Efficiency reviewer: no material findings (measured — encode/expand
costs negligible vs walk I/O, no timing regression vs base).
@kshitijk4poor
kshitijk4poor enabled auto-merge (rebase) August 6, 2026 12:02
@kshitijk4poor
kshitijk4poor merged commit 863e313 into NousResearch:main Aug 6, 2026
40 checks passed
This was referenced Aug 6, 2026
This was referenced Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management P2 Medium — degraded but workaround exists sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data tool/terminal Terminal execution and process management type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants