Skip to content

fix(guard): don't cat binaries via lifecycle-guard remote fallback - #78824

Closed
yang-fireworkhq wants to merge 1 commit into
NousResearch:mainfrom
yang-fireworkhq:fix/lifecycle-guard-binary-remote-fallback
Closed

fix(guard): don't cat binaries via lifecycle-guard remote fallback#78824
yang-fireworkhq wants to merge 1 commit into
NousResearch:mainfrom
yang-fireworkhq:fix/lifecycle-guard-binary-remote-fallback

Conversation

@yang-fireworkhq

Copy link
Copy Markdown

Bug Description

Running a terminal command that references an existing binary by its absolute path (e.g. --version on a large native CLI installed under a full path) makes the terminal-tool lifecycle guard hang for ~30 minutes, then fail with:

ValueError: embedded null byte

Concrete symptoms: a terminal tool call that should return in <1s instead hangs the whole agent turn for 20–30 minutes and eventually errors out. Two occurrences of this exact signature were observed in production logs (1769s and 1944s tool-call durations), both triggered by commands that referenced a 275MB binary by full path.

Root Cause

tools/terminal_tool.py runs contains_gateway_lifecycle_command_or_referenced_script(command, ..., read_remote_script=_read_script_in_env) before executing every command. The referenced-script walk (cron/lifecycle_guard.py) yields absolute executable paths from the command, then calls _read_referenced_script on each.

_read_referenced_script deliberately skips binaries — it reads the first chunk, sees a NUL byte, and returns (None, False) ("nothing to scan", #76762). The bug: the caller can't distinguish that from "path missing", so when read_remote_script is supplied it falls back to:

env.execute(f"cat {shlex.quote(script_path)}")

For a 275MB ELF binary this returns the entire decoded machine code, which the walk then recursively tokenizes and scans: ~800k junk segments, each doing Path()/os.open() work, recursing into any real paths embedded in the binary — a ~30 minute hang that finally crashes when a NUL-laden junk path reaches os.open (ValueError: embedded null byte). The existing #76762 fix only covered the resolve() site, not this os.open path through the remote fallback.

Fix

cron/lifecycle_guard.py

  • Only invoke the remote fallback when the local path is genuinely absent (resolved.exists() and is_file()). A NUL-skipped binary is "nothing to scan", not "missing" — so existing binaries are never cat-ed.
  • Wrap _read_referenced_script in try/except (OSError, ValueError) so a NUL-laden junk path can never crash the guard, mirroring the existing resolve() guard.

tools/terminal_tool.py (defense in depth)

  • _read_script_in_env: cap env.execute output at 1MB and return None when the first 4KB contains a NUL byte, so a genuinely remote large binary (SSH/Modal/Daytona) can't trigger the same explosion.

Remote-backend behavior is preserved: a genuinely missing path still goes through read_remote_script.

How to Verify

  1. python -m pytest tests/hermes_cli/test_gateway_restart_loop.py -o 'addopts=' -q → 84 passed
  2. Regression test test_existing_binary_path_does_not_trigger_remote_fallback fails on main (the fallback spy is invoked with the binary path) and passes with this fix.
  3. Manual: contains_gateway_lifecycle_command_or_referenced_script("<path-to-large-binary> --version", cwd="/", read_remote_script=<spy>) returns in milliseconds with zero spy calls on the fixed code; the pre-fix code calls the spy with the binary path.

Test Plan

  • Added regression tests (existing binary path → no remote fallback; missing path → fallback still fires)
  • Existing tests still pass (84/84 in the guard test module)
  • Manual verification of the fix (see above)

Risk Assessment

Low — the change only narrows when the remote cat fallback fires (existing local file → never) and hardens two crash sites. The genuinely-missing-path case (the only one remote backends rely on) is covered by a dedicated test.

A command referencing an existing binary by absolute path (e.g. a
275MB native CLI binary invoked via its full path with --version)
made the terminal-tool lifecycle guard hang for ~30 minutes and
then crash with ValueError: embedded null byte.

_contains_unsafe_gateway_action treats _read_referenced_script's
None return for NUL-skipped binaries as 'path missing', so when
read_remote_script is supplied (as terminal_tool always does) it
cats the entire binary through the environment and recursively
scans decoded machine code. For a 275MB binary that is millions of
junk segments — a ~30 minute hang before the first NUL-laden junk
path crashes os.open.

- Only use the remote fallback when the local path is genuinely
  absent; a NUL-skipped binary is 'nothing to scan', not 'missing'.
- Tolerate ValueError from _read_referenced_script (mirrors the
  existing resolve() guard) so a junk path can never crash the guard.
- Defense in depth in _read_script_in_env: cap the env.execute
  output at 1MB and skip output whose first chunk contains NUL
  bytes, so a genuinely remote large binary cannot trigger the
  same explosion.

Adds regression tests: an existing binary path must not invoke the
remote fallback; a genuinely missing path still does.
@alt-glitch alt-glitch added type/bug Something isn't working comp/cron Cron scheduler and job management tool/terminal Terminal execution and process management sweeper:risk-automation Sweeper risk: may affect CI, automerge, label sync, or maintainer automation needs-decision Awaiting maintainer decision before any implementation P2 Medium — degraded but workaround exists labels Aug 4, 2026
@kshitijk4poor

Copy link
Copy Markdown
Collaborator

Closing as superseded by #80258, which fixes this whole bug class architecturally rather than per-callsite: path candidates are sanitized once at the ingestion boundary (NUL/empty/unexpandable tokens rejected before any OS call), text from any read_remote_script callback is sanitized at the recursion boundary (NUL = binary = nothing to scan; >1 MiB = fail closed), the remote fallback read is bounded at the source (head -c, so oversized binaries never cross the wire), and the public guard is total by construction — an unexpected walk failure logs and falls back to the direct-scan verdict instead of breaking every terminal command.

Your report and fix targeted a real member of this class — thank you. The per-callsite patches kept leaving sibling frames exposed (#76762#77703#77780#78256 each crashed one frame away from the previous fix), which is why we went with the boundary fix instead of merging the fragments individually. #80258 carries regression tests for the NUL-path, binary-callback, oversized-read, unset-HOME, and walk-crash cases plus an adversarial never-raises sweep.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management needs-decision Awaiting maintainer decision before any implementation P2 Medium — degraded but workaround exists sweeper:risk-automation Sweeper risk: may affect CI, automerge, label sync, or maintainer automation tool/terminal Terminal execution and process management type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants