fix(guard): don't cat binaries via lifecycle-guard remote fallback - #78824
fix(guard): don't cat binaries via lifecycle-guard remote fallback#78824yang-fireworkhq wants to merge 1 commit into
Conversation
A command referencing an existing binary by absolute path (e.g. a 275MB native CLI binary invoked via its full path with --version) made the terminal-tool lifecycle guard hang for ~30 minutes and then crash with ValueError: embedded null byte. _contains_unsafe_gateway_action treats _read_referenced_script's None return for NUL-skipped binaries as 'path missing', so when read_remote_script is supplied (as terminal_tool always does) it cats the entire binary through the environment and recursively scans decoded machine code. For a 275MB binary that is millions of junk segments — a ~30 minute hang before the first NUL-laden junk path crashes os.open. - Only use the remote fallback when the local path is genuinely absent; a NUL-skipped binary is 'nothing to scan', not 'missing'. - Tolerate ValueError from _read_referenced_script (mirrors the existing resolve() guard) so a junk path can never crash the guard. - Defense in depth in _read_script_in_env: cap the env.execute output at 1MB and skip output whose first chunk contains NUL bytes, so a genuinely remote large binary cannot trigger the same explosion. Adds regression tests: an existing binary path must not invoke the remote fallback; a genuinely missing path still does.
|
Closing as superseded by #80258, which fixes this whole bug class architecturally rather than per-callsite: path candidates are sanitized once at the ingestion boundary (NUL/empty/unexpandable tokens rejected before any OS call), text from any Your report and fix targeted a real member of this class — thank you. The per-callsite patches kept leaving sibling frames exposed (#76762 → #77703 → #77780 → #78256 each crashed one frame away from the previous fix), which is why we went with the boundary fix instead of merging the fragments individually. #80258 carries regression tests for the NUL-path, binary-callback, oversized-read, unset-HOME, and walk-crash cases plus an adversarial never-raises sweep. |
Bug Description
Running a terminal command that references an existing binary by its absolute path (e.g.
--versionon a large native CLI installed under a full path) makes the terminal-tool lifecycle guard hang for ~30 minutes, then fail with:Concrete symptoms: a
terminaltool call that should return in <1s instead hangs the whole agent turn for 20–30 minutes and eventually errors out. Two occurrences of this exact signature were observed in production logs (1769s and 1944s tool-call durations), both triggered by commands that referenced a 275MB binary by full path.Root Cause
tools/terminal_tool.pyrunscontains_gateway_lifecycle_command_or_referenced_script(command, ..., read_remote_script=_read_script_in_env)before executing every command. The referenced-script walk (cron/lifecycle_guard.py) yields absolute executable paths from the command, then calls_read_referenced_scripton each._read_referenced_scriptdeliberately skips binaries — it reads the first chunk, sees a NUL byte, and returns(None, False)("nothing to scan", #76762). The bug: the caller can't distinguish that from "path missing", so whenread_remote_scriptis supplied it falls back to:For a 275MB ELF binary this returns the entire decoded machine code, which the walk then recursively tokenizes and scans: ~800k junk segments, each doing
Path()/os.open()work, recursing into any real paths embedded in the binary — a ~30 minute hang that finally crashes when a NUL-laden junk path reachesos.open(ValueError: embedded null byte). The existing #76762 fix only covered theresolve()site, not thisos.openpath through the remote fallback.Fix
cron/lifecycle_guard.pyresolved.exists() and is_file()). A NUL-skipped binary is "nothing to scan", not "missing" — so existing binaries are nevercat-ed._read_referenced_scriptintry/except (OSError, ValueError)so a NUL-laden junk path can never crash the guard, mirroring the existingresolve()guard.tools/terminal_tool.py(defense in depth)_read_script_in_env: capenv.executeoutput at 1MB and returnNonewhen the first 4KB contains a NUL byte, so a genuinely remote large binary (SSH/Modal/Daytona) can't trigger the same explosion.Remote-backend behavior is preserved: a genuinely missing path still goes through
read_remote_script.How to Verify
python -m pytest tests/hermes_cli/test_gateway_restart_loop.py -o 'addopts=' -q→ 84 passedtest_existing_binary_path_does_not_trigger_remote_fallbackfails onmain(the fallback spy is invoked with the binary path) and passes with this fix.contains_gateway_lifecycle_command_or_referenced_script("<path-to-large-binary> --version", cwd="/", read_remote_script=<spy>)returns in milliseconds with zero spy calls on the fixed code; the pre-fix code calls the spy with the binary path.Test Plan
Risk Assessment
Low — the change only narrows when the remote
catfallback fires (existing local file → never) and hardens two crash sites. The genuinely-missing-path case (the only one remote backends rely on) is covered by a dedicated test.