fix(terminal): stop the silent output blackout when stdout fd >= FD_SETSIZE - #553
Merged
Conversation
…ETSIZE
The terminal tool intermittently returned `{"output": "", "exit_code": 0}`
for every command — including a bare `echo` — while `execute_code` kept
working. In the same window `write_file` failed its own post-write
verification with "wrote N chars, read back 0 chars ... The write did not
persist". Three occurrences (2026-08-07, 2026-08-09 x2); each ended at
"a gateway restart cleared it", never root-caused.
Root cause: `BaseEnvironment._wait_for_process`'s drain thread polled the
subprocess stdout pipe with `select.select()`. select(2) cannot represent a
file descriptor at or above FD_SETSIZE (1024) and raises
`ValueError: filedescriptor out of range in select()`. The drain loop
swallowed that with a bare `break`, so the collector stayed empty and the
result was an empty capture with the command's real exit code — identical
in shape to a command that legitimately printed nothing.
This explains every observed symptom:
* Intermittent, and worsens with gateway uptime — fds only cross 1024 after
enough accumulate. The tree already carries two fixed fd-leak regressions
in long-running gateways (NousResearch#69567, gateway/delivery_ledger), and the
gateways run with RLIMIT_NOFILE=4096, well above FD_SETSIZE.
* `write_file` degrading simultaneously — `file_operations._exec` runs
through `env.execute()` -> this same drain. Its `cat` read-back returned
"", so the verification honestly reported a mismatch.
* `execute_code` unaffected — separate runtime, not this drain.
* A gateway restart cures it — the new process starts with low fds.
* It can also self-clear — fds get released, and a later spawn lands back
below 1024.
* `env -i /bin/bash --noprofile --norc` also blank — the shell was always
healthy; the fault is downstream of it.
Fix, in two parts:
1. Replace `select()` with `poll()` in both drain loops
(`environments/base.py`, `process_registry.py`). poll(2) takes an array
of pollfd structs and has no FD_SETSIZE ceiling, so the failure mode
cannot recur. EINTR is retried rather than treated as end-of-stream.
Windows keeps the existing blocking-read path (it has neither call for
pipes).
2. Fail LOUD instead of silent. The drain records any abnormal abort reason
and `_finalize_wait_result` prepends an `OUTPUT CAPTURE FAILED` marker,
sets a `drain_error` key, and logs at error level. A capture we failed to
read must never again be indistinguishable from a command that printed
nothing.
Verification:
* RED-proof: reverting only `tools/environments/base.py` makes 4 of the 6
new tests fail, with the exact production signature
`assert 'HIGH_FD_MARKER_OK' in ''`.
* End-to-end against the real tool paths (`LocalEnvironment.execute` and
`ShellFileOperations.write_file`) with fds burned past 1024: the
unpatched tree exits 1 and reproduces BOTH production symptoms verbatim,
including "wrote 29 chars, read back 0 chars ... The write did not
persist"; the patched tree exits 0 with output and content intact.
* 572 passed / 0 failed across tests/tools -k
"terminal or environment or process_registry or file_op or file_tool".
* ruff clean on all three changed files.
This was referenced Aug 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
The terminal tool intermittently returns
{"output": "", "exit_code": 0}for every command — including a bareecho— whileexecute_codekeeps working. In the same windowwrite_filefails its own post-write verification withwrote N chars, read back 0 chars ... The write did not persist.This has happened at least three times (2026-08-07, 2026-08-09 x2). Each instance ended at "a gateway restart cleared it". It was never root-caused, and the empty-output-with-exit-0 shape is actively dangerous: it is indistinguishable from a command that legitimately printed nothing, so an empty
lsreads as "the directory is empty" and a mutating command that never ran reads as success.Root cause
BaseEnvironment._wait_for_process's drain thread polled the subprocess stdout pipe withselect.select():select(2) cannot represent a file descriptor at or above
FD_SETSIZE(1024 on Linux and macOS). Above it, Python raisesValueError: filedescriptor out of range in select(). That barebreakswallowed the exception, the collector stayed empty, and the result was an empty capture carrying the command's real exit code.This accounts for every documented symptom:
gateway/delivery_ledger), and gateways run withRLIMIT_NOFILE=4096— well above FD_SETSIZE.write_filedegrades at the same timefile_operations._exec→env.execute()→ this same drain. Itscatread-back returned"", so the verification honestly reported a mismatch. The write itself was fine.execute_codeunaffectedenv -i /bin/bash --noprofile --norcalso blankThe fix
1.
poll()instead ofselect()in both drain loops (tools/environments/base.py,tools/process_registry.py). poll(2) takes an array ofpollfdstructs and has no FD_SETSIZE ceiling, so the failure mode cannot recur.EINTRis now retried rather than treated as end-of-stream. Windows keeps the existing blocking-read path (it supports neither call on pipes).2. Fail loud instead of silent. The drain records any abnormal abort reason;
_finalize_wait_resultprepends anOUTPUT CAPTURE FAILEDmarker, sets adrain_errorkey on the result, and logs at error level. Even if some future capture failure slips through, it can never again masquerade as "the command printed nothing".Part 2 is deliberately independent of part 1 — it is the general guard against this whole class.
Verification
RED-proof — reverting only
tools/environments/base.py(keeping the new tests):That
assert ... in ''is the production signature, reproduced on demand.End-to-end through the real tool paths (
LocalEnvironment.executeandShellFileOperations.write_file, fds burned past 1024). Unpatched tree:Both production symptoms reproduced verbatim, including the exact
read back 0 charsstring. Patched tree, same harness:Regression suites —
572 passed, 5 skipped, 0 failedacrosstests/tools/ -k "terminal or environment or process_registry or file_op or file_tool".ruff checkclean on all three changed files.Notes for review
FD_SETSIZEbefore running, so it cannot pass vacuously if fd pressure fails to materialise; it skips instead.process_registry.pygot the same treatment as the sibling call path — it had the identicalselect()+ bare-breakshape and would silently drop a background process's entire output under the same conditions.drain_errorkey is additive; existing consumers readingoutput/returncodeare unaffected.