Fix #2884: raise SDK reader buffer (the real #2804 crash fix); reframe per-tool caps as cost discipline

docs/development/STRUCTURE.md

@@ -270,7 +270,7 @@ shared/
 │   ├── command.py          # build_agent_command() for orchestrator-spawned containers
 │   ├── result.py           # AgentResult dataclass
 │   ├── tool_interceptor.py # Pre-execution file write checks (Write/Edit/NotebookEdit) against role restrictions
-│   └── tool_output_cap.py  # Predictive PreToolUse cap for built-in CC tools (Read/Grep): denies calls likely to overflow the 1 MB SDK buffer before they run; tunable via EGG_TOOL_OUTPUT_CAP / EGG_READ_CAP_BYTES (#2876)
+│   └── tool_output_cap.py  # Predictive PreToolUse cap for built-in CC tools (Read/Grep): denies calls whose model-bound result is likely to be excessive (cost/context discipline, NOT the buffer-crash fix — that's the raised reader buffer in client.py, #2884); tunable via EGG_TOOL_OUTPUT_CAP / EGG_READ_CAP_BYTES (#2876)
 ├── egg_anchor/             # Agent anchor mechanism for post-compaction state recovery
 │   ├── __init__.py         # Public API exports
 │   ├── models.py           # Pydantic models (AgentAnchor, AnchorMeta, ProgressItem, Decision, BRCState)

docs/reference/agent-recovery.md

@@ -123,23 +123,33 @@ In concurrent (BRC) mode, all agents are wrapped with a shell script that handle
 
 The `is_buffer_overflow()` function is checked **before** `is_transient_crash()`. It greps the captured agent output log for the Claude Agent SDK's `CLIJSONDecodeError` marker (`"exceeded maximum buffer size"`) and, when found, exits the wrapper immediately without consuming any restart budget.
 
-The SDK has a 1 MB JSON message-reader buffer cap; a tool result that exceeds it kills the agent with exit 255. This failure is **deterministic** — re-running the agent against the same codebase produces the same oversized payload and hits the same crash. Retrying is therefore wasteful. The wrapper logs:
+The upstream Claude Agent SDK ships a 1 MiB JSON message-reader buffer; egg raises it to 32 MiB on this path (see the next section, [#2884](https://github.com/jwbron/egg/issues/2884)), so the cap that's actually in effect is much higher than the SDK default. A tool result that exceeds *the configured cap* — whatever it is — kills the agent with exit 255. This failure is **deterministic** — re-running the agent against the same codebase produces the same oversized payload and hits the same crash. Retrying is therefore wasteful. The wrapper logs:
 
 ```
 Agent crashed on Claude Agent SDK buffer overflow (issue #2804). Deterministic failure; retry budget would be wasted. NOT restarting.
 ```
 
 The agent output is captured by piping stdout and stderr through `tee` into a temporary log file (`AGENT_OUTPUT_LOG`, created via `mktemp`). This log is truncated at the start of each agent run so old crash signatures don't bleed into subsequent runs.
 
-> **Note:** The buffer-overflow marker string is synchronized between the wrapper's `grep` and the `_BUFFER_OVERFLOW_MARKER` constant in `shared/egg_agent/client.py`. If a future `claude-agent-sdk` release changes the wording, the wrapper silently falls back to burning the transient-crash retry budget. See [#2823](https://github.com/jwbron/egg/issues/2823) for the follow-up to pin this against the installed SDK. The real fix is tool-layer truncation of oversized payloads ([#2805](https://github.com/jwbron/egg/issues/2805)); this is the fail-fast path until that lands.
+> **Note:** The buffer-overflow marker string is synchronized between the wrapper's `grep` and the `_BUFFER_OVERFLOW_MARKER` constant in `shared/egg_agent/client.py`. If a future `claude-agent-sdk` release changes the wording, the wrapper silently falls back to burning the transient-crash retry budget. See [#2823](https://github.com/jwbron/egg/issues/2823) for the follow-up to pin this against the installed SDK. With the reader buffer raised (next section, [#2884](https://github.com/jwbron/egg/issues/2884)) this fail-fast is now a rare backstop — it fires only if a single stream message exceeds the generous raised buffer — not the common path it was when the cap was 1 MiB.
+
+### SDK Reader Buffer (the crash-prevention layer)
+
+Source: `_DEFAULT_SDK_MAX_BUFFER_BYTES` / `_sdk_max_buffer_bytes()` in `shared/egg_agent/client.py`, wired as `ClaudeAgentOptions.max_buffer_size`.
+
+This is what actually prevents the [#2804](https://github.com/jwbron/egg/issues/2804) crash. egg's Agent SDK reader decodes the CLI's stream-json output into a JSON buffer; a single message larger than `max_buffer_size` raises `CLIJSONDecodeError` and kills the agent (exit 255). The SDK default is 1 MiB.
+
+The crucial point ([#2884](https://github.com/jwbron/egg/issues/2884)): **the messages that overflow are not model-bound.** Claude Code attaches the **entire original file** to every `Edit`/`Write` result as transcript metadata (`toolUseResult.originalFile`) that the model never sees — only egg's reader decodes it. So a routine ~2 KB edit to the 1.1 MB, 25k-line `orchestrator/routes/pipelines.py` emits a >1 MB stream message and crashes the reader, even though the model's `tool_result` is just a bounded `cat -n` snippet. (The original #2884 framing guessed the *edit snippet* scaled with file size; the CLI's `fBB` snippet builder bounds it to `new_string` lines + 8, so the culprit is the result *envelope* metadata, not the snippet.)
+
+egg raises `max_buffer_size` to **32 MiB** (default; override with `EGG_SDK_MAX_BUFFER_BYTES`). This costs **no model context or tokens** — the oversized field never reaches the model, and egg logs at most `_MAX_TOOL_CONTENT_LOG_LEN` of any result — only transient reader memory for one message. It cannot let an oversized payload reach the model either: model-bound result sizes are bounded independently (the predictive caps below, the MCP `@tool` caps [#2805](https://github.com/jwbron/egg/issues/2805), and Claude Code's own Bash truncation). 32 MiB covers source files far larger than anything in this repo while still bounding a runaway/malformed stream; the fail-fast above is the clean backstop for anything beyond it.
 
 ### Predictive Output Cap (PreToolUse)
 
 Source: `shared/egg_agent/tool_output_cap.py`, wired in `shared/egg_agent/client.py`.
 
-The fail-fast above is a backstop, not prevention. egg caps its own MCP `@tool` payloads at the tool boundary ([#2805](https://github.com/jwbron/egg/issues/2805)), but **built-in** Claude Code tools (`Read`, `Grep`, `Edit`, `Bash`) run inside the CLI and can't be wrapped. [#2876](https://github.com/jwbron/egg/issues/2876) bounds those via a **PreToolUse** hook that fires *before* the tool runs and denies calls whose result is likely to overflow — for example a whole-file `Read` of the 1.1 MB, 24k-line `orchestrator/routes/pipelines.py` that crashed the [#2777](https://github.com/jwbron/egg/issues/2777) slice-1 coder. PostToolUse can't help here: the payload has already crossed the channel that crashes the reader ([#2810](https://github.com/jwbron/egg/issues/2810) dropped that approach).
+These caps are **model-context/cost discipline, not crash prevention** (the reader buffer above is the crash fix). What they bound is the volume a tool sends *to the model*: a whole-file `Read` returns the file's content to the model (the 1.1 MB `pipelines.py` ≈ ~275k tokens), and a whole-repo content `Grep` dumps every matching line — both wasteful. **Built-in** Claude Code tools (`Read`, `Grep`, `Bash`) run inside the CLI and can't be wrapped the way egg caps its own MCP `@tool` payloads ([#2805](https://github.com/jwbron/egg/issues/2805)), so [#2876](https://github.com/jwbron/egg/issues/2876) bounds them via a **PreToolUse** hook that fires *before* the tool runs and denies calls whose model-bound result is likely to be excessive, telling the agent how to narrow the call. (`Edit`/`Write` are deliberately *not* capped here: their model-bound result is the small snippet, and their crash vector was the reader-buffer metadata, fixed above — not anything a per-call cap could see.)
 
-Current heuristics — predictive, so expect some false positives/negatives, with the fail-fast as the backstop when a prediction misses:
+Current heuristics — predictive, so expect some false positives/negatives:
 
 | Tool | Denied when | Deny reason points at |
 |------|-------------|------------------------|
@@ -148,7 +158,7 @@ Current heuristics — predictive, so expect some false positives/negatives, wit
 | `Read` (image/notebook) | Target image/notebook > `EGG_READ_CAP_BYTES` (returned whole; `offset`/`limit`/`pages` don't bound it) | images: avoid reading whole, use Bash (`file`/`stat`) for metadata; notebooks: inspect cells with `jq` (e.g. `jq '.cells[].source'`) |
 | `Grep` | `output_mode=content`, no `head_limit`, **and** no `path`/`glob` scope (whole-repo content dump) | `head_limit`, a `path`/`glob` scope, or `output_mode=files_with_matches` |
 
-The hook is **always-on** (the overflow hits every route, including first-party Opus). Set `EGG_TOOL_OUTPUT_CAP=false` (or `0`/`no`/`off`) to disable; set `EGG_READ_CAP_BYTES` to tune the `Read` threshold (a set-but-invalid value — non-integer or non-positive — is logged and ignored in favour of the default).
+The hook is **always-on** (excess model-bound output is wasteful on every route, including first-party Opus). Set `EGG_TOOL_OUTPUT_CAP=false` (or `0`/`no`/`off`) to disable; set `EGG_READ_CAP_BYTES` to tune the `Read` threshold (a set-but-invalid value — non-integer or non-positive — is logged and ignored in favour of the default).
 
 ### Transient Exit Codes
 

docs/reference/agent-tools.md

@@ -214,29 +214,36 @@ round-trip.
 
 ### Output-size cap (`EGG_TOOL_OUTPUT_CAP_BYTES`, #2805)
 
-Every egg-owned tool result is bounded before it crosses the Claude
-Agent SDK's hard 1 MB JSON reader buffer — an oversized result kills the
-agent with exit 255 (#2804). The cap is applied at two chokepoints: the
+Every egg-owned tool result is bounded as **model-context/cost
+discipline** before it crosses the Claude Agent SDK reader — a runaway
+result would otherwise dump tens of thousands of tokens to the model in
+a single tool call. The cap is applied at two chokepoints: the
 orchestrator MCP server (`handle_tool_call` → `cap_result_dict`) and
 every sandbox `@tool` wrapper (`invoke_handler` → `cap_text`), both via
-the shared `shared/egg_tool_output.py` helper.
+the shared `shared/egg_tool_output.py` helper. The cap is **not** the
+crash-prevention layer for the SDK reader (the upstream 1 MiB buffer was
+the original concern, but egg raises that to 32 MiB at
+`ClaudeAgentOptions.max_buffer_size`, #2884 — see
+[Agent Recovery → SDK Reader Buffer](agent-recovery.md#sdk-reader-buffer-the-crash-prevention-layer)).
 
 | Variable | Default | Effect |
 |----------|---------|--------|
 | `EGG_TOOL_OUTPUT_CAP_BYTES` | `102400` (100 KB) | Max serialized size of a single tool result. Output above the cap is replaced with a structured head-preview marker (`_egg_truncated`) that names how to narrow the call, or — for unpaginated content like a full checkpoint transcript — spilled to a temp file (`_egg_output_spilled`) the agent can `Read`/`grep`, with a small inline preview. |
 
-The default leaves ~10× headroom under the 1 MB buffer. A non-positive
-or non-integer value is **ignored with a logged warning** (the operator
-is not left believing a cap is in effect when it isn't); the helper
-falls back to the 100 KB default. The orchestrator measures the cap
-against `indent=2`-serialized JSON (matching what its MCP server ships),
-so raising the cap toward the buffer size stays safe.
+At ~4 B/token for prose/JSON, the 100 KB default ≈ ~25k tokens — a
+sensible upper bound for a single model-bound tool result. A
+non-positive or non-integer value is **ignored with a logged warning**
+(the operator is not left believing a cap is in effect when it isn't);
+the helper falls back to the 100 KB default. The orchestrator measures
+the cap against `indent=2`-serialized JSON (matching what its MCP server
+ships), so raising the cap stays safe against the reader buffer above it
+either way.
 
 **Built-in tool cap (complementary):** The cap above covers egg-owned MCP
 `@tool` payloads. Built-in Claude Code tools (`Read`, `Grep`, etc.) run
 inside the CLI and can't be wrapped the same way. [#2876](https://github.com/jwbron/egg/issues/2876)
-adds a PreToolUse hook that predicts overflows *before* the tool runs and
-denies the call with a narrowing hint. See
+adds a PreToolUse hook that predicts when a result would be excessive
+*before* the tool runs and denies the call with a narrowing hint. See
 [Agent Recovery → Predictive Output Cap](agent-recovery.md#predictive-output-cap-pretooluse)
 for the heuristic table and the `EGG_TOOL_OUTPUT_CAP` / `EGG_READ_CAP_BYTES`
 operator knobs.

orchestrator/consensus_wrapper.py

@@ -126,9 +126,12 @@
 
 # Capture agent stdout+stderr so the wrapper can post-mortem the run.
 # Used by is_buffer_overflow() to detect the Claude Agent SDK
-# message-reader 1MB JSON buffer crash (issue #2804) which is
-# deterministic — retrying just hits the same overflow and burns
-# the restart budget for no gain.
+# message-reader JSON buffer crash (issue #2804) which is deterministic —
+# retrying just hits the same overflow and burns the restart budget for
+# no gain. With the reader buffer raised to 32 MiB on the egg path (#2884,
+# see shared/egg_agent/client.py::_DEFAULT_SDK_MAX_BUFFER_BYTES) this is
+# a rare backstop rather than the common path it was at the 1 MiB SDK
+# default, but the wrapper still has to fail-fast when it does fire.
 #
 # Use ``mktemp`` for the default path so a co-tenant on the same host
 # cannot pre-create a symlink at a predictable ``/tmp/agent-output-$$``
@@ -179,12 +182,11 @@
     return ${{PIPESTATUS[0]}}
 }}
 
-# Detect the Claude Agent SDK 1 MB JSON message-reader overflow
-# signature in the most recent agent run. Issue #2804. The overflow
-# is deterministic: re-running the agent against the same codebase
-# hits the same oversized tool result, so the wrapper must NOT
-# consume retry budget on this failure class. Returns 0 (true) if
-# the marker was logged, 1 otherwise.
+# Detect the Claude Agent SDK JSON message-reader overflow signature in
+# the most recent agent run. Issue #2804. The overflow is deterministic:
+# re-running the agent against the same codebase hits the same oversized
+# tool result, so the wrapper must NOT consume retry budget on this
+# failure class. Returns 0 (true) if the marker was logged, 1 otherwise.
 #
 # The substring matches CLI output from claude_agent_sdk emitted on
 # the buffer overflow path. If a future SDK bump changes the
@@ -194,8 +196,12 @@
 # test_script_marker_matches_client_constant and the
 # test_buffer_overflow_*_aborts_without_retry pair) exercise the
 # wrapper against a synthetic log to keep this honest, but do not
-# pin against the installed SDK. The real fix is tool-layer
-# truncation (#2805); this is the fail-fast path until that lands.
+# pin against the installed SDK. The real fix for the overflow class
+# is the raised reader buffer (#2884, see
+# shared/egg_agent/client.py::_DEFAULT_SDK_MAX_BUFFER_BYTES = 32 MiB);
+# this fail-fast is the clean backstop for anything beyond it. The
+# per-tool MCP @tool caps (#2805) and Read/Grep predictive caps (#2876)
+# are independent model-context/cost discipline — not the crash fix.
 is_buffer_overflow() {{
     [ -f "$AGENT_OUTPUT_LOG" ] || return 1
     grep -q "exceeded maximum buffer size" "$AGENT_OUTPUT_LOG" 2>/dev/null