fix(sandbox): keep Claude Code current so built-in model aliases resolve (#3136) - #3137
Conversation
…lve (#3136) The sandbox image's Claude Code install layer was keyed on the literal 'stable' channel name, so BuildKit never re-ran it and the deployed CC froze at whatever stable pointed to when the layer first built. That build predates the 'fable' alias agent_model_resolution.py uses as the tier-3 default for refine/plan roles, so those agents crash-looped at spawn (160+ failed invocations on the first issue-3077 run). Three changes to the CC install: - Switch the default channel to 'latest': the stable channel lags model launches (2.1.153, current stable, has no fable support at all), and egg's built-in defaults track new model families at launch. - Cache-bust via a remote ADD of the channel manifest: BuildKit re-checks the URL content every build, so the install layer re-runs exactly when the channel moves to a new version. - Build-time alias gate: fail the image build if the installed binary lacks the model families used as built-in defaults (fable, opus), instead of discovering the gap at spawn time in a live pipeline. Auth isn't mounted at build time, so the gate greps the binary for the versioned 'claude-<family>-*' IDs the alias maps to. Also bump claude-agent-sdk 0.1.65 -> 0.2.97 (pin range >=0.2.97,<0.3). The full surface egg uses — query, ClaudeAgentOptions fields, create_sdk_mcp_server, @tool, the CLIJSONDecodeError hierarchy and the #2804 'exceeded maximum buffer size' marker — was validated against 0.2.97. Fixes #3136
|
Follow-up for the event-pump no-backoff/no-escalation observation filed as #3138. |
There was a problem hiding this comment.
The core fix is sound and well-reasoned: changing the channel default to latest, cache-busting via ADD of the channel manifest, and the build-time alias gate together close the silent-failure gap that caused #3136. CI is green and the manual SDK-surface validation is documented.
Non-blocking findings below — please address the first one before merge if practical, the others are mop-up.
Findings
1. docs/guides/per-agent-models.md:109-117 is now actively misleading
The "Minimum sandbox Claude Code version" callout says:
The sandbox's
CLAUDE_CODE_VERSIONbuild-arg insandbox/Dockerfiledefaults tostable, which satisfies this on a fresh build.
After this PR both clauses are wrong:
- the default is now
latest, notstable; - per your own analysis in the PR body, "the stable channel itself doesn't have
fableyet — current stable is 2.1.153, whose binary contains zeroclaude-fablestrings", so the "which satisfies this on a fresh build" claim was already false. A reader troubleshooting an operator-side override would currently be told to trust stable.
This is the natural place to fix it — suggested rewrite:
> **Minimum sandbox Claude Code version.** The `fable` / `fable[1m]`
> aliases require Claude Code ≥ 2.1.170. The sandbox's
> `CLAUDE_CODE_VERSION` build-arg in `sandbox/Dockerfile` defaults to
> `latest` (#3137 — `stable` lagged the fable launch and crash-looped
> refine/plan agents per #3136). A build-time gate in the Dockerfile
> fails the image build if the installed binary doesn't know `fable`
> or `opus`, so a stale pin surfaces at build time rather than at
> spawn. If you need to pin to an older `CLAUDE_CODE_VERSION` that
> predates the alias, also set a repo-level `default_agent_model: opus`
> AND temporarily drop `fable` from the Dockerfile gate.
2. tests/sandbox/egg_agent_tools/test_sdk_surface.py still references the old pin
Two stale spots after the bump:
- Line 7:
pin the SDK to '>=0.1.65,<0.2'— should be the current pin. - Line 56:
restore the ``claude-agent-sdk>=X,<Y`` form(the docstring above is fine but the embedded example on line 56 —range like '>=0.1.65,<0.2'— should also be updated).
The test is still passing because the regex on line 54 is range-agnostic, but you're editing the surrounding pin in this PR, and the docstring drift is a small additional ask.
3. SDK smoke test undersells the bump (pre-existing gap, but more salient now)
The PR description claims manual validation of a much larger surface (every ClaudeAgentOptions field egg sets, the message/block types, CLIJSONDecodeError + _BUFFER_OVERFLOW_MARKER, query, etc.). The committed CI guard is still only two hasattr checks for create_sdk_mcp_server and tool.
A 0.3 bump's "manual validation" requirement will rest on whoever does that bump remembering to re-run the same checks by hand. Worth pinning the longer surface — at minimum add hasattr checks for the symbols shared/egg_agent/client.py:205-221 imports, so a 0.3 release that drops CLIJSONDecodeError / PermissionResultAllow / HookMatcher / ToolPermissionContext / query / etc. fails CI loudly rather than silently. The #2823 follow-up on the exceeded maximum buffer size marker would naturally land alongside this.
Non-blocking because the production CI tests cover the end-to-end behavior, but the smoke-test guard isn't doing what the comment on pyproject.toml:13-14 ("will fail CI loudly on API drift") implies.
4. SDK 0.1 → 0.2 behavior changes the PR description doesn't call out
Two from the upstream changelog that brush against egg's code paths and are worth a sanity-check before merge:
- MCP servers connect in the background by default (≥0.2.x). With
MCP_CONNECTION_NONBLOCKINGdefaulting to non-zero, slow servers report"pending"during init.shared/egg_agent/client.py:319-358registers the in-process SDK MCP servers viaoptions.mcp_servers = …. In-process SDK MCP servers should not be "slow" in the stdio sense, but the change is documented as a behavior shift — please confirm the in-process path is not affected, or pinMCP_CONNECTION_NONBLOCKING=0if it is. The system-prompt nudge is also load-bearing for tool discovery; a background connection that completes after the first model turn would defeat it. - TodoWrite → Task tools* (per 0.2.x release notes).
sandbox/agent-config/rules/overseer.md:201listsTodoWriteas a working-window heuristic. After this bump, agents no longer emitTodoWritecalls — they emitTaskCreate/TaskUpdate. The overseer's "active tool calls" heuristic now misses what should be one of its strongest signals of legitimate work. Worth a one-line rule update in this PR or a follow-up.
5. Cache-bust ADD pins to the latest manifest regardless of requested channel
The ADD https://downloads.claude.ai/claude-code-releases/latest … line fires on every move of the latest channel. When the operator passes CLAUDE_CODE_VERSION=stable (the previous default), the cache will be busted on each latest bump even though stable hasn't moved — wasteful but functionally fine, as your comment acknowledges for the concrete-version case.
A stronger version would parameterize the manifest URL on the requested channel (e.g. ADD https://downloads.claude.ai/claude-code-releases/${CLAUDE_CODE_VERSION} when the arg is a channel name, skip the ADD when it's a concrete version). Not blocking — listed for completeness because the current comment doesn't quite cover this case.
6. Build-time gate lacks an opt-out
The gate is hardcoded to fable and opus. An operator who legitimately wants to pin to an older CC and set default_agent_model: opus to opt out of fable cannot do so without editing the Dockerfile — the gate fails first. Probably fine for now (and the operator can override agent_models per-pipeline once the image is built); an ARG SKIP_ALIAS_GATE= knob would make this configurable without code edits. Worth a one-line note in the gate's error message that operators can comment out the gate to bypass.
7. Cache-busting depends on the CDN preserving HTTP cache headers
Verified live — https://downloads.claude.ai/claude-code-releases/latest returns proper ETag and Last-Modified headers (cache-control: public,no-cache,max-age=0), and BuildKit's ADD <URL> cache key incorporates those, so the busting works today. If the CDN ever drops those headers (config change, hosting migration), the busting would silently degrade to "cached forever per URL string." Worth a defense-in-depth note in the cache-bust comment for future debugging, and possibly a --checksum would be even stronger — but the upstream content is not stable (it's the channel pointer), so --checksum is the wrong tool here.
Worth recording as follow-ups
- The
make buildpath always uses defaults (make build→docker buildwith no--build-arg, no--no-cache). This PR makes the channel defaultlatest, so everymake buildrun will dutifully pick up the newest CC. That's the intended behavior post-#3136, but it does mean builds are not reproducible across timestamps. Worth recording in the PR body for the merger / future incident response. - The #2823 follow-up on
_BUFFER_OVERFLOW_MARKERbecomes more pressing with a major SDK bump; the marker check inorchestrator/consensus_wrapper.py:155is still string-coupled to whatever the SDK happens to raise.
— Authored by egg
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
- docs/guides/per-agent-models.md: rewrite the "minimum sandbox CC version" callout — previous text said default was 'stable' and that 'stable' satisfied the fable requirement, both wrong after this PR and the second already-false before. Point at the build-time gate + the operator workaround. - tests/sandbox/egg_agent_tools/test_sdk_surface.py: bump the stale '>=0.1.65,<0.2' references in the docstring and example to the current pin; parametrize a new hasattr check across every symbol shared/egg_agent/client.py:205-221 imports (Assistant/System/User/ Result/SystemMessage, Tool*Block, query, ClaudeSDKError, CLIJSONDecodeError, CLINotFoundError, HookMatcher, PermissionResult*, ProcessError) so a 0.3 release that drops any of them fails CI rather than crash-looping agent spawn at runtime. - shared/egg_agent/client.py: setdefault MCP_CONNECTION_NONBLOCKING=0 before the SDK launches the CC subprocess. The 0.1→0.2 SDK/CLI bump flipped the default to non-blocking; egg's in-process SDK MCP servers don't connect over stdio so they're unaffected, but the egg-ddg stdio fallback registered on the LiteLLM→non-Anthropic path does — and the SYSTEM_PROMPT_NUDGE that steers tool discovery is load-bearing on the first turn. - sandbox/agent-config/rules/overseer.md: TodoWrite was renamed to TaskCreate / TaskUpdate in CC 2.x — update the "evidence of legitimate work" heuristic to list both so the overseer doesn't misclassify normal task-tracking calls as a stall. - sandbox/Dockerfile: expand the cache-bust comment to call out the hardcoded 'latest' URL (intentional, wasteful for stable but functionally fine) and the CDN-header dependency BuildKit uses for the ADD cache key. Add a hint to the alias gate's error message that operators pinning an older CC can comment the loop out to bypass. Authored-by: egg
|
Thanks for the careful review. Per-item disposition (commit 602b98d): 1.
|
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
No agent-mode design concerns. This is a sandbox image / SDK pin fix — Dockerfile channel switch, cache-bust, build-time alias gate, SDK 0.1.65→0.2.97 bump, and an MCP_CONNECTION_NONBLOCKING=0 default for the spawned CC. No prompt construction, output-format requirements, post-processing pipelines, or direct-API surfaces are introduced, and the alias gate reinforces the model-alias convention (checks for fable/opus family support, not pinned version IDs).
— Authored by egg
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Re-review of commit 602b98d (delta from b303540). All seven findings from the prior review have been substantively addressed — not just papered over. Spot-checked each fix end-to-end:
- Finding 1 (
docs/guides/per-agent-models.md:109-118): applied verbatim. Callout now nameslatest, points at the build-time gate, and gives operators a path off the gate. - Finding 2 (stale
0.1.65intest_sdk_surface.py): docstring (line 7) and example string (line 102) both bumped to>=0.2.97,<0.3. - Finding 3 (smoke test undersells the bump):
_EGG_AGENT_CLIENT_SYMBOLScovers all 15 runtime imports fromshared/egg_agent/client.py:219-235plusCLIJSONDecodeError(referenced via comment for the #2804 buffer-overflow marker). Parametrized — a 0.3 drop of any one of them now fails CI loudly per symbol rather than crash-looping at spawn. The disposition on_BUFFER_OVERFLOW_MARKER(defer to #2823 because it's a string in an error message, not an exported symbol) is correct. - Finding 4a (
MCP_CONNECTION_NONBLOCKING):os.environ.setdefault("MCP_CONNECTION_NONBLOCKING", "0")atshared/egg_agent/client.py:210correctly:- Sets the var before the import block, so it's in
os.environwhen the SDK'ssubprocess_cli.py:401buildsinherited_env. - Uses
setdefaultso an operator-set shell value is preserved;options.envoverrides via the SDK's merge (subprocess_cli.py:405), so the in-functionenv=param also wins. - The env var is real — the bundled CC binary references
process.env.MCP_CONNECTION_NONBLOCKINGdirectly. Confirmed against the unpacked 0.2.97 wheel. - The egg-ddg stdio MCP server is registered conditionally at
client.py:454for the LiteLLM→non-Anthropic path (ANTHROPIC_CUSTOM_MODEL_OPTION), so the load-bearing scenario is real.
- Sets the var before the import block, so it's in
- Finding 4b (
TodoWrite→TaskCreate/TaskUpdate):overseer.md:201lists both new tools and the legacyTodoWritefor older CC builds — correct because the build-gate bypass keeps that path live. - Finding 5 (cache-bust comment): expanded with the three cases (
latest/stable/ concrete) and the rationale for keeping the URL hardcoded. The "would 404 on concrete-version branch" argument for not parameterizing is correct. - Finding 6 (gate bypass): error message now hints at commenting the
for family in …loop out. Lighter than anARG SKIP_ALIAS_GATEknob, and aligned with the docs callout. Acceptable. - Finding 7 (CDN-header dependency): defense-in-depth paragraph added, naming
ETag/Last-Modified/cache-controland pointing at--no-cachefor debugging.
New code is solid
The MCP_CONNECTION_NONBLOCKING fix is the only behaviour change of substance in this delta, and the reasoning is correct on every dimension I traced:
- Subprocess env propagation — SDK inherits parent env at
subprocess_cli.py:401. - Idempotency —
setdefaultis safe under concurrent invocations. - Override precedence — shell-set value preserved by
setdefault; per-callenv=preserved by the SDK's**options.envmerge. - In-process SDK MCP path unaffected — confirmed by the producer's analysis that
_handle_sdk_mcp_requestservices those in-band.
Non-blocking nits
These are small enough to fix in this PR or leave for next.
1. Stale line-range reference in the new smoke test
tests/sandbox/egg_agent_tools/test_sdk_surface.py:23:
# Sourced from ``shared/egg_agent/client.py:205-221`` and the
# ``CLIJSONDecodeError`` reference in the same block (#2804 marker).
The imports are now at lines 219-235 (the new MCP_CONNECTION_NONBLOCKING block bumped them down ~13 lines). Worth updating to 219-235 so a future reader can navigate to the source list without grep.
2. Slight mismatch between the docs bypass and the gate error-message bypass
docs/guides/per-agent-models.md:117-118 tells operators to "temporarily drop fable from the Dockerfile gate" (drop one family). sandbox/Dockerfile:300-302 tells operators to "comment this 'for family in ...' loop out to bypass the gate" (disable the gate entirely). Both work; both are intentional. Worth aligning the language so the operator sees consistent instructions whichever surface they hit first — either "edit the gate to drop the family you're skipping" everywhere, or "comment the loop out" everywhere.
3. Annotation-only SDK types are not in the smoke-test list
shared/egg_agent/client.py:24 imports HookContext, HookInput, HookJSONOutput under TYPE_CHECKING. With from __future__ import annotations these don't crash at runtime if the SDK drops them — but mypy / type-checking would. The smoke test deliberately scopes to runtime ImportError, which is the right primary concern, so I'd leave this alone unless you want a stronger guard for the type-check path.
4. No direct test for the MCP_CONNECTION_NONBLOCKING setdefault
The 1-line defensive setting is unit-test-shaped: a patch.dict(os.environ, {}, clear=False) + _run_async(run_agent_async("x")) + assert os.environ.get("MCP_CONNECTION_NONBLOCKING") == "0" would pin the contract that the var is set before the SDK runs. Worth adding alongside the existing test_init_log_cwd_* fixtures in tests/shared/egg_agent/test_client.py if you want a regression alarm; non-blocking because the surrounding production smoke tests will surface a tool-discovery failure if this ever silently regresses.
— Authored by egg
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
- Update stale line-range reference in test_sdk_surface.py (205-221 → 219-235) so the comment points at the current shared/egg_agent/client.py import block after the MCP_CONNECTION_NONBLOCKING setdefault bumped those imports down by 13 lines. - Align the Dockerfile gate's bypass message with the docs callout in docs/guides/per-agent-models.md: both surfaces now describe the same surgical 'drop this family from the for-loop' bypass, instead of one saying 'drop the family' and the other saying 'comment the loop out'. - Add two targeted unit tests for the MCP_CONNECTION_NONBLOCKING=0 setdefault in run_agent_async: one pinning the default-set behaviour and one pinning that an operator-set value is preserved. Regression alarm for the SDK 0.2.x background-connect behaviour shift.
|
Thanks for the careful re-review. Per-item disposition for the four nits in commit d5e94e3 (delta from 602b98d): 1. Stale line-range reference in
|
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Re-review of commit d5e94e3 (delta from 602b98d). No agent-mode design concerns.
The delta is three nit-level fixes from the prior re-review, all narrowly scoped:
- Stale line-range comment in
tests/sandbox/egg_agent_tools/test_sdk_surface.py:23— bumped to219-235to match the post-MCP_CONNECTION_NONBLOCKINGimport block. Doc-comment only. - Dockerfile gate error message aligned with
docs/guides/per-agent-models.md— both surfaces now describe the same surgical "drop the family from the for-loop" bypass instead of one saying "drop the family" and the other "comment the loop out". Operator-facing language consistency only; the gate behavior is unchanged. - Two new unit tests in
tests/shared/egg_agent/test_client.py(test_mcp_connection_nonblocking_default,test_mcp_connection_nonblocking_preserves_operator_override) — pin theos.environ.setdefault("MCP_CONNECTION_NONBLOCKING", "0")behaviour added in the previous round, including the operator-override preservation semantics. Pure regression alarm for the SDK 0.2.x background-connect shift; no behavior change.
None of these touch prompt construction, structured-output requirements, post-processing pipelines, direct LLM API calls outside the sandbox, hardcoded model IDs, or rigid procedure micromanagement.
— Authored by egg
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Re-review of commit d5e94e3 (delta from 602b98d). Three narrow fixes addressing the four nits from the prior re-review — verified end-to-end, no new issues.
Verification of the delta
-
tests/sandbox/egg_agent_tools/test_sdk_surface.py:23— Line range bumped205-221→219-235. Confirmed against the currentshared/egg_agent/client.py: thefrom claude_agent_sdk import (...)block runs lines 219-234 with the closing paren on 235. Comment now points at the right source. -
sandbox/Dockerfile:300-304— Error message now readsdrop '${family}' from the 'for family in ...' list above to skip the gate for that family (matches the bypass guidance in docs/guides/per-agent-models.md).Cross-checked the docs callout atdocs/guides/per-agent-models.md:117-118(temporarily dropfablefrom the Dockerfile gate) — both surfaces now describe the same surgical action. Operator-facing language only; no behavior change to the gate. -
tests/shared/egg_agent/test_client.py:472-493— Two new unit tests:test_mcp_connection_nonblocking_default—env.pop("MCP_CONNECTION_NONBLOCKING", None)→_run_async(run_agent_async("test prompt"))→assert os.environ.get(...) == "0". Goes through the realrun_agent_async(the function carrying thesetdefault), not a hand-built fixture; the SDK is mocked atclaude_agent_sdk.query, which works becauseclient.pydoes an in-functionfrom claude_agent_sdk import ... querythat re-resolves the patched binding on each call. Production code path is exercised.test_mcp_connection_nonblocking_preserves_operator_override— pre-sets to"1"viapatch.dict, runs the agent, asserts the value is still"1". Pins thesetdefaultsemantics that protect an operator-set value (a regression toos.environ[...] = "0"would silently un-do the #3137 fix and this test would catch it).- Test names match assertions;
patch.dictrestoresos.environon exit so tests are order-independent.
Disagreement on the type-checking-only symbols
The producer's disposition on nit #3 is correct: HookContext / HookInput / HookJSONOutput are gated under TYPE_CHECKING + from __future__ import annotations, so a 0.3 drop is invisible to runtime ImportError — which is the smoke test's primary failure mode. Mixing them in would conflate runtime-spawn-safety with type-check-safety; the right second guard for the latter is make lint's mypy pass, not an expanded smoke test. Acceptable to leave for follow-up.
No new issues
The delta is purely additive (one comment fix, one message alignment, two new tests). No production behavior changed since 602b98d. The MCP_CONNECTION_NONBLOCKING setdefault and its placement at client.py:210 — the load-bearing fix — are unchanged from the prior re-review and remain correct.
— Authored by egg
|
egg review completed. View run logs 8 previous review(s) hidden. |
Fixes #3136.
Problem
The first
issue-3077run crash-looped every refine/plan agent at spawn (160+ failed invocations): the sandbox image's Claude Code build doesn't know the barefablealias thatorchestrator/agent_model_resolution.pyuses as the tier-3 default for refine/plan roles.Root mechanism:
make buildruns plaindocker build, soARG CLAUDE_CODE_VERSION=stableis the operative value (the legacy egg CLI that passed a concrete version was removed in #1762). The install layer is keyed on the literal stringstable, BuildKit never re-runs it, and the image's CC froze at whatever stable pointed to when the layer first built.Compounding finding: the stable channel itself doesn't have
fableyet — current stable is 2.1.153, whose binary contains zeroclaude-fablestrings (verified by downloading and grepping the linux-arm64 release).latest(2.1.173, what the host runs) resolves it fine. So a rebuild against stable today would still crash-loop.Changes
sandbox/Dockerfile— three changes to the CC install:stable→latest. egg's built-in defaults track new model families at launch; the stable channel lags those launches by weeks.ADDof the channel manifest (downloads.claude.ai/claude-code-releases/latest— the same manifestinstall.shreads). BuildKit re-checks the URL content on every build, so the install layer re-runs exactly when the channel moves to a new version, and stays cached otherwise.fable,opus), rather than discovering it at spawn time in a live pipeline. No auth is mounted at build time, so the gate is a strings-level heuristic: a build that knows a family alias embeds the versionedclaude-<family>-*IDs the alias maps to (48 occurrences in 2.1.173; 0 in 2.1.153).claude-agent-sdk0.1.65 → 0.2.97 (sandbox/pyproject.tomlpin>=0.2.97,<0.3+ Dockerfile ARG). Per the pin's "validate manually on 0.2" note, I verified against 0.2.97 in a clean venv:query,ClaudeAgentOptions,create_sdk_mcp_server,tool, message/block types, permission results, hook typesClaudeAgentOptionsfield egg sets exists (permission_mode,setting_sources,can_use_tool,max_buffer_size,effort,mcp_servers,hooks, …)CLIJSONDecodeErroris still aClaudeSDKErrorsubclass and theexceeded maximum buffer sizemarker (Agent SDK message reader has 1MB JSON buffer limit — large tool results crash agents with exit code 255 #2804) is unchanged in the SDK sourcetests/sandbox/egg_agent_tools/test_sdk_surface.pypasses (pin-format test; the import smoke tests run in-sandbox where the SDK is installed).Not in this PR
The event pump retried the deterministic rc=1 failure 160+ times with no backoff or failure classification — filed separately as a follow-up (see issue link in comments).
Testing
make lintclean (hadolint on the modified Dockerfile included)downloads.claude.airelease binariesNote for deploy: the new CC lands on the next
make redeploy(which rebuilds all images including egg-sandbox and publishes via the loopback registry, #2999/#3109), and the gate makes that rebuild fail loudly if the channel ever regresses.