Repository navigation
fix(sandbox): retry the SRT launch past a transient bridge-socket failure - #1327
Conversation
SRT gives socat 600 ms to bind both bridge sockets and Tend's proxy config is the branch that spawns two of them, so a slow runner loses the whole run before the agent starts. Retry the launch up to three times. Closes #1326
tend-agent
left a comment
There was a problem hiding this comment.
The retry itself checks out against SRT 0.0.76 as installed: initialize's network catch clears initializationPromise and managerContext before throwing, so re-entry is the supported path; the bridge-timeout branch SIGTERMs both socat processes itself, so a retried attempt inherits no orphans; and each attempt derives fresh randomBytes(8) socket paths, so there is no stale-socket collision. The 600 ms figure and the two-socat branch are both as the PR body describes.
One finding, on the tests rather than the change. The three new cases run sandbox_runtime.mjs as a real subprocess, and main()'s first statement is if (process.platform !== "linux") throw new Error("Tend SRT requires Linux") — so off Linux every one of them fails before reaching the retry loop they exist to cover: assert completed.returncode == 0 fails, and counts["initialize"] == LAUNCH_ATTEMPTS raises KeyError against the empty state file. The stand-in lifecycle's /usr/bin/bash is the second non-portable dependency. Nothing else in the suite executes a Linux-only path — test_prepare_agent_workspace.py shells out to git, test_sandbox.py to sys.executable, and generator/tests/__init__.py resolves bash through shutil.which specifically so the suite survives macOS's 3.2 — so this is the first uv run pytest failure a developer off Linux would hit, on a file where the other seven tests still pass.
I'm pushing a skipif for the three cases rather than leaving it as a suggestion, since this is the bot's own PR.
sandbox_runtime.mjs refuses to run on a non-Linux platform as main()'s first statement, so the three new tests that drive it as a real subprocess fail before reaching the retry loop they cover. The stand-in lifecycle's /usr/bin/bash is the second non-portable dependency.
Version bump, lockfile, and the `## 0.2.12` section the release workflow publishes verbatim as the GitHub Release notes. 17 commits since 0.2.11. The headline is the sandbox layout: the agent runs in the job's own checkout and home through a copy-on-write view, so what a consumer's `setup:` steps prepare reaches the agent without being rebuilt in `sandbox_setup:`, and the job environment crosses into the sandbox in a `0600` file rather than on `sudo`'s command line ([#1333](#1333), [#1334](#1334)). Alongside it: `/tmp` is writable inside the sandbox as scratch private to the run ([#1324](#1324)), CI sessions ignore the bot's claude.ai skills, plugins and connectors ([#1328](#1328)), the SRT launch retries a transient bridge-socket failure ([#1327](#1327)), and a cancelled gating check polls as unverified rather than green ([#1323](#1323)). Each bullet was written from the commit's diff rather than its subject. No real agent session has run through the new view yet (CI's `test-sandbox` lifecycle runs stub `claude`/`codex` binaries), so the release's live check after tagging is the first one that does. > _This was written by Claude Code on behalf of max-sixty_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Regenerates the nine `tend-*` workflows against tend 0.2.13 (from 0.2.11). The committed diff is only the pinned action ref and the `uv` setup version (`0.12.13` → `0.12.17`); everything below rides in the action and the plugin it loads. ## What changes for this repo **A cancelled gating check no longer reads as green.** `poll_pr_checks.py` now treats only `SUCCESS`, `NEUTRAL`, and `SKIPPED` as green and exits 2 — unverified — on anything else ([max-sixty/tend#1323](max-sixty/tend#1323)). Cancelled runs are routine here: main serializes the complete suite, so a newer commit replaces the one waiting for its turn, and `running-tend` already carries a rule for reading that shape. Under 0.2.11 a session polling its own push could report such a check as verified; it now has to say it could not verify. **CI sessions ignore the bot's claude.ai account.** Skills, plugins, and MCP connectors enabled on the `leaf-agent` account no longer load into sessions that push and post as it ([max-sixty/tend#1328](max-sixty/tend#1328)). Sessions run with this repo's skills and tend's own, and nothing else. **`/tmp` is writable inside the sandbox**, as scratch private to the run ([max-sixty/tend#1324](max-sixty/tend#1324)). A tool that hard-codes a `/tmp` path no longer needs an override; `$TMPDIR` stays the documented place to write. **The sandbox launch retries a transient bridge-socket failure**, up to three attempts ([max-sixty/tend#1327](max-sixty/tend#1327)). SRT gives socat 600 ms to bind, and a slow runner previously lost the whole job before the agent started. Three plugin-side fixes also land: `review-runs` can now cancel a run parked in GitHub's `waiting` state that holds its concurrency group without ever concluding ([#1320](max-sixty/tend#1320)); `ground-claims` reads a past run's code at that run's commit rather than at the current default branch ([#1319](max-sixty/tend#1319)); and `run-tend` corrects its list of instruction paths restored to the base version on a PR ([#1311](max-sixty/tend#1311)). ## No configuration change needed 0.2.12 moved the agent into the job's own checkout through a copy-on-write view, which would have changed where dependency setup belongs. 0.2.13 reverts it ([#1339](max-sixty/tend#1339)): `setup:` and `sandbox_setup:` behave as they did at 0.2.11, so `.config/tend.yaml` is unchanged here. This repo pins no `model:`, so the harness default applies as before. Full comparison: [0.2.11...0.2.13](max-sixty/tend@0.2.11...0.2.13). Co-authored-by: leaf-agent <318509791+leaf-agent@users.noreply.github.com>
Regenerates the `tend-*` workflows with tend 0.2.13 (from 0.2.11). Mechanically the diff is the action pin `max-sixty/tend/claude@0.2.11` → `@0.2.13` in the eight workflows that run the agent, the `astral-sh/setup-uv` pin `0.12.13` → `0.12.17` in the two that install uv directly (`tend-mention`, `tend-notifications`), and a regenerated header comment in all nine. The upgrade that matters for this repo is in CI polling. `poll_pr_checks.py` previously treated any gating check that settled without a red conclusion as green — so a required check belonging to a **cancelled** or **stale** run, or a `COMPLETED` check carrying no conclusion at all, let the bot report a PR as green when nothing had actually verified it. 0.2.13 splits those out as `unverified` and gives `poll` a distinct exit code for them (max-sixty/tend#1323). Practically: fewer "checks went green" claims on PRs whose required jobs were cancelled, at the cost of some polls ending unverified rather than green. The rest, in rough order of how visible it is here: - **Sandbox reliability** — the sandbox runtime now retries past a transient bridge-socket failure at launch (max-sixty/tend#1327). That failure is one of the shapes that produces a `tend-outage` issue, so it should show up as fewer of those. - **Sandbox environment** — root `/tmp` is writable inside the sandbox (max-sixty/tend#1324); the `run-tend` skill's note that it was read-only is gone. This repo's `running-tend` skill still restates that note, so it goes stale when this merges — I'll follow up separately rather than mix a skill edit into the generated-workflow diff. - **Instruction-path pinning** — the `run-tend` skill's description of which paths get restored from the PR base, and on which events, was wrong in 0.2.11; `.agents/` is now named alongside `.claude/`, and the event list is corrected (max-sixty/tend#1311). The restore behavior itself is unchanged. - **Auth hygiene** — the bot's claude.ai account is kept out of CI sessions (max-sixty/tend#1328). - **Harness pins** — claude-code 2.1.278 (max-sixty/tend#1330), uv 0.12.17 (max-sixty/tend#1331). `.config/tend.yaml` pins no model, so the harness-default check found nothing to adjust. Full comparison: max-sixty/tend@0.2.11...0.2.13 <details><summary>Copy-on-write sandbox view: landed and reverted within this range</summary> max-sixty/tend#1333 and max-sixty/tend#1334 moved the agent into the job's own checkout and home through a copy-on-write view; max-sixty/tend#1339 reverted both before 0.2.13. Nothing from that work is in this diff — noted so a later reader doesn't chase it out of the commit list. </details> Co-authored-by: worktrunk-bot <254187624+worktrunk-bot@users.noreply.github.com>
SRT's
initializeLinuxNetworkBridgewaits for socat to bind the bridge sockets across five probes on ani * 100ms backoff — sleeps of 0, 100, 200 and 300 ms, so 600 ms in total — and Tend's network config is the branch that spends it fastest.sandbox_runtime.mjssetshttpProxyPortfromTEND_PROXY_PORTand leavessocksProxyPortunset, which makes SRT'sneedLocalProxytrue and resolve SOCKS to its own mux port. The two ports then differ, a second socat is spawned, and both sockets have to appear inside that one budget. When a loaded runner misses it,SandboxManager.initializethrows, nothing retries, and the run is spent before the agent exists: zero turns, no comment posted, no session log.initializeSandboxnow retries the launch up to three times, with a growing backoff and an awaitedreset()between attempts. Retrying is safe at exactly this point and nowhere later — SRT clears its own state on an initialization error so the call can be re-entered, each attempt names fresh random socket paths so there is no stale-socket collision, and nothing has run inside the sandbox yet, so there is no partial work to reconcile. A sandbox that genuinely cannot come up still fails on SRT's own message and the same exit code, so failure reporting and outage tracking are unchanged; the cost of the retry is at most three seconds.The reproduction drives the real
sandbox_runtime.mjsunder a stand-inSandboxManagerinjected throughTEND_SRT_ENTRY, the same variable the action sets, so it exercises the launch path rather than a copy of it. Before the change, one inducedinitializefailure ends the process at exit 1 with the reported message and the lifecycle never spawned; after it, the lifecycle runs and the process exits 0. Two more tests hold the edges: the retries stay bounded and surface SRT's original diagnosis on exhaustion, and areset()that itself rejects — which SRT's own error path tolerates by never observing it — is logged rather than allowed to replace the launch error.On the rule against retries.
CLAUDE.mdsays not to add retries "for rare event shapes or small runner-time savings", and to keep the actions as linear plumbing. What this failure costs is not runner time but the run's entire outward action — the review that never gets posted — and the remedy is three lines of loop in already-tested code rather than a new gate or a second implementation layer. Widening the budget upstream in SRT would be the better fix; it is not one a pin bump reaches, since the loop is byte-identical atv0.0.77and onmain.Source the analysis was checked against
Read from
anthropics/sandbox-runtimeatc392e6cf9f8df957c66d9ab1461e2cfa99b1ab5d, the commit tagv0.0.76resolves to, which is thesandbox_runtime_versionboth harness actions pin:src/sandbox/linux-sandbox-utils.ts— themaxAttempts = 5/i * 100loop, its per-callrandomBytes(8)socket paths, and the separateLinux bridge process died unexpectedlybranch this failure is not.src/sandbox/sandbox-manager.ts—needLocalProxy, which puts a consumer that sets onlyhttpProxyPorton the two-socat path.sandbox-manager.tsagain, at the initializationcatch: it clearsinitializationPromiseandmanagerContext"so initialization can be retried", and callsreset()without awaiting the promise or observing its rejection.Closes #1326 — automated triage