Skip to content

fix(sandbox): retry the SRT launch past a transient bridge-socket failure - #1327

Merged
max-sixty merged 2 commits into
mainfrom
fix/issue-1326
Sep 20, 2026
Merged

max-sixty merged 2 commits into
mainfrom
fix/issue-1326

Conversation

@tend-agent

Copy link
Copy Markdown
Collaborator

SRT's initializeLinuxNetworkBridge waits for socat to bind the bridge sockets across five probes on an i * 100 ms backoff — sleeps of 0, 100, 200 and 300 ms, so 600 ms in total — and Tend's network config is the branch that spends it fastest. sandbox_runtime.mjs sets httpProxyPort from TEND_PROXY_PORT and leaves socksProxyPort unset, which makes SRT's needLocalProxy true and resolve SOCKS to its own mux port. The two ports then differ, a second socat is spawned, and both sockets have to appear inside that one budget. When a loaded runner misses it, SandboxManager.initialize throws, nothing retries, and the run is spent before the agent exists: zero turns, no comment posted, no session log.

initializeSandbox now retries the launch up to three times, with a growing backoff and an awaited reset() between attempts. Retrying is safe at exactly this point and nowhere later — SRT clears its own state on an initialization error so the call can be re-entered, each attempt names fresh random socket paths so there is no stale-socket collision, and nothing has run inside the sandbox yet, so there is no partial work to reconcile. A sandbox that genuinely cannot come up still fails on SRT's own message and the same exit code, so failure reporting and outage tracking are unchanged; the cost of the retry is at most three seconds.

The reproduction drives the real sandbox_runtime.mjs under a stand-in SandboxManager injected through TEND_SRT_ENTRY, the same variable the action sets, so it exercises the launch path rather than a copy of it. Before the change, one induced initialize failure ends the process at exit 1 with the reported message and the lifecycle never spawned; after it, the lifecycle runs and the process exits 0. Two more tests hold the edges: the retries stay bounded and surface SRT's original diagnosis on exhaustion, and a reset() that itself rejects — which SRT's own error path tolerates by never observing it — is logged rather than allowed to replace the launch error.

On the rule against retries. CLAUDE.md says not to add retries "for rare event shapes or small runner-time savings", and to keep the actions as linear plumbing. What this failure costs is not runner time but the run's entire outward action — the review that never gets posted — and the remedy is three lines of loop in already-tested code rather than a new gate or a second implementation layer. Widening the budget upstream in SRT would be the better fix; it is not one a pin bump reaches, since the loop is byte-identical at v0.0.77 and on main.

Source the analysis was checked against

Read from anthropics/sandbox-runtime at c392e6cf9f8df957c66d9ab1461e2cfa99b1ab5d, the commit tag v0.0.76 resolves to, which is the sandbox_runtime_version both harness actions pin:

  • src/sandbox/linux-sandbox-utils.ts — the maxAttempts = 5 / i * 100 loop, its per-call randomBytes(8) socket paths, and the separate Linux bridge process died unexpectedly branch this failure is not.
  • src/sandbox/sandbox-manager.ts — needLocalProxy, which puts a consumer that sets only httpProxyPort on the two-socat path.
  • sandbox-manager.ts again, at the initialization catch: it clears initializationPromise and managerContext "so initialization can be retried", and calls reset() without awaiting the promise or observing its rejection.

Closes #1326 — automated triage

SRT gives socat 600 ms to bind both bridge sockets and Tend's proxy
config is the branch that spawns two of them, so a slow runner loses the
whole run before the agent starts. Retry the launch up to three times.

Closes #1326

@tend-agent tend-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The retry itself checks out against SRT 0.0.76 as installed: initialize's network catch clears initializationPromise and managerContext before throwing, so re-entry is the supported path; the bridge-timeout branch SIGTERMs both socat processes itself, so a retried attempt inherits no orphans; and each attempt derives fresh randomBytes(8) socket paths, so there is no stale-socket collision. The 600 ms figure and the two-socat branch are both as the PR body describes.

One finding, on the tests rather than the change. The three new cases run sandbox_runtime.mjs as a real subprocess, and main()'s first statement is if (process.platform !== "linux") throw new Error("Tend SRT requires Linux") — so off Linux every one of them fails before reaching the retry loop they exist to cover: assert completed.returncode == 0 fails, and counts["initialize"] == LAUNCH_ATTEMPTS raises KeyError against the empty state file. The stand-in lifecycle's /usr/bin/bash is the second non-portable dependency. Nothing else in the suite executes a Linux-only path — test_prepare_agent_workspace.py shells out to git, test_sandbox.py to sys.executable, and generator/tests/__init__.py resolves bash through shutil.which specifically so the suite survives macOS's 3.2 — so this is the first uv run pytest failure a developer off Linux would hit, on a file where the other seven tests still pass.

I'm pushing a skipif for the three cases rather than leaving it as a suggestion, since this is the bot's own PR.

sandbox_runtime.mjs refuses to run on a non-Linux platform as main()'s
first statement, so the three new tests that drive it as a real
subprocess fail before reaching the retry loop they cover. The stand-in
lifecycle's /usr/bin/bash is the second non-portable dependency.
@max-sixty
max-sixty merged commit 30e1e36 into main Sep 20, 2026
11 checks passed
@max-sixty
max-sixty deleted the fix/issue-1326 branch September 20, 2026 15:09
@max-sixty max-sixty mentioned this pull request Sep 21, 2026
max-sixty added a commit that referenced this pull request Sep 21, 2026
Version bump, lockfile, and the `## 0.2.12` section the release workflow
publishes verbatim as the GitHub Release notes.

17 commits since 0.2.11. The headline is the sandbox layout: the agent
runs in the job's own checkout and home through a copy-on-write view, so
what a consumer's `setup:` steps prepare reaches the agent without being
rebuilt in `sandbox_setup:`, and the job environment crosses into the
sandbox in a `0600` file rather than on `sudo`'s command line
([#1333](#1333),
[#1334](#1334)). Alongside it:
`/tmp` is writable inside the sandbox as scratch private to the run
([#1324](#1324)), CI sessions
ignore the bot's claude.ai skills, plugins and connectors
([#1328](#1328)), the SRT launch
retries a transient bridge-socket failure
([#1327](#1327)), and a cancelled
gating check polls as unverified rather than green
([#1323](#1323)).

Each bullet was written from the commit's diff rather than its subject.
No real agent session has run through the new view yet (CI's
`test-sandbox` lifecycle runs stub `claude`/`codex` binaries), so the
release's live check after tagging is the first one that does.

> _This was written by Claude Code on behalf of max-sixty_

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
max-sixty pushed a commit to max-sixty/leaf that referenced this pull request Sep 21, 2026
Regenerates the nine `tend-*` workflows against tend 0.2.13 (from
0.2.11). The committed diff is only the pinned action ref and the `uv`
setup version (`0.12.13` → `0.12.17`); everything below rides in the
action and the plugin it loads.

## What changes for this repo

**A cancelled gating check no longer reads as green.**
`poll_pr_checks.py` now treats only `SUCCESS`, `NEUTRAL`, and `SKIPPED`
as green and exits 2 — unverified — on anything else
([max-sixty/tend#1323](max-sixty/tend#1323)).
Cancelled runs are routine here: main serializes the complete suite, so
a newer commit replaces the one waiting for its turn, and `running-tend`
already carries a rule for reading that shape. Under 0.2.11 a session
polling its own push could report such a check as verified; it now has
to say it could not verify.

**CI sessions ignore the bot's claude.ai account.** Skills, plugins, and
MCP connectors enabled on the `leaf-agent` account no longer load into
sessions that push and post as it
([max-sixty/tend#1328](max-sixty/tend#1328)).
Sessions run with this repo's skills and tend's own, and nothing else.

**`/tmp` is writable inside the sandbox**, as scratch private to the run
([max-sixty/tend#1324](max-sixty/tend#1324)). A
tool that hard-codes a `/tmp` path no longer needs an override;
`$TMPDIR` stays the documented place to write.

**The sandbox launch retries a transient bridge-socket failure**, up to
three attempts
([max-sixty/tend#1327](max-sixty/tend#1327)).
SRT gives socat 600 ms to bind, and a slow runner previously lost the
whole job before the agent started.

Three plugin-side fixes also land: `review-runs` can now cancel a run
parked in GitHub's `waiting` state that holds its concurrency group
without ever concluding
([#1320](max-sixty/tend#1320)); `ground-claims`
reads a past run's code at that run's commit rather than at the current
default branch ([#1319](max-sixty/tend#1319));
and `run-tend` corrects its list of instruction paths restored to the
base version on a PR
([#1311](max-sixty/tend#1311)).

## No configuration change needed

0.2.12 moved the agent into the job's own checkout through a
copy-on-write view, which would have changed where dependency setup
belongs. 0.2.13 reverts it
([#1339](max-sixty/tend#1339)): `setup:` and
`sandbox_setup:` behave as they did at 0.2.11, so `.config/tend.yaml` is
unchanged here. This repo pins no `model:`, so the harness default
applies as before.

Full comparison:
[0.2.11...0.2.13](max-sixty/tend@0.2.11...0.2.13).

Co-authored-by: leaf-agent <318509791+leaf-agent@users.noreply.github.com>
max-sixty pushed a commit to max-sixty/worktrunk that referenced this pull request Sep 21, 2026
Regenerates the `tend-*` workflows with tend 0.2.13 (from 0.2.11).
Mechanically the diff is the action pin `max-sixty/tend/claude@0.2.11` →
`@0.2.13` in the eight workflows that run the agent, the
`astral-sh/setup-uv` pin `0.12.13` → `0.12.17` in the two that install
uv directly (`tend-mention`, `tend-notifications`), and a regenerated
header comment in all nine.

The upgrade that matters for this repo is in CI polling.
`poll_pr_checks.py` previously treated any gating check that settled
without a red conclusion as green — so a required check belonging to a
**cancelled** or **stale** run, or a `COMPLETED` check carrying no
conclusion at all, let the bot report a PR as green when nothing had
actually verified it. 0.2.13 splits those out as `unverified` and gives
`poll` a distinct exit code for them (max-sixty/tend#1323). Practically:
fewer "checks went green" claims on PRs whose required jobs were
cancelled, at the cost of some polls ending unverified rather than
green.

The rest, in rough order of how visible it is here:

- **Sandbox reliability** — the sandbox runtime now retries past a
transient bridge-socket failure at launch (max-sixty/tend#1327). That
failure is one of the shapes that produces a `tend-outage` issue, so it
should show up as fewer of those.
- **Sandbox environment** — root `/tmp` is writable inside the sandbox
(max-sixty/tend#1324); the `run-tend` skill's note that it was read-only
is gone. This repo's `running-tend` skill still restates that note, so
it goes stale when this merges — I'll follow up separately rather than
mix a skill edit into the generated-workflow diff.
- **Instruction-path pinning** — the `run-tend` skill's description of
which paths get restored from the PR base, and on which events, was
wrong in 0.2.11; `.agents/` is now named alongside `.claude/`, and the
event list is corrected (max-sixty/tend#1311). The restore behavior
itself is unchanged.
- **Auth hygiene** — the bot's claude.ai account is kept out of CI
sessions (max-sixty/tend#1328).
- **Harness pins** — claude-code 2.1.278 (max-sixty/tend#1330), uv
0.12.17 (max-sixty/tend#1331).

`.config/tend.yaml` pins no model, so the harness-default check found
nothing to adjust.

Full comparison:
max-sixty/tend@0.2.11...0.2.13

<details><summary>Copy-on-write sandbox view: landed and reverted within
this range</summary>

max-sixty/tend#1333 and max-sixty/tend#1334 moved the agent into the
job's own checkout and home through a copy-on-write view;
max-sixty/tend#1339 reverted both before 0.2.13. Nothing from that work
is in this diff — noted so a later reader doesn't chase it out of the
commit list.

</details>

Co-authored-by: worktrunk-bot <254187624+worktrunk-bot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sandbox launch dies on SRT's 600 ms bridge-socket wait, and nothing retries it

2 participants