[teamleaderleo]

Leo, word for word (2026-10-26): "this needs to retry :("

subsession-card's ask, daemon half (a): "A turn that fails on a transient upstream error (HTTP 503, 529 overloaded, "no non-exhausted ... accounts", "overloaded_error", or a connection reset or EOF to the gateway) retries by itself with backoff, then gives up. It resumes the same session and turn, never a new session. Emit a state on the session or turn that the webview can draw. Non-transient errors (auth, a bad request, a refused permission) never retry."

Why these turns never came back

hub/turn_errors.rs read every one of them as an account limit, so resume_cause returned nothing and the turn sat failed:

• is_limit_error matched any 401 in the text. The SR2 gateway's errors name its port, (127.0.0.1:4016) / sr2-gateway…:4016, so every SR2 error read as a sign-in failure. That includes the 503 "no non-exhausted claude accounts" error and a plain ECONNRESET to the gateway.
• is_limit_error also matched overloaded, so a 529 overloaded_error and Codex's "Selected model is at capacity. overloaded" went to the fallback-profile path, or to nothing when there was no fallback.

What changed (PR 1 of 2; _acpmux/retry is PR 2)

• turn_errors: new is_capacity_error (HTTP 503 or 529, overloaded, "at capacity", "no non-exhausted … accounts") and pub fn transient_cause(error) -> Option<&'static str>. It returns capacity, transport, or None for limits, sign-in and anything else. hb-retry-subagent shares this classifier. resume_cause calls it and keeps its own agent_exited branch. Status codes (401, 429, 503, 529) now match only as whole numbers. overloaded is no longer a limit. Transport also covers unexpected eof and early eof.
• auto_resume: capacity errors go through the existing continuation path in the same session (prompt id P~resume-n, the harness's own resume), with a continuation text that names the capacity error.
• Session row (session/list _meta.acpmux, _acpmux/sessions, _acpmux/session_changed): retry: {attempt, of, reason, nextAt, promptId, turnId}, where reason is capacity | transport | agent_exited and nextAt is epoch ms. It is null when nothing waits, and it clears when the attempt fires, is skipped (Stop, close, a turn that took the chat) or runs out. New protocol types RetrySummary and RetryReason. The IR and the webview's generated types are regenerated.
• Events: auto_resume_scheduled gains of, reason and nextAt. auto_resume_exhausted gains reason. Watchers now hear all three auto_resume_* records as _acpmux/session_changed, so the row's retry comes and goes live.
• The wait's timer also wakes when its mark goes, so Stop or close records the skip and clears the row at once instead of up to 20 minutes later.

Field names match hb-acpmux's contract (retry-contract.md). subagentId? on the row is hb-retry-subagent's to add.

Decisions

• A capacity error does not try the fallback profile first. The SR2 503 comes from the pool that Claude's fallback profile points at, so hopping lands on the same exhausted pool. A 529 is the provider's overload whichever account asks. A hop also costs a fresh agent start (or a transcript rehydrate when the store differs) on what is usually a 15-second blip. Retrying in place at 10 s recovers soonest. Usage limits (429, "hit your limit", quota) and sign-in failures keep going to the fallback, unchanged.
• The default delays stay at 10 s, 1 min, 5 min and 20 min, each staggered up to a fifth. Claude Code has already retried internally before it reports the 503, so a shorter first wait would mostly hit the same full pool. When the pool really is out ("next account frees up in 50m"), the later attempts cover it.

Stop conditions (timers)

The one timer is each chat's existing auto-resume wait. It now ends at its delay or when its mark is dropped (Wake's Drop notifies it): Stop, close, a newer turn taking the chat, or the attempts running out. In every case the task then records the skip or fires once and exits. No new observer or loop.

Tests (red first)

• Red at 1be746f: turn_errors::capacity_errors_are_continued_in_place and a_dropped_gateway_connection_is_continued failed (an ECONNRESET naming the gateway's :4016 read as a 401). So did the integration tests auto_resume::capacity::* ("never scheduled") and the reason assertion added to automatic_continuations_stop_after_the_configured_attempts.
• Unit: the exact SR2 503 texts (including "retry after 2401s" and "429s" variants), the 529 overloaded_error JSON and "Selected model is at capacity." are capacity. 429 / "hit your limit" / usage limit / 401 / "Not logged in" / invalid API key / 400 invalid_request_error / permission refusals stay None.
• Integration (fake agent cap-fail-once: answers with the SR2 503): the turn continues in the same session with the same agent session id and prompt id p-cap~resume-1. While it waits, retry shows in hub.session_summary, in session/list and in _acpmux/sessions. It is null after the attempt fires. Stop clears it at once, and a watcher hears auto_resume_scheduled with retry and then auto_resume_skipped with retry: null. A prompt sent meanwhile clears the row, and the wait is skipped as superseded.
• Green at 20c03fa (Blacksmith fast-checks run 40253941821): cargo fmt -p acpmux makes no diff; cargo test -p acpmux --lib passes 813 tests, protocol_ir_is_current among them; --test hub_integration passes 411; clippy --all-targets is clean. The webview's bun run acpmux-protocol:check is current (run 40253470646).

Review

Codex gpt-6-astra (xhigh) review: SHIP at 98178f2 after three rounds (round 1: a refused attempt vanished without a watcher event, bad requests could retry through EOF words, status-less overloads lost route fallback; round 2: the agent_exited shortcut bypassed the refusal guard; each fixed red-then-green).

No capture: no UI changes here. The webview lane draws retry from these fields.

Compatible with #23921 (restart continuations): it reuses split_attempt, SUFFIX and MAX_ATTEMPTS unchanged, and this PR leaves the doc lines that PR edits untouched.

Agent session: big-red
🤖 Generated with Claude Code

[teamleaderleo]

Owned by hb-acpmux on big-red · session

[blacksmith-sh]

Found 10 test failures on Blacksmith runners:

Failures

Test	View Logs
/tmp/tmpk3pg_ctt/summary-missing.json (live):/summary-missing.json (live):	View Logs
/tmp/tmpyw31d80t/summary-fail.json (live):/summary-fail.json (live):	View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/ [new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows	View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames	View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/ [tabs] debug.tab_switch timed 0 of 40 switches	View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames	View Logs
bench OTHER_SWIFT_FLAGS is 'OTHER_SWIFT_FLAGS=$(inherited) -DCMUX_BENCH_HOOKS -DCMUX_BL OCKING_CHECKS'/ bench OTHER_SWIFT_FLAGS is 'OTHER_SWIFT_FLAGS=$(inherited) -DCMUX_BENCH_HOOKS -DCMUX_BL OCKING_CHECKS'	View Logs
bench OTHER_SWIFT_FLAGS is 'OTHER_SWIFT_FLAGS=$(inherited) -DCMUX_BENCH_HOOKS -DCMUX_BL OCKING_CHECKS'/ bench OTHER_SWIFT_FLAGS is 'OTHER_SWIFT_FLAGS=$(inherited) -DCMUX_BENCH_HOOKS -DCMUX_BL OCKING_CHECKS'	View Logs
test_the_nightly_is_covered (main.NightlyStepCoverageTests.test_the_nightly_is_cove red)/ test_the_nightly_is_covered (main.NightlyStepCoverageTests.test_the_nightly_is_cove red)	View Logs
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl ights_validate)/ test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl ights_validate)	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[teamleaderleo]

Bead cx-o4pcj created for this PR by the cmux coordinator.

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Gallery before/after for 98178f2: 16 states rendered, 3 images differ, 1889 more touched states not rendered. Images are in the run's artifact, not on this public PR.

gh run download 40257609997 -R manaflow-ai/cmux -n gallery-advisory-25232-98178f25f853fc695b82f7bcc610296ea74f5b6f (then read gallery-review.json)

States

State	Theme	Status	Diff	Picked by
agent-pane.chat-width/standard	Apple System Colors Light	unchanged	0%	imports types.ts (2 hops)
agent-pane.chat-width/standard	Apple System Colors	unchanged	0%	imports types.ts (2 hops)
agent-pane.chat-width/wide	Apple System Colors Light	unchanged	0%	imports types.ts (2 hops)
agent-pane.chat-width/wide	Apple System Colors	unchanged	0%	imports types.ts (2 hops)
agent-pane.chat-width/wide-blocks	Apple System Colors Light	changed	1.53%	imports types.ts (2 hops)
agent-pane.chat-width/wide-blocks	Apple System Colors	unchanged	0%	imports types.ts (2 hops)
agent-pane.composer-floats/scrolled-to-end	Apple System Colors Light	unchanged	0%	imports types.ts (2 hops)
agent-pane.composer-floats/scrolled-to-end	Apple System Colors	unchanged	0%	imports types.ts (2 hops)
agent-pane.composer-floats/composer-grown	Apple System Colors Light	unchanged	0%	imports types.ts (2 hops)
agent-pane.composer-floats/composer-grown	Apple System Colors	unchanged	0%	imports types.ts (2 hops)
agent-pane.docked-chat/new-chat	Apple System Colors Light	unchanged	0%	imports types.ts (2 hops)
agent-pane.docked-chat/new-chat	Apple System Colors	unchanged	0%	imports types.ts (2 hops)
agent-pane.docked-chat/after-turn	Apple System Colors Light	unchanged	0%	imports types.ts (2 hops)
agent-pane.docked-chat/after-turn	Apple System Colors	unchanged	0%	imports types.ts (2 hops)
agent-pane.docked-chat/working	Apple System Colors Light	unchanged	0%	imports types.ts (2 hops)
agent-pane.docked-chat/working	Apple System Colors	unchanged	0%	imports types.ts (2 hops)
agent-pane.docked-chat/typing-in-dock	Apple System Colors Light	unchanged	0%	imports types.ts (2 hops)
agent-pane.docked-chat/typing-in-dock	Apple System Colors	unchanged	0%	imports types.ts (2 hops)
agent-pane.docked-chat/long-title	Apple System Colors Light	unchanged	0%	imports types.ts (2 hops)
agent-pane.docked-chat/long-title	Apple System Colors	unchanged	0%	imports types.ts (2 hops)
agent-pane.gutter-select/from-left-margin	Apple System Colors Light	unchanged	0%	imports types.ts (2 hops)
agent-pane.gutter-select/from-left-margin	Apple System Colors	unchanged	0%	imports types.ts (2 hops)
agent-pane.gutter-select/from-right-margin	Apple System Colors Light	unchanged	0%	imports types.ts (2 hops)
agent-pane.gutter-select/from-right-margin	Apple System Colors	unchanged	0%	imports types.ts (2 hops)
agent-pane.inspector/chat-menu	Apple System Colors Light	unchanged	0%	imports types.ts (2 hops)
agent-pane.inspector/chat-menu	Apple System Colors	unchanged	0%	imports types.ts (2 hops)
agent-pane.inspector/wire-log	Apple System Colors	changed	0.0181%	imports types.ts (2 hops)
agent-pane.inspector/wire-log	Apple System Colors Light	changed	0.0141%	imports types.ts (2 hops)
agent-pane.orchestration-lab/local-queue	Apple System Colors Light	unchanged	0%	imports types.ts (2 hops)
agent-pane.orchestration-lab/local-queue	Apple System Colors	unchanged	0%	imports types.ts (2 hops)
agent-pane.orchestration-lab/build-box	Apple System Colors Light	unchanged	0%	imports types.ts (2 hops)
agent-pane.orchestration-lab/build-box	Apple System Colors	unchanged	0%	imports types.ts (2 hops)

_{Pick other states with a Gallery: entry.id, entry.id/variant line in the PR body.}

red: cmux-next checks (god files, concurrency, crash safety, l10n); no finished base run on 0fd73db to compare; do not merge until green; see lane-rules

[github-actions]

Shipped in:

• cmux NEXT nightly 1.0.0-nightly.4029396825001 (5392ef3326)
• cmux NEXT nightly 1.0.0-nightly.4030749448501 (b781e3b4a0)