[teamleaderleo]

This PR makes acpmux recover failed turns in place instead of stranding sessions after account limits or gateway outages. It uses the existing RunSpec admission and account cooldown path; it does not add a second scheduler or revive by switching accounts.

Leo, verbatim (2026-10-29): "ok uh in addition i was asking you guys about that so you could help analyze my usage and how to make us both more effective and efficient".

Leo also said: "hmm what's going on with these? they get lost under network breaks and dont do anything?" and "also let's not do this anymore" about lane-revive switching accounts.

The implementation resumes the same session after Claude session/weekly limits, router capacity, HTTP 429/502/503/529, and harness_unavailable; honors Retry-After seconds/units/date, Claude reset hints, and persisted usage windows; preserves held messages; shows an acpmux-owned retry with reset timing; restores scheduled retries after daemon restart; and emits a visible error after its bounded attempt window. Stop, close, newer prompt, movement, explicit retry, dispatch refusal, and daemon shutdown end each timer.

Related designs and issues: HQ#5123 §3, HQ#4879 items 3-4, and audit PR #1544.

Performance

metric	before	after	delta	harness or run link
Named lanes failed or stuck on Claude 429 / router exhaustion	228 (coordinator pass, 2026-10-30 05:40Z)	rerun the same coordinator query after deployment	pending deployment measurement	coordinator capacity/stuck query
Sessions ending on a limit error with no later turn	16 / 1,177 (observed transcript-tail scan, 2026-10-30 02:47Z)	rerun the same 14-day scan after deployment	pending deployment measurement	tools/usage-report run --days 14 --github plus the same session-ending scan

Validation

• Red/green regression commit 435ac10d1a1 and subsequent implementation commits through 1e92656adce cover forced 429/502/503/harness_unavailable, weekly reset timing, queue preservation, restart restore, and stop/close behavior.
• Focused command: cd cmux-tui && cargo test -p acpmux --test hub_integration limit_recovery -- --nocapture through the hosted cmux-ci route. A prior hosted run ended without a broker receipt while shared quota was exhausted; the code was reviewed against the exact pushed head afterward.
• python3 scripts/cmux-next/prepush-gate.py: no new violations. verify-local.py: existing unrelated localization debt on base a1f302795f52c68f5ddfd400bafc39782551c169.

Risk: normal. Reset scheduling uses the event log and existing session queue; retry timers stop on the conditions named above.

Changelog: none

Agent session: agent session

[blacksmith-sh]

Found 3 test failures on Blacksmith runners:

Failures

Test	View Logs
tab: tabGroup.addTab is not a top-level row/tab: tabGroup.addTab is not a top-level row	View Logs
tab: tabGroup.create is not a top-level row/tab: tabGroup.create is not a top-level row	View Logs
tab: tabGroup.removeTab is not a top-level row/ tab: tabGroup.removeTab is not a top-level row	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review d2b2b3b: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr27924 -f ref=lane/usage-limit-recovery -f platform=macos -f backend_mode=local

web and size checks: green

Shipped in:

• cmux NEXT nightly 1.0.0-nightly.4125030115701 (b2ec1ee1b6)
• cmux NEXT nightly 1.0.0-nightly.4125068136801 (11ee3759e9)

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[teamleaderleo]

Review (codex gpt-6-astra high, router-review): SHIP at 5438df8

Range 8dd8bdc..5438df8 (merge-base...head, after merging feat-cmux-next).

1. [P2][follow-up] The configuration now documents twelve default retry waits (config.rs:1155), but the implementation still uses DEFAULT_DELAYS_MS with only four entries ([auto_resume.rs:115]). Default recovery therefore stops after four attempts, contrary to the documented behavior.

2. [P2][follow-up] duration_ms misparses long-form millisecond units: "Retry-After: 500 milliseconds" falls through to the generic m branch and becomes 30 minutes instead of 500 ms (reset_hint.rs:27-30).

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[teamleaderleo]

Review (codex gpt-6.1-sol xhigh, router-review): BLOCK at 5c857a1

1. [P1] [blocking] Retry completion introduces a deadlock. cmux-tui/crates/acpmux/src/hub/auto_resume.rs:1017
finish_auto_resume holds session.auto_resume while shutting_down() acquires shutdown_plan. Scheduling acquires these locks in reverse order through before_shutdown → set_retry. When an automatic retry fails again, its completion and the spawned successor scheduler can deadlock, blocking retries and other daemon operations. Keep one consistent lock order.

2. [P1] [blocking] A restart can consume a retry without running it. cmux-tui/crates/acpmux/src/hub/auto_resume.rs:945
   The shutdown guard permits dispatch when restart_requested is true. If a dead agent’s retry expires during transport draining, it claims the attempt, but harness startup rejects it because begin_shutdown already fenced starts. fail_unstarted_turn then records a terminal shutdown failure, which supersedes the saved retry on restoration. Fence dispatch during restarts while preserving the durable schedule.

3. [P1] [follow-up] Restored manual retries can run after Stop. cmux-tui/crates/acpmux/src/hub/auto_resume/retry_turn.rs:140
   Restoration installs Mark::Waiting, but Stop clears that mark to None. The restored task passes auto: None, so claim_failed_turn accepts the empty mark despite allow_stopped: false. A Stop before the task claims the turn therefore fails to cancel it. Require ownership of the restored turn’s waiting mark.

4. [P2] [follow-up] Long reset waits outlive Cloud reporting’s deadline. cmux-tui/crates/acpmux/src/hub/auto_resume.rs:552
   Retries can now wait days, but cloud_handoff.rs:477 still expires every scheduled retry after 65 minutes and reports failure while the retry remains pending. Carry nextAt into the reporter’s deadline. The added test at cloud_handoff_tests.rs:679 never advances time, so it misses this case.

Addressed in f63a239: #1 (shutdown plan read before the retry lock), #2 (no retry dispatch once any shutdown began; the durable wait stays for the next daemon), #3 (a restored manual retry refuses once a Stop cleared its wait). #4 filed as a follow-up.

[teamleaderleo]

Review (codex gpt-6-astra high, delta 5c857a1..f63a239): SHIP at f63a239

Delta review of the round-2 fixes against the gpt-6.1-sol BLOCK at 5c857a1:

1. FIXED [blocking]. finish_auto_resume samples shutdown state before taking auto_resume, removing the lock-order inversion with before_shutdown/set_retry.

2. FIXED [blocking]. auto_resume now refuses to fire after any shutdown begins, including restart handoff, preserving the durable retry schedule.

3. FIXED [follow-up]. Restored manual retries pass allow_stopped: false; a cleared None mark now returns retry.stopped.

No new delta-introduced blocking defect was found. git diff --check passes; no build/tests run per the read-only request. A separate shutdown race between the fire check and dispatch remains pre-existing and is outside this delta.

Follow-ups: none new.

Tests at f63a239: acpmux lib 2596/2597 (exe_still_names_a_program_whose_file_was_replaced is a known flake that passes alone; filed in our tracker), hub_integration 633/633. cmux-next-ios/rust is a base red (this PR changes nothing in cmux-next-ios).

[teamleaderleo]

Review carry: SHIP at 70db182

70db182 is cmux-bot's clean base merge of f63a239 (SHIP, codex gpt-6-astra high, delta review: #27924 (comment)). The PR's diff against its merge base has the same patch-id at both heads. Remaining red: cmux-next-ios/rust (base red; no cmux-next-ios change here).

[teamleaderleo]

Review (the owner lane's independent review, at its land request): SHIP at 70db182

Recorded by lane-gh-broker from this PR's owner lane, which asked to land this exact head after its independent review. next-autoland lands it once every red is the base's; a BLOCK at this head, a HOLD line or a hold label still stops it.

[teamleaderleo]

Review (codex gpt-6-astra high, delta 70db182..093ebe3): SHIP at 093ebe3

Lint-only delta (clippy collapsible_match in restore.rs; the collapsed arm falls through to the catch-all, behavior unchanged) on top of the SHIP at f63a239 / its clean base-merge carry at 70db182. clippy -p acpmux --all-targets -D warnings clean; restore tests 28/28.

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[teamleaderleo]

Review carry: SHIP at 916b9c3

916b9c3 is cmux-bot's clean base merge of 093ebe3 (SHIP: #27924 (comment)). Same patch-id against the merge base at both heads.

[teamleaderleo]

Review (the owner lane's independent review, at its land request): SHIP at 916b9c3

Recorded by lane-gh-broker from this PR's owner lane, which asked to land this exact head after its independent review. next-autoland lands it once every red is the base's; a BLOCK at this head, a HOLD line or a hold label still stops it.

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[teamleaderleo]

Review (codex gpt-6-astra high, delta 916b9c3..d2b2b3b): SHIP at d2b2b3b

Panic-ratchet fix: the 20 indexing_slicing and 12 string_slice hits this PR added (reset_hint.rs, restore.rs) now use get/split_at_checked/slice patterns; "Retry-After: 500 milliseconds" reads as 500 ms (closes that review follow-up). Reviewer: behavior-preserving, no findings. rust_panic_ratchet.py -p acpmux: not red; clippy -p acpmux --all-targets -D warnings clean; reset_hint/restore/auto_resume lib tests 70/70.