Skip to content

cmux NEXT: Settings > Retry After Errors sets the automatic retry loop (on/off, seconds between tries, tries per error) - #31709

Merged
teamleaderleo merged 1 commit into
feat-cmux-nextfrom
lane/ask-082c-font-retry-rail
Nov 2, 2026
Merged

teamleaderleo merged 1 commit into
feat-cmux-nextfrom
lane/ask-082c-font-retry-rail

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Leo-ask: ask-082cdbe456

Leo, passing on Lawrence's notes (2026-10-28): "leo make sure font size everywhere (like text input area) is customizable", "should have retry button + configurable timer loop retry", "needs more left padding".

This PR builds the one part that was still missing: the configurable timer loop retry. Where the other parts stand:

  • Font size in the text input area: done in fix(next): let composer content follow the chat font size #26523 (merged 2026-10-28), which makes the composer follow Settings > Agent Chat > Chat Font Size (agentPane.display.font.size).
  • Retry button: done in 9e836df (2026-10-29). Continue or Retry leads the rate-limit and capacity error card.
  • Left padding on the prompt rail: handled in its own small PR.

What changes

acpmux already retries a turn that a rate limit, a capacity error or a lost connection ended (#25232, #27924), waiting 10 s, 1 min, 5 min and 20 min. Until now, only acpmux's own config.json (autoResumeDelaysMs) could change that loop. Three new settings under Settings > General > Retry After Errors now set it:

key default meaning
agents.autoRetry.enabled true Off: no automatic retry. The card's Retry still works.
agents.autoRetry.intervalSeconds 0 (0–3600) 0 keeps today's growing waits. N waits N seconds before every try.
agents.autoRetry.attempts 4 (1–12) Tries per failed turn.
  • acpmux reads them from the settings file at each retry (hub/auto_resume/settings.rs, through user_settings, as Sub-sessions does), so a change applies to the next failed turn. autoResumeDelaysMs in config.json, when set, still comes first, so the dev slot's [] keeps working. The defaults give exactly today's waits, and a test pins the section JSON's defaults and ranges to the daemon's constants.
  • Every reader of the loop goes through effective_delays: the turn retry, a restored wait, a chain of restarts, held agent messages and subagent retries.
  • The schema source is schemas/settings/source/agent-retry.json, generated into Swift by gen-settings-sources.py. Through the schema, the settings appear in Settings, the palette, cmux settings and MCP. MCP refuses all three as user-only (SettingsSchema+AgentPolicy), like sessions.resumeAgents. settings-schema.json and the settings reference are regenerated by base autofix.
  • Strings: 7 new keys in the CmuxNextSettings catalog, translated into all 21 locales.
  • The phone parity row is in cmux-next-ios/verify/parity-mac.md: the loop runs in acpmux, so phone chats get it, but the keys can only be edited on the Mac.

Timers: no new timer. The existing per-attempt sleep reads its wait from these settings, and Stop or close still ends it (auto_resume.rs).

risk: normal (a settings row plus where the daemon reads its existing waits).

Changelog

  • Added: Settings > Retry After Errors: turn automatic retries after a rate limit or lost connection on or off, and set the seconds between tries and how many run.

Observed / Not run

  • Ran: gen-settings-sources.py (writes the generated Swift), rustfmt, the pre-push gate. The settings.rs unit tests and the acpmux auto_resume tests run through cmux-ci; their result is posted below.
  • Not run yet: a tagged-build check of the palette row, cmux settings get|set agents.autoRetry.intervalSeconds and the MCP refusal.

Agent session: agent session

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Makes acpmux's automatic retry loop after rate limits, capacity errors, or lost connections configurable from Settings > Retry After Errors, instead of only via the daemon's config.json.

  • Adds three agents.autoRetry.* keys: enabled (on/off), intervalSeconds (0 keeps today's growing waits), and attempts (1–12 per failed turn), read from the settings file at each retry.
  • autoResumeDelaysMs in config.json still takes precedence when set, so the dev slot's [] (retry off) keeps working.
  • Disabling auto-retry still leaves the error card's Retry button working; changes apply to the next failed turn, and Stop or closing the chat still ends the loop.
  • All three keys are user-only in MCP, appearing in Settings, the palette, and cmux settings; the phone gets the retry behavior but has no editor for the keys.
  • The schema source in schemas/settings/source/agent-retry.json generates the Swift descriptors, with settings strings translated into all 21 locales.

Migration

  • No action needed; defaults reproduce exactly today's waits (10 s, 1 min, 5 min, then 20 min, 4 attempts).

Written for commit 1fc7836. Summary will update on new commits.

View guided diff Turn on auto-fix

…/off, seconds between tries, tries per error)

Lawrence via Leo (ask-082cdbe456): "should have retry button + configurable timer loop retry".
The error card's Retry has led capacity failures since 9e836df; acpmux already retries rate
limits, capacity and lost connections on its own (#25232, #27924), but only its config.json
could change the loop. agents.autoRetry.{enabled,intervalSeconds,attempts} in the settings file
now set it, read at each retry; config.json's autoResumeDelaysMs still comes first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-5224b647da981c2d3c39
@github-actions github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Nov 2, 2026
@blacksmith-sh

blacksmith-sh Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Found 12 test failures on Blacksmith runners:

Failures

Test View Logs
/tmp/tmp15alwjku/summary-missing.json (live):/summary-missing.json (live): View Logs
/tmp/tmpz2pjbno1/summary-fail.json (live):/summary-fail.json (live): View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows
View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs
main.DeveloperIdAppTests.test_download_prints_the_active_profile_only/
test_download_prints_the_active_profile_only
View Logs
stubs: /System/Library/Frameworks/Security.framework/Versions/A/Security has no stub/
Security has no stub
View Logs
stubs: CoreFoundation.tbd: install-name /System/Library/Frameworks/
CoreFoundation.framework/Versions/A/CoreFoundation, missing 1: ['_CFRetain']/
CoreFoundation, missing 1: ['_CFRetain']
View Logs
stubs: CoreFoundation.tbd: install-name /System/Library/Frameworks/
CoreFoundation.framework/Versions/A/CoreFoundation, missing 1: ['_log2']/
CoreFoundation, missing 1: ['_log2']
View Logs

...and 2 more test failures. View all on Blacksmith

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (subagent claude-opus-5-5 medium): SHIP at 1fc7836

No blocking findings. With no settings file, or a file without these keys, every path (turn retry, restore, restart chain, held messages, subagent retry) keeps today's DEFAULT_DELAYS_MS and attempt count. No file is read under the config lock. Contention in restart_resume still gives the default attempts. The generated Swift matches (gen-settings-sources.py --check). Each key is in one agent-policy table, the xcstrings parse, and all 7 keys have 21 locales.

Follow-ups:

  • restore.rs: after wait_startup(), a restored wait still checks only config.json's []. If Settings retries are turned off during the startup wait, that one wait still fires.
  • Make the acpmux unit tests that rely on unset autoResumeDelaysMs independent of the host's settings file (CMUX_NEXT_CONFIG_FILE pointed at a temp file).
  • The settings file is read with a blocking read on async workers. Consider spawn_blocking or an mtime cache.

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Not verified: macOS jobs did not run: compile and app tests were skipped on 1fc7836cd7; ci-status is green without them.

CI passes on 1fc7836cd7 (run 41382183008 attempt 1).
macOS jobs did not run: compile and app tests were skipped: Fast static checks skipped (the macos job needs both to succeed). Re-run CI once they pass, before merging.

Written by scripts/ci/classify_failures.py (ci-failure-attribution.yml); signatures are its SIGNATURES table. A machine verdict is the runner's fault, not this PR's; yours means the failing file is one this PR changes, also red on main that main's latest full suite fails the same way, seen on other PRs that it failed on another pull request's run lately.

red: cmux-next checks (god files, concurrency, crash safety, l10n); no finished base run on fbaa7a5 to compare; do not merge until green; see lane-rules

Default changes

This pull request changes 12 defaults at 1fc7836cd783 (what everyone gets without configuring anything):

  • tunable-source.CmuxNextSettings/AgentRetrySettings.swift:parse: (none) -> "0a4cf1e260d46866176820d9c1ddc0202b471e52d3d8171f3659b1663067f89b"
  • tunable-source.CmuxNextSettings/CmuxConfigSnapshot.swift:parse: "42b7249d36451bb5b87b5b804555236506252030aeb061928c20134978d18eb2" -> "d1ee289af2553fdd99aedad540a615ab3ea745334082a2b96fd840189e6876f0"
  • tunable-source.CmuxNextSettings/Schema/SettingsSchema+AgentRetry.generated.swift:autoRetryAttemptsDefault: (none) -> "cc5416a82eb34314cd2d3b19cfd5d01ca2ee10d740c1dfde02c4f59ae5ada8e1"
  • tunable-source.CmuxNextSettings/Schema/SettingsSchema+AgentRetry.generated.swift:autoRetryAttemptsPath: (none) -> "f8df405e40ff2bd350e4117b9ad5f1781fcab1b5c99566f2bb4b0c21338d830e"
  • tunable-source.CmuxNextSettings/Schema/SettingsSchema+AgentRetry.generated.swift:autoRetryAttemptsRange: (none) -> "97cdd8a22a9642b03f9d2d9181aeb3ac8f87dc6ae8963b8540ca86f6b31bd067"
  • tunable-source.CmuxNextSettings/Schema/SettingsSchema+AgentRetry.generated.swift:autoRetryEnabledDefault: (none) -> "67ff8e608ad6a75337bf8afdd4c9a77a4a6cb8238519387a1475df0fe830ba60"
  • tunable-source.CmuxNextSettings/Schema/SettingsSchema+AgentRetry.generated.swift:autoRetryEnabledPath: (none) -> "73d33f2c8bb0b0dbc6519cb9a7bd675f4e61246aebc274b02c8912c53cc6c4d0"
  • tunable-source.CmuxNextSettings/Schema/SettingsSchema+AgentRetry.generated.swift:autoRetryIntervalDefault: (none) -> "e8b56d6d0f319df2b28e442d68b406fbb591bf11a1f0220cb68c1aeb099c2046"
  • tunable-source.CmuxNextSettings/Schema/SettingsSchema+AgentRetry.generated.swift:autoRetryIntervalPath: (none) -> "73c666717b6fab8fd9921f78ddd91f961634927d41f8fefaf8d0f59e32fca9a2"
  • tunable-source.CmuxNextSettings/Schema/SettingsSchema+AgentRetry.generated.swift:autoRetryIntervalRange: (none) -> "c028bdd9b971b539d5efa6b8857bb9a0638100fa710b73c8ce9f13244c580fb1"
  • tunable-source.CmuxNextSettings/Schema/SettingsSchema+AgentRetry.generated.swift:descriptors: (none) -> "8cb1cf5e0e91fbfa917d651af8f818ede7d7a4e88c51db389b47bf39c853c9f4"
  • tunable-source.CmuxNextSettings/Schema/SettingsSchema.swift:all: "97afc91c4d604c283fb296263f0892aa22224da255a2e93b270c395da21e8e6c" -> "7565a04ad833b58578f4a711494a642470b097c7b27786cd9ee33bb9aed422de"

Signed off on this commit by review (subagent claude-opus-5-5 medium).

Merge receipt: merged as 030cb46514b1 with these defaults changed, signed off by review (subagent claude-opus-5-5 medium).

Regenerate the snapshot with python3 scripts/ci/defaults_guard.py snapshot.

Shipped in:

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Checks for 1fc7836:

  • acpmux: cargo test -p acpmux --lib -- auto_resume subagent_retry digest restart_resume passed, 119 tests (run 41382203013). The 6 new auto_resume::settings tests passed (run 41382856730).
  • Swift: cmux-ci compile-check --package CmuxNext passed (210 s, job af5a61ed5f7c6560ff235262) on this head merged with the current base. The base's stale Cargo.lock, which made every compile check fail, was fixed in base red: cmux-tui/Cargo.lock follows fs4's move between crates #31717.

@teamleaderleo
teamleaderleo merged commit 030cb46 into feat-cmux-next Nov 2, 2026
175 of 208 checks passed
@teamleaderleo
teamleaderleo deleted the lane/ask-082c-font-retry-rail branch November 2, 2026 03:39
lawrencecchen added a commit that referenced this pull request Nov 2, 2026
… (panic ratchet, #31709)

#31709 (030cb46, "Settings > Retry After Errors sets acpmux's retry loop")
added DEFAULT_DELAYS_MS[n.min(DEFAULT_DELAYS_MS.len() - 1)] in
hub/auto_resume/settings.rs, one more clippy::indexing_slicing hit in acpmux.
With the #30726 hits fixed, acpmux is still 687 > 686 (hq run 41386195677).

get(n), else last(), is the same wait: past the defaults the last one repeats.
The array is a non-empty const, so the MIN_DELAY_MS fallback never applies.
@lawrencecchen

Copy link
Copy Markdown
Contributor

Base red from this PR (author: Leo, teamleaderleo). settings_delays in cmux-tui/crates/acpmux/src/hub/auto_resume/settings.rs reads DEFAULT_DELAYS_MS[n.min(DEFAULT_DELAYS_MS.len() - 1)], one more clippy::indexing_slicing hit in acpmux (the Rust panic ratchet is red on feat-cmux-next). #31773 changes it to DEFAULT_DELAYS_MS.get(n).or(DEFAULT_DELAYS_MS.last()), with the same waits. Please run cmux-ci run --class light --script scripts/ci/cmux-tui-rust-check.sh --arg panic --os linux --ref <head sha> before you merge acpmux code.

lawrencecchen added a commit that referenced this pull request Nov 2, 2026
…ic ratchet 697 > 686, #30726, #31709) (#31773)

* base red: acpmux process pages read rows without an index (panic ratchet indexing_slicing 696 > 686, #30726)

#30726 (fbaa7a5, "byte-bound background-process history with opt-in pages")
added ten clippy::indexing_slicing hits in acpmux hub/processes.rs: the page
window self.processes[start..end] in ProcessTable::page_snapshot and nine
serde_json indexes row["..."] in the too-large row of process_page_values.
With #31695 merged, acpmux is 696 > baseline 686, so the Rust panic ratchet
is red on feat-cmux-next.

The window is now processes.get(start..end): start is an ID's slot + 1 (at most
len) and end = min(start + limit, len), so the range is always in bounds and the
fallback never applies. The row fields use row.get(..), which serializes a
missing field as null, the same as an index on a missing key. The output does
not change. No baseline change and no allow.

Fleet panic step: RED at tip 7da73fe (indexing_slicing 697 > 686, hq run
41384159581) and at 4a94625 with the get_mut fix (696 > 686, hq run
41384159646, hits at processes.rs:327 and 384-388).

* base red: acpmux retry delays read the default table without an index (panic ratchet, #31709)

#31709 (030cb46, "Settings > Retry After Errors sets acpmux's retry loop")
added DEFAULT_DELAYS_MS[n.min(DEFAULT_DELAYS_MS.len() - 1)] in
hub/auto_resume/settings.rs, one more clippy::indexing_slicing hit in acpmux.
With the #30726 hits fixed, acpmux is still 687 > 686 (hq run 41386195677).

get(n), else last(), is the same wait: past the defaults the last one repeats.
The array is a non-empty const, so the MIN_DELAY_MS fallback never applies.

---------

Co-authored-by: cmux-bot[bot] <348531757+cmux-bot[bot]@users.noreply.github.com>
teamleaderleo added a commit that referenced this pull request Nov 2, 2026
…t-retry.json changes (#31883)

#31709 made acpmux read schemas/settings/source/agent-retry.json, but
cmux-tui-spec.yml's push and pull_request path filters didn't list it, so
tests/test_cargo_test_verdict.py fails on base.


Agent-Session: https://cmux.com/deeplink/agent?id=aid-fbe0da5ff5fd5f3e6eba

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

This branch was previously deployed

1 inactive deployment
ci-routing-agent — 1fc7836c Deployed Nov 2, 2026 by teamleaderleo via namespace autofix #11865
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants