[teamleaderleo]

Status at 05367c0: Codex (gpt xhigh) passes this head. The acpmux watchdog sends its signal by the daemon's audit token (proc_signal_with_audittoken, looked up at run time because macOS 14.0 and 14.1 lack it), so a pid handed to another process is never signalled however long the app stalls.

Leo-ask: ask-98502f19a4
Leo-ask: ask-48bbaec1d4
Leo-ask: ask-e8a08a017c

Leo: "OUR SHIT IS HARDLY EVER GOING TO GO IDLE. IT JUST WONT. YOU'LL JUST HAVE TO ACCEPT IT BREAKING AT SOME POINT BUT IT RESUMES WHATEVER IT INTERRUPTED. END OF STORY FOR THE DAEMON UPDATE."

So no wait here is open-ended: a stale cmux-tui waits at most 2 minutes for a quiet moment and a stale acpmux with agent hosts at most 10 minutes, then each is handed off between requests. #24026 then restarts a stale local acpmux at once and shows its build in Settings > Updates.

Summary

After a cmux NEXT update (nightly, beta or release), the app's own daemons could stay on the old build. A local acpmux with no agent hosts was kept running indefinitely ("its agents would end"). A cmux-tui found stale after launch waited for the manual Restart button. A daemon that was alive but hung had to be restarted by hand. This PR is the local-app half of keeping every daemon on the bundled build:

• Stale acpmux. At every lookup (launch and each reconnect), a daemon of another build is handed off once no session is running a turn or waiting on a permission. This now includes daemons with no agent hosts; their chats resume from disk at the next prompt.
  • While it is busy, the app waits on a one-shot _acpmux/watch. A daemon with agent hosts waits at most 10 minutes; the wait races a deadline on that cap, so a long turn that only streams output can't hold it past the cap.
  • Health check: the next daemon must report the bundled build within 5 s. It gets one retry, then the "acpmux did not start" banner shows.
• Hung acpmux. If 3 status calls miss their deadline across 20 s, a 5 s confirm runs, then SIGTERM. SIGKILL follows after a 10 s exit wait. This happens at most once per pid. Agent hosts survive SIGTERM and the next daemon adopts them.
  • The daemon is known by its pid and start time (proc_pidinfo), checked again right before each signal and only after the kernel exit watch is on the pid. A pid reused by another process counts as the daemon's exit and is never signalled.
• Stale cmux-tui. QuietDaemonUpdate checks every handshake against the bundled build. The build is read again each time, so an in-place app update is noticed. A stale daemon is handed off with the existing lossless handOffForUpdate once quiet:
  • Quiet means the person has been idle 60 s or the app is in the background, no menu or drag is in progress, no app request is in flight, and the owner has no CLI, MCP, chief-chat or TUI client attached.
  • It waits for that for at most 2 minutes from when the daemon was found stale. After that, only an app request in flight or a menu, drag or resize loop defers it, retrying every 5 s. An attached TUI exits and reattaches; an attached cmux chief chat exits (code 3) and a CLI or MCP call in flight fails, and both can be run again.
  • Health check: the new daemon must report the bundled build within 90 s. It gets one retry, then a "Couldn't restart cmux-tui" toast.
• Hung cmux-tui owner. After 2 or more stuck server ensure results over 30 s, the owner is replaced (--replace-unresponsive) with no confirm. This happens once per owner process and at most 3 times per launch. An owner answering server.busy is never replaced.

Each behaviour has a default-on Debug Settings toggle: agent.replaceStaleAcpmuxWhenQuiet, agent.restartUnresponsiveAcpmux, cmuxTUI.restartStaleWhenQuiet, cmuxTUI.restartUnresponsive. The rules are written up in plans/cmux-next/server-updates-DESIGN.md (rule 2a/2b and the status table).

Coming next, in separate PRs:

• acpmux host update with rollback, idle check, arch check and migration of hand-written units;
• automatic updates of ssh peers and cloud machines to the app's pinned build;
• a daemon-capabilities file for web-ota compatibility.

Testing

Tests added:

• CmuxNextAgentPaneTests
  • AcpmuxQuietHandoffTests: a stale daemon without agent hosts, busy then quiet, the agent-host cap, the health check with retry and banner; aBusyDaemonWithAgentHostsIsHandedOffAtTheCapWithNoEvent (the cap fires with no event and no reconnect, and not before).
  • AcpmuxUnresponsiveWatchTests: the miss window, confirm, once per pid, toggle off. AcpmuxUnresponsiveIdentityTests: a reused pid is never signalled; a daemon that exits after its identity check gets no signal (aDaemonThatExitsAfterItsCheckGetsNoSignal); a signal whose token names an earlier process with the pid is refused by the kernel (aSignalForAnotherProcessWithThePidIsRefused).
• CmuxNextDaemonTests
  • UnresponsiveOwnerWatchTests, IdleOwnerRestartRetryTests.
  • QuietUpdateKeepsTerminalTests: a live cmux-tui in a temp session. The quiet restart waits for a CLI call and the terminal keeps running.
• CmuxNextAppTests
  • QuietDaemonUpdateTests: covers an old daemon next to an app updated while running, a successor of the wrong build or one that never connects, anAttachedClientBetweenTurnsHoldsTheRestartOnlyUntilTheCap and pastTheCapARequestInFlightStillDefersIt.
  • DaemonAutoRestartTests, plus ServerUpdateNoticeTests.anotherBuildWithNothingMissingIsAPlainNotice.

Observed / Not run

• CI's first test run found that the quiet cmux-tui restart could never fire. The daemon's own agent-screen detector holds two kindless connections forever, and the idle probe counted each kindless local client as a CLI call. Now a kindless client counts as a call only for its first 60 s (4f7d43f).
• On cmux16 (macOS 27), swift build --build-tests of the package passes at the new head:
  • QuietUpdateKeepsTerminalTests passed 3/3 and QuietDaemonUpdateTests 5/5 (at 4f7d43f);
  • AcpmuxQuietHandoffTests, AcpmuxQuietHandoffDecisionTests, AcpmuxUnresponsiveWatchTests, AcpmuxUnresponsiveRestartTests and base's AcpmuxStrayDaemonTests passed 3/3.
• Those runs found three fixture bugs in this PR's own tests (90860f5). CI never finished those suites because its test helper died first.
  • The stand-in daemon's ready line used a ws:// URL, which the launcher refuses.
  • macOS 27 kills a copied /usr/bin/nc, so the copy is now signed ad hoc.
  • The old daemon's exit event can come before the process table shows it gone, so that check now waits (bounded).
• CI's UnresponsiveOwnerWatchTests, IdleOwnerRestartRetryTests, QuietUpdateKeepsTerminalTests and OwnerRestartWaitsForChatTests passed in the daemon-tests job.
• At c38a347 on cmux16: the named suites above passed, 49 tests in 8 suites. ClassicUpgradeSimulationTests was not run there: the build guard refused it for low disk, so CI's app-tests job is its run.
• Not run outside CI: the watchdog commits after c38a347 (f8f10fc: the kqueue exit watch before the identity check; d969314 and 05367c0: the audit-token signal). cmux16 had too little disk for another package build, so CI's package-test lane is their only run. aSignalForAnotherProcessWithThePidIsRefused expects the kernel's refusal, so it would fail on macOS 14.0 or 14.1, where the token call is missing and nothing is sent; no CI runner uses those versions.
• Never run against a live daemon: no daemon on any of Leo's machines was restarted while developing this.

No capture: there's no visible UI to show. The change is daemon upkeep (handoffs, restarts and watchdogs) behind the Debug toggles.

Changelog

Changed: cmux NEXT moves its local acpmux and cmux-tui daemons to the app's own build after an update, at a quiet moment or after at most a few minutes, and restarts either one if it stops answering

Proof

No UI change beyond the existing toasts and banner; behaviour is covered by the tests above.

Checklist

• Behavior changes have added or updated tests
• Localization: the Debug toggle titles (one renamed to "Update a stale cmux-tui automatically") are debug-only. The cmux-tui failure reasons are English inside the localized "Couldn't restart %1$@: %2$@" (follow-up).
• Reviewed with a subagent before merge

Agent session: Air-Blue

🤖 Generated with Claude Code

[teamleaderleo]

Owned by hb-acpmux on big-red · session

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review d022ab1: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr23199 -f ref=daemon-autoupdate -f platform=macos -f backend_mode=local

Passes: CI passes on d022ab1d96.

CI passes on d022ab1d96 (run 39866587787 attempt 1).

Written by scripts/ci/classify_failures.py (ci-failure-attribution.yml); signatures are its SIGNATURES table. A machine verdict is the runner's fault, not this PR's; yours means the failing file is one this PR changes, also red on main that main's latest full suite fails the same way, seen on other PRs that it failed on another pull request's run lately.

red: cmux-next checks (god files, concurrency, crash safety, l10n); no finished base run on 84d5d32 to compare; do not merge until green; see lane-rules

Tour of da84a7b3b6 (run): no media. The tour job failed before the tour ran (build or setup); the run log has the output.

[teamleaderleo]

Bead cx-1idoh created for this PR by the cmux coordinator.

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[teamleaderleo]

codex-review: SHIP at a1432fe

• Reviewed the current diff, its focused tests, and the affected call paths; no correctness bug found.

[teamleaderleo]

Review (independent subagent): SHIP at 020c8bd

Rebase of a1432fe (reviewed SHIP) onto feat-cmux-next. The reviewer checked that all 34 files have the same changed lines as before the rebase. The one conflict, TunableCatalog.all in DebugSettingsService.swift, keeps every base entry and adds DaemonTunables.all once. No base commit adds a caller of an API this PR changes.

[teamleaderleo]

cmux-next guards / defaults-snapshot is red on 20 of 23 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

[teamleaderleo]

Review (independent subagent): SHIP at ac9d19e

Second rebase of 020c8bd (reviewed SHIP) onto feat-cmux-next. Commits 2 to 5 are identical in the range-diff. Commit 1 differs only in TunableCatalog.all, which is base's list (now with BrowserPageTunables.all) plus DaemonTunables.all once. In all 34 files the PR's added and removed lines hash the same as before. No base code added since calls an API this PR changed or adds a type this PR adds.

[teamleaderleo]

Review (independent subagent): SHIP at da84a7b

On top of ac9d19e (reviewed SHIP). 4f7d43f fixes a real bug that CI's first test run found. OwnerIdleProbe counted the daemon's own agent-screen-detector connections (kindless and permanent) as CLI calls in flight, so the quiet restart could never fire. Now a kindless local client holds it only while connected_seconds < 60, which the reviewer checked against client.list's wire format (a decimal string). The trade-off: a kindless call older than a minute, such as cmux agents wait, gets a retryable error at the handoff; terminals survive. c36a5ec and da84a7b make the busy-owner test wait for, and require, its first probe, and add the doc cross-references. On cmux16, QuietUpdateKeepsTerminalTests passed 3/3 and QuietDaemonUpdateTests 5/5.

[blacksmith-sh]

Found 11 test failures on Blacksmith runners:

Failures

Test	View Logs
/tmp/tmp6_1iep77/summary-fail.json (live):/summary-fail.json (live):	View Logs
/tmp/tmpvxbpuqdi/summary-missing.json (live):/summary-missing.json (live):	View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/ [new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows	View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames	View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/ [tabs] debug.tab_switch timed 0 of 40 switches	View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames	View Logs
convention violations found (ERROR lines above)./ convention violations found (ERROR lines above).	View Logs
test_base_passes (main.NoticesPairTests.test_base_passes)/ test_base_passes (main.NoticesPairTests.test_base_passes)	View Logs
test_every_submodule_free_checkout_on_an_owned_runner_resets_submodules (main.Reuse dWorkspaceSubmodules.test_every_submodule_free_checkout_on_an_owned_runner_resets_submo dules)/ test_every_submodule_free_checkout_on_an_owned_runner_resets_submodules (main.Reuse dWorkspaceSubmodules.test_every_submodule_free_checkout_on_an_owned_runner_resets_submo dules)	View Logs
test_every_submodule_free_checkout_on_an_owned_runner_resets_submodules (main.Reuse dWorkspaceSubmodules.test_every_submodule_free_checkout_on_an_owned_runner_resets_submo dules) (job='web-bundles')/ test_every_submodule_free_checkout_on_an_owned_runner_resets_submodules (main.Reuse dWorkspaceSubmodules.test_every_submodule_free_checkout_on_an_owned_runner_resets_submo dules) (job='web-bundles')	View Logs

...and 1 more test failures. View all on Blacksmith

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[teamleaderleo]

cmux-next crash-class rules / cmux-next crash-class rules is red on 10 of 10 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

[teamleaderleo]

cmux-next guards / nightly-guards / workflow-guard-tests / release-notary is red on 19 of 19 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

[teamleaderleo]

cmux-next / cmux-next checks (god files, concurrency, crash safety, l10n) › Checks job structure is red on 14 of 14 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

[teamleaderleo]

cmux-next / cmux-next checks (god files, concurrency, crash safety, l10n) › No fleet host names in gallery fixtures is red on 17 of 17 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

[teamleaderleo]

cmux-next / cmux-next checks (god files, concurrency, crash safety, l10n) › No new SwiftUI (macOS UI is AppKit) is red on 15 of 17 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

[teamleaderleo]

Closing: rebuilt on current feat-cmux-next as smaller PRs, now merged:

• cmux NEXT: restart this app's acpmux when it stays alive but stops answering #25379, restarts this app's own acpmux when it is alive but stops answering (merged b3e7b4f)
• cmux NEXT: restart the local cmux-tui by itself when it stays alive but never answers #25382, replaces a cmux-tui owner that stops answering, with no confirm (merged a4b5a4e)
• cmux NEXT: the words for updating other machines' daemons to the app's build #25386, the update words and types (merged 2828ede)

Base already moves a stale local acpmux and cmux-tui to the bundled build (AcpmuxVersionHandoff.handOffIfStale, #25219; DaemonLauncher.handOffIfStale). Following Leo's order ("OUR SHIT IS HARDLY EVER GOING TO GO IDLE ... IT RESUMES WHATEVER IT INTERRUPTED"), the quiet-wait handoff (AcpmuxQuietHandoff) was not rebuilt.