Repository navigation
cmux NEXT: a keep quit ends within 4 s whatever any step does; Restart to Update never leaves cmux invisible - #22919
Conversation
Bugbot needs on-demand usage enabledBugbot uses usage-based billing for this team and requires on-demand usage to be enabled. A team admin can enable on-demand usage in the Cursor dashboard. |
Correctness review (subagent), and resolutionReviewed head 84f3ee1 (before the fixes below). Current head 03fc2a5.
Checked fine by the reviewer: the keep deadline is armed only after every question is answered; The |
11c3df7 to
25f9a34
Compare
|
Review follow-up after the fixes: the step limit moved out of CmuxNextApp (it was 23 lines over the app size cap) into CmuxNextControl as |
25f9a34 to
a59630b
Compare
|
cmux status (one comment, edited in place; each line is updated by the job that owns it) Review App of this head (app.zip, kept 1 day)
Gallery before/after for
States
Pick other states with a Passes: CI passes on CI passes on Written by web and size checks: green Default changesThis pull request changes 1 default at
Needs a review. It passes with an independent review receipt on this commit: a comment or review whose first line is Merge receipt: merged as Regenerate the snapshot with Shipped in:
|
|
|
A tagged app builds a day's session (terminals, Chromium tabs with beforeunload pages, a WebKit tab, an agent chat), quits it by the interactive, scripted and signal paths and launches again. It fails when a quit takes longer than its bound, a launch misses its window or interactive bound, the launch's main thread stalls past its bound, or an app quit runs a page's beforeunload. sample records every quit and launch. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A step in cmux-scheme-compile runs scripts/cmux-next/quit-launch-bound.py on the built app, and the cmux-next quit and launch bound job is its red check (the perf gate's pattern, so a slow quit never fails the app-producing job). On a host over the launch load threshold the launch bounds only warn; the quit bound always holds. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…wedged
The wedged mode arms debug.quit {wedge_engine_stop: true} before the
interactive quit: the quit's engine stop blocks the main thread for good,
and the app must still end within the quit bound by its own deadline. On
the base the deadline is 15 s after the engines begin to stop, so this
mode is red there.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… to Update hides the windows only when Sparkle relaunches Slow quit and update, 2026-10-20 (nightly 3951982493001, cx-p00l3). Restart to Update: the click hid every window (transparent, click-through) before Sparkle took the install. That install never reached the installer: Sparkle gave up after its 7 s first-message timeout (unified log: windows hidden 19:28:43.545, back 19:28:50.736, the quit Apple event only after a second click at 19:29:09.8; the quit itself then took 0.29 s). For 7 s the app showed nothing. The windows now leave the screen at updaterWillRelaunchApplication, which Sparkle calls only on the path that sends its installer to stage 2 and quits the app (a rollback relaunch calls it too: intended, the quit follows at once). Quit: a quit that keeps the sessions now ends 4 s after it is decided, whatever any step does (QuitDeadline.keepQuitLimit, armed at the decision, after every question was answered). Each save step has its own limit (QuitDeadline.step: remote snapshots and agent images 300 ms; browser records, windows and sidebar 500 ms); a windows step past its limit stops saving at once, and the window records now flush before the incognito close. The engines' close wait is 2 s (was 3 s; Chromium gives beforeunload about 1 s) and an End quit ends 3 s after its engines begin to stop (was 15 s). Each quit step writes a quit.* launch mark. The quit origin and choice log at notice level, so they stay on disk. Measured (scripts/cmux-next/quit-launch-bound.py on mac26, 32 terminals, 5 browser tabs, an agent pane): quit with the engine stop wedged 15.04 s before, 4.04 s after (4 s deadline); other quits unchanged (0.11-2.5 s). Real Sparkle update (update-harness): click to the old app's exit 85 ms before, 63 ms after; windows hide at will_relaunch (+22 ms) instead of before the hand-off to Sparkle (+24 ms). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
271aeab to
b0315e9
Compare
|
Gate receipt for head b0315e9 (gate-run.sh, all green): CmuxNext suites QuitSessionsTests, MainThreadWatchdogTests, MainThreadWatchdogPolicyTests, SilentBackgroundUpdateTests, UpdateCheckFeedbackTests, UpdateReadyCardLayoutTests, RestorePreviousLaunchTests, CrashNoticeTests (cmux-ci, Xcode 26.6); Release compile (check-release-compile.sh); check-concurrency; check-app-size. Behavior check on mac26 at a59630b (same code, rebased since): every quit and launch within its bounds, wedged quit 4.03 s. Labeled ready-to-land for next-autoland (the workflow file moves on the base faster than a direct gated push). |
|
Check classification: |
|
Review (Codex gpt-6-astra xhigh): SHIP at b0315e9. No findings. It conflicts with feat-cmux-next, so it needs a rebase to land; the rebased head needs a fresh review. @lawrencecchen |
Conflicts resolved:
- QuitCoordinator: kept feat's unsaved registry, Cloud prepareForTermination
and Home cache flushes, and the PR's step limits and quit marks. Home's
quit flush in the save steps runs as its own step ("home_cache", 500 ms);
the engine stop still awaits Home's flush, as on feat. The deadline uses
feat's reason form, named for both arm points.
- QuitDeadline: feat's reason-based fault line with the PR's limits.
- UpdaterService+App: feat's relaunch-behind, relaunch blockers and
UpdateWindowHandoff(unsaved:), with the PR's move of the window hide from
the click to willRelaunch.
- cmux-next.yml: feat's side. The PR's "Quit and launch bound" CI step and
quit-launch-bound job are dropped: no new CI jobs, steps or gates
(CLAUDE.md, Lawrence 2026-10-23). scripts/cmux-next/quit-launch-bound.py
stays, to run by hand or on the fleet.
|
Conflict resolver (cmux-ad merge order, 2026-10-31): this PR is not pushed. The author must make the changes below. I merged
I did not use a skip reason, because these files are in this PR. The |
…ext-slowquit-resolve
…l; no static mark sink The quit's step limiter was a static entry point with a static mark sink (`QuitDeadline.stepMark`, a global mutable static). It is now an instance, QuitStepLimiter, that QuitCoordinator builds with its launch-mark sink and an injectable clock. The save steps, their order and their limits (QuitSaveSteps) and the quit's step sequence (QuitSteps, was App QuitCompletion) move into CmuxNextControl beside QuitDeadline; QuitCoordinator only fills them from AppServices. The Restart to Update hide timing is documented on UpdateWindowHandoff. CmuxNextApp lines: -20 against feat (was +57). QuitStepLimiterTests (CmuxNextControlTests): a step that ends in time is marked, a hung step passes its limit and the quit goes on, and the save steps run in order with the window saves stopped right after the windows step.
…nc closure default)
…eam iterator cannot be advanced on the main actor The fleet compile of 2cf3bab (job 0d9400b1ab624f3ed7e3e1ca) refused Marks.nextLimit: 'cannot call mutating async function next() on actor-isolated property iterator'.
|
Fixes for the third FIX review at ce9cd46 (#22919 (comment)). Head 6f77b0a. Commits: red Focused command:
Note 3 (browser records): a closed CEF page returns no navigation list ( Not run: the quit-launch-bound bench and the update harness at this head (GUI host); a live quit with a chat page whose draft is still settling. |
|
Review (claude-opus-review-subagent-4): FIX at 6f77b0a Independent review of the full diff from merge-base a718e9f to 6f77b0a. I also read the repair Findings1. The PR causes its own red check: "Web bundle key inputs" (must fix)
2. Note (no change required for SHIP): a late save now uses the engine stop's time
3. Nits
Checked and fine
After finding 1 is fixed (two path lines), I expect SHIP. Findings 2 and 3 are optional. |
…path lines; no new gate, job or step) Review of #22919 at 6f77b0a, finding 1: composerDraft.ts imports schemas/agent-pane/wake-state.json, so a change to it must rebuild and publish the web bundle. This adds "schemas/agent-pane/*" to the existing BUNDLE_INPUTS list in scripts/ci/cmux_next_route.py and "schemas/agent-pane/**" to the existing on.push.paths filter of cmux-next-web-bundle-publish.yml, next to schemas/sync-motion. It adds a path to two existing filters only: no new CI gate, job, step or test. tests/test_web_bundle_key_inputs.sh: "web-bundle-key inputs: ok" (it failed before).
… a closed page has no session to save (red) Review of #22919 at 6f77b0a, note 2: the wait for late saves before the engine stop ran to 200 ms before the deadline, so a hung save left Chromium no time to close its pages (CefShutdown's last cookie and storage flush lost). aHungSaveLeavesTheEnginesTheirCloseWait fails here: the engines have not stopped once only engineCloseWait is left. Reserving that time lets a late browser-records save run on beside the engine close, where a closed page's session keeps only the shown page; aClosedPageHasNoSessionToSave fails here (BrowserRestoredSession answers for a closed page). Nit: aHungCloudStepHoldsNoSave waits on a signal, not a 2 ms poll.
…ve; a closed page has no session to save Review of #22919 at 6f77b0a, note 2 and nits. QuitSteps waits for late saves before the engine stop only for what the deadline leaves less QuitDeadline.engineCloseWait (and the reply margin), so Chromium still closes its pages and CefShutdown flushes cookies and storage. A save still late then runs on beside the engine stop, and the wait before AppKit's reply covers it. BrowserRestoredSession answers no session for a closed page (as its other entry points already do), so a late browser-records save writes nothing over the good record the daemon holds. The QuitSteps doc now says what an End quit does: its deadline is armed only as the engines stop, so a save still late after endQuitSaveWait runs on beside the engine stop.
|
Fixes for the fourth FIX review at 6f77b0a (#22919 (comment)). Head c99b21a. Commits:
Static checks on the agent VM (no build):
Fleet (
The time budget is in the PR body ("Keep-quit time budget"). Late saves before the engines get until 1.8 s after the decision (4.0 − 0.2 − 2.0). A chat-composer step that starts within 0.5 s ends inside that wait. A later one runs on beside the engine close, and the last wait covers it until 3.8 s. Not run: the quit-launch-bound bench and the update harness at this head, because they need a GUI host. |
|
Review (claude-opus-review-subagent-5): FIX at c99b21a Independent review of the full diff from merge-base a718e9f to c99b21a, of the repair Findings1. The closed-page guard is checked only before the scroll measure, so a measure in flight when the engine closes the page still writes the shown-page-only session (must fix)
Notes (do not block)
Checked and fine
CI at c99b21a
After finding 1 is fixed (one more |
… the page is still open The engine can close the page during the await; reading the navigation list after it gave a session with only the shown page, which a quit's late save wrote over the good record.
|
Review (claude-opus-review-subagent-6): SHIP at a1ec641 Independent review of the full diff from merge-base a718e9f to a1ec641, of the repair FindingsNone that block. The repair (finding 1 of review 5)
Earlier findings did not come backThe only changes after c99b21a are the two files above. Checked again on this head:
Notes (do not block)
|
|
Fleet evidence for a1ec641 (review 5, finding 1: a page closed during its scroll measure):
|
…se-quit-flush Port of #28275 onto feat (lead decision 05:20Z), keeping feat's state-file and quit work and adding only what this PR adds beyond it: - AgentUnsentDrafts: feat's #28204 undelivered drafts and unreadable-file set-aside, on this PR's per-app AgentPaneStateFiles (no static registry). - QuitCoordinator: feat's #22919 QuitStepLimiter, time-left reply margin, launch marks and engineCloseWait, plus this PR's drains of the state files around the engine close, the testable quitSteps() (was steps(), which now names feat's limiter) and closeEngines seam, and the deadline end that records cut-off files. - AppDelegate/AppServices: feat's openedURLs, windowRecord, peer approver and injected key window; this PR's secondSignalExit (no blocking flush in prepare) and the state-files owner and loss record. - Paired phones: feat's #25437 MobileHostPairedStore now owns the directory's one MobileHostPairedDevicesFile (configure(directory:lastQuitCutOff: cutOffSettled:)); the iPhone host takes it from the store for the quit watch, so the shared-list transactions and this PR's cut-off accounting use one file object and one queue (an instance enqueue that counts as saving), not two.
Summary
Incident 2026-10-20 (nightly 3951982493001): Lawrence clicked Restart to Update. The app looked frozen. Process 5342 exited 28 s after the click; launch looked slow as well. Bead cx-p00l3.
Root cause (unified log,
log showof pid 5342 and the Sparkle tools):UpdateWindowHandoff.hideatwillInstallStaged).SPUInstallerDriverFIRST_INSTALLER_MESSAGE_TIMEOUT, "An error occurred while starting the installer"). The installerAutoupdate[6160]had staged the update at 19:00:14 and logged nothing in this window.Handling Quit AppleEvent(Sparkle's agent).CEF shutdown complete,replyToApplicationShouldTerminate:YES. The quit itself took 0.29 s.So the 28 s was not the quit: it was 7 s of no visible windows, then the failure sheet and a second click. The launch delay is the daemon's serial host adoption (follow-up cx-p00l3.1).
Changes
updaterWillRelaunchApplication(Sparkle calls it only on the path that sends its installer to stage 2 and quits the app), not at the click, and only for a relaunch the person asked for (hidesForRelaunch, called when no automatic install is in flight). A click whose install fails keeps the windows on screen with the Installing state, then the failure. An install nobody clicked (Sparkle's automatic driver, the relaunch behind the person's app) leaves the windows as they are, as before; a declined relaunch behind and an automatic failure still callinstallAbandoned. ThewillInstallStagedhook, which nothing set any more, is gone.QuitDeadline.keepQuitLimit). An End quit has no deadline until its engines begin to stop (below). Each save step has a limit (QuitSaveStepsin CmuxNextControl, run by aQuitStepLimiterinstance thatQuitCoordinatorowns): browser records 300 ms, windows 500 ms, sidebar 300 ms, Home's cache 400 ms, agent chat composers 400 ms, then terminal frames, agent images and remote snapshots 200 ms each. The person's own data saves first. Cloud's quit work (Codex broker stop, VM credential revokes) starts first and runs beside the saves, so a slow network never delays a save. If every step hangs they take 2.6 s (worstCase; Cloud's in-step wait is 100 ms, the late-step wait gives it the rest); with the 500 ms quit mark (QuitDeadline.quitMarkLimit), the 300 ms "Don't ask again" write and the 200 ms reply margin, 400 ms remain.QuitStepLimiter.finishOverruns,QuitDeadline.timeLeft; async, never a main-thread wait). An End quit also waits up to 2 s for a late save (not Cloud's step) before its sessions and daemon stop. In a keep quit, late saves get what the deadline leaves less the engines' 2 s close wait and the reply margin, so Chromium still closes its pages and flushes cookies and storage; a save still late then runs on beside the engine stop, and the wait before AppKit's reply covers it. A closed page has no session to save (BrowserRestoredSession), so a late browser-records save writes nothing over the good record the daemon holds (written after each navigation). In an End quit the deadline is armed only as the engines stop: a save still late after its 2 s End wait runs on beside the engine stop (its daemon has stopped). Chat pages are asked for their composer state within 1 s (AgentPaneRelaunchWake.askLimit, as on feat): the page first waits up to 500 ms for its typed draft to reach acpmux (draftSettleMsinschemas/agent-pane/wake-state.json, whichcomposerDraft.tsand the host's test both read), and the quit waits for the composer step past its 400 ms limit. A page that does not answer in time is logged (agent-pane.relaunch-wake). Engines' close wait 2 s (was 3 s); an End quit ends 3 s after its engines begin to stop (was 15 s). The window records flush before the incognito close, and a windows step past its limit stops saving at once. Each step writes aquit.*launch mark.scripts/cmux-next/quit-launch-bound.py(behavior check: real tagged app, control socket, launch-mark pipe, loopback pages with beforeunload handlers,sampleof every quit and launch). It fails on a quit over 6 s, a launch over 3 s to the first window or 8 s to the first live terminal frame (frommain()), a launch main-thread stall over 2 s, or a restored session with fewer workspaces or tabs than before the quit. This PR adds no CI step or job (repo rule: no new CI gates); run it yourself on a GUI host.Numbers (mac26, 8 workspaces / 32 terminals, 5 browser tabs, 1 agent pane)
cmux app quit)debug.quit wedge_engine_stop)will_relaunch(+22 ms)main(), which comes about 1.5 s after process start)daemon.ensure8.4 sSamples:
sampleevery 2 s during each quit and launch are in the job artifacts (nx-remote mac26 jobs 1020-200458-a88a5f base, 1020-204732-4fdd44 this PR). Base wedged quit main thread, every sample:semaphore_wait_trap <- QuitDeadline.wedgeMainThread <- closure #6 in QuitCoordinator.completeuntil the old 15 s deadline.Keep-quit time budget (4 s from the decision)
QuitDeadline.quitMarkLimit), "Don't ask again" ≤ 0.3 s, then the saves in order: browser records 0.3, windows 0.5, sidebar 0.3, Home's cache 0.4, chat composers 0.4, terminal frames 0.2, agent images 0.2, remote snapshots 0.2, Cloud 0.1 (2.6 s if every step hangs).engineCloseWait) = 1.8 s after the decision. Then the engine stop, then the wait for every step still running until 3.8 s, then AppKit's reply.theComposerAskFitsTheKeepQuitDeadline). Only CefShutdown itself, which blocks the main thread, can delay its last page answers.Cloud credential cleanup at quit (trade-off)
On feat, a quit awaited
cloud.prepareForTermination()to its end before any save: the Codex broker's stop (SIGTERM, up to 2 s grace) and one revoke per VM it delivered to (each bounded at 3 s, about four or five app-server round trips:isLive, readaccess.json, read the mode file, remove, restore or remove). With no keep deadline, a slow network delayed the whole quit by that much.Now Cloud's cleanup starts first and gets all the time the quit has left: the save steps' time, the engine stop, then what is left of the 4 s keep deadline (End quit: the 3 s engine-stop deadline) less 200 ms. On a normal network that is enough for the broker stop and a few revokes. When it is not (a slow network, several VMs, a broker that ignores SIGTERM), the deadline ends the process first, and
access.jsonand theownmode file stay on the VM until the delivered token expires. That is the trade-off of a quit that always ends: the token's expiry is the backstop.plans/cmux-next/OWNERS.mdhas no row for the Cloud broker (CmuxNextCloud/Broker/); its author (#26170, #26212) is asked for an OK in a PR comment.Risk tier
risk:high(updater). No default-off setting: both changes fix user-visible regressions (the click-time hide that left cmux invisible for 7 s in the incident, and a wedged quit that took 15 s), whichdocs/ci/risk-tiers.md("Ship risky behavior dark") exempts. Automatic installs keep feat's behavior (no hide).Review of c0756ab (FIX) and its fixes
740d5c33d96, fix0088e20ec76.bbfcfffe854, fix0088e20ec76.restore-journey.py(4 s), theUpdaterServicedoc of the removed hook.Review of 0088e20 (FIX) and its fixes
c2e92cb8e49, fix19e931e1183,f95bdd0a0ef(the "Don't ask again" limit moves intoQuitSteps),ce9cd46ad51(Cloud's in-step wait 100 ms, so the limits fit with the reply margin).QuitDeadline.quitMarkLimit(the boundquitBeganpasses); a test coversinstallAbandonedon the automatic failure branch.Review of ce9cd46 (FIX) and its fixes
schemas/agent-pane/wake-state.json; a page that does not answer is logged. Tests:theQuitAsksLongerThanThePageSettlesItsDraft(reads the JSON),theComposerAskFitsTheKeepQuitDeadline..limitmark and the manual clock's sleeper.49563554e39, fix654ab7183ff, merge oforigin/feat-cmux-next2cf3babb845(the branch was 187 commits behind, and the web lint judged feat's own newer files; after the merge it passes).Review of 6f77b0a (FIX) and its fixes
schemas/agent-pane/*added toBUNDLE_INPUTSinscripts/ci/cmux_next_route.pyandschemas/agent-pane/**toon.push.pathsofcmux-next-web-bundle-publish.yml. These are two path lines in two existing filters: no new CI gate, job, step or test.bash tests/test_web_bundle_key_inputs.sh:web-bundle-key inputs: ok(it failed before);python3 -m unittest tests.test_cmux_next_route: 111 tests OK.engineCloseWait; a closed page has no session to save. Tests:aHungSaveLeavesTheEnginesTheirCloseWait,aClosedPageHasNoSessionToSave.aHungCloudStepHoldsNoSavewaits on a signal; theQuitStepsdoc and this description now say what an End quit does.d49e188e628, red243c9d3333f, fixc99b21a4ff9.beforeunload
App quit does not show a page's question, but Chromium's forced close (
cmux_shim_close_all,CloseBrowser(true)) still runs each page's beforeunload handler: the test pages' beacons arrive about 0.5 s after close-all. The quit waits for it at most the 2 s close wait. Follow-up cx-p00l3.2 (renderer fast shutdown in the shim).Changelog
Fixed: Restart to Update never leaves cmux invisible while the installer starts, and a quit that keeps your sessions always ends within 4 seconds.
Observed / Not run
scripts/cmux-next/quit-launch-bound.pyon mac26 (GUI host) with fleet-built apps: base 7819daf + check (red: wedged quit 15.04 s), this PR (green, 2 rounds of interactive, scripted, signal, wedged; session census kept: 9 workspaces, 32 terminals, 5 browser tabs, 1 conversation, 1 page).--handoff --workspaces 28on base and this PR: shows the daemon adoption delay (not changed here).scripts/cmux-next/update-harness/job.sh, real Sparkle update between two signed copies) on base and this PR on cmux-lawrence-2: both relaunched.🤖 Generated with Claude Code