Skip to content

fix(quit): state files have one owner each and no quit blocks the main thread on them - #28275

Merged
lawrencecchen merged 23 commits into
feat-cmux-nextfrom
fix/arch-rise-quit-flush
Nov 1, 2026
Merged

lawrencecchen merged 23 commits into
feat-cmux-nextfrom
fix/arch-rise-quit-flush

Conversation

@lawrencecchen

@lawrencecchen lawrencecchen commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This PR fixes the arch-ratchet tip rises of #27749 and #25535. These state files flush at quit and had no reviewed exception:

  • The statics move to instances. AppServices now owns one AgentPaneStateFiles registry and gives it to AgentUnsentDrafts and AgentPaneScrollAnchors. The registry rule is the same as before: one writer per path. The paired-phones file (MobileHostPairedDevicesFile) is now a class. IPhoneHostService owns one instance per state directory, and its queue and the record of the list in force are members of that instance.

  • No quit blocks the main thread on these files. This PR removes all three allowBlocking(.terminate) flushes. The quit awaits the files with AppServices.stateFilesDrained(). The wait holds no thread and has no limit of its own; the quit awaits it only after its deadline is armed:

    • Forced quit (a second quit signal): before, ForcedQuitExit prepare ran its flush before the 160 ms deadline was armed, so it could wait 2 s. Now the exit arms the deadline first. Then, in save, it waits for the state files and for the Home cache write at the same time.

    • Normal quit: before, applicationWillTerminate and IPhoneHostService.stop() could wait 2 s each, so 4 s in all. Now QuitCoordinator's engine-stop step does this, in order:

      1. It arms QuitDeadline.engineStopLimit (15 s).
      2. It drains the files.
      3. It closes the engines (closeEngines, a seam for tests).
      4. It drains the files again.

      The second drain catches a draft that a closing page saves (the composer's page-hide flush). The prepareWindows drain and the bare 2 s limit are gone (R9 review, R-L1).

  • A lost draft is named and told (R9 review, R-L5). If a quit deadline ends the process while a state file is still saving, the deadline's end (AgentPaneStateLoss.deadlineEnd) does two things:

    • It logs a .fault with the file names only, never content.
    • It records the names in <state>/agent-state-lost.json.

    The next launch reads and deletes that record. If a new chat's unsent text was lost, it shows a toast: "Text typed in a new chat just before cmux quit was not saved." The string is localized in 21 locales. Both deadline reasons now name the state files. Before this PR, such a loss was silent. The paired-phones list is in the loss record too, and it fails closed, but the deadline's end never calls the Keychain. A hung securityd, the case QuitDeadline exists for, must not hold the process's end.

    • MobileHostPairedDevicesFile counts its saves and withdrawals that are queued or running.
    • The iPhone host owns a MobileHostPairedQuitWatch that gives the deadline's end its current file from any thread.
    • If a save is still in flight when the deadline fires, the end writes a marker (paired-devices.cut-off, beside the file), names the file in the fault and the record, and ends.
    • If the marker cannot be written (a full disk), the end logs a fault. The record then names the file with the note paired-devices.cut-off (not written).
    • At the next launch, the new file object fails closed when the marker exists OR the last quit's record names the file. It checks this first on its own queue, before any read. The launch reads the record once (AgentPaneStateLoss.lastQuitNames()), so this check and the notice share the same names in either order. Nothing stored counts in that run. It clears the Keychain seal, so the stored list no longer matches its seal and every phone must pair again (the digest rule of load). A removed phone does not get its access back.
    • The fact must stay on disk until the seal is cleared:
      • The record keeps naming the paired list after the launch reads it. It names only that list, so a lost draft is told once.
      • A quit deadline in that run writes the paired name beside its own names.
      • The paired file tells the record (pairedCutOffSettled()) once the fact is safe without it: its marker is on disk, or its seal is cleared.
    • When only the record named the save, the launch writes the marker BEFORE the clear. A failed write is a fault of its own, and the record keeps the fact.
    • The clear counts as a save in flight, so a quit deadline during a hung clear marks and names the file again.
    • If the Keychain refuses the clear, the list stays refused for that run and a fault is logged. The fault names what keeps the fact for the next launch: the marker, or the record when no marker could be written.
    • The second toast, "A change to your paired iPhones was cut off when cmux quit. Pair your iPhones again." (21 locales), shows only when that launch actually cleared the seal.
    • The unsent-draft toast shows at once. The pair-again toast comes when its own Keychain check answers, so a slow Keychain never holds the draft toast back (showNotices).
    • AppServices.quitDeadlineEnd() builds the one deadline end that both quits use.
  • One path for the drain. QueueDrain is deleted. Each file awaits its own serial queue (AgentPaneStateFile.saved()), so the drain needs no timer and no race.

  • Testable quit steps: QuitCoordinator.steps() builds the QuitSteps that complete runs. deadline and closeEngines are replaceable, so a test runs the coordinator's own steps without ending its process or starting Chromium.

  • Line budget: SignalRelay moves from CmuxNextApp/Quit to CmuxNextControl/Diagnostics, next to ForcedQuitExit and QuitDeadline. The loss record and its notice live in CmuxNextAgentPane. With this, CmuxNextApp's line count does not rise (+0).

Sites fixed (paths under Packages/macOS/CmuxNext/Sources/)

Row Site Added by
global_mutable_statics CmuxNextAgentPane/AgentPaneStateFile.swift:31 static let files = Mutex<...> d02ea17 (#27749)
global_mutable_statics CmuxNextMobile/MobileHost/MobileHostPairedDevices.swift:188 static let queue 06d4a46 (#25535)
global_mutable_statics CmuxNextMobile/MobileHost/MobileHostPairedDevices.swift:197 static let inForce 06d4a46 (#25535)
blocking_allow_escapes CmuxNextApp/AppDelegate.swift:42 flush in ForcedQuitExit prepare d02ea17 (#27749)
blocking_allow_escapes CmuxNextApp/AppDelegate.swift:514 flush in applicationWillTerminate d02ea17 (#27749)
blocking_allow_escapes CmuxNextMobileHost/IPhoneHostService.swift:387 flush in stop() 06d4a46 (#25535)

AppDelegate.swift:514 was first listed as an exception candidate. An independent exception review then asked for it to be fixed with an async wait in the .terminateLater phase, so this PR fixes it too.

Ratchet delta (arch_counts.py --delta <merge-base 5a3b7e1cef4> HEAD)

global_mutable_statics             -3 {'CmuxNextAgentPane': -1, 'CmuxNextMobile': -2}
blocking_allow_escapes             -3 {'CmuxNextApp': -2, 'CmuxNextMobileHost': -1}
app_lines                          +0

All other rows: +0. arch-ratchet.py --base origin/feat-cmux-next --head HEAD says "no architecture count rises". This PR does not change arch-ratchet.json (--update was not run).

Testing

No clock decides any quit test. A test helper, DrainGate, holds a state file's queue with a save, and only the start of a drain lets that save go:

  • AgentPaneStateFiles(drainBegan:) is an injected init hook, not a static. The drain calls it as it begins, and the gate releases the held save from it (or from the test's own defer, after its checks).
  • So a quit step that never drains never sees the draft written, on any host. A step that drains always does.
  • The gate also records the order of the quit steps.

The quit tests build their services through one helper, QuitStateFilesTests.services. It makes NSApplication.shared first, answers no key window, and puts in a QuitDeadline whose end does nothing. A test process has no app object; the root fix of that trap is its own PR, fix/arch-rise-appservices-no-nsapp-trap.

  • QuitStateFilesTests.theForcedQuitArmsItsDeadlineBeforeItWaitsForTheStateFiles: the step order is ["deadline armed", "drain began"], and the draft is on disk when the forced exit ends.
  • QuitStateFilesTests.theQuitAwaitsAQueuedDraftWithoutBlockingTheMainThread: the main actor releases the held save while the quit waits. A wait that blocked the main thread would never end, and .timeLimit(.minutes(1)) fails the test.
  • QuitStepsStateFilesTests runs QuitCoordinator's own stopBrowserEngines:
    • theEnginesCloseAfterTheQueuedDraftIsOnDisk: two drains happen, and the draft is on disk when the engines close.
    • aDraftSavedWhileTheEnginesCloseIsOnDiskWhenTheQuitEnds: a draft queued during the engines' close is on disk when the step returns.
  • AgentPaneStateLossTests:
    • The deadline's end records the files still saving, and the next launch takes the record once.
    • A cut-off paired list is recorded, and its notice shows only when the seal was cleared.
    • Nothing still saving records nothing.
    • A lost scroll place gets no notice.
  • MobileHostPairingTests.aRemovalCutOffByAQuitDeadlineIsNotTrustedAfterARelaunch: a removal's save is held at the seal, before its own clear, when the deadline's end runs. The test checks four things:
    • The end names the paired file.
    • The end makes zero seal calls. The test seal counts every call of digest, setDigest and clearDigest.
    • The next launch clears the seal before its first read.
    • That read trusts no phone.
  • MobileHostPairingTests.aCutOffRemovalWithoutItsMarkerIsNotTrustedAfterARelaunch: an injected disk refuses the marker. The record names the file with the note, no marker is on disk, and the next launch, told only by the record, trusts no phone and clears the seal.
  • MobileHostPairingTests.aRecordOnlyLaunchWhoseClearHangsStillLeavesTheListUntrusted: a launch told only by the record has its seal clear held. A quit deadline during that clear finds .marked, and the launch after it, with no record, trusts no phone.
  • MobileHostPairingTests.aRecordOnlyLaunchWithNoMarkerAndARefusedClearKeepsTheRecord: a launch with no marker (the disk refuses it) and a refused clear never tells the record the fact is safe, and the launch after it still trusts no phone.
  • AgentPaneStateLossTests.theRecordKeepsNamingACutOffPairedListUntilItIsSettled: after the launch reads the record, it names only the paired list. A deadline in that run keeps that name beside its own, and only settling removes it.
  • MobileHostPairingTests.aRemovalTakesTheOldDigestAwayAfterTheSavesBeforeIt and aRemovalNeverWaitsOnTheCallersThread now release their held save on a signal, after the removal is queued, not after 0.3 s or 0.6 s. The second no longer measures time: a removal that waited on the caller's thread would never return, and its time limit fails the test.
  • AgentPaneStateLossTests.thePairedCheckAndTheNoticeShareOneReadOfTheRecord: the notice reads the record first, and the paired check still sees it.
  • QuitStateFilesTests.theLossRecordNamesThePairedListByItsFileName: the record's paired-phones name matches the file's own name.
  • AgentPaneStateFileTests: a file counts as saving until its saves ran; drained() waits for queued saves and holds no thread.
  • MobileHostPairingTests: one file object per run. GatedSeal became HeldSeal, which holds the same file's queue.
  • The tests in ChatScrollRestore*, UnsentDraftTests and RestoredNewTabPageTitleTests build their own registry.

Observed / Not run

  • Observed: prepush-gate.py reports no new violations. arch-ratchet.py reports no rises. lint-ios-package-conventions.sh --namespace-fix reports no violations. localization_catalog.py check reports nothing for the new key.
  • Earlier failed fleet runs (not evidence):
    • Compile a1714da97296c3cf467069ff failed on a test isolation error, fixed in 57dfaa1.
    • Green 04224ccd89106f08f50cc908 at 07f1b95 crashed on NSApp.keyWindow.
    • Green eb20257db2553e2e4a4dd378 at 57dfaa1 crashed on NSApp.isActive.
    • Both crashes are fixed by NSApplication.shared in the test helper.
  • Compile 6c6043c82f095749e151f624 at c85dcca failed: a public method named an internally imported type (CmuxToastCenter). Fixed in 96a1de8.
  • Compile f064b3161ab4caa8918617ec at 96a1de8 passed. The R9 re-review then found the Keychain call on the deadline thread, fixed in 67afa2c. The steps queued at 96a1de8 were cancelled.
  • Compile 8400b1b5d3dcc4763a67535a at 67afa2c passed. The R9 review then found two silent-loss cases (a failed marker write, and the draft toast waiting on the Keychain), fixed in fc54bef. The proof steps at 67afa2c were cancelled.
  • Compile c822c59f4aaedeabf6bb007c at fc54bef failed: a public init's default argument called an internally imported function (blockingWrite). Fixed in e22c234. I checked every public declaration this PR adds or changes for default arguments or types from internal imports, and found no other case.
  • Compile 8d21112a25c2cd9435417c71 at e22c234 passed. The R9 review then found that the record-only path could still trust a cut-off removal: a hung clear, then a second end, left no record and no marker. Fixed in d6f29ce. The proof steps at e22c234 were cancelled.
  • Compile dcbec3f24cd1d802ceb3bd20 at d6f29ce passed. Proof results at d6f29ce. They are evidence for that head only, not for the current head:
    • Green ce744edd3717933ee9dfd3c6 passed. The four suite runs had 3, 2, 5 and 29 tests, and the build completed.
    • Red 97a826e11fa23297d8fc74da (on fix/arch-rise-quit-flush-redproof-6) failed on exactly the 9 expected tests, each on its assertions, with no crash:
      • theForcedQuitArmsItsDeadlineBeforeItWaitsForTheStateFiles
      • both QuitStepsStateFilesTests tests
      • theDeadlineRecordsTheFilesStillSavingAndTheNextLaunchTakesThem
      • aCutOffPairedListIsRecordedAndTold
      • thePairedCheckAndTheNoticeShareOneReadOfTheRecord
      • aCutOffRemovalWithoutItsMarkerIsNotTrustedAfterARelaunch
      • aRemovalCutOffByAQuitDeadlineIsNotTrustedAfterARelaunch
      • aRecordOnlyLaunchWhoseClearHangsStillLeavesTheListUntrusted The R9 review then found that the record was deleted at the launch's first read, so a failed marker write at launch plus a refused or hung clear could still lose the fact. Fixed in 626d97e, with the two timed tests made signal-driven.
  • Not run yet: the compile of this head, and the fleet red (on fix/arch-rise-quit-flush-redproof-7) and green of QuitStateFilesTests, QuitStepsStateFilesTests, AgentPaneStateLossTests and MobileHostPairingTests.

Changelog

  • Changed: quitting cmux never blocks on saving a new chat's unsent text or the paired iPhones. If a quit ends before they are saved, the next launch says so, and a cut-off iPhone change makes every iPhone pair again.

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Fixes the arch-ratchet tip rises for the agent-pane state files (cmux#27749) and the paired-phones file (cmux#25535): each file now has one owner, and no quit blocks the main thread waiting on them.

  • Replaces the process-global statics with instances: AppServices owns one AgentPaneStateFiles registry, and IPhoneHostService owns one MobileHostPairedDevicesFile per state directory, keeping the one-writer-per-path rule.
  • Removes all three allowBlocking(.terminate) flushes. The forced quit arms its 160 ms deadline before awaiting the files; QuitCoordinator's engine stop arms its 15 s deadline, drains the files, closes the engines, and drains again so a closing page's saved draft is on disk. The first drain is bounded by the deadline's time left less the reply margin and the engine close's wait, so it can't starve CefShutdown; an overrun runs beside the close and is awaited before the quit ends. The drains hold no thread.
  • A quit deadline that ends the process while a file still saves logs a fault (names only) and records them in <state>/agent-state-lost.json; the next launch toasts a lost unsent draft.
  • A cut-off paired-phones save fails closed: the deadline's end writes only a marker and never calls the Keychain, so a hung securityd cannot hold the process's end; the next launch clears the seal before its first read so no stored list counts after relaunch, and earns its own toast. Both toasts localize in 21 locales.
  • When only the record names the cut-off save, the launch writes the marker before clearing, and the record keeps naming the list until the marker is on disk or the seal is cleared, so a refused or hung clear can't lose the fact. The launch's clear counts as a save in flight, so a deadline during a hung clear marks it again.
  • QuitCoordinator builds its steps via quitSteps() with a replaceable closeEngines seam, so the quit tests run the coordinator's own engine stop with a queued draft.
  • Moves SignalRelay to CmuxNextControl/Diagnostics to keep CmuxNextApp's line count from rising.
  • The quit tests hold a file's queue with a DrainGate that the registry's drainBegan hook releases when a drain starts, so no clock decides them. The test AppServices initializer takes the state files so the tests can hold one, and their services answer no key window because a test process has no NSApp. Draft, scroll-restore, and title tests build their own AgentPaneStateFiles, sharing one per test so a fresh instance reads after its queued saves, as the static registry made it do.

Written for commit 2fdc8e5. Summary will update on new commits.

View guided diff Turn on auto-fix

Takes back #27749: its quit flush blocked the main thread twice (allowBlocking(.terminate) { AgentPaneStateFile.flushAll() } in ForcedQuitExit prepare before the 160 ms forced deadline was armed, and again in applicationWillTerminate) with no reviewed arch-ratchet exception (reward-hacking audit, arch-ratchet tip rises). The drafts are still saved. The quit now awaits the same files after its deadline is armed, with the main thread free. A draft that the deadline cuts off is recorded and told at the next launch.

No capture: the one new UI is an existing toast (CmuxToast) with new text. It shows only at the launch after a quit deadline ended the process while a draft was still saving, and a capture tour cannot reach that state without killing a quit mid-save.

… the agent panes' state files

The forced quit (ForcedQuitExit, a second SIGTERM/SIGINT/SIGHUP while a quit runs) flushed the agent
panes' state files in `prepare`, under allowBlocking(.terminate), before its 160 ms deadline was armed:
with a save queued behind a busy disk the exit waited up to 2 s on the main thread, and its own deadline
did not bound that wait.

This commit extracts the exit's steps into AppDelegate.secondSignalExit (same steps, injectable deadline
and end) and adds QuitStateFilesTests.theForcedQuitArmsItsDeadlineBeforeItWaitsForTheStateFiles. It is
red on this commit: the deadline is armed about 2 s after the start, and the exit ends before the queued
draft is on disk. The fix is the next commit.

Not run here (no Swift build on this VM): the fleet runs it.
…n thread on them

Rise audit (arch-ratchet tip rises with no reviewed exception) of cmux#27749 and #25535:

- AgentPaneStateFile's process registry (`static let files = Mutex<...>`) becomes the
  AgentPaneStateFiles instance that AppServices owns and passes to AgentUnsentDrafts and
  AgentPaneScrollAnchors (composition root, PATTERNS.md). Same one-writer-per-path rule.
- MobileHostPairedDevicesFile's `static let queue` and `static let inForce` become members of one
  file object that IPhoneHostService owns per state directory (it built a new value per use before).
  This run's record of the list in force is now per file object; a relaunch is a new object.
- The three allowBlocking(.terminate) flushes are gone:
  - ForcedQuitExit `prepare` (AppDelegate): the flush ran before the 160 ms deadline was armed, so a
    second quit signal could wait 2 s. The exit now arms the deadline first and awaits the state files
    beside the Home cache's write in `save`.
  - applicationWillTerminate (AppDelegate) and IPhoneHostService.stop(): up to 2 s each, 4 s in all.
    The quit's `.terminateLater` phase (QuitCoordinator) now awaits AppServices.stateFilesDrained() in
    prepareWindows and again in stopBrowserEngines, next to home.flushCacheForQuit(): one 2 s limit
    for all the files, holding no thread.
- QueueDrain (CmuxNextWakeups): waits for serial queues with one continuation and one DemandTimer
  deadline, so no thread blocks and a quit with no backlog costs no timer fire.
- SignalRelay moves from CmuxNextApp/Quit to CmuxNextControl/Diagnostics (beside ForcedQuitExit and
  QuitDeadline), so CmuxNextApp's lines do not rise.

Ratchet (arch_counts --delta): global_mutable_statics -3, blocking_allow_escapes -3, app_lines -31.

Regression: QuitStateFilesTests.theForcedQuitArmsItsDeadlineBeforeItWaitsForTheStateFiles is red on
the previous commit and should be green here; theQuitAwaitsAQueuedDraftWithoutBlockingTheMainThread
covers the normal quit's wait. Not run here (no Swift build on this VM): the fleet runs them.
Owners told: Performance Lead (Wakeup primitives, statics), PL-threads (the terminate allowance).
@lawrencecchen lawrencecchen added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 30, 2026
@cursor

cursor Bot commented Oct 30, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 2fdc8e5: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr28275 -f ref=fix/arch-rise-quit-flush -f platform=macos -f backend_mode=local

web and size checks: green

Shipped in:

@blacksmith-sh

blacksmith-sh Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Found 13 test failures on Blacksmith runners:

Failures

Test View Logs
/tmp/tmp7xcfd3tw/summary-missing.json (live):/summary-missing.json (live): View Logs
/tmp/tmpigb5bp2a/summary-fail.json (live):/summary-fail.json (live): View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows
View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs
main.DeveloperIdAppTests.test_download_prints_the_active_profile_only/
test_download_prints_the_active_profile_only
View Logs
convention violations found (ERROR lines above)./
convention violations found (ERROR lines above).
View Logs
stubs: /System/Library/Frameworks/Security.framework/Versions/A/Security has no stub/
Security has no stub
View Logs
stubs: CoreFoundation.tbd: install-name /System/Library/Frameworks/
CoreFoundation.framework/Versions/A/CoreFoundation, missing 1: ['_CFRetain']/
CoreFoundation, missing 1: ['_CFRetain']
View Logs

...and 3 more test failures. View all on Blacksmith: run 41278346468, run 41278346331

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@lawrencecchen

Copy link
Copy Markdown
Contributor Author

Review (independent review subagent, arch-rise): SHIP at 1869061
perf claim: none
compile: c3a700cfb361196f01759346 passed

What I checked. I did not edit, build or run Swift. HEAD, the worktree and the PR headRefOid are all 1869061. Job c3a700cf compiled this ref (CmuxNext, app, cmux-tui), and its log shows that QuitStateFilesTests.swift, MobileHostPairingTests.swift and the CmuxNextAgentPaneTests module compiled.

  • Saves at quit. Every normal quit goes through QuitCoordinator.shouldTerminate (.terminateLater), and QuitCompletion.run always runs prepareWindows and stopBrowserEngines. Both steps now await stateFilesDrained(), before the engine-stop deadline is armed. The only .terminateNow path has no services, so it has no state files. A forced quit (second signal) runs prepare (no disk wait), arms its deadline, then awaits the state files and the Home cache write together.
  • Behavior changes, judged:
    1. In a forced quit, a queued save now has 160 ms, not 2 s, to finish. This agrees with the ForcedQuitExit contract: a second signal does not wait for the stuck quit. The files are small JSON writes (a few ms when the disk is idle). Accepted.
    2. A save that starts after the drain in stopBrowserEngines (during the CEF shutdown or in applicationWillTerminate) is not awaited now. Before, the blocking flush in applicationWillTerminate waited for such a save. The agent pane is a WKWebView, so a late chat.unsentDraft or chat.scrollAnchor message can arrive during the CEF shutdown. The risk is low: the write is a few ms on an idle queue, and QuitSteps already says that nothing the user asked for comes after the engine stop. The Home cache has the same rule. Accepted.
    3. A normal quit can now await 2 s twice (prepareWindows and stopBrowserEngines). The main thread is not held during these waits. Accepted.
  • QueueDrain. The continuation is stored before the deadline is scheduled and before a marker goes on a queue, so no wakeup is lost. finish takes the continuation under the lock and clears it, so it resumes only once. It uses only queue.async and short Mutex locks, so it does not block main. The DemandTimer task runs on the global executor, so the limit bounds the wait while main waits. cancel() on finish breaks the countdown/timer retain cycle. It obeys the idle-wakeups rule: one one-shot DemandTimer, with no polling and no sleep.
  • Instances. AppServices.agentPaneStateFiles is the only registry for real files. AgentUnsentDrafts (AgentTabs+Wiring) and AgentPaneScrollAnchors (AppDelegate) both receive it. The default in AgentTabStore has file: nil, so it never writes. MobileHostPairedDevicesFile is made only in IPhoneHostService.dependencies.didSet, once for each state directory. The single IPhoneHostService is at AppServices.swift:142. dependencies is set after followAccount, as before, so the time when pairedFile becomes non-nil does not change.
  • SignalRelay. The move adds only public and @MainActor on the C handler constant. CmuxNextApp is main-actor by default and CmuxNextControl is not, so @MainActor keeps the same isolation. The assume_isolated_total row moves -1/+1 between the two modules (net 0).
  • AgentPaneScrollAnchors.shared. Only AgentPageProvider.scrollAnchor reads it, from asynchronous page messages. It is set in the same main-thread turn as AppServices init, before any page message can arrive.
  • Tests. On 82d2f13, prepare blocks for up to 2 s on the held queue before armDeadline, so armed < 500 ms fails. On the head, armDeadline runs at once, and the drain marker runs after the draft write on the same serial queue, so both expectations pass. HeldSeal is not weaker: the hold now runs on the same file's queue, and the held save writes the old list again. If the removal did not wait for that save, seal.digest() != digest(old) would fail, and the old gate could not detect this. In the flush test, drained is now an #expect.
  • Ratchet: arch_counts.py --delta shows global_mutable_statics -3 and blocking_allow_escapes -3, and no row goes up. arch-ratchet.py reports "no architecture count rises". prepush-gate.py reports "no new violations". The PR adds no CI job, step or gate, and it does not change arch-ratchet.json.

Two reds are caused by this PR. Each one needs only a description change, so this receipt stays valid (the head does not change):

  • risk evidence holds the PR because it takes back cmux NEXT: agent pane state files save off the main thread #27749's change (AppDelegate.swift) from the last 7 days. Add the line Takes back #27749: <why>.
  • pr-capture-check: QuitCoordinator.swift and QuitSignal.swift are on native UI paths. Add No capture: <why> (this PR makes no visible change).

Notes (they do not block):

  • No test shows that QuitCoordinator calls stateFilesDrained(). The second test calls it directly. If someone removes either await in QuitCoordinator, all tests still pass. On the base, the old flush in applicationWillTerminate also had no test.
  • The red and green fleet runs of the regression test are not recorded yet. The PR says "still to do".
  • QueueDrain does again part of what FirstOutcome does (one resume, with a DemandTimer limit). It could use FirstOutcome.
  • The gate gives an owner warning for CmuxNextWakeups. Tell the Performance Lead about QueueDrain.
  • The queue com.cmuxterm.app.next.agent-pane.state-file is not in threading-model.md. This was already true on the base.
  • The other reds (Sidebar-Look, String tables, literal state roots in acpmux and the nightly-guards) are base findings. This PR does not touch those files.

@lawrencecchen lawrencecchen added ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds and removed ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds labels Oct 30, 2026
…tor members

`steps()` builds the QuitSteps that `complete` runs, and `stopEngines` is the engine stop (the
deadline, then CefShutdown) that stopBrowserEngines ends with. Same steps in the same order; a test
can now run a step through the coordinator with its own engine stop. No behavior change.
…efore it goes on

QuitStepsStateFilesTests runs the coordinator's own steps with a new chat's draft save queued behind
a busy disk (a held file queue, released after 1 s):
- theSaveStepEndsWithTheQueuedDraftOnDisk: prepareWindows returns with the draft on disk.
- theEnginesStopAfterTheQueuedDraftIsOnDisk: stopBrowserEngines reaches the engine stop with the
  draft on disk.
Each fails when its step's `await services.stateFilesDrained()` is removed (the red-proof ref).
Not run here (no Swift build on this VM): the fleet runs them.
…FirstOutcome

One path for "first of an event and a DemandTimer limit wins" (FirstOutcome): QueueDrain keeps only
the count of queues still to answer; its own continuation and countdown class are gone.
@lawrencecchen

Copy link
Copy Markdown
Contributor Author

Review (independent review subagent, arch-rise): SHIP at 07f1b95
perf claim: none
compile: 3f8265b992de37f6d386e0ef passed

Scope: the three commits after the earlier SHIP at 1869061 (b976b02, 25a49ee, 07f1b95). The earlier findings stand for the code that review read.

  • b976b02 (QuitCoordinator.steps(), stopEngines): the steps and their order do not change. stopBrowserEngines still does the Home cache write, then stateFilesDrained(), then arms the deadline, ends the hang assertions, does the debug wedge and calls cef.shutdown(). QuitAttempts() is still made once for each completion. Both new members are module-internal, in the same way as unsaved ("tests replace it"). Nothing is visible outside CmuxNextApp. There is no retain cycle: the lazy closure holds self weakly, and complete() keeps the coordinator alive while the steps run.
  • 07f1b95 (QueueDrain on FirstOutcome<Bool>): no wakeup is lost, because FirstOutcome.value() returns at once after an earlier resolve. Nothing can resume twice, because only the first resolve wins. The DemandTimer limit is scheduled before the queue blocks start, and the local deadline stays alive until cancel() after the await, so the limit still ends the wait. Nothing blocks the main thread. The PR body is correct that FirstOutcome alone cannot do the drain: it resolves on the first value, and the drain needs N answers.
  • 25a49ee (QuitStepsStateFilesTests): the tests check behavior through the coordinator's real steps. When the hold is released at 1 s, the green result is reliable. Each test fails without its stateFilesDrained() call only if the step's other work takes less than 1 s. This is probably true for a test-host AppServices, and the red-proof run will show it. Non-blocking: the tests do not check this condition, so on a host with a high load they can pass without the call. Suggested follow-up: give the drain an observable start (an injected drain clock or hook) and release the hold from it, or at least add a timed warm-up run of the step with #require that it takes less than 500 ms.
  • PR body: it has the Takes back #27749: ... line and the No capture: ... line, and it describes this head. arch-ratchet.py shows no rises, and the head adds no CI job or gate.

`AgentPaneStateFiles(drainBegan:)` is told each time `drained(within:)` begins, before it waits;
AppServices takes the registry through its init (default: a plain one). A test holds a file's queue
and releases it only there, so only a real drain lets a quit step go on: no wall-clock hold decides a
test (lead rule R5). `stateFilesDrained` returns whether every file's saves ran in time.
…lock decides the quit tests

The quit tests held a file's queue and let it go on a timer (1 s), or measured wall time, so a slow
test host could decide them (lead rule R5). DrainGate holds the queue until the registry is told a
drain began (`AgentPaneStateFiles(drainBegan:)`), and records the order of the quit's steps:
- QuitStepsStateFilesTests (both): a step that never awaits the state files never writes the draft,
  and the gate saw no drain.
- theForcedQuitArmsItsDeadlineBeforeItWaitsForTheStateFiles: the order is "deadline armed", then
  "drain began", and the exit ends with the draft on disk.
- theQuitAwaitsAQueuedDraftWithoutBlockingTheMainThread: the main actor lets the held save go once
  the drain began; a wait that blocked it would reach its limit with the draft still queued.
…t in the test process)

The green step 04224ccd at 07f1b95 crashed both quit suites before any assertion:
AppServices.keyWindowSource reads `NSApp.keyWindow`, and `NSApp` is nil in a test process that never
made NSApplication.shared; the quit steps ask for the key window (WindowManager.swift:143). The
tests' services answer nil instead, with no app object.
…em off the main actor)

The compile a1714da9 at d77dbff refused QuitStateFilesTests.swift:85: the drainBegan closure runs
in the drain, off the main actor, and called DrainGateLog.append, main-actor by the test target's
default isolation. Both helper classes are nonisolated now (a Mutex and semaphores inside).
@lawrencecchen

Copy link
Copy Markdown
Contributor Author

Review (independent review subagent, arch-rise): SHIP at 57dfaa1
perf claim: none
compile: f4ac4587d8dbc52fd51452e0 passed

Scope: the delta 07f1b95..57dfaa1 (0aaaccd seam, 9355de1 tests, d77dbff key-window fix, 57dfaa1 isolation fix), on top of the earlier SHIP at 07f1b95.

  • Seam: AgentPaneStateFiles(drainBegan:) is an injected init closure (default {}), with no new static. AppServices.init(agentPaneStateFiles:) makes a new default object for each call. Production (AppDelegate.swift:134) gets the plain registry, which is the real one because the hook is its only option. stateFilesDrained() -> Bool has @discardableResult. All callers are correct: QuitCoordinator 215/238, AppDelegate 51 (_ = await files), and the test.
  • No clock decides the 4 quit tests (fixes the weak point R5 from the earlier review). Only the drain hook or the test's own defer releases the held save. With no drain call, tests 1, 3 and 4 fail on every run (step order and draft checks). Test 2 fails if the wait blocks the main actor, because the release must run on the main actor. The hook runs before QueueDrain waits, and the forced-quit test injects armDeadline, so neither limit can make a broken run green.
  • All 4 tests build their services through queuedDraft, which sets keyWindowSource = { nil }.
  • DrainGate and DrainGateLog are nonisolated final class: Sendable. Their state is Sendable (semaphore, FirstOutcome, Mutex-guarded log). The hook closure captures only local Sendable values, made in a nonisolated init. CmuxNextCompat and CmuxNextWakeups are in the test target's dependencies.
  • arch-ratchet.py --base origin/feat-cmux-next --head HEAD: no architecture count rises.
  • The PR body describes the head and keeps the "Takes back cmux NEXT: agent pane state files save off the main thread #27749" and "No capture" lines.

Notes (do not block): if stateFilesDrained stopped draining the agent-pane files, test 2 would hang at gate.drainBegan() and not fail with an assertion (the suite has no .timeLimit). It never passes falsely. Under a host stall of more than 2 s, a correct run can go red. The fleet red/green runs of the quit tests were still running at review time.

…cking wait has a time limit

The green step eb20257d at 57dfaa1 crashed both quit suites before any assertion: a quit step
reads `NSApp.isActive` (NotificationCenterService+ProgramStatusSeen.swift:57), and `NSApp` is nil in
a test process with no app object. The one helper makes NSApplication.shared first, as the other
AppServices tests do; the root fix of those reads is fix/arch-rise-appservices-no-nsapp-trap.
theQuitAwaitsAQueuedDraftWithoutBlockingTheMainThread gets a 1 minute time limit, so a wait that
never ends fails the test instead of the run.
lawrencecchen added a commit that referenced this pull request Oct 30, 2026
…ition root too

The second trap (#28275 green eb20257db2553e2e4a4dd378, NotificationCenterService+ProgramStatusSeen.swift:57):
AppServices.init starts notifications, whose viewed-program-status observation calls viewedTabs, which
read NSApp.isActive. Any test that built AppServices and then awaited (each quit step awaits) trapped.

- AppServices.init takes appActiveSource with no default; AppDelegate passes { NSApp.isActive }.
- The notifications' six reads (viewedTabs, isViewed, the program-status visibility, arrive, the
  hand-off and sandbox-run banners) ask services.appActiveSource().
- The test target's init reads the application only if the test made one (NSApp?.keyWindow,
  NSApp?.isActive ?? false), so tests behave as before and never unwrap the global.
- AppServicesKeyWindowSourceTests: the recording test also checks the app-active lookups and awaits the
  started observation's first ask of the injected source (no clock).
…lose; a lost draft is named and told

R9 review of #28275 (reviewer 0053c0be):
- R-L1 (bare 2 s): the state files' drain has no limit of its own. The quit awaits it only after its
  deadline is armed: QuitCoordinator's engine stop now arms QuitDeadline.engineStopLimit (15 s), drains,
  closes the engines (`closeEngines`, a seam for tests) and drains again, so a page that closes and
  saves its draft (the composer's page-hide flush) is on disk when the quit ends. The forced quit
  awaits it under its 160 ms deadline. The prepareWindows drain is gone (the engine stop covers it).
  QueueDrain is deleted: each file awaits its own queue (`AgentPaneStateFile.saved()`).
- R-L5 (silent loss): when a quit deadline ends the process while a state file still saves, its end
  (`AgentPaneStateLoss.deadlineEnd`) logs a fault naming the files (no content) and records them in
  `<state>/agent-state-lost.json`; the next launch takes the record and, for a new chat's unsent text,
  shows a toast ("Text typed in a new chat just before cmux quit was not saved.", 21 locales). Both
  deadline reasons name the state files.
- Tests (no clock decides any): QuitStepsStateFilesTests (the engines close after the queued draft is
  on disk; a draft saved while the engines close is on disk when the quit ends), AgentPaneStateLossTests
  (the deadline records the files still saving, the next launch takes them once, the notice is only for
  a lost draft), AgentPaneStateFileTests (a file is saving until its saves ran).
Not run here (no Swift build on this VM): the fleet runs them.
…se-quit-flush

Port of #28275 onto feat (lead decision 05:20Z), keeping feat's state-file and
quit work and adding only what this PR adds beyond it:
- AgentUnsentDrafts: feat's #28204 undelivered drafts and unreadable-file
  set-aside, on this PR's per-app AgentPaneStateFiles (no static registry).
- QuitCoordinator: feat's #22919 QuitStepLimiter, time-left reply margin,
  launch marks and engineCloseWait, plus this PR's drains of the state files
  around the engine close, the testable quitSteps() (was steps(), which now
  names feat's limiter) and closeEngines seam, and the deadline end that records
  cut-off files.
- AppDelegate/AppServices: feat's openedURLs, windowRecord, peer approver and
  injected key window; this PR's secondSignalExit (no blocking flush in prepare)
  and the state-files owner and loss record.
- Paired phones: feat's #25437 MobileHostPairedStore now owns the directory's
  one MobileHostPairedDevicesFile (configure(directory:lastQuitCutOff:
  cutOffSettled:)); the iPhone host takes it from the store for the quit watch,
  so the shared-list transactions and this PR's cut-off accounting use one file
  object and one queue (an instance enqueue that counts as saving), not two.
…ith it); PhoneAdmissionStoreTests need no forgetThisRun

Fleet compile 6ade1fafcc18 at b062435: QuitCoordinator.swift:215 had no
'choice' in quitSteps(), and two feat tests called the static per-path
forgetThisRun, which the per-file inForce record replaces: the restored store's
new list file remembers no digest, as a relaunch does.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

8 issues found across 30 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="plans/cmux-next/threading-model.md">

<violation number="1" location="plans/cmux-next/threading-model.md:130">
P2: This update documents agent-pane state-file work without registering its queue, so the threading model still omits `com.cmuxterm.app.next.agent-pane.state-file`. Add that queue to the Agents registration list and describe its utility serial ownership.</violation>
</file>

<file name="Packages/macOS/CmuxNext/Sources/CmuxNextAgentPane/AgentPaneStateFile.swift">

<violation number="1" location="Packages/macOS/CmuxNext/Sources/CmuxNextAgentPane/AgentPaneStateFile.swift:46">
P2: The `defer` can move `ran` backwards when concurrent writes reach the serial queue out of number order. Keep this marker monotonic, otherwise a completed file remains reported as saving and can produce a false lost-draft notice at the next quit.</violation>
</file>

<file name="Packages/macOS/CmuxNext/Tests/CmuxNextAgentPaneTests/AgentPaneStateLossTests.swift">

<violation number="1" location="Packages/macOS/CmuxNext/Tests/CmuxNextAgentPaneTests/AgentPaneStateLossTests.swift:64">
P3: This test does not cover the either-order behavior it claims to verify: it always reads through `take()` before calling `lastQuitCutOffPairedPhones()`. Add a case that calls the paired check first, and exercise the notice read path as well.</violation>
</file>

<file name="Packages/macOS/CmuxNext/Sources/CmuxNextMobile/MobileHost/MobileHostPairedStore.swift">

<violation number="1" location="Packages/macOS/CmuxNext/Sources/CmuxNextMobile/MobileHost/MobileHostPairedStore.swift:110">
P2: This queues a read-only reload as a save, so a quit deadline that overlaps a reload can mark `paired-devices.json` cut off even though no save was interrupted. The next launch then clears the pairing seal and forces every phone to pair again; track read transactions separately from saves, or only mark the file when a save/withdrawal is actually pending.</violation>
</file>

<file name="Packages/macOS/CmuxNext/Sources/CmuxNextMobile/MobileHost/MobileHostPairedDevices.swift">

<violation number="1" location="Packages/macOS/CmuxNext/Sources/CmuxNextMobile/MobileHost/MobileHostPairedDevices.swift:313">
P1: This check is not atomic with admitting work to the file queue. A quit deadline can observe no save just before a removal is queued, exit without recording `paired-devices.json`, and leave the pre-removal paired list trusted after relaunch; synchronize deadline admission with queue accounting or close the queue before taking the cut-off snapshot.</violation>

<violation number="2" location="Packages/macOS/CmuxNext/Sources/CmuxNextMobile/MobileHost/MobileHostPairedDevices.swift:357">
P2: Ignoring marker-removal failure can leave a stale `paired-devices.cut-off` that invalidates every subsequent pairing on relaunch. Handle failed cleanup or make a later successful save retire the marker before allowing the marker to trigger recovery again.</violation>
</file>

<file name="Packages/macOS/CmuxNext/Sources/CmuxNextControl/Diagnostics/ForcedQuitExit.swift">

<violation number="1" location="Packages/macOS/CmuxNext/Sources/CmuxNextControl/Diagnostics/ForcedQuitExit.swift:40">
P1: The deadline callback performs blocking filesystem writes before `_exit`, so a stalled state directory can prevent the hard deadline from ending a stuck quit. Record the loss through a bounded/nonblocking path, or guarantee the deadline thread reaches the process exit independently of that recording.</violation>
</file>

<file name="Packages/macOS/CmuxNext/Sources/CmuxNextAgentPane/AgentPaneStateLoss.swift">

<violation number="1" location="Packages/macOS/CmuxNext/Sources/CmuxNextAgentPane/AgentPaneStateLoss.swift:71">
P2: `pairedCutOffSettled()` forgets the pending marker before knowing that record cleanup succeeded. If `write` fails, later settlement calls no-op and the record remains forever, causing repeated fail-closed recovery on every launch. Make the write report success and clear `state.pending` only after cleanup succeeds.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

/// when one is queued or running, it writes the cut-off marker. It never calls the Keychain (a hung
/// securityd must not hold the process's end): the next launch clears the seal (``distrustIfCutOff(recorded:)``).
public func markCutOffIfSaving() -> CutOff {
guard isSaving else { return .notSaving }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: This check is not atomic with admitting work to the file queue. A quit deadline can observe no save just before a removal is queued, exit without recording paired-devices.json, and leave the pre-removal paired list trusted after relaunch; synchronize deadline admission with queue accounting or close the queue before taking the cut-off snapshot.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At Packages/macOS/CmuxNext/Sources/CmuxNextMobile/MobileHost/MobileHostPairedDevices.swift, line 313:

<comment>This check is not atomic with admitting work to the file queue. A quit deadline can observe no save just before a removal is queued, exit without recording `paired-devices.json`, and leave the pre-removal paired list trusted after relaunch; synchronize deadline admission with queue accounting or close the queue before taking the cut-off snapshot.</comment>

<file context>
@@ -247,35 +280,104 @@ public struct MobileHostPairedDevicesFile: Sendable {
+    /// when one is queued or running, it writes the cut-off marker. It never calls the Keychain (a hung
+    /// securityd must not hold the process's end): the next launch clears the seal (``distrustIfCutOff(recorded:)``).
+    public func markCutOffIfSaving() -> CutOff {
+        guard isSaving else { return .notSaving }
+        do {
+            try writeMarker(cutOffMarker)
</file context>

self.save = save
self.armDeadline = armDeadline ?? { QuitDeadline(reason: "a forced quit began its Home cache write").arm(after: $0) }
self.armDeadline = armDeadline ?? {
QuitDeadline(reason: "a forced quit began its last writes (the Home cache, the agent panes' state files)", end: deadlineEnd).arm(after: $0)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: The deadline callback performs blocking filesystem writes before _exit, so a stalled state directory can prevent the hard deadline from ending a stuck quit. Record the loss through a bounded/nonblocking path, or guarantee the deadline thread reaches the process exit independently of that recording.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At Packages/macOS/CmuxNext/Sources/CmuxNextControl/Diagnostics/ForcedQuitExit.swift, line 40:

<comment>The deadline callback performs blocking filesystem writes before `_exit`, so a stalled state directory can prevent the hard deadline from ending a stuck quit. Record the loss through a bounded/nonblocking path, or guarantee the deadline thread reaches the process exit independently of that recording.</comment>

<file context>
@@ -25,16 +25,20 @@ public final class ForcedQuitExit {
         self.save = save
-        self.armDeadline = armDeadline ?? { QuitDeadline(reason: "a forced quit began its Home cache write").arm(after: $0) }
+        self.armDeadline = armDeadline ?? {
+            QuitDeadline(reason: "a forced quit began its last writes (the Home cache, the agent panes' state files)", end: deadlineEnd).arm(after: $0)
+        }
         self.end = end ?? { exit(0) }
</file context>

host's paired-phone list and its Keychain seal, utility, serial: its file and Keychain calls block this queue,
never main; one item per save, read or withdrawal, each from a person's action or an account change, so no back-pressure;
quit waits at most 2 s for it under `terminate`)
the quit awaits it holding no thread, with the agent panes' state files (`AppServices.stateFilesDrained`), after its deadline is armed; that deadline records the files still saving, `AgentPaneStateLoss`)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This update documents agent-pane state-file work without registering its queue, so the threading model still omits com.cmuxterm.app.next.agent-pane.state-file. Add that queue to the Agents registration list and describe its utility serial ownership.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At plans/cmux-next/threading-model.md, line 130:

<comment>This update documents agent-pane state-file work without registering its queue, so the threading model still omits `com.cmuxterm.app.next.agent-pane.state-file`. Add that queue to the Agents registration list and describe its utility serial ownership.</comment>

<file context>
@@ -127,7 +127,7 @@ matches the label's literal text; `<x>` matches an interpolation). Its owner row
   host's paired-phone list and its Keychain seal, utility, serial: its file and Keychain calls block this queue,
   never main; one item per save, read or withdrawal, each from a person's action or an account change, so no back-pressure;
-  quit waits at most 2 s for it under `terminate`)
+  the quit awaits it holding no thread, with the agent panes' state files (`AppServices.stateFilesDrained`), after its deadline is armed; that deadline records the files still saving, `AgentPaneStateLoss`)
 - Browser: `com.cmuxterm.next.browser-host.provider.write`, `cmux browser-host provider reader`,
   `com.cmuxterm.chromium-host.read`, `com.cmuxterm.chromium-host.exit`, `com.cmuxterm.app.next.cef-children`,
</file context>

}
queue.async { [self] in
guard newest.withLock({ $0 == number }) else { return }
defer { numbers.withLock { $0.ran = number } }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The defer can move ran backwards when concurrent writes reach the serial queue out of number order. Keep this marker monotonic, otherwise a completed file remains reported as saving and can produce a false lost-draft notice at the next quit.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At Packages/macOS/CmuxNext/Sources/CmuxNextAgentPane/AgentPaneStateFile.swift, line 46:

<comment>The `defer` can move `ran` backwards when concurrent writes reach the serial queue out of number order. Keep this marker monotonic, otherwise a completed file remains reported as saving and can produce a false lost-draft notice at the next quit.</comment>

<file context>
@@ -51,12 +38,13 @@ public nonisolated final class AgentPaneStateFile: Sendable {
         }
         queue.async { [self] in
-            guard newest.withLock({ $0 == number }) else { return }
+            defer { numbers.withLock { $0.ran = number } }
+            guard numbers.withLock({ $0.newest == number }) else { return }
             let data: Data?
</file context>
Suggested change
defer { numbers.withLock { $0.ran = number } }
defer { numbers.withLock { $0.ran = max($0.ran, number) } }

}
let selected = state.generation
MobileHostPairedDevicesFile.enqueue { [self] in
file.enqueue { [self] in

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This queues a read-only reload as a save, so a quit deadline that overlaps a reload can mark paired-devices.json cut off even though no save was interrupted. The next launch then clears the pairing seal and forces every phone to pair again; track read transactions separately from saves, or only mark the file when a save/withdrawal is actually pending.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At Packages/macOS/CmuxNext/Sources/CmuxNextMobile/MobileHost/MobileHostPairedStore.swift, line 110:

<comment>This queues a read-only reload as a save, so a quit deadline that overlaps a reload can mark `paired-devices.json` cut off even though no save was interrupted. The next launch then clears the pairing seal and forces every phone to pair again; track read transactions separately from saves, or only mark the file when a save/withdrawal is actually pending.</comment>

<file context>
@@ -97,7 +107,7 @@ public final class MobileHostPairedStore: Sendable {
                 }
                 let selected = state.generation
-                MobileHostPairedDevicesFile.enqueue { [self] in
+                file.enqueue { [self] in
                     var list = file.load(userID: user)
                     let removals = self.state.withLock { state in
</file context>

}
return
}
try? FileManager.checked.removeItem(at: cutOffMarker)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Ignoring marker-removal failure can leave a stale paired-devices.cut-off that invalidates every subsequent pairing on relaunch. Handle failed cleanup or make a later successful save retire the marker before allowing the marker to trigger recovery again.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At Packages/macOS/CmuxNext/Sources/CmuxNextMobile/MobileHost/MobileHostPairedDevices.swift, line 357:

<comment>Ignoring marker-removal failure can leave a stale `paired-devices.cut-off` that invalidates every subsequent pairing on relaunch. Handle failed cleanup or make a later successful save retire the marker before allowing the marker to trigger recovery again.</comment>

<file context>
@@ -247,35 +280,104 @@ public struct MobileHostPairedDevicesFile: Sendable {
+            }
+            return
+        }
+        try? FileManager.checked.removeItem(at: cutOffMarker)
+        cutOffSettled()
+        clearedForCutOff.withLock { $0 = true }
</file context>

state.withLock { state in
guard !state.pending.isEmpty else { return }
let pending = state.pending
state.pending = []

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: pairedCutOffSettled() forgets the pending marker before knowing that record cleanup succeeded. If write fails, later settlement calls no-op and the record remains forever, causing repeated fail-closed recovery on every launch. Make the write report success and clear state.pending only after cleanup succeeds.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At Packages/macOS/CmuxNext/Sources/CmuxNextAgentPane/AgentPaneStateLoss.swift, line 71:

<comment>`pairedCutOffSettled()` forgets the pending marker before knowing that record cleanup succeeded. If `write` fails, later settlement calls no-op and the record remains forever, causing repeated fail-closed recovery on every launch. Make the write report success and clear `state.pending` only after cleanup succeeds.</comment>

<file context>
@@ -0,0 +1,145 @@
+        state.withLock { state in
+            guard !state.pending.isEmpty else { return }
+            let pending = state.pending
+            state.pending = []
+            let onDisk = record.flatMap { try? Data(blockingContentsOf: $0) }.flatMap { try? JSONDecoder().decode([String].self, from: $0) } ?? []
+            write(onDisk.filter { !pending.contains($0) && !Self.isPaired($0) })
</file context>

defer { try? FileManager.default.removeItem(at: directory) }
AgentPaneStateLoss(directory: directory).deadlineEnd(files: AgentPaneStateFiles(), alsoCutOff: { [AgentPaneStateLoss.pairedPhonesFileName] }, end: {})()
let launch = AgentPaneStateLoss(directory: directory)
let names = await launch.take()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: This test does not cover the either-order behavior it claims to verify: it always reads through take() before calling lastQuitCutOffPairedPhones(). Add a case that calls the paired check first, and exercise the notice read path as well.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At Packages/macOS/CmuxNext/Tests/CmuxNextAgentPaneTests/AgentPaneStateLossTests.swift, line 64:

<comment>This test does not cover the either-order behavior it claims to verify: it always reads through `take()` before calling `lastQuitCutOffPairedPhones()`. Add a case that calls the paired check first, and exercise the notice read path as well.</comment>

<file context>
@@ -0,0 +1,95 @@
+        defer { try? FileManager.default.removeItem(at: directory) }
+        AgentPaneStateLoss(directory: directory).deadlineEnd(files: AgentPaneStateFiles(), alsoCutOff: { [AgentPaneStateLoss.pairedPhonesFileName] }, end: {})()
+        let launch = AgentPaneStateLoss(directory: directory)
+        let names = await launch.take()
+        #expect(names == [AgentPaneStateLoss.pairedPhonesFileName])
+        #expect(launch.lastQuitCutOffPairedPhones(), "the paired check missed the record the notice read first")
</file context>

…eat's #28321 test init replaced the one QuitStateFilesTests called)

Suites step 4e05c54c690b at 729a0b7: QuitStateFilesTests.swift:33 passed
agentPaneStateFiles to feat's test-only convenience init, which had no such
argument.
@cursor

cursor Bot commented Nov 1, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

…er test (the per-app writer registry)

Compile 7b999930c9b2 at 0ff4259: UnsentDraftTests.swift built
AgentUnsentDrafts(file:) with no files. Each test now gives its instances one
AgentPaneStateFiles, so a fresh instance reads after the queued save as the
static registry made it do. A grep of the package finds no other caller of a
changed initializer without its new argument.
@cursor

cursor Bot commented Nov 1, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@lawrencecchen

Copy link
Copy Markdown
Contributor Author

Review (claude-review-subagent-cw): BLOCK at e79b754

Scope: git diff b062435158c^2 e79b754facb (the PR's net change over feat), git show --remerge-diff b062435158c, and the three fix commits 729a0b7, 0ff4259, e79b754. Read only. I did not compile or run tests.

Blocking finding

  1. The first state-file drain takes the engines' reserved time and has no limit (Packages/macOS/CmuxNext/Sources/CmuxNextApp/Quit/QuitCoordinator.swift:251-252).
    • Feat's cmux NEXT: a keep quit ends within 4 s whatever any step does; Restart to Update never leaves cmux invisible #22919 QuitSteps.run gives the engines a fixed reserve. It stops waiting for late saves at timeLeft - engineCloseWait (Packages/macOS/CmuxNext/Sources/CmuxNextControl/Quit/QuitSteps.swift:91-92). When a save is late, stopBrowserEngines starts with only engineCloseWait + replyMargin left.
    • The merge puts await services.stateFilesDrained() in front of closeEngines(). This drain has no limit. It also waits on the paired-phones queue, and that queue makes Keychain calls (MobileHostPairedDevicesFile.save, withdrawTrust, the launch distrustIfCutOff, and the store's reload read).
    • Result: under disk load (the doc says up to about 1 s per save), Chromium gets less than its 2 s close wait, so CefShutdown can lose its cookie and storage flush. If securityd hangs (the cx-ccjnn failure that QuitDeadline was made for), closeEngines() never starts. Then the deadline ends the process with no Chromium close and no CefShutdown flush. This is true for keep quits (keepQuitLimit) and for End quits (engineStopLimit). Feat's guarantee was that the engines always get their reserve. Before this PR, the paired-list flush ran after the engines, in applicationWillTerminate.
    • The first drain does not make the files safer. Each file's queue writes in the background whether or not the main actor waits. The deadline ends the process, not the drain. Chromium's 10 s shutdown watchdog fires after the 3 s and 4 s deadlines, so the comment's reason does not apply. The second drain (line 254) already holds AppKit's reply until the files are written, and the deadline end records what is still saving.
    • Fix: do one of these. (a) Remove the first drain, keep the drain after closeEngines(), and change QuitStepsStateFilesTests.theEnginesCloseAfterTheQueuedDraftIsOnDisk to check that the draft is on disk when the step ends. (b) Limit the first drain to max(0, timeLeft - engineCloseWait), for example as a steps.run(...) step that the engines do not wait on after its limit. Option (a) is simpler.

Checked and correct

  • Keep-quit deadline: deadline.arm(after: QuitDeadline.keepQuitLimit) is still the first action in complete (QuitCoordinator.swift:170), before updater.prepareForQuit() and crashRecovery.quitBegan(). The deadline is made in init with services.quitDeadlineEnd(). quit is a lazy property, so agentPaneStateFiles, agentPaneStateLoss and iphone.pairedQuitWatch exist when the deadline is made.
  • quitSteps(_ choice:): endLocalSessions calls endSessionsAndStop(choice). Feat's steps limiter, timeLeft(margin: replyMargin), the quit.saved and quit.engines_stopped launch marks, and cef.shutdown(timeout: engineCloseWait) (now in closeEngines) are all kept.
  • AgentUnsentDrafts: feat's carry of #27963: preserve drafts through reconnect #28204 undelivered map and unreadable-file set-aside are not changed. Only init changed, to files.at. In production, AgentPaneStateFiles has one owner (AppServices). The drafts and the scroll anchors get it from there. The AgentTabStore default uses file: nil, so it writes nothing.
  • AppDelegate and AppServices: feat's openedURLs, windowRecord.watchLaunch, peerApprover and the injected key-window and app-active sources are kept. The PR's secondSignalExit (no blocking flush in prepare), agentPaneStateFiles and agentPaneStateLoss are present. forcedQuitExit is lazy and is first used at the second signal, so services is set when quitDeadlineEnd() is read.
  • No allowBlocking(.terminate) flush and no AgentPaneStateFile.at, flushAll, static forgetThisRun, static enqueue or static per-path inForce stay in Packages/ (git grep).
  • Paired phones: the only MobileHostPairedDevicesFile( in Sources is in MobileHostPairedStore.configure (MobileHostPairedStore.swift:93). AppServices makes one IPhoneHostService, and that makes one store, so production has one file object per directory. The first configure comes from the dependencies didSet in wire, after iphone.pairedQuitRecord is set (AppServices+Cloud.swift:104). No Sources caller uses attach(directory:), so the file always gets the cut-off closures. Every store transaction (reload, forget, submit) and the forgotten sign-out saves run through the file's queue and its saving count. Thus pairedSavesDrained() and markCutOffIfSaving() cover them.
  • Relaunch semantics: a relaunch makes a new file object with inForce == nil, so only the Keychain seal counts. This is the same result as feat's per-path static after forgetThisRun. Because production has one object per directory, the per-object record is the same as the old per-process record. The two changed tests in PhoneAdmissionStoreTests (a new store, so a new file) still check that a removed phone is not trusted again and that an older sealed file put back is refused.
  • Localization: each of the two new strings has 21 translated languages.

Non-blocking findings

  1. MobileHostPairedDevicesFile.enqueue counts each transaction as saving (MobileHostPairedDevices.swift:238). This includes reload's read when no removal is pending (MobileHostPairedStore.swift:110). If a read is stuck in the Keychain when the quit deadline fires, the next launch marks the list as cut off, clears the seal, and every phone must pair again. This fails closed, so it is safe, but it is a false positive that the person sees. loadInBackground is not counted, so the two reads are not consistent. Recommendation: count only operations that write.
  2. Doc comment in the wrong place (IPhoneHostService+Pairing.swift:27-30): the loadPaired doc ("Loads the list of the signed-in user...") is now above configurePairedFile. Move it down to loadPaired.
  3. pairedFile changed from private(set) to internal var (IPhoneHostService.swift:141). Only configurePairedFile sets it. Keep it private(set) so that nothing else in the module can replace the quit-watched file.
  4. A save queued after the second drain (for example, a page message that arrives late on the main queue after closeEngines(), or anything in applicationWillTerminate) is not waited for. Feat's blocking flushAll in applicationWillTerminate covered that time. The atomic write prevents a corrupt file, so the only cost is one lost save. Accept it or note it in the PR body.
  5. AgentPaneScrollAnchors.shared is still a global mutable static (AppDelegate.swift:141). This was already on feat. It is now set after AppServices init. It is not a second writer, because it uses services.agentPaneStateFiles.

Observed / Not run

  • Observed: I read the diffs and the remerge-diff, and I ran git grep for the removed statics and for the paired-file constructors and callers.
  • Not run: no Swift build or test (not allowed on this machine). I depend on the fleet compile that the fix commits refer to. I did not run the QuitStepsStateFilesTests, QuitStateFilesTests, UnsentDraftTests or MobileHostPairingTests suites.

…he time they do not need

Review of e79b754 (comment 6554069134): an unbounded drain before
closeEngines could use up feat's engineCloseWait (a hung Keychain call on the
paired-phones queue, a loaded disk), so CefShutdown lost its flush. The first
drain is now a QuitStepLimiter step limited to the deadline's time left less
the reply margin and engineCloseWait; past it, it runs beside the close and the
quit awaits it in finishOverruns. The drain after the close is unchanged.
@lawrencecchen

Copy link
Copy Markdown
Contributor Author

Review (claude-review-subagent-cw): SHIP at 2fdc8e5

Scope: git show 2fdc8e53b54 (the only commit after e79b754; it changes only QuitCoordinator.swift), checked against feat's QuitSteps.run and QuitStepLimiter. Read only. I did not compile or run tests.

The blocker from comment 6554069134 is fixed

  • The engines keep engineCloseWait. The first drain is now steps.run("state_files", limit: spare) with spare = deadline.timeLeft(margin: replyMargin + engineCloseWait) ?? .zero (Packages/macOS/CmuxNext/Sources/CmuxNextApp/Quit/QuitCoordinator.swift:250-251). timeLeft clamps to zero. The deadline is always armed when this line runs: line 242 arms it, or the keep quit's earlier arm stays. When the step ends or passes its limit, at least engineCloseWait + replyMargin is left for closeEngines(). The only exception is the main-actor delay of the limit timer, which is small. A hung Keychain call on the paired-phones queue now costs at most spare and does not stop Chromium's close and CefShutdown flush.
  • The overrun does not delay the reply past feat's margin. A step that passes its limit goes into overruns. The last call in QuitSteps.run is limiter.finishOverruns(within: timeLeft() ?? .zero), with no excluding (Packages/macOS/CmuxNext/Sources/CmuxNextControl/Quit/QuitSteps.swift:96). It runs after stopBrowserEngines() returns, so it covers this step. timeLeft keeps replyMargin. The earlier finishOverruns calls (lines 79 and 92) run before this step exists, so they do not see it.
  • A .zero limit is handled correctly. run starts the step task, the zero timer resumes the waiter, the step is recorded as an overrun with the quit.state_files.limit mark, and the quit goes on to closeEngines(). If finishOverruns then gets a zero limit, it marks quit.overruns.limit and leaves the step to the deadline. No path waits on the drain without a limit before the engines close.
  • QuitStepsStateFilesTests should still pass. The test services replace the deadline with one whose end does nothing (QuitStateFilesTests.swift:36), so arming it with engineStopLimit gives spare of about 0.8 s. The step's task calls drained(), which logs "drain began" and releases the held save. The drain after the close logs the second event, so there are still two events. In test 1, the draft is on disk before closeEngines when the drain ends within 0.8 s. In test 2, the drain after the close covers the draft that the close queues.

Non-blocking findings

  1. The drain after the close still has no limit (QuitCoordinator.swift:254). It runs before the last finishOverruns. If the paired-phones queue hangs, this drain waits until the deadline, so the reply margin is not kept and the deadline's end (which records the files still saving) ends the process instead of AppKit's reply. This was already true before this commit, and the deadline still bounds the quit and records the loss. To make it the same as the first drain, run it as a step too: steps.run("state_files.after", limit: deadline.timeLeft(margin: replyMargin) ?? .zero). The last finishOverruns then covers it.
  2. Test 1 now has a time limit. theEnginesCloseAfterTheQueuedDraftIsOnDisk passes only if the gated drain ends within about 0.8 s. On a heavily loaded fleet Mac this can flake. Also, the suite doc says "No clock decides these tests" (QuitStepsStateFilesTests.swift:11), and that is now false for this test. Update the doc, or give the test services a deadline with a longer limit.
  3. These items from comment 6554069134 are still open and do not block: a reload read counts as a save (MobileHostPairedDevices.swift:238), the loadPaired doc comment is in the wrong place (IPhoneHostService+Pairing.swift:27-30), and pairedFile should be private(set) (IPhoneHostService.swift:141).

Observed / Not run

  • Observed: I read the diff from e79b754 to 2fdc8e5, QuitStepLimiter.run and finishOverruns, the order of QuitSteps.run, and how QuitStateFilesTests sets up its deadline. I ran git grep for consumers of the quit launch marks and found none that a new quit.state_files mark would break.
  • Not run: no Swift build or test on this machine. The author says the fleet compile and the suites are queued at this head. This SHIP depends on those results.

@lawrencecchen

Copy link
Copy Markdown
Contributor Author

Conflict wave (lead decision 2026-11-01 05:20Z: "port onto feat"). This PR now keeps feat's #28204 drafts work and the #22919 quit limits, and it adds only what it adds beyond them:

  • b062435, the merge: AgentUnsentDrafts uses feat's undelivered/unreadable logic on this PR's per-app AgentPaneStateFiles. QuitCoordinator keeps feat's QuitStepLimiter, reply margin, launch marks and engineCloseWait, inside this PR's closeEngines seam, with the state-file drains around it. AppDelegate/AppServices keep feat's openedURLs, windowRecord, peerApprover and injected sources, and this PR's secondSignalExit, state files and loss record. Paired phones: feat's MobileHostPairedStore owns the directory's one list file, and the iPhone host takes it for the quit watch. So the store's transactions and this PR's cut-off accounting share one file and one queue.
  • 729a0b7, 0ff4259, e79b754: API and test fixes that the fleet compile found (quitSteps(choice), the test AppServices init, feat's tests on the per-app writer registry).
  • 2fdc8e5: the review of e79b754 (comment 6554069134) blocked an unbounded drain before the engines close. The drain is now a limited quit step, so the engines always keep engineCloseWait.

Fleet evidence at 2fdc8e5:

  • Compile (cmux-ci compile-check --package CmuxNext --app, job cd6de22ca17bee3f20ba358f): CmuxNext passed (208 s), app passed (83 s).
  • Suites (step b6561bef24320c16481b3dd1, aws-m4pro-4): QuitStateFilesTests, QuitStepsStateFilesTests, UnsentDraftTests, AgentPaneStateFileTests, AgentPaneStateLossTests, MobileHostPairingTests, PhoneAdmissionTests, PhoneAdmissionStoreTests, ChatScrollRestoreTests, ChatScrollRestoreDefaultPathTests, RestoredNewTabPageTitleTests, ForcedQuitExitTests: all 12 passed, 102 tests ran.
  • Review: SHIP at this head (comment 6554094456). Non-blocking notes from it: the drain after the close is still unbounded (it is bounded by the deadline, which records what it cuts off), and test 1 now depends on a 0.8 s window.

@lawrencecchen lawrencecchen added the ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds label Nov 1, 2026
@lawrencecchen
lawrencecchen merged commit fe2bcde into feat-cmux-next Nov 1, 2026
181 of 196 checks passed
@lawrencecchen
lawrencecchen deleted the fix/arch-rise-quit-flush branch November 1, 2026 06:18
austinywang added a commit that referenced this pull request Nov 3, 2026
feat-cmux-next landed the same goal as this PR's Swift change:
#27749 saves the agent panes'
drafts and scroll places off the main thread (one serial queue per file,
latest wins), and #28275 gives
each file one owner and has both quit paths await the saves under the
quit deadline, holding no thread.

The conflicts (12 files) resolve to the base: StateFileWriter,
AgentPanePersistenceQuit, their tests and the App wiring are dropped,
with the app_lines +8 allowance they needed. That also removes the two
blocking findings of the review at 6aaafd0 (a seal that refused the
quit's own late draft, and an unbounded drain before CefShutdown): the
base never seals and bounds its wait.

Kept: guardrails.py --sites (the whole-tree count) and its test. It now
diffs against git's built-in empty tree instead of writing an empty tree
and commit object (review note 5).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was previously deployed

1 inactive deployment
ci-routing-agent — 2fdc8e53 Deployed Nov 1, 2026 by lawrencecchen via namespace autofix #10763
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant