Skip to content

cmux NEXT: agent pane state files save off the main thread - #27749

Merged
teamleaderleo merged 8 commits into
feat-cmux-nextfrom
leo/next-main-thread-io
Oct 30, 2026
Merged

teamleaderleo merged 8 commits into
feat-cmux-nextfrom
leo/next-main-thread-io

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Leo (2026-10-29): "well somehow at "100% cpu" usage claude code desktop still seems fast / how does that work? how can cmux next also do that? let's do that"

and: "but i also wonder how much of this is simply putting the desktop at higher priority?" ... "the gui of the desktop app i mean"

This is the app side of the QoS work; the process side is #27587.

Where the main thread stalls under load

The measurements in #27587 (scripts/measure/qos-load-bench.sh, cmux-mac-mini, 10 cores) answer Leo's question:

  • CPU is not the problem. A thread at the main thread's QoS runs every 120 Hz frame on time at 100% CPU (0 late of 21,600). cmux already runs at 46, like Claude Desktop's 47-54, so raising the GUI's priority gains nothing.
  • Disk on the main thread is the problem. Under agent-like disk load (git, rg, find, dd), a 64 KiB Data.write(options: .atomic) save took 1.0 s at p95, and 44% of them took longer than a frame, whatever priority the load ran at. Uncached reads and fsyncs stall too (p99 86 ms; fsync p50 33 ms).

An audit of the main thread (every file under Packages/macOS/CmuxNext/Sources) found few such waits: daemon socket I/O already runs on actors, and blocking calls go through checked wrappers. Ranked by how often they hit the interactions Leo named:

main-thread wait trigger how often this PR
AgentUnsentDrafts.save: atomic write of every tab's draft typing in a new chat every settled edit (300 ms throttle) moved off main
AgentPaneScrollAnchors.save: atomic write of up to 500 places scrolling a transcript every settled scroll moved off main
AgentPaneScrollAnchors first read switching to or opening a chat once per run still a read, now ordered behind queued saves
AgentPaneBrowserChoices.list: LaunchServices queries, Info.plist reads, icon renders for 16 apps opening a preview card's "Open in" menu each menu open next
RemoteRdStreamTransport queue.sync per key and pointer event remote browser input each event next
TerminalOutputLane queue.sync on close closing a terminal tab each close next
icon(forFile:) in drag and Open With menus file drags, menus each next

What this does

AgentPaneStateFile (new, CmuxNextAgentPane) writes one small JSON state file on its own serial utility queue:

  • Saves keep call order. A save that a newer one replaced before it ran is skipped.
  • JSON encoding runs on the queue too, so the main thread only hands over a value.
  • read() waits for the saves already queued for that file, so a store made right after a save reads it back. Each path has one writer.
  • Quit (applicationWillTerminate and the forced-quit path) waits for queued saves under the reviewed allowBlocking(.terminate) allowance.

The unsent-drafts store and the scroll-place store use it. No new timer or observer: a queue runs only when a save is queued, and quit drains it.

Performance

metric before after delta harness or run link
main-thread time per draft save, under agent disk load, p95 1.0 s one enqueue (no disk on main) -100% of the disk wait scripts/measure/qos-load-bench.sh (#27587), atomic_write_us
main-thread time per draft save, idle, p50 0.8 ms one enqueue -100% of the disk wait same
saves that block the main thread longer than a frame, under load 44% (132 / 300) 0 -100% same

The "after" column holds by construction (the main thread never touches the file). App-level input-to-frame numbers for typing and scrolling under load are not here: they need the GUI bench on a capture mini (the every-action sweep in scripts/measure/blocking-probe.sh under synthetic load), which is the next step.

Testing

  • AgentPaneStateFileTests: a save runs off the main thread and a read sees it, a burst ends with the last value, nil removes the file, flushAll waits, and unsent drafts and scroll places survive a fresh store.
  • UnsentDraftTests and ChatScrollRestoreDefaultPathTests (existing) still read back right after a write.
  • scripts/cmux-next/check-concurrency.sh: ok.

No UI changes, so there are no screenshots.

No capture: nothing on screen changes; two small state-file saves move from the main thread to a background queue.

Agent session: agent session

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Moves the agent panes' state file saves off the main thread so typing and scrolling no longer stall on disk I/O under agent load.

  • AgentUnsentDrafts and AgentPaneScrollAnchors now save through a new AgentPaneStateFile, which encodes and atomically writes on one serial utility queue per file, keeps call order, and skips saves superseded before they run. A throwing encode leaves the file as the last save that ran left it; nil removes it.
  • Reads wait for the file's queued saves; only a read from inside that file's own save reads the disk directly, and each file queue has its own on-queue key so a save can't read another file's saves out of order. App quit flushes pending saves under the terminate allowance, bounded by a shared deadline that reports whether all drained.
  • Under agent disk load the old main-thread save took 1.0 s at p95 and 44% of saves ran over a frame; the main thread now only enqueues the save.

Written for commit 828c4f0. Summary will update on new commits.

View guided diff Turn on auto-fix

The composer saves a new chat's unsent draft every settled edit (300 ms
while typing) and the transcript saves its scroll place every settled
scroll, both as an atomic write on the main thread. Under agent disk load
that write took 1.0 s at p95 on a fleet mini (scripts/measure/
qos-load-bench.sh). AgentPaneStateFile writes them on one utility queue per
file, latest first, encoding included; a read waits for queued writes, and
quit flushes them under the terminate allowance.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by an agent session · session

@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Oct 30, 2026
…e failure

Review fixes: read() on the file queue reads directly instead of waiting on
itself; flushAll waits once for every queue against one shared deadline and
reports whether it drained; a throwing encode keeps the last good file while
nil still removes it; tests hold the queue so saves really overlap, and check
the generation skip and the flush deadline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@blacksmith-sh

blacksmith-sh Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Found 9 test failures on Blacksmith runners:

Failures

Test View Logs
--from must install a complete prebuilt tree/
--from must install a complete prebuilt tree
View Logs
/tmp/tmp0qb6_o5h/summary-fail.json (live):/summary-fail.json (live): View Logs
/tmp/tmpfckji1pb/summary-missing.json (live):/summary-missing.json (live): View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows
View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt
epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)/
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt
epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)
View Logs
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)/
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

The throwing-encode test queued its good save and the throwing one together,
so the newer one replaced the good save before it ran. Read between them, and
say in the docs that a throw leaves the file as the last save that ran.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ue read

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review fix: one key value on every file queue let a save for one file read
another file directly, skipping that file's queued saves. Each file now has
its own key. A test holds file B's queue and checks that a read of B from
inside a save of A waits for B's queued save.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

red: cmux-next checks (god files, concurrency, crash safety, l10n); no finished base run on b6554ad to compare; do not merge until green; see lane-rules

… reads it

Review fix: B was released right after A queued its save, so the old shared
key could pass. The test now waits until A has reached its read of B and
checks that the read does not finish while B is held.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…under it

Review fix: the hold released itself after 10 s, the same time the test gave
A to reach its read, so a starved runner could let the old shared key pass.
The hold now lasts 60 s, and the test asserts that A reached its read within 30 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (Codex gpt-6-astra high, fast, review router): SHIP at be44791

The review took four rounds:

  • 712f0b3: BLOCK. flushAll had no bound. read() could deadlock on the file's own queue. An encode failure deleted the file. The tests never forced overlap.
  • baba105: BLOCK. One shared on-queue key let a save of one file read another file directly. Two tests could race.
  • 6f4c7ac: BLOCK. The test's queue hold could expire before the check that needed it.
  • be44791: SHIP, no findings.

The fixes:

  • flushAll waits on every file queue against one shared deadline and reports whether it drained.
  • Each file queue has its own DispatchSpecificKey, so a read never waits on its own queue.
  • A throwing encode keeps the file as the last completed save left it, and nil still removes it.
  • The tests hold the queue so saves really overlap.
  • The crash window (the queue backlog) is written down in the type's docs.

828c4f0 is a clean merge of the current base, which now includes the JoinConsent compile fix #27758. Nothing was resolved by hand.

Fleet evidence:

  • 828c4f0: cmux-ci compile-check --app passed. AgentPaneStateFileTests passed 9/9; UnsentDraftTests and ChatScrollRestoreDefaultPathTests also passed.
  • Red/green: with the old shared static key (scratch commit 0f4c4a9), aReadOfAnotherFileFromInsideASaveWaitsForThatFilesSaves fails both of its expectations. At the head it passes.

@teamleaderleo
teamleaderleo marked this pull request as ready for review October 30, 2026 05:12
@teamleaderleo
teamleaderleo merged commit d02ea17 into feat-cmux-next Oct 30, 2026
168 of 180 checks passed
@teamleaderleo
teamleaderleo deleted the leo/next-main-thread-io branch October 30, 2026 05:13
@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

lawrencecchen added a commit that referenced this pull request Oct 30, 2026
The slice head 2b2fc8e (= 43037147a0b's tree: slice + trunk sync c09bb74) merges the feat tip, 130 new
commits. 58 files conflicted:
- 0 reuse the previous round's resolutions: this round's base is the previous feat tip a14725a, so no hunk had
  all three sides equal to 5f39f9e's inputs.
- Regenerated, not hand-merged: the 4 CmuxMessagesLab patches (Fixture, Layout, RowDrawing, WindowView) from the
  merged vendored sources against the pinned upstream reconstructed from the side whose patch was consistent (feat's
  for 3, perf's for WindowView; feat 65de604 edited WindowView.swift without rewriting its patch).
- 54 resolved by porting feat's change into perf's shape:
  - Package.swift (CmuxNext: Floating tests get CmuxNextDesign with perf's Wakeups deps; CmuxMobileKit: Model gets
    feat's Resources with perf's CmuxSafeConcurrency).
  - Lineage parents #24136: SidebarSessionTreeFeed (recordedParents and per-host links on perf's ServiceObservation),
    RemoteAcpmuxWire (host/linksAlias notes posted on perf's mainWork; test-only inits take them too),
    AgentTabRemoteHost, AgentTabs+Wiring, AgentSessionWorkspace (the record is one completion task on main), tests.
  - Artifact sandbox #26216: perf's page-host PageArtifactFrame already serves the sandboxed endpoint; feat's
    navigation tests ported to PageNavigation, frame-ancestors expectation added, artifactFrameURL keeps only the
    page host, webviews tests on perf's URLs.
  - pane.commandAvailability (#25405) and chief.face (#27914) beside perf's keyToPaint and Wiring init.
  - State files off main #27749: AgentUnsentDrafts (perf's ObservedMap) and ScrollAnchors (blockingDecode) on
    AgentPaneStateFile.
  - AgentPaneTransport.close: perf's close reason plus feat's host-request failure and trust reset.
  - CloudService: perf's file, plus #27104 signedOutCreation (scope, policy, stop, signOut in +Machines) and #26740
    autosize/sizePick; listIsCurrent moved to +AppServer as feat did.
  - debug.object_menu into perf's DebugDriverControl; Home fixture presence/faces beside perf's field.
  - Palette overlay window #26951 into perf's paletteBuilt.
  - Per-window Home #27880 in perf's dockTab helper; active team (1bb1b96) on perf's cloudAuth.
  - Busy refusal (a930474, 2a9b813) in perf's LineTransport split (+State, +Route, framed write).
  - Launch snapshot sidebar doc (dc38061): DaemonLaunchSnapshot.sidebarDoc, shown once settled
    (whenSettled), its connect follow a ServiceObservation; the Rust test moved to launch_snapshot_tests.rs.
  - Hover card corners (bc66475) with perf's interactive flag; Chief pet state (#25671) on perf's
    ServiceObservation and AppearanceSignals; Chief picture crop (#27933) on perf's lanes and optional store;
    Home presence (#25578) as a FrameObservation; sidebar tab look (#27530) via SidebarLookFollower; settleOpenings
    (#24374) with perf's snap; Omnibar imports.
  - acpmux: inbox digest senders without slicing; withdraw_queued async with from_agents_only; cmux-tui-core and
    platform: connection leases beside perf's modules and InlineSocket (bounded_outbound.rs takes back the
    `platform::transport` import perf had dropped, which feat's claim_watched_connection names).
  - webviews: App/NewTab/NewTabScreen (retry, slash commands, at), settings keys moved to cardRows.ts.
- Feat's tests that call perf-changed inits fixed: TabConcentricCornerTests (input:), HomeChatPresenceFacesTests
  (deadlines:, frame delivery), RemoteChildSessionsTests (main run loop for posted notes).

Generators: cargo metadata in cmux-tui, cmux-tui/fuzz, Native/DiffSidecar and optchat-chief changed no lock;
gen-features.py --check, gen-chromium-host-wire.py --check and --check-ops, sync-resource-api.py --check pass.
webviews: bun run check passes. schemas/settings/settings-schema.json stays perf's (generated; the autofix
regenerates it): until then feat's chiefLook.test.tsx misses the home.bubbleColors rows.
teamleaderleo added a commit that referenced this pull request Oct 31, 2026
Merge origin/feat-cmux-next (05730c3) into the carry of #27963.
NewTabScreen's conflict keeps both the PR's takeDraft prop and base's
startNotice.

Fixes the blocking review findings at 168707b:

- AgentUnsentDrafts lost the PR's own changes (draft(for:), typed(_:for:),
  merged(typed:kept:), the undelivered set and the unreadable-file set
  aside) when an earlier base merge took #27749's off-main-thread store.
  They are back on top of AgentPaneStateFile: the set aside runs on the
  file's queue right before the first save, and a failed move throws so
  the save leaves the file as it is. UnsentDraftTests compiles again; its
  set-aside test now reads through a fresh store (which waits for queued
  saves) before it lists the directory.
- A New Tab page with typed text replaced by the chat view before the
  handshake (a host that failed early) dropped the text: onLeaveDraft
  returned unless a Sandboxed run took the view. Before the handshake,
  with no submit queued for it, the draft now goes to the chat view's
  composer. The classic NewTabPage gets the same onLeaveDraft/takeDraft
  handoff (Agent kind only: a terminal command or an address has no chat
  composer to go to). The stand-in flag now stays set until the page is
  back after a handshake, so the page that comes back takes the draft
  (before, takeDraft only ran while the host was still failing).

Test: bun test test/new-tab-draft-host-failure.test.tsx: 0 pass / 2 fail
with onLeaveDraft's old early return and no classic handoff; 2 pass on
this head.

Agent-Session: aid-3ef145a002811735c5ae
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
austinywang added a commit that referenced this pull request Nov 3, 2026
feat-cmux-next landed the same goal as this PR's Swift change:
#27749 saves the agent panes'
drafts and scroll places off the main thread (one serial queue per file,
latest wins), and #28275 gives
each file one owner and has both quit paths await the saves under the
quit deadline, holding no thread.

The conflicts (12 files) resolve to the base: StateFileWriter,
AgentPanePersistenceQuit, their tests and the App wiring are dropped,
with the app_lines +8 allowance they needed. That also removes the two
blocking findings of the review at 6aaafd0 (a seal that refused the
quit's own late draft, and an unbounded drain before CefShutdown): the
base never seals and bounds its wait.

Kept: guardrails.py --sites (the whole-tree count) and its test. It now
diffs against git's built-in empty tree instead of writing an empty tree
and commit object (review note 5).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was previously deployed

1 inactive deployment
ci-routing-agent — 828c4f0c Deployed Oct 30, 2026 by teamleaderleo via namespace autofix #7174
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant