Repository navigation
fix(cmux-tui, cmux NEXT): the daemon answers a sidebar move promptly; a timeout toast says why in words - #34628
Merged
Conversation
…daemon answers a sidebar move; a timeout toast says why in words Leo: "idk what happened here", on "Move did not apply: set-personal-workspace: timed out: workspace.place (no reply within 2.0 seconds)". The roster catch-up in journal maintenance read the non-output tail with a range on the (kind, sequence) index's first column, which cannot seek on sequence: each page walked every non-output entry the active journal kept (5.6M on the nightly registry), so the agent_roster step took 5-8 s every tick, and requests behind it (workspace.place, put-frontend-projection, history.trail.record, new-conversation-tab) missed their 2 s deadline. The read now seeks once per kind (a skip scan over the same index) inside a bounded sequence window: the same page in 5 ms instead of 330 ms warm. A command that misses its deadline now shows "cmux's background service was busy and did not answer in time. Try again." (21 languages) in place of the command names; callers keep the raw error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-3bab30503f0b9a7059e5
…stead of 'Try again' (review follow-up) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-3bab30503f0b9a7059e5
teamleaderleo
added
the
leo-named-fix
Bugbot needs on-demand usage enabledBugbot uses usage-based billing for this team and requires on-demand usage to be enabled. A team admin can enable on-demand usage in the Cursor dashboard. |
teamleaderleo
temporarily deployed
to
ci-routing-agent
November 4, 2026 02:37 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
Collaborator
Author
|
Review (subagent claude-opus-5-5 medium): SHIP at 561fe9c Checked: Follow-ups: [follow-up] the timeout copy said "Try again" while the change may still apply. Fixed at 3f34094, a copy-only change. [follow-up] the history feed saves on every page across long runs of output. [follow-up] the distinct-kinds read could be cached per catch-up loop. |
Collaborator
Author
|
Owned by an agent session · session |
Contributor
|
cmux status (one comment, edited in place; each line is updated by the job that owns it) Review App of this head (app.zip, kept 1 day)
Not verified: macOS jobs did not run: compile and app tests were skipped on CI passes on Written by Shipped in:
|
teamleaderleo
temporarily deployed
to
ci-routing-agent
November 4, 2026 02:41 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
teamleaderleo
temporarily deployed
to
sentry-dsyms
November 4, 2026 03:02 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
teamleaderleo
temporarily deployed
to
artifacts
November 4, 2026 03:02 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
teamleaderleo
added a commit
that referenced
this pull request
Nov 4, 2026
Conflict in ActionWorkFailure+Daemon.swift with #34628, which gives a missed deadline a reason in words. Both kept: the reason stays on the failure, and notice(title:) shows nothing for a missed deadline (Leo: no chatty unactionable toasts). A close that may still happen (#30915) keeps its notice. Sidebar-Look: no visual change Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-cb01117327fc358e9ba1
teamleaderleo
temporarily deployed
to
artifacts
November 4, 2026 03:05 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
This was referenced Nov 4, 2026
Merged
teamleaderleo
added a commit
that referenced
this pull request
Nov 4, 2026
Leo: "app's calls to the daemon were timing out at 2 s. shouldn't be this short. no reason for this." cmux-tui's control-plane default (DaemonConnection.defaultRequestTimeout) goes from 2 s to 30 s, the heartbeat's not-answering window (15 s x 2): a busy daemon answers late (#34628's 5-8 s journal step failed requests it still answered), and a daemon that misses that long is the heartbeat's not-answering case with its Restart offer. Snapshots (10 s) and the loopback forward client follow it. The app's acpmux client reads and writes use one AcpmuxEnvironment.answerDeadline (30 s) in place of their 2 s defaults. Quit and shutdown bounds keep their 2 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-47ff611049541adb5070
teamleaderleo
added a commit
that referenced
this pull request
Nov 4, 2026
…i and acpmux clients) (#35075) * test(daemon): a daemon that answers in 3 s is answered, not timed out (red on base) Leo: "shouldn't be this short. no reason for this." on the app's calls to the daemon timing out at 2 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-47ff611049541adb5070 * fix(cmux NEXT): the app waits 30 s for a daemon's answer, not 2 s Leo: "app's calls to the daemon were timing out at 2 s. shouldn't be this short. no reason for this." cmux-tui's control-plane default (DaemonConnection.defaultRequestTimeout) goes from 2 s to 30 s, the heartbeat's not-answering window (15 s x 2): a busy daemon answers late (#34628's 5-8 s journal step failed requests it still answered), and a daemon that misses that long is the heartbeat's not-answering case with its Restart offer. Snapshots (10 s) and the loopback forward client follow it. The app's acpmux client reads and writes use one AcpmuxEnvironment.answerDeadline (30 s) in place of their 2 s defaults. Quit and shutdown bounds keep their 2 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-47ff611049541adb5070 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 4, 2026
Merged
lawrencecchen
added a commit
that referenced
this pull request
Nov 4, 2026
…e (cmux-ad m11 2) (#35329) * bench: the daemon's request latency through a journal step, and head-of-line (cmux-ad m11 2) Two scenarios for cmux-tui/scripts/bench-daemon-latency.py (what scripts/measure/perf-10x-daemon.sh runs), not in the default list. The set DOGFOOD BENCH runs per build: cmux-ci run --class exclusive --measurement \ --script scripts/measure/perf-10x-daemon.sh --ref SHA \ --arg=scenarios=small,journal-step,head-of-line --timeout 3600 - journal-step: 80 agents' real hook helpers fill the journal while an app connection (put-frontend-projection, list-workspaces, identify) and a churn connection (create and close terminals) run through --journal-ticks maintenance ticks (30 s apart in a release build). It reports each kind's p50/p95/p99/max, the requests over 30 ms, and the daemon's slow-maintenance lines. This is #34628's 5-8 s stall, for which #35075 raised the app's deadline to 30 s. - head-of-line: the same load, with an identify that shares one connection with the churn, against the same identify on its own connection. These are the data for per-request dispatch, an open question for PL-daemon. * bench: start the daemon as the app does, with --terminal-reap-grace-seconds 30 Without the flag a close leaves the terminal running unplaced (reaping is opt-in, main.rs), so journal-step's ~350 creates and closes and then head-of-line's terminals ran the Mac out of PTYs (511 on macOS): "PTY capacity exhausted" on aws-m4pro-9 (727887e6) and on aws-m4pro-8 (795b1221) at the same point, with ~550 terminal hosts still alive at the end of the step. The app's DaemonLauncher passes 30 s; at most 16 closed terminals then wait it out. --reap-grace-seconds sets it; a negative value leaves the flag out. * bench: each churn scenario ends its agents; a refused connection or a dead worker fails the run On cmux17-perf (fleet 6aca4d0c, b34023f1) head-of-line recorded nothing on its shared connection: journal-step's 80 agents were still running, so 160 agents' hook helpers and CLI calls took every connection the daemon gives terminal processes, and the daemon refused the bench's own connections at its limit (MAX_SERVER_CONNECTIONS). The refusal line has no id, so Conn.call read on to the close; the worker threads died, and the run still exited 0 with n=0. - journal-step and head-of-line start their agents with a stop file and end them (the stop file, then their surfaces closed) before the next scenario. - Conn.call says "refused: <code>: <message>" for the daemon's refusal line. - A head-of-line worker that fails stops the scenario, and an error in either scenario fails the run after its numbers are printed. Nothing is retried. * bench: head-of-line measures the daemon's queue, not the bench's lock Review FIX at 4acfccb: head-of-line held its own lock across each shared call, so the shared identify went out only after the create's reply; its ~100 ms p95 was the bench's wait. A plain identify never waits in the daemon's queue (connection_frame.rs answers it on the reader), and a create leaves the connection's dispatch thread for the terminal pool. - head-of-line keeps several requests in flight per connection (PipelinedConn): a request goes out at once, one reader thread per connection matches each reply by id, and each request is timed from its send to its reply. One connection sends close-surface (it runs on the connection's dispatch thread) and right behind it list-workspaces (queued behind the close) and identify (answered on the reader: the control); a second connection sends the same two at the same moment. Every reply goes through must(). - journal-step: a worker whose connection fails to open, or that gets an error reply, stops the scenario and fails the run; a failed request is no longer counted as a sample. create-terminal is reported apart, with create-terminal-idle (20 creates before the agents start), and kept out of "all" and over-30ms. - The window holds --journal-ticks whole ticks: 30 s per tick, plus --journal-tick-seconds (default 10) for each tick and for the one running when the window opens. - The agents scenario ends its agents with a stop file, as journal-step does. - The docstring says which requests skip the queue, that the reap grace changes the default scenarios' new-tab numbers, and that a tick under 500 ms logs no line.
cmux-bot Bot
pushed a commit
that referenced
this pull request
Nov 4, 2026
…x-ad m11 2) (#35334) * cmux-tui: requests wait for no roster catch-up, test (red, cmux-ad m11 2) A journal maintenance tick's roster catch-up holds the roster fold lock for its whole journal tail read and snapshot write (agent_roster_fold.rs advance_agent_roster), and request paths take that lock: a terminal close (lifecycle.rs purge_terminal_side_tables), every agent journal ingress (journal.rs finish_journal_ingress -> fold_agent_roster) and a direct agent report's echo (agent_reports.rs). Requests on one connection run in order on its one dispatch thread, so one of them waiting there held all later requests of the app: the 5-8 s step of #34628, and #35075's 2 s -> 30 s app deadline. The test pauses a catch-up mid-tail (the existing page seam) and asserts that a workspace close and an agent report finish within 500 ms. Red here: both wait for the catch-up. (cherry picked from commit c45ef0b) * cmux-tui: the fold-hold test reports a working agent (AgentState has no Running) (red) c45ef0b did not compile (fleet 77b55788: E0599, no variant Running); the agent states are working, blocked, idle, done and unknown. (cherry picked from commit aab2df2) * cmux-tui: no request waits for a roster fold another thread runs (cmux-ad m11 2) The test is the commit before this one. Request paths (a terminal close, every agent journal ingress, a direct agent report's echo) took the roster fold lock, which a maintenance tick's catch-up held for its whole journal tail read and snapshot write: 5-8 s on #34628's nightly registry, and one request waiting there held every later request of its connection (one dispatch thread per connection), which is why workspace.place, new-conversation-tab, window_record.put and others timed out together and #35075 raised the app's deadline to 30 s. Now a request records what it needs (fold through its commit, or retire a closed terminal) behind a short lock of its own (`Mux::agent_roster_requests`) and `try_lock`s the fold: when another thread holds it, the request returns at once, and the holder does the work before it lets go (it reads the requests again after each page and after it lets go, so none is left behind). The catch-up keeps the only blocking wait, on the maintenance thread. A retirement now waits until the roster has folded the journal head, so the closed terminal's own earlier events cannot bring its entry back. Uncontended, a request folds exactly as before. (cherry picked from commit e9ba248) (cherry picked from commit ec3b321) * cmux-tui: the roster's request queue lives in its host, so mux.rs does not grow mux.rs is over its 1000-line budget on the base (1114 lines); the first cut added a Mux field. The queue now sits in AgentRosterHost, whose lock is the short in-memory lock every fold already takes per page. (cherry picked from commit 830d8b7) (cherry picked from commit b4364bb) * cmux-tui: the fold-hold test drives the request paths that take the fold Red2 (fleet 99500a98, hq run 41813886487, at aab2df2) passed on the old code: a workspace close does not purge its terminals on the request thread (tree_close.rs ends with purge_surface_side_tables only), and a direct report's journal echo may not reach the fold. The test now uses the two paths that do take the fold lock: an agent hook through append_journal_ingress (session.journal.append) and the terminal purge every terminal close runs (purge_terminal_side_tables). It also asserts what they asked for is done once the catch-up lets go: the roster at the journal head, the closed terminal's entry gone, the hook folded. (cherry picked from commit 9a8f124) * cmux-tui: the fold-hold test uses the imported HashMap (unused_qualifications) (cherry picked from commit c68a3e4) * threading-model: per-request dispatch on a daemon connection, an open question for PL-daemon (cmux-ad m11 2) Lead condition 2: one slow request on a connection's single dispatch thread holds every later request of that connection. Not changed here; the bench's head-of-line scenario (scripts/measure/perf-10x-daemon.sh --arg=scenarios=head-of-line, its own scripts-only PR) gives the data. * test(cmux-tui-core): roster work a request records is never left behind, and no close or hook reads the journal or saves the roster on its own thread (#35334 review F1, F2; red) Regression tests for the review of #35334 (they fail on the code before the fix): - F1: a hook recorded while another thread runs a roster pass that makes no progress (a retirement of a terminal with no entry, the roster at the head) is left behind: that pass returns without reading the requests again. The reviewer's repro (f65ccd3), without reading internal request state. - F2 (perf-lead.md D1): a terminal close or an agent hook that finds the fold free reads the journal tail and saves the snapshot on its own request thread. One test holds whichever thread reads the first page and puts a 500 ms budget on the close and the hook; one records the thread of every roster journal read and snapshot save and asks that all are the roster's owner thread (cmux-agent-roster). - P3: the catch-up test asks for Some("working"), not is_some(). Test seams (cfg(test)): a pause after the head read (from the review), the threads of the roster's journal reads and saves, and a settle wait (nothing to wait for here: every request does or hands off its work before it returns). * fix(cmux-tui-core): one roster owner thread does every fold and snapshot save; a request only records its work (#35334 review F1, F2) The review of #35334 found two defects in its hand-off: a request recorded while another thread ran a pass that made no progress was left behind (F1), and a terminal close or a hook that found the fold free read the journal and saved the snapshot on its own request thread (F2, perf-lead.md D1). The root fix the review recommended: one background owner. - mux/agent_roster_worker.rs: the `cmux-agent-roster` thread owns the fold. A request (hook ingress, terminal close, direct report echo) records what it needs under the owner's own lock (the highest sequence asked, the closed terminal) in O(1) and wakes it; it never folds, reads the journal or saves. The owner takes every recorded request before each pass and looks again after it, so nothing is left behind (F1); no request thread reads or writes for the roster (F2). It sleeps on a condvar (no polling) and holds only a weak mux reference between passes. - The maintenance tick's catch-up asks the owner for a pass through the journal head and waits for it, so the tick's checkpoint and seal still come after the roster caught up. Shutdown lets the owner finish the work recorded before it (retirements are saved) and waits at most 2 s. - mux/agent_roster_fold.rs `run_roster_pass`: a retirement runs only after a head read in the same pass and a fold that reached it; when the head or a page cannot be read, it waits for the next pass (P3 F5). A pass that moved nothing saves nothing and serializes nothing (P3 F3). A panicking pass does not end the owner. - The live agent count of the activity stream is read again after a pass that changed the roster (the request that changed it answered first). - The roster host goes back to the base shape (no request queue in it). - Default QoS, as the journal writer, maintenance and history feed threads (threading-model.md): its passes hold the roster lock, the writer's connection lock and (plugin deltas) the registry lock that requests take. Tests: the three regression tests of the previous commit go green; tests that read the roster right after a write now wait for the owner first (`settled`), since the owner folds after the request answered. Red (the previous commit on the code before this fix): fleet step 1dc37bc961cd75fa606dd127 (aws-m4pro-4), 3 of 5 red, also alone. * test(cmux-tui): the sidebar projection tests wait until the roster lists a reported agent The agent roster's owner thread folds a report right after the report answered (#35334), and the sidebar reads the roster, so a test that reports an agent and then renders or navigates its row waits for the roster first. These tests build against cmux-tui-core without its test seams, so they wait on the public list_agents read. * test(cmux-tui-core): the roster regression tests check the hook's own sequence, not the journal head A hook commits its record and then its projection's resource change, so the journal head is one record past what the hook asks the roster to fold. With one owner, nothing asks for that later record until the next request or the maintenance catch-up, so the F1 and F2 tests now check that the roster folded through the hook's own sequence and that the saved snapshot is at the roster's cursor. Their first failing check on the code before the fix is unchanged. * threading-model: the head-of-line probe of the repaired bench (#35329) and its numbers The probe is a list-workspaces sent right after a close-surface on the same connection (pipelined), against the same request on a connection of its own, with identify as the control (cmux18-perf, fleet ce07a34858265193980f9cdf). * test(cmux-tui): the roster wait helper names SurfaceId, Instant and Duration as the module imports them clippy -D warnings (unused_qualifications) refused the fully qualified paths. * threading-model: the roster owner stays at the default priority because a throttled holder of locks that requests take is a priority inversion
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Leo, on the toast "Move did not apply: set-personal-workspace: timed out: workspace.place (no reply within 2.0 seconds)": "idk what happened here".
Cause
The app's log at the failed move shows every request to the home cmux-tui daemon timing out together (window_record.put, workspace.place, close-workspace, new-conversation-tab, put-frontend-projection, create-terminal). In the same minutes, the daemon logged
slow journal maintenance: 5-8 s (slowest agent_roster)on every tick, about once a minute. So the daemon was blocked; the move itself was not slow.The roster catch-up reads the journal tail except
terminal.outputwithkind < 'terminal.output' AND sequence > ?on the(kind, sequence)index. A range on the first column cannot seek on the second, so each 512-record page walked every non-output entry the active journal held: 5.6M on this nightly registry. The fold lock and the I/O sit in front of agent hook appends and other requests, which then missed the 2 s deadline.Fix
query_session_journal_without_output_afterseeks once per kind: a recursive skip scan lists the distinct kinds from the same index (61 here), then joins a(kind = ?, sequence in window)seek for each. A bounded sequence window (8,192) caps how many matches one page sorts. A short page reports the window's end as covered. Every caller already loops until it reaches the head (roster fold, roster restore, history feed).message. This covers every action toast, not just Move.Performance
Risk: SQL read path of the roster and history folds. The new test pages through 16k records with a run of output longer than one window and checks that each non-output record comes back exactly once, in order.
Tests:
non_output_read_pages_every_other_kind_across_seek_windows(Rust) andDaemonTimeoutToastCopyTests(Swift).Follow-up (not in this PR): the nightly registry is 58 GB with a 15 GB WAL. That growth is its own issue.
Agent session: agent session
🤖 Generated with Claude Code
Open in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need.Summary by cubic
Fixes the daemon answering slowly by making the roster catch-up's journal read seek once per kind, and replaces raw command names in timeout toasts with a human-readable reason.
Bug Fixes
(kind, sequence)index's first column, so each page walked every non-output entry the active journal kept (5.6M), blocking the daemon for 5–8 s per maintenance tick and making requests miss their 2 s deadline. It now lists distinct kinds via a skip scan and seeks per kind inside a bounded sequence window (8,192), cutting a 512-record page from 330 ms to 5 ms on a warm nightly registry.set-personal-workspace: timed out: workspace.place. Callers keep the raw error inmessage.Written for commit 3f34094. Summary will update on new commits.