Repository navigation
cmux-tui: agents in terminals hold at most 48 of the daemon's 64 connections - #27719
Conversation
…ed, #27698) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… connections (#27698) A process inside one of the session's terminals (an agent, its hooks, cmux run in a pane) now also takes a terminal slot when it connects to the local socket; once terminals hold 48, the next one reads the same server.busy line as before. The other 16 connections stay for the app and its other clients, so a burst of agents can no longer lock the app out of its own daemon (a nightly-next session refused 55,162 connections in 2.5 hours, and the app could not save window state or close workspaces). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
cmux status (one comment, edited in place; each line is updated by the job that owns it) Review App of this head (app.zip, kept 1 day)
|
…oop (#27698 review) The agent share was decided on the connection thread through the session state lock: while the app held that lock (closing workspaces), connecting agents each held a general slot waiting on it, and could fill the app's reserve. The accept loop now reads the peer itself, without a daemon lock and before a slot is taken: it runs in a cmux terminal when its session's leader is a child of a `cmux-tui __terminal-host` (one peer, its session leader and that leader's parent; nothing enumerated). A terminal refusal now moves the connection from accepted to refused instead of counting it in both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…never waiting on state (#27698 review) Review of the process-read version: on macOS a terminal's shell starts through login(1), which runs as root, so its parent could not be read and no agent there was capped; adopted and held terminals were missed too, and each accept read a foreign process's argv. Now the accept loop matches the peer's process session (getsid) against the session's own terminals, as the request-origin rules do, read with try_lock: exact while the state lock is free, and from the last read while another thread holds it, so the accept loop never waits. The cmux-tui-platform changes are reverted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…own terminals skip the read (#27698 review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Correctness review (subagent, four rounds) and resolutionRound 1, at 2c28545: FIX
Round 2, at 7f3f3e7: FIX
Round 3, at ad92fb1: FIX
Round 4, at 344942f: SHIP Primary agent's own check at the same head: I read the full diff. Each permit is released exactly once, on every path. A refused terminal connection drops its general slot before the busy line is written. Remote-entry and WebSocket admission are unchanged. Parked connections during restore go through Open, by scope (not covered here):
|
|
Found 2 test failures on Blacksmith runners: Failures
|
|
CI note for the hosted focused runs at 344942f (runs 40979418663 and 40979417257):
|
|
The principled replacement for this split is up: #28343. It gives the app its own daemon socket ( PL-daemon (owner of |
The base landed 21918fb (a hung-up local client gives its connection slot back, ConnectionLease). Keep both: a terminal connection's share is now a flag on its ConnectionLease, claimed before the lease is made (claim_watched_terminal_connection), so a terminal client that hangs up gives back its terminal slot with its main one, and a terminal claim at its share first reclaims hung-up clients, as try_claim does. Adds a test that a hung-up terminal client frees its terminal slot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Sweep fix: merge conflict with |
Conflicts: diagnostics.rs keeps feat's ConnectionStats.socket next to the PR's terminal_active; signaled_mutex.rs keeps feat's #[track_caller] try_lock with telemetry and the PR's removal of #[cfg(test)] (the accept loop uses it).
Bugbot needs on-demand usage enabledBugbot uses usage-based billing for this team and requires on-demand usage to be enabled. A team admin can enable on-demand usage in the Cursor dashboard. |
…26-11-01 05:20Z) The app socket, the RLIMIT_NOFILE budget and the removal of the 2 s write stall leave this PR: #27719 (fbb9e39) already keeps 16 of the 64 connections for the app by capping terminal clients at 48, and the lead chose to build on that cap. The tree is feat-cmux-next again here; the next commits add only the per-terminal fair share inside the terminal lane.
…thers (red) Builds on #27719's 48-of-64 terminal cap (lead decision 2026-11-01 05:20Z). The accept loop already knows which terminal a peer runs in (its process session); this commit carries that id to the terminal claim and the lease, with no behavior change, and adds the test. Red here: terminal 101 takes all 48 terminal connections and terminal 102 finds none.
…nal lane On top of #27719's 48-of-64 terminal cap (lead decision 2026-11-01 05:20Z): a terminal's first connection gets in whenever the lane has room; a further one only while that terminal then holds fewer than the lane has left free. One terminal opening connections in a loop stops at 24 of the 48, and every other terminal still finds room. The share and the count change together under one lock; a hung-up client's reclaim gives back its terminal's share too. A connection whose terminal is unknown keeps the lane bound alone.
…lane (on top of #27719's 48-of-64 cap) (#28343) * test(cmux-tui): the app keeps its daemon connections under terminal load (red, #27698 B1) Austin's nightly-next (2026-10-29): about 80 terminal agents held all 64 daemon connections, 55,162 were refused, the app's among them. Red tests, with behavior-neutral seams (ConnectionBudget at the old 64, ConnectionLane/ConnectionSlot, app_lane_socket_path) so they compile on the base: - an_app_connection_is_served_while_the_session_socket_is_full - one_terminal_opening_many_connections_cannot_starve_another - the_connection_budget_follows_the_open_file_limit - a_client_that_stops_reading_for_a_while_keeps_its_connection - AppLaneConnectionTests (Swift): the app connects on <socket>.app * fix(cmux-tui): the app has its own daemon socket; per-client fair shares; budget from RLIMIT_NOFILE; no 2 s write kill (#27698 B1) Austin's nightly-next (2026-10-29): about 80 terminal agents held all 64 daemon connections; 55,162 were refused in 2.5 h, the app's among them. The Austin's-day A/B on #28159 showed 64/64 with ~3,700-4,250 refusals per 6-min run. - The app's own socket, `<session socket>.app` (app_lane_socket_path), bound under the same start lock. Each socket has its own budget, so terminals, agents, hooks and the CLI (session socket) can never take the app's connections. The app (LineTransport) connects there first and falls back to the session socket (older daemon, stale file, long path). Chromium: the browser's own IPC channels never compete with renderer connections (render_process_host_impl.cc gives each renderer its own). - A fair share per client on the session socket: the client is the peer's process session (getsid), so a terminal's shell, agent, hooks and cmux calls count as one. A client may take a connection only while it then holds fewer than the socket has left free: one terminal stops at half the lane, and every new client still finds room. Remote entry and WebSocket clients keep the lane budget only (no identity at accept). - The budget comes from the soft RLIMIT_NOFILE the owner already raises at start (owner_start, raise_open_file_limit: hard limit, at most kern.maxfilesperproc / OPEN_MAX): half of the descriptors left after start, 8 per slot (4 descriptors per connection, 2 connections per slot with the hung-up pool), half per socket; on macOS also bounded by kern.num_taskthreads (2 threads per connection). 64 and MAX_SERVER_CONNECTIONS are gone; the parked-connection bound follows it. - No write deadline: STREAM_WRITE_TIMEOUT (2 s) closed a connection whose client read late (a stalled app main thread lost its connection and resynced). Now a client is ended only when its replies overflow the bounded control queue (QueuedSink evicts and shuts the socket); the WebSocket ping never waits on a peer that does not read, and the ping clock shuts a dead peer's socket. Replaces the stopgap split of #27719 (48/64 for terminals). * test(cmux-tui): the slow-reader test declares the stall as a signal, not a 3 s sleep (R5, #27698 B1) StalledSocket models the kernel in memory: a daemon write waits while the client's buffer is full, and one that waits when the test declares the stall fails as SO_SNDTIMEO does if a write deadline is set (red on the base: 2 s STREAM_WRITE_TIMEOUT), else keeps waiting (green). The app-lane test reads with no deadline: the daemon answers or refuses with one line. * test(cmux-tui): read the child's budget anywhere in its line (libtest prints its own prefix first) (#27698 B1) the_connection_budget_follows_the_open_file_limit failed on both the red and the green fleet runs (16894a02, 3830e38f) at connection_lanes.rs:123: the child printed "connection-budget=30" after libtest's "test <name> ... " on the same line, and the parser only took a line that starts with it. * fix(cmux-tui): the app socket gets its own outbound budgets; clean up .app first; clippy (review of 6aaecf6, #27698 B1) Review FIX (landing/reviews/6aaecf6a9207...md): - Blocker: clippy -D warnings on changed files: SynchronizedTcpStream:: set_write_timeout is dead with no write deadline; same-type casts, unnecessary qualifications and a redundant clone in the tests. - The two sockets shared one RenderService, whose outbound byte budgets a session-socket client that stops reading now holds until its queue overflows (no 2 s write kill). The app's replies could then fail on a full pool and the app's connection close: #27698 by another path. The app socket now has its own RenderService. - cleanup removes <socket>.app before the session socket, so a daemon that binds right after the first removal keeps its fresh .app. * test(cmux-tui): the app's replies never wait for the session socket's outbound bytes (review of 6aaecf6, #27698 B1) Regression test for the app socket's own RenderService (0a0796c): spend the session socket's outbound and control budgets, as clients that stopped reading hold them now that no write deadline frees them; a session-socket identify then gets no reply (the test checks it proves something), and the app socket's identify is still answered. ServedSession::new builds the session's two budget sets in one place. Red proof: cp-scratch/pa-b1-red-budget (6aaecf6 + this test, with ServedSession::new serving both sockets from one RenderService). * test(cmux-tui): a one-connection lane still admits a client; the daemon starts without its app socket (red, #27698 B1) Lead review of 5631798, two defects for Austin's load: - every_lane_admits_a_client_at_the_smallest_budget: at the smallest budget (one connection per lane) the fair share refuses every client. - the_session_socket_serves_when_the_app_socket_cannot_be_bound: a directory nobody may remove at <socket>.app makes the whole daemon fail to start. * fix(cmux-tui): a client's first connection always gets in while the lane has room; the daemon starts without its app socket (#27698 B1) - The fair share admitted a further connection only while the client then held fewer than the lane had left free, so a lane of one (the smallest budget an fd limit derives) refused every client. A client's first connection now gets in whenever the lane has room. - A failure to bind <socket>.app (a path that can't be removed, Windows) failed the whole start. The app socket is an addition: the daemon now logs why and serves the session socket alone; the app's LineTransport already falls back to it. * revert(cmux-tui): narrow this PR to build on #27719 (lead decision 2026-11-01 05:20Z) The app socket, the RLIMIT_NOFILE budget and the removal of the 2 s write stall leave this PR: #27719 (fbb9e39) already keeps 16 of the 64 connections for the app by capping terminal clients at 48, and the lead chose to build on that cap. The tree is feat-cmux-next again here; the next commits add only the per-terminal fair share inside the terminal lane. * test(cmux-tui): one terminal cannot take the terminal lane from the others (red) Builds on #27719's 48-of-64 terminal cap (lead decision 2026-11-01 05:20Z). The accept loop already knows which terminal a peer runs in (its process session); this commit carries that id to the terminal claim and the lease, with no behavior change, and adds the test. Red here: terminal 101 takes all 48 terminal connections and terminal 102 finds none. * fix(cmux-tui): a terminal's connections get a fair share of the terminal lane On top of #27719's 48-of-64 terminal cap (lead decision 2026-11-01 05:20Z): a terminal's first connection gets in whenever the lane has room; a further one only while that terminal then holds fewer than the lane has left free. One terminal opening connections in a loop stops at 24 of the 48, and every other terminal still finds room. The share and the count change together under one lock; a hung-up client's reclaim gives back its terminal's share too. A connection whose terminal is unknown keeps the lane bound alone. * test(cmux-tui): the fair-share test proves a released terminal's share is cleared; the share is "no more than" the free lane (review 6553892057) * fix(cmux-tui): contains_peer is test-only now (clippy dead_code; the accept loop asks terminal_of_peer) clippy step b7e7a3f2c43e at 8fc1e8a: method contains_peer is never used.
![Fix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/fix-with-codesmith-light-3x.png)
Fixes #27698
Summary
The session daemon admits at most 64 connections, and every client shares them. With about 80 agents running in terminals, terminal-bound clients filled every slot. The daemon then refused the app's own connections with
server.busy: one nightly-next session refused 55,162 connections in about 2.5 hours. The app couldn't save window state or close workspaces, and the CLI failed the same way.Now a local-socket connection from a process in one of the session's terminals (an agent, its hooks,
cmuxrun in a pane) also takes a terminal slot. Terminals may hold at most 48 (MAX_TERMINAL_CONNECTIONS, 3/4 of 64). The next terminal-bound connection reads the sameserver.busyline and gives its slot back, so the other 16 stay for the app and its other local clients.How a terminal peer is found (
server/terminal_sessions.rs,TerminalSessions):getsid) against the session's own terminals: theprocess_id()ofsurfacesandterminal_catalog, the set the request-origin rules use.Mux::try_with_state(StateMutex::try_lock, now available outside tests). While another thread holds the state lock, it answers from the last read, so the accept loop never waits on it.A terminal refusal moves the connection from
acceptedtorefusedinserver-stats. Remote-entry and WebSocket connections are unchanged.Testing
terminal_connections_leave_slots_for_the_app(server/tests/outbound_and_scheduler.rs): fills the terminal share; one more terminal connection is refused and releases its slot; non-terminal connections still get the rest. It also checks the accepted and refused counts.terminal_sessions::tests: malformed keys aren't terminal-bound. With the state lock held, a hit answers from the last read and a miss doesn't wait; a free lock reads again.Hosted runs (
./scripts/verify-cmux-tui-hosted.sh --filter …):terminal_connections_leave_slots_for_the_appterminal_connections_leave_slots_for_the_appterminal_sessions::testsObserved / Not run
server-statsnumbers (limit 64, peak 64, accepted 15,144, refused 55,162).setsidinto their own session; Windows (no session id, so nothing is capped there).Review (claude-subagent): SHIP at 344942f. The four review rounds and their resolutions are in the PR comments.
Changelog
Fixed: Agents running in terminals can no longer use up every daemon connection and lock cmux NEXT out of its own session (workspaces would not close).
🤖 Generated with Claude Code