Skip to content

cmux-tui: agents in terminals hold at most 48 of the daemon's 64 connections - #27719

Merged
cmux-bot[bot] merged 10 commits into
feat-cmux-nextfrom
27698-daemon-connection-reserve
Oct 31, 2026
Merged

cmux-bot[bot] merged 10 commits into
feat-cmux-nextfrom
27698-daemon-connection-reserve

Conversation

@austinywang

@austinywang austinywang commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #27698

Summary

The session daemon admits at most 64 connections, and every client shares them. With about 80 agents running in terminals, terminal-bound clients filled every slot. The daemon then refused the app's own connections with server.busy: one nightly-next session refused 55,162 connections in about 2.5 hours. The app couldn't save window state or close workspaces, and the CLI failed the same way.

Now a local-socket connection from a process in one of the session's terminals (an agent, its hooks, cmux run in a pane) also takes a terminal slot. Terminals may hold at most 48 (MAX_TERMINAL_CONNECTIONS, 3/4 of 64). The next terminal-bound connection reads the same server.busy line and gives its slot back, so the other 16 stay for the app and its other local clients.

How a terminal peer is found (server/terminal_sessions.rs, TerminalSessions):

  • The accept loop, after claiming a slot, matches the peer's process session (getsid) against the session's own terminals: the process_id() of surfaces and terminal_catalog, the set the request-origin rules use.
  • A peer in the last set read answers at once.
  • A miss reads the terminals again, with Mux::try_with_state (StateMutex::try_lock, now available outside tests). While another thread holds the state lock, it answers from the last read, so the accept loop never waits on it.
  • No other process is read. It works for login(1)-started macOS shells, adopted terminals and held terminals.

A terminal refusal moves the connection from accepted to refused in server-stats. Remote-entry and WebSocket connections are unchanged.

Testing

  • terminal_connections_leave_slots_for_the_app (server/tests/outbound_and_scheduler.rs): fills the terminal share; one more terminal connection is refused and releases its slot; non-terminal connections still get the rest. It also checks the accepted and refused counts.
  • terminal_sessions::tests: malformed keys aren't terminal-bound. With the state lock held, a hit answers from the last read and a miss doesn't wait; a free lock reads again.

Hosted runs (./scripts/verify-cmux-tui-hosted.sh --filter …):

Commit Filter test (linux) macOS
048e26e (test only) terminal_connections_leave_slots_for_the_app failure (red), run 40977996955 queued
344942f (head) terminal_connections_leave_slots_for_the_app success, run 40979418663 or 40979417257 queued
344942f (head) terminal_sessions::tests success, run 40979418663 or 40979417257 queued

Observed / Not run

  • Observed: the nightly-next server-stats numbers (limit 64, peak 64, accepted 15,144, refused 55,162).
  • Not run:
    • macOS hosted test jobs: queued at the time of writing.
    • No live burst against a tagged build.
    • No socket-level test drives a real terminal peer through the accept loop.
  • Not covered: WebSocket and remote-entry connections; helpers that call setsid into their own session; Windows (no session id, so nothing is capped there).

Review (claude-subagent): SHIP at 344942f. The four review rounds and their resolutions are in the PR comments.

Changelog

Fixed: Agents running in terminals can no longer use up every daemon connection and lock cmux NEXT out of its own session (workspaces would not close).

🤖 Generated with Claude Code

austinywang and others added 2 commits October 29, 2026 17:08
…ed, #27698)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… connections (#27698)

A process inside one of the session's terminals (an agent, its hooks, cmux
run in a pane) now also takes a terminal slot when it connects to the local
socket; once terminals hold 48, the next one reads the same server.busy line
as before. The other 16 connections stay for the app and its other clients,
so a burst of agents can no longer lock the app out of its own daemon (a
nightly-next session refused 55,162 connections in 2.5 hours, and the app
could not save window state or close workspaces).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 7bbba7d: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr27719 -f ref=27698-daemon-connection-reserve -f platform=macos -f backend_mode=local

@github-actions github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 30, 2026
austinywang and others added 4 commits October 29, 2026 17:15
…oop (#27698 review)

The agent share was decided on the connection thread through the session
state lock: while the app held that lock (closing workspaces), connecting
agents each held a general slot waiting on it, and could fill the app's
reserve. The accept loop now reads the peer itself, without a daemon lock and
before a slot is taken: it runs in a cmux terminal when its session's leader
is a child of a `cmux-tui __terminal-host` (one peer, its session leader and
that leader's parent; nothing enumerated). A terminal refusal now moves the
connection from accepted to refused instead of counting it in both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…never waiting on state (#27698 review)

Review of the process-read version: on macOS a terminal's shell starts through
login(1), which runs as root, so its parent could not be read and no agent
there was capped; adopted and held terminals were missed too, and each accept
read a foreign process's argv. Now the accept loop matches the peer's process
session (getsid) against the session's own terminals, as the request-origin
rules do, read with try_lock: exact while the state lock is free, and from the
last read while another thread holds it, so the accept loop never waits. The
cmux-tui-platform changes are reverted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…own terminals skip the read (#27698 review)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@austinywang

Copy link
Copy Markdown
Contributor Author

Correctness review (subagent, four rounds) and resolution

Round 1, at 2c28545: FIX

  • Medium: the terminal check ran on the connection thread through the session state lock. While the app held that lock, connecting agents each held a general slot waiting on it and could fill the app's reserve. Resolved: the decision moved into the accept loop, before the connection's thread, and it never waits on the state lock.
  • Medium: the peer was classified twice per connection. Resolved: the accept-loop check no longer goes through the state lock.
  • Low: a terminal refusal was counted as both accepted and refused. Resolved: it now moves from accepted to refused, and the test asserts both.

Round 2, at 7f3f3e7: FIX

  • High: the process-read classification (session leader's parent argv) missed macOS shells started through login(1), whose parent can't be read with proc_pidinfo. It also missed adopted and holder terminals, and it read a foreign process's argv on every accept. Resolved: the cmux-tui-platform changes are reverted. Instead, TerminalSessions matches getsid(peer) against the session's own terminals (surfaces and terminal_catalog process_id(), the same set classify_local_peer uses). It reads that set with try_lock.

Round 3, at ad92fb1: FIX

  • High: StateMutex::try_lock was #[cfg(test)], so non-test builds didn't compile. Resolved: it's now available outside tests. A try can't deadlock, so it skips the rank check.
  • Low: a poisoned lock fell back silently. Resolved: try_with_state now reads the recovered state.
  • Advisory: rescanning on every accept costs a lot during a hook storm. Resolved: a peer already in the last set answers without a rescan; only a miss rescans.

Round 4, at 344942f: SHIP
Review (claude-subagent): SHIP at 344942f

Primary agent's own check at the same head: I read the full diff. Each permit is released exactly once, on every path. A refused terminal connection drops its general slot before the busy line is written. Remote-entry and WebSocket admission are unchanged. Parked connections during restore go through ServedSession::serve once.

Open, by scope (not covered here):

  • WebSocket and remote-entry connections aren't capped.
  • Helpers that call setsid aren't recognised as terminal-bound.
  • Windows has no session id, so nothing is capped there.
  • No socket-level test drives a real terminal peer through the accept loop.

@blacksmith-sh

blacksmith-sh Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Found 2 test failures on Blacksmith runners:

Failures

Test View Logs
test_a_picked_signing_label_has_a_rerun_alternative (main.RunsOnFallbackTests.test_
a_picked_signing_label_has_a_rerun_alternative)/
test_a_picked_signing_label_has_a_rerun_alternative (main.RunsOnFallbackTests.test_
a_picked_signing_label_has_a_rerun_alternative)
View Logs
test_no_runs_on_chain_ends_on_the_reserved_side_label (main.RunsOnFallbackTests.tes
t_no_runs_on_chain_ends_on_the_reserved_side_label)/
test_no_runs_on_chain_ends_on_the_reserved_side_label (main.RunsOnFallbackTests.tes
t_no_runs_on_chain_ends_on_the_reserved_side_label)
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@austinywang austinywang added the ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds label Oct 30, 2026
@austinywang

Copy link
Copy Markdown
Contributor Author

CI note for the hosted focused runs at 344942f (runs 40979418663 and 40979417257):

  • test (linux) passed.
  • lint (linux) failed on the clippy baseline ratchet: acpmux: clippy::indexing_slicing 709 > baseline 697. This PR doesn't touch acpmux; its final diff is limited to cmux-tui-core. The test-only commit 048e26e (on the older base) fails the same lint, so this is a base finding, not this PR's.
  • macOS jobs were cancelled after that lint failure, so there's no macOS test result yet.

@lawrencecchen

Copy link
Copy Markdown
Contributor

The principled replacement for this split is up: #28343. It gives the app its own daemon socket (<session socket>.app, its own budget), gives each terminal a fair share on the session socket (by process session), derives the budget from RLIMIT_NOFILE instead of 64, and removes the 2 s write-stall kill. When #28343 lands it deletes the 48/64 split here, so keep this PR as the stopgap until then.

PL-daemon (owner of cmux-tui-core): this changes connection admission in diagnostics/connection_budget.rs, server/listen.rs and server/restoring.rs, plus the outbound write path (bounded_outbound.rs, line_connection.rs, websocket_listener.rs). R9 rows: 3 (no setrlimit-derived cap, R-L1), the known MAX_SERVER_CONNECTIONS 64 / MAX_PARKED 32 (R-L1, R-L2), and 4 (STREAM_WRITE_TIMEOUT 2 s, R-L5).

The base landed 21918fb (a hung-up local client gives its connection
slot back, ConnectionLease). Keep both: a terminal connection's share is
now a flag on its ConnectionLease, claimed before the lease is made
(claim_watched_terminal_connection), so a terminal client that hangs up
gives back its terminal slot with its main one, and a terminal claim at its
share first reclaims hung-up clients, as try_claim does. Adds a test that a
hung-up terminal client frees its terminal slot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@austinywang

Copy link
Copy Markdown
Contributor Author

Sweep fix: merge conflict with feat-cmux-next (710 behind) in diagnostics.rs, bounded_outbound.rs, listen.rs and tests/outbound_and_scheduler.rs. The base landed 21918fb (a hung-up local client gives back its connection slot, ConnectionLease, for #27669). Resolution keeps both: ConnectionPermitLease/TerminalPermitLease/into_terminal are replaced by a terminal flag on ConnectionLease, claimed before the lease is built (claim_watched_terminal_connection, chosen in ServedSession::serve from TerminalSessions::contains_peer). A terminal client that hangs up gives back its terminal slot along with its main one, and a terminal claim at its 48 share first reclaims hung-up clients the way try_claim does. Added hung_up_terminal_clients_give_back_their_terminal_slots; kept both sides' tests. rustfmt run locally; not compiled here. New head cbe8a2d. The head needs CI and a review receipt before relabel.

Conflicts: diagnostics.rs keeps feat's ConnectionStats.socket next to the
PR's terminal_active; signaled_mutex.rs keeps feat's #[track_caller] try_lock
with telemetry and the PR's removal of #[cfg(test)] (the accept loop uses it).
@lawrencecchen lawrencecchen removed the ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds label Oct 31, 2026
@cursor

cursor Bot commented Oct 31, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@lawrencecchen lawrencecchen added the ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds label Oct 31, 2026
@cmux-bot
cmux-bot Bot temporarily deployed to ci-routing-agent October 31, 2026 15:03 Inactive
@cmux-bot
cmux-bot Bot merged commit fbb9e39 into feat-cmux-next Oct 31, 2026
214 of 215 checks passed
@cmux-bot
cmux-bot Bot deleted the 27698-daemon-connection-reserve branch October 31, 2026 15:30
lawrencecchen added a commit that referenced this pull request Nov 1, 2026
Conflicts take feat's side (#27719's 48-of-64 terminal cap, #28041's hook
queue). Lead decision 2026-11-01 05:20Z: this PR builds on #27719 instead of
replacing it; the next commits narrow it to the per-terminal fair share.
lawrencecchen added a commit that referenced this pull request Nov 1, 2026
…26-11-01 05:20Z)

The app socket, the RLIMIT_NOFILE budget and the removal of the 2 s write
stall leave this PR: #27719 (fbb9e39) already keeps 16 of the 64
connections for the app by capping terminal clients at 48, and the lead chose
to build on that cap. The tree is feat-cmux-next again here; the next commits
add only the per-terminal fair share inside the terminal lane.
lawrencecchen added a commit that referenced this pull request Nov 1, 2026
…thers (red)

Builds on #27719's 48-of-64 terminal cap (lead decision 2026-11-01 05:20Z).
The accept loop already knows which terminal a peer runs in (its process
session); this commit carries that id to the terminal claim and the lease,
with no behavior change, and adds the test. Red here: terminal 101 takes all
48 terminal connections and terminal 102 finds none.
lawrencecchen added a commit that referenced this pull request Nov 1, 2026
…nal lane

On top of #27719's 48-of-64 terminal cap (lead decision 2026-11-01 05:20Z): a
terminal's first connection gets in whenever the lane has room; a further one
only while that terminal then holds fewer than the lane has left free. One
terminal opening connections in a loop stops at 24 of the 48, and every other
terminal still finds room. The share and the count change together under one
lock; a hung-up client's reclaim gives back its terminal's share too. A
connection whose terminal is unknown keeps the lane bound alone.
lawrencecchen added a commit that referenced this pull request Nov 1, 2026
…lane (on top of #27719's 48-of-64 cap) (#28343)

* test(cmux-tui): the app keeps its daemon connections under terminal load (red, #27698 B1)

Austin's nightly-next (2026-10-29): about 80 terminal agents held all 64
daemon connections, 55,162 were refused, the app's among them.

Red tests, with behavior-neutral seams (ConnectionBudget at the old 64,
ConnectionLane/ConnectionSlot, app_lane_socket_path) so they compile on
the base:
- an_app_connection_is_served_while_the_session_socket_is_full
- one_terminal_opening_many_connections_cannot_starve_another
- the_connection_budget_follows_the_open_file_limit
- a_client_that_stops_reading_for_a_while_keeps_its_connection
- AppLaneConnectionTests (Swift): the app connects on <socket>.app

* fix(cmux-tui): the app has its own daemon socket; per-client fair shares; budget from RLIMIT_NOFILE; no 2 s write kill (#27698 B1)

Austin's nightly-next (2026-10-29): about 80 terminal agents held all 64
daemon connections; 55,162 were refused in 2.5 h, the app's among them.
The Austin's-day A/B on #28159 showed 64/64 with ~3,700-4,250 refusals
per 6-min run.

- The app's own socket, `<session socket>.app` (app_lane_socket_path),
  bound under the same start lock. Each socket has its own budget, so
  terminals, agents, hooks and the CLI (session socket) can never take
  the app's connections. The app (LineTransport) connects there first and
  falls back to the session socket (older daemon, stale file, long path).
  Chromium: the browser's own IPC channels never compete with renderer
  connections (render_process_host_impl.cc gives each renderer its own).
- A fair share per client on the session socket: the client is the peer's
  process session (getsid), so a terminal's shell, agent, hooks and cmux
  calls count as one. A client may take a connection only while it then
  holds fewer than the socket has left free: one terminal stops at half
  the lane, and every new client still finds room. Remote entry and
  WebSocket clients keep the lane budget only (no identity at accept).
- The budget comes from the soft RLIMIT_NOFILE the owner already raises at
  start (owner_start, raise_open_file_limit: hard limit, at most
  kern.maxfilesperproc / OPEN_MAX): half of the descriptors left after
  start, 8 per slot (4 descriptors per connection, 2 connections per slot
  with the hung-up pool), half per socket; on macOS also bounded by
  kern.num_taskthreads (2 threads per connection). 64 and
  MAX_SERVER_CONNECTIONS are gone; the parked-connection bound follows it.
- No write deadline: STREAM_WRITE_TIMEOUT (2 s) closed a connection whose
  client read late (a stalled app main thread lost its connection and
  resynced). Now a client is ended only when its replies overflow the
  bounded control queue (QueuedSink evicts and shuts the socket); the
  WebSocket ping never waits on a peer that does not read, and the ping
  clock shuts a dead peer's socket.

Replaces the stopgap split of #27719 (48/64 for terminals).

* test(cmux-tui): the slow-reader test declares the stall as a signal, not a 3 s sleep (R5, #27698 B1)

StalledSocket models the kernel in memory: a daemon write waits while the
client's buffer is full, and one that waits when the test declares the
stall fails as SO_SNDTIMEO does if a write deadline is set (red on the base:
2 s STREAM_WRITE_TIMEOUT), else keeps waiting (green). The app-lane test
reads with no deadline: the daemon answers or refuses with one line.

* test(cmux-tui): read the child's budget anywhere in its line (libtest prints its own prefix first) (#27698 B1)

the_connection_budget_follows_the_open_file_limit failed on both the red
and the green fleet runs (16894a02, 3830e38f) at connection_lanes.rs:123:
the child printed "connection-budget=30" after libtest's "test <name> ... "
on the same line, and the parser only took a line that starts with it.

* fix(cmux-tui): the app socket gets its own outbound budgets; clean up .app first; clippy (review of 6aaecf6, #27698 B1)

Review FIX (landing/reviews/6aaecf6a9207...md):
- Blocker: clippy -D warnings on changed files: SynchronizedTcpStream::
  set_write_timeout is dead with no write deadline; same-type casts,
  unnecessary qualifications and a redundant clone in the tests.
- The two sockets shared one RenderService, whose outbound byte budgets
  a session-socket client that stops reading now holds until its queue
  overflows (no 2 s write kill). The app's replies could then fail on a
  full pool and the app's connection close: #27698 by another path. The
  app socket now has its own RenderService.
- cleanup removes <socket>.app before the session socket, so a daemon
  that binds right after the first removal keeps its fresh .app.

* test(cmux-tui): the app's replies never wait for the session socket's outbound bytes (review of 6aaecf6, #27698 B1)

Regression test for the app socket's own RenderService (0a0796c):
spend the session socket's outbound and control budgets, as clients that
stopped reading hold them now that no write deadline frees them; a
session-socket identify then gets no reply (the test checks it proves
something), and the app socket's identify is still answered.
ServedSession::new builds the session's two budget sets in one place.
Red proof: cp-scratch/pa-b1-red-budget (6aaecf6 + this test, with
ServedSession::new serving both sockets from one RenderService).

* test(cmux-tui): a one-connection lane still admits a client; the daemon starts without its app socket (red, #27698 B1)

Lead review of 5631798, two defects for Austin's load:
- every_lane_admits_a_client_at_the_smallest_budget: at the smallest
  budget (one connection per lane) the fair share refuses every client.
- the_session_socket_serves_when_the_app_socket_cannot_be_bound: a
  directory nobody may remove at <socket>.app makes the whole daemon fail
  to start.

* fix(cmux-tui): a client's first connection always gets in while the lane has room; the daemon starts without its app socket (#27698 B1)

- The fair share admitted a further connection only while the client
  then held fewer than the lane had left free, so a lane of one (the
  smallest budget an fd limit derives) refused every client. A client's
  first connection now gets in whenever the lane has room.
- A failure to bind <socket>.app (a path that can't be removed, Windows)
  failed the whole start. The app socket is an addition: the daemon now
  logs why and serves the session socket alone; the app's LineTransport
  already falls back to it.

* revert(cmux-tui): narrow this PR to build on #27719 (lead decision 2026-11-01 05:20Z)

The app socket, the RLIMIT_NOFILE budget and the removal of the 2 s write
stall leave this PR: #27719 (fbb9e39) already keeps 16 of the 64
connections for the app by capping terminal clients at 48, and the lead chose
to build on that cap. The tree is feat-cmux-next again here; the next commits
add only the per-terminal fair share inside the terminal lane.

* test(cmux-tui): one terminal cannot take the terminal lane from the others (red)

Builds on #27719's 48-of-64 terminal cap (lead decision 2026-11-01 05:20Z).
The accept loop already knows which terminal a peer runs in (its process
session); this commit carries that id to the terminal claim and the lease,
with no behavior change, and adds the test. Red here: terminal 101 takes all
48 terminal connections and terminal 102 finds none.

* fix(cmux-tui): a terminal's connections get a fair share of the terminal lane

On top of #27719's 48-of-64 terminal cap (lead decision 2026-11-01 05:20Z): a
terminal's first connection gets in whenever the lane has room; a further one
only while that terminal then holds fewer than the lane has left free. One
terminal opening connections in a loop stops at 24 of the 48, and every other
terminal still finds room. The share and the count change together under one
lock; a hung-up client's reclaim gives back its terminal's share too. A
connection whose terminal is unknown keeps the lane bound alone.

* test(cmux-tui): the fair-share test proves a released terminal's share is cleared; the share is "no more than" the free lane (review 6553892057)

* fix(cmux-tui): contains_peer is test-only now (clippy dead_code; the accept loop asks terminal_of_peer)

clippy step b7e7a3f2c43e at 8fc1e8a: method contains_peer is never used.

This branch was previously deployed

3 inactive deployments
artifacts — 1487a27e Deployed Oct 31, 2026 by cmux-bot[bot] via add the Windows daemon to the tree #90320
ci-routing-agent — 1487a27e Deployed Oct 31, 2026 by github-actions[bot] via cmux-next macOS placement #54663
sentry-dsyms — 1487a27e Deployed Oct 31, 2026 by cmux-bot[bot] via upload cmux-tui dSYMs to Sentry #90320
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants