Repository navigation
perf(cmux-tui-core): the input lane is the daemon's default; CMUX_TUI_RDS_INPUT=0 keeps the old path for one nightly (R-DAEMON-STATE S1.7) - #35413
Merged
Conversation
…MON-STATE S1 red) A key line from an attached view goes through the real connection path (the line handler, the connection's dispatcher, the send handler) while another thread holds one shared lock for 300 ms. #31940 took Mux.state off the key; the key still waits for the client registry lock (five takes per line), the link peers lock, the host proxy authors lock, the surface admission lock, the creating-input lock, the client sizing lock and the activity lock. Their other holders are every connect, attach, resize and request of every client. Mux.state and the workspace registry rows are controls (pass on the base). Two keys go in a row, so a lock taken after the PTY write (sizing, activity) shows as a delay of the second key on the connection's dispatcher. Test-only hold hooks (cfg(test)) in creating_input.rs, terminal_sizing.rs, activity.rs and agent_session_attach.rs. No product change. Design: handoff/rewrites/R-DAEMON-STATE/DESIGN.md section 2.6 (stage S1).
…N-STATE S1 step 1 red) The previous commit's census test held nine shared locks in turn; this commit splits it by fix step. Step 1 is the client registry lock (ClientRegistry.state), which a key line takes about nine times (handoff checks, remote-client and remote-device checks, the request's actor, the kept terminal, the reply's incognito filter) while every connect, disconnect, attach and detach of every client holds it. The test turns on the stage switch CMUX_TUI_RDS_INPUT (rds_switch.rs, inert in this commit) and drives two keys through the real connection path while another thread holds the lock for 300 ms. A second test keeps Mux.state and the workspace registry off the key on the new path (#31940). The hold hooks of the other locks leave with this commit; they come back with their fix steps.
…ished facts, never its lock (R-DAEMON-STATE S1 step 1) A key line took the client registry lock (ClientRegistry.state) about nine times: the daemon handoff checks, the remote-client and remote-device checks, the request's actor, the kept terminal of the key and the reply's incognito filter. Every connect, disconnect, attach, detach and capability change of every client holds that lock, so an agent storm held back the person's keys (red at 2472251: 300.4 ms behind a 300 ms hold; STATUS 9131). The registry is now a Published value (published.rs): writers still serialize on its mutex, and a guard that changed the value publishes an immutable ClientFacts projection (handoff state; per connection its transport, remote device flag, base actor and kept input terminals) with arc-swap when it drops, under the mutex, so projections follow the order of the changes. With the stage switch CMUX_TUI_RDS_INPUT=1, the line and key paths read the projection with one atomic load: daemon_handoff_in_progress/committed, transport_of (so is_remote_client), is_remote_device (refuse_line, line_to_recheck, prune_reply), connection_actor and attached_input_surface. A poisoned registry still fails closed (the projection comes in the error, as Mutex::lock gives its guard). With the switch off nothing changes. A projection costs O(connections) under the registry lock per change, bounded by the 64-connection cap; stage S2 makes it O(changed). arc-swap 1.9 is already in the lockfile and the shipped notices (through iroh-dns); cmux-tui-core now names it. Tests: rds_input_lane (a key waits for no client registry lock; the session and workspace registry locks stay off; a respawn under the same id sends the next key to the new shell), published.rs unit tests (a change publishes on drop, a read publishes nothing, a reader never waits for a writer, a poisoned value fails closed). The two remote-relay poison tests poison through the new type.
… stage switch per mux (R-DAEMON-STATE S1 step 1) The hosted run of f1ea1b6 (https://github.com/manaflow-ai/cmux/actions/runs/41655768031) still failed both lane tests: the key waited 300 ms for the client registry and, new, for Mux.state. Two causes: - The key still used the kept terminal of #31940. That cache is validated by process-wide route generations, and cargo test runs the tests of one binary as threads of one process: another test's mux moved the same generation slot (ids start at 1 in every mux), the kept terminal looked stale, and the key fell back to the locked lookup and then to remember_attached_input (registry lock). The stage switch was a process-wide static too. - So the lane must not depend on a cache with process-wide invalidation at all. Now, with CMUX_TUI_RDS_INPUT=1: - The session's three routing tables (surfaces, terminal_catalog, terminal_catalog_by_runtime) are copy-on-write behind an Arc. A state guard that changed one publishes a RouteSnapshot (three Arc clones) with arc-swap when it drops, under the lock, so publications follow the changes; the snapshot it replaced is freed after the lock is released. Mux::surface reads the snapshot with one atomic load. A respawn, close, move, rollback or projection is seen once its guard is released, as a reader that waited for the lock would see it. Nothing is kept per client, so nothing is invalidated. - input_surface on the lane: the published routes give the terminal, the registry's published facts say whether the client types into it (types_into: the person's input QoS mark). attached_input_surface and remember_attached_input stay only on the old path. - The switch is per mux (rds_switch::InputLaneSwitch, read once from the environment when the mux is built; Mux::set_input_lane_for_test in tests), and it also decides whether anything is published: off, the registry builds no projection and the state holds no route snapshot, so a route write never copies a shared table and the off arm costs what the base costs. Cost on the lane: a guard's first route write copies that table once (O(routes)), bounded by one copy per changing guard; State::clone no longer deep-copies the three tables. Stage S2 makes both O(log n).
…an old-path control (R-DAEMON-STATE S1 step 3 red) Every request of every connection reserves its retained bytes from one daemon-wide budget (ServerSurfaceOperationAdmission) under one mutex before it is queued, and releases them under it after it ran. The S1 census measured a key waiting 300.3 ms behind a 300 ms hold of that lock (hosted run 41648095321). The lane test now has a control: the same two keys with stage S1 off must wait for the held lock, so a lane result can never come from a hold that does not reach the key path (a vacuous green). The control is added to the client registry test too. A test-only hook (cfg(test)) holds the admission lock. Red here: with the lane on, the key still reserves through the mutex.
…s on the input lane (R-DAEMON-STATE S1 step 3) Every request of every connection reserves its retained bytes (and every clear-history its worker) from one daemon-wide budget before it is queued and releases after it ran. Under one mutex, a key waited for any other connection's reservation or release (red at 5d0e4a0: 300 ms behind a 300 ms hold). With CMUX_TUI_RDS_INPUT=1 the two counters are atomics with compare-and-swap reservations (fetch_update: add only while the sum stays within the capacity; release saturates at zero), so nothing waits. The limits and the refusals are the same as the mutex arm's: a worker is refused at 16 reserved, bytes when the sum would pass 16 MiB. Each permit releases to the arm it reserved from. With the switch off the mutex arm runs as before; it goes with the switch when the stage passes (S1.8). The arm is fixed when the mux is built (ServerSurfaceOperationAdmission::new(publish_routes)); tests set it with Mux::set_input_lane_for_test before any request. The two scheduler tests that counted reserved workers through the mutex read reserved_workers() (both arms).
…roxy authors lock (R-DAEMON-STATE S1 step 2 red) Two more shared locks on the person's key line (DESIGN.md Appendix A.2 steps 3, 7, 12 and 13): the link peers map (RemoteRelayState.peers: the remote-client check and the request's actor) and the host proxy authors (AgentSessions.proxy_authors: the proxy-link check on the reader thread). The S1 census measured a full 300 ms wait on each (hosted run 41648095321: 300.5 ms and 300.4 ms). The test holds each lock 300 ms on another thread while two keys go through the real line path. It first proves the hold reaches the old path (stage switch off: the key waits), then expects the stage's path not to wait. Red on this commit: the stage does not publish either value yet.
… (R-DAEMON-STATE S1 red) A published value's guard publishes its projection in Drop::drop, which also runs while a panic unwinds, before its MutexGuard field drops and marks the mutex poisoned. A writer that panics in the middle of a change so publishes the half-made value, and in the moment between the publication and the poisoning a reader gets it as a good value: a fail-open window for readers that fail closed on a poisoned value (transport_of, peer_checked). Red on this commit: the test sees the half-made value 9 in the poisoned projection.
…panicked change publishes no routes (R-DAEMON-STATE S1 red) Two defects the #34014 review found on the input lane (both only with CMUX_TUI_RDS_INPUT on): - RouteTable::try_reserve returns early when the map has room, also when a published route snapshot shares the map. The insert after the durable commit then copies the whole table (Arc::make_mut), an allocation that ResourceMutationPlan::apply forbids ("Plan builders reserve all needed capacities before returning"). - The session state guard publishes the routes of a change that a panic cut short: Drop::drop runs during the unwind, before the MutexGuard field poisons the mutex, so the lane routes by half-made tables. Red on this commit: the insert changes the map's pointer, and the routes snapshot is replaced.
…the S1.1 review follow-ups (R-DAEMON-STATE S1 step 2) A person's key line asked two more shared locks (DESIGN.md Appendix A.2 steps 3, 7, 12, 13): the link peers map (the remote-client check and the request's actor) and the host proxy authors (the proxy-link check on the reader thread). Every remote bind, disconnect and revocation sweep holds the first; every proxy bind and every client's disconnect holds the second. - published.rs moves to cmux-tui-util (daemon primitives, no daemon state) so cmux-tui-remote-access can use it. RemoteRelayState.peers becomes a Published copy of the records; peer_checked reads it with no lock when the mux's stage switch is on (Published::publishing), and a poisoned map still fails closed. AgentSessions.proxy_authors publishes the same way; proxy::author reads it. A disconnect of a client without a record publishes nothing. - Review follow-ups of #34014 (review comment 6612546773): - RouteTable::try_reserve copies a map a published snapshot shares, before the durable commit, so the insert after the commit allocates nothing (ResourceMutationPlan::apply). - A change a panic cuts short is never published, by the state guard (routes) or by a PublishGuard: Drop skips the publication when a panic began while the guard held the lock (std's own poisoning test), so no reader sees half-made tables, not even before the mutex is poisoned. - A PublishGuard frees the projection it replaced after the unlock (as the state guard does). - The two poison tests of the relay run on both arms. - ClientFacts and types_into move to server/client_registry/facts.rs (god-file warning). Same switch, CMUX_TUI_RDS_INPUT, default off: with it off nothing is published and every read takes its lock as on the base.
…no session lock behind one (R-DAEMON-STATE S1 step 4 red) Every key to a live terminal first asks whether creating input of its terminal must go first (wait_for_creating_input_flush, DESIGN.md Appendix A.2 step 16). It takes the daemon-wide creating-input lock even when no terminal is being created, and that lock's holders (begin_creating_input, knows_terminal_input_target, finish_creating_input) look up the live terminal under it, which takes the session lock. The S1 census measured a full 300 ms wait on the creating-input lock (hosted run 41648095321: 300.3 ms). Two holders, each with an old-path control: the creating-input lock itself, and the session lock held while a reader waits for it under the creating-input lock (with another terminal being created, so the key cannot skip the lock). Red on this commit: the stage does not change these paths yet.
… is creating; the lock becomes a leaf (R-DAEMON-STATE S1 step 4) On the input lane (CMUX_TUI_RDS_INPUT, per mux, default off): - CreatingInputs keeps the number of creating terminals in an atomic, stored under its lock after every insert and removal (Release). A key reads it (Acquire) and returns at once when it is zero: no queue can be ahead of it, because a terminal's queue is added before the terminal goes live and removed only after every queued byte was written to it. - The creating-input lock's holders look up the live terminal in the published routes instead of under the session lock, so no holder waits for the session lock while it holds the creating-input lock, and a key that takes it never waits behind one. The routes carry the host-id catalog index for this: terminal_catalog_by_host becomes a copy-on-write route table in the RouteSnapshot, and one lookup (route_table::live_terminal) serves the session state and the snapshot. A commit publishes its routes before it releases the session lock, and finish runs after the commit, so begin, knows and finish give the answers they gave under the session lock. Off, every path takes the locks it took on the base; the by-host index is copy-on-write on both arms, as the other three route tables are since step 1.
… control fails on a lost key (#34146 review follow-ups) The #34146 review (comment 6613345630) noted: - Every budget test used the mutex arm, so an atomic arm that admitted everything would have passed. The two mux-wide limit tests now run on both arms, and a new test checks the same limits on both: the 17th worker, the whole byte budget, one byte over, zero bytes, a sum that overflows, and released reservations coming back. - The old-path control passed when a key never arrived (Duration::MAX >= BUDGET); it now fails on a lost key. - ServerSurfaceOperationState and the admission's mutex are private (nothing outside the file reads them), and the doc says which requests reserve (pings, identify, v2 resource lines and reader-thread frame handlers do not).
…(clippy unused_qualifications) The lint (linux) job's clippy -D warnings failed on mux/tests/signaled_mutex.rs:120 (TerminalPublicId is in scope); the #34174 review found it behind the base's acpmux red in the same job.
…lished tables (R-DAEMON-STATE S1 red) The #34171 review noted: after a panic poisons Mux::state, the next guard that recovers the poison may publish the half-made route tables, and the lane's Mux::surface reads the published routes with no poison check, where the old path fails (it unwraps the lock). Red on this commit: with the lane on, a surface read after the poisoning returns from the published routes instead of failing. The relay poison tests now poison published values through a helper in the test file.
…oisoned state; the poison helper is test code (#34171 review) Review comment 6614044759 (FIX at 8f7d405): - published.rs: poison_for_test moved out of the test module with its panic! respelled as panic_any, which hides a hit from the crash ratchet's count. It is test code again: a private helper in the published tests and one in the relay tests, with plain panic!. - Mux::surface on the lane reads the published routes only while the session state is not poisoned; a poisoned state takes the old read and fails as it does (StateMutex::is_poisoned). A guard that recovers the poison may publish the half-made tables of the change that panicked. - Published::load debug-asserts that the value publishes: only a publishing value has a current projection. - The module doc says what load gives on a poisoned value; the caller decides whether to fail closed. The unused_qualifications error at mux/tests/signaled_mutex.rs:120 is fixed in 10f1669.
…ly on an unpoisoned state; its comment states the lane's linearization (#34174 review) Review comment 6613998735 (SHIP, notes): the begin_creating_input comment still said the check takes the state under the creating lock; on the lane it reads the published routes, and its answer is one the old path can give (a commit in progress has not made the id live, as when the old path takes the state before it). The lane's live lookup now takes the old read on a poisoned state, as Mux::surface does since bb0265f.
…ity lock; a key takes the grid back after another view's click (R-DAEMON-STATE S1 step 5 red) After its PTY write a key notes activity in the terminal's sizing (the newest view sets the grid under the default policy): it takes the daemon-wide client sizing lock twice (the note and the size-state publication), then the VM activity stream's lock. Every attach, detach, resize and size report of every terminal holds the sizing lock, and a PTY resize runs under it. The S1 census measured a full 300 ms wait on each (hosted run 41648095321: 300.2 and 300.3 ms). The first test holds each lock 300 ms (with an old-path control each). The second keeps the fix honest: two views of one terminal, the default policy; the other view's focus click takes the grid, and the next key from the typing view must take it back, on both arms. Red on this commit: the first test (the stage does not change these paths yet).
…settled newest; the person's input time is an atomic (R-DAEMON-STATE S1 step 5) On the input lane (CMUX_TUI_RDS_INPUT, per mux, default off): - Each terminal's sizing entry publishes, on the terminal's runtime (shared by every placement), the view a key may skip its sizing note for: the sole newest participant, a client's own view, only while the entry is settled (no grid hold, no handheld with a viewport, and the engine holds the grid or the applied grid and participants are its decision). Noting activity for that view changes no decision under any policy, and the size state carries no activity, so the key compares (client, placement) with one atomic load and takes no sizing lock. - The publication is automatic: Mux::client_sizing is a SizingMutex whose guard republishes, on drop and under the lock, each entry its holder read or wrote (SizingEntries marks them; an apply changes Cells through a shared borrow, so reads count). A removed entry clears its slot; a change a panic cuts short publishes nothing. - Another view's activity makes that view the newest and republishes, so the next key from the first view takes the slow path once and takes the grid back (the second test pins this on both arms). - ActivityStream::note_user_input stores the time in an atomic; it takes the stream lock only while a subscriber exists, once per event the worker sends (a wake_pending flag the worker clears before it reads the time). - TerminalSizingEngine::sole_latest (cmux-terminal-sizing) with a unit test; AppliedSeat::of shares the seat rule between the apply and the settled check; set_workspace_size_policy reads its state before the note (no index). Off, every key takes the sizing and activity locks as on the base; the marks and the guard check one bool.
…ook's cfg(test) (dead code and 2 expect hits in production) dfe81bf put hold_client_sizing_for_test between pin_latest_size_policy_for_test's doc and its #[cfg(test)], so pin_latest_size_policy_for_test compiled into production: clippy -D warnings dead_code, and the panic ratchet's cmux-tui-core expect_used 175 > 173 (hosted lint job 126202464363).
…ts poison check; two test doc comments on their items (#34171 review 2) Review comment 6615927915 (SHIP at 0fb7c43, notes): Mux::surface checked is_poisoned() and then loaded the routes; between the two another thread could poison the state and a third, recovering the poison, publish half-made tables. Poison never clears on the session state, so routes loaded before an unpoisoned check were published before any poisoning: load first, then check. Two doc comments in remote_relay/tests.rs sat above the wrong item.
…content is already here (590cb11); tree unchanged
lawrencecchen
temporarily deployed
to
ci-routing-agent
November 4, 2026 23:10 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
lawrencecchen
temporarily deployed
to
ci-routing-agent
November 4, 2026 23:10 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
lawrencecchen
temporarily deployed
to
sentry-dsyms
November 4, 2026 23:21 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
lawrencecchen
temporarily deployed
to
artifacts
November 4, 2026 23:21 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
lawrencecchen
temporarily deployed
to
artifacts
November 4, 2026 23:25 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
…wner (#35413 review; red) Review of #35413 at afadf63 (SHIP, should-fix 4): with S1.7 the input lane is on by default in every cmux-tui daemon, and the app's fallback switch for the one nightly before S1.8 is CMUX_NEXT_TUI_RDS_INPUT=0. The app passes it to its own daemon (DaemonDiagnosticKnobs), but the Chief owner's `server ensure` environment (DaemonLauncher.chiefEnvironment) keeps only a short list of variables plus the R-LIFECYCLE keepers switch, so the Chief daemon runs the lane with no way to turn it off. The test mirrors theOwnerKeepersSwitchReachesTheChiefOwner. It fails on this code: the Chief environment has no CMUX_TUI_RDS_INPUT.
…ner (#35413 review) The test is the commit before this one (red: the Chief owner's environment had no CMUX_TUI_RDS_INPUT, so `CMUX_NEXT_TUI_RDS_INPUT=0` on the app's launch turned the lane off in the app's daemon only, and the Chief daemon ran the lane with no off switch for the fallback night). DaemonDiagnosticKnobs now names the switches the Chief owner's launch passes on (chiefOwnerDaemonNames: the R-LIFECYCLE keepers switch, as before, and the R-DAEMON-STATE input lane switch), and DaemonLauncher.chiefEnvironment passes each one that the app's launch set. Nothing else reaches the Chief owner: the test still checks that CMUX_NEXT_TRACE_FLOWS does not.
lawrencecchen
temporarily deployed
to
ci-routing-agent
November 5, 2026 00:18 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
lawrencecchen
temporarily deployed
to
ci-routing-agent
November 5, 2026 00:18 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
lawrencecchen
temporarily deployed
to
ci-routing-agent
November 5, 2026 00:19 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
Author
|
Review (claude-reviewer-rds-s17b): SHIP at c8900c4 Independent review subagent, two rounds: SHIP at afadf63 (the own diff, the cascade merges of #35765 and #35872, the 11 feat commits on the key path since the last SHIP, the default flip with both fixes), then this SHIP of the delta afadf63..c8900c4.
|
lawrencecchen
added a commit
that referenced
this pull request
Nov 5, 2026
… per connection and sends without a copy (#35413 s17ga) The test is the commit before this one (red on macOS: 100 keys to one connection opened 100 kernel event queues, one per key's room check). s17ga (STATUS 13245, 13254, 13302) kept the S1 input lane off. On one boundary for both arms, ON was worse at p50 beyond the OFF spread. The whole difference was the write call on the key's own request thread: 0.019-0.020 ms with the inline write, against 0.005 ms for the old path's direct write. The inline write did three things the old path did not: 1. poll(POLLOUT, 0) before the send. On XNU each poll() opens a kernel event queue, registers a knote under the socket lock and frees both. 2. Two heap copies of the key (Frame::new(payload.to_vec()), encode_frame). 3. One send of a frame it had just built. Now (terminal_host_runtime/shared/attachment/room_check.rs): - macOS: each host connection keeps one kernel event queue, with the socket's EVFILT_WRITE registered once at its first key. Each key reads it with kevent64(KEVENT_FLAG_IMMEDIATE): one call, no allocation, no sleep. The filter's data is the socket's free room; the key goes inline only when it holds the whole frame. EV_EOF or EV_ERROR is no room, so the key goes to the port and its reconnect path. A failed kqueue() falls back to poll(). The connection's attachment owns the queue and shares it with every route clone; it closes with the attachment, and a reconnect makes a new one. - Linux: no check. MSG_DONTWAIT is honoured, and a Unix stream socket takes a frame this small whole or not at all. - Every Unix target: the frame header is built on the stack (terminal_host_protocol::input_frame_header, pinned to encode_frame's output by the_inline_frame_header_is_the_encoders_header). The header and the key go out in one sendmsg of two iovecs, with no heap copy of the key. The order rules, the size limit and the fallbacks to the port are the same. The room check still runs under the connection's writer guard, so nothing takes the room before the send.
…1 input lane is the default, #35413)
lawrencecchen
temporarily deployed
to
ci-routing-agent
November 5, 2026 01:42 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
lawrencecchen
temporarily deployed
to
ci-routing-agent
November 5, 2026 01:42 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
lawrencecchen
temporarily deployed
to
ci-routing-agent
November 5, 2026 01:43 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
|
Found 6 test failures on Blacksmith runners: Failures
|
![Fix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/fix-with-codesmith-light-3x.png)
Contributor
Author
|
Review (claude-reviewer-rds-s17b): SHIP at f69479f Independent review subagent, third round on this PR (SHIP at afadf63, SHIP at c8900c4, this SHIP at f69479f).
|
Contributor
Author
|
Base finding, not from this PR: |
lawrencecchen
added
the
ready-to-land
lawrencecchen
removed
the
ready-to-land
lawrencecchen
added a commit
that referenced
this pull request
Nov 5, 2026
…line key write #35872) into fix/p0-typing-s1-8-delete-old-path S1.7 landed on feat as one squash, so feat holds the S1.7 branch's content but not its commits, and the merge base git picks (0f4febb) is older than every S1 change. With that base, each line S1.8 set back to its pre-S1 form looks unchanged on this side, and the merge silently put the S1.7 version back (for example old_path_mux in server/tests/input_lane.rs). So this merge is made against 6bfdf88, the S1.7 head this branch already merged (git merge-tree --merge-base): this side brings only S1.8's own changes, feat brings everything after that S1.7 head. Checked after the merge: no deleted name is back (the switch, the kept terminal, the route generations, the publishing and lock-free flags), and git diff against feat touches the same 34 S1.8 files plus the two below. Resolved: - rds_switch.rs: deleted (feat's copy is the S1.7 branch's). - DaemonDiagnosticKnobs.swift: feat moved the lane knob to inputLaneDaemonName and passed it to the Chief owner (chiefOwnerDaemonNames). Both lane parts are deleted; chiefOwnerDaemonNames keeps the R-LIFECYCLE keepers switch only. DaemonLauncher.chiefEnvironment's doc no longer names the lane switch, and theInputLaneSwitchReachesTheChiefOwner (it tested that knob) is deleted. - surface/tests/hosted.rs: feat made hosted_typing_surface and wait_until pub(super) for input_port_inline.rs and opts the held-writer tests out of the inline write; kept with S1.8's hosted_typing_surface(name) (no lane argument). - surface/tests/input_port_inline.rs (new on feat): its four hosted_typing_surface calls drop the lane argument. - surface/input_port.rs and terminal_host_runtime/shared/attachment.rs: the inline write and its room check stay exactly as on feat (attachment.rs has no diff against feat; input_port.rs differs only in S1.8's doc lines).
lawrencecchen
added a commit
that referenced
this pull request
Nov 5, 2026
…e per connection and sends without a copy (#35413 s17ga) (#35988) * test(cmux-tui): a connection's keys share one kernel queue for their room check (#35413 s17ga; red on macOS) The s17ga typing A/B for #35413 (STATUS 13245, 13254) kept the S1 input lane off: daemon part p50 was +0.016 ms with ON, against an OFF round-to-round spread of 0.009 ms, with both ON rounds above both OFF rounds. The whole difference was the key's write call (daemon.send.write -> written): 0.019 and 0.020 ms at p50 with ON, 0.005 ms with OFF. The inline key write (#35872) checks for room with poll(POLLOUT, 0) before its send. On XNU each poll() opens a kernel event queue, registers a knote on the socket under the socket's lock, and frees both again (sys_generic.c poll_nocancel; the #35872 reviewer's note 3). The test sends 100 keys to one idle connection and counts the kernel event queues that their room checks opened, with a test-only per-thread counter at the room check. It fails on this code: 100 keys open 100 queues, one per key. macOS only (Linux's poll opens no queue). * fix(cmux-tui): the inline key write checks room with one kernel queue per connection and sends without a copy (#35413 s17ga) The test is the commit before this one (red on macOS: 100 keys to one connection opened 100 kernel event queues, one per key's room check). s17ga (STATUS 13245, 13254, 13302) kept the S1 input lane off. On one boundary for both arms, ON was worse at p50 beyond the OFF spread. The whole difference was the write call on the key's own request thread: 0.019-0.020 ms with the inline write, against 0.005 ms for the old path's direct write. The inline write did three things the old path did not: 1. poll(POLLOUT, 0) before the send. On XNU each poll() opens a kernel event queue, registers a knote under the socket lock and frees both. 2. Two heap copies of the key (Frame::new(payload.to_vec()), encode_frame). 3. One send of a frame it had just built. Now (terminal_host_runtime/shared/attachment/room_check.rs): - macOS: each host connection keeps one kernel event queue, with the socket's EVFILT_WRITE registered once at its first key. Each key reads it with kevent64(KEVENT_FLAG_IMMEDIATE): one call, no allocation, no sleep. The filter's data is the socket's free room; the key goes inline only when it holds the whole frame. EV_EOF or EV_ERROR is no room, so the key goes to the port and its reconnect path. A failed kqueue() falls back to poll(). The connection's attachment owns the queue and shares it with every route clone; it closes with the attachment, and a reconnect makes a new one. - Linux: no check. MSG_DONTWAIT is honoured, and a Unix stream socket takes a frame this small whole or not at all. - Every Unix target: the frame header is built on the stack (terminal_host_protocol::input_frame_header, pinned to encode_frame's output by the_inline_frame_header_is_the_encoders_header). The header and the key go out in one sendmsg of two iovecs, with no heap copy of the key. The order rules, the size limit and the fallbacks to the port are the same. The room check still runs under the connection's writer guard, so nothing takes the room before the send. * test(cmux-tui): after a full host socket drains, the next key goes inline again; the header test covers Paste (#35988 review) Review of #35988 (SHIP at 0fffc79), should-fix 1: no test sent a key after the socket was full and the host then read everything. On macOS that is the one state the per-connection room-check queue keeps between keys: if its write filter did not report room again, every later key on the connection would go to the port and nothing would say so. keys_to_a_host_that_stopped_reading_never_block_their_requests now ends by waiting for an idle port, holding the port's writer and sending one more key: it must reach the host inline. The order test uses the same pattern. Note 3: the_inline_frame_header_is_the_encoders_header also covers MessageKind::Paste, which reaches the inline write too.
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
R-DAEMON-STATE stage S1 step 7 (handoff/rewrites/R-DAEMON-STATE/DESIGN.md, section 3, S1.7). S1.6b (#35082), S1.6c parts 1 and 2 (#35280, #35375) and S1.6d (#35460, the input port's writer at user-interactive QoS) are on feat.
The input lane (stage S1: the person's keys read published routes and facts, never a shared lock, and go to the terminal's own input port) becomes the daemon's default.
CMUX_TUI_RDS_INPUT=0(or the app knobCMUX_NEXT_TUI_RDS_INPUT=0) keeps the old path for one nightly; S1.8 then deletes the old path and the switch. The switch is read once, when a daemon starts: a daemon that already runs keeps its lane. To fall back, set the knob on the app's launch, then restart both daemons: "Restart cmux-tui..." restarts the app's own daemon; the Chief daemon restarts with the next app build or install-key change, or when it is stopped. Daemons outside the app get the lane by default too, each with its own switch in its own environment: Cloud session hosts (they inheritCMUX_TUI_*from the cmux-host agent), the Mac server roles (cmux host run, off by default) and SSH remotes (daemon ensurein the remote environment).rds_switch.rs:InputLaneSwitch::from_valueis on unless the value is0,false,offorno(any case, spaces ignored; review). The first switch a process reads prints one line to stderr (cmux-tui: input lane on, CMUX_TUI_RDS_INPUT unsetor... off (the old path), CMUX_TUI_RDS_INPUT="0"). A daemon thatserver ensurestarts has its stderr on /dev/null, so that line reaches no daemon log; the arm of a running daemon is proven by its environment (ps -wwE), as the A/B sets do.server/tests/input_lane.rs: the kept-terminal tests of the old path pin the switch off (old_path_mux), so the fallback keeps its coverage while it ships. Every other test now runs on the lane.DaemonDiagnosticKnobs.swiftandDaemonLauncher.swift: the Chief owner (cmux-chief-*) is a cmux-tui daemon too, so it also gets the lane by default; itsserver ensureenvironment now passes the app'sCMUX_NEXT_TUI_RDS_INPUTon asCMUX_TUI_RDS_INPUT, as it passes the R-LIFECYCLE keepers switch, so the fallback switch turns the lane off in both daemons (review of afadf63; testtheInputLaneSwitchReachesTheChiefOwner, red first).an_activity_subscriber_hears_the_persons_keys_on_both_armsandthe_sizing_note_skip_sets_the_grids_the_old_path_sets(latest, smallest, largest and fixed policies; Mac and phone views; a relayed sub-view).Why now: the S1 typing A/B (STATUS 11230)
Build 2 (S1.1-S1.6a), cmux21, OFF/ON/OFF/ON, typing into a traced terminal under the L6 load. The daemon part of each key is from the app's socket submit to the host's PTY write, in ms:
The OFF keys over 4 ms waited in their own
sendhandler or in the reader (the shared-lock waits that S1 removes). The earlier "ON does not meet A4" (STATUS 10595) came from the test's echo program, which wrote a log line before each echo under the 15 GB memory load (STATUS 11019). It was not the daemon.Caveat: the L1 load in those rounds reached 108.9-112.4 calls/s, under the 114/s rule of STATUS 7761. The kit then ran 12 L1 workers (cp/typing-under-load-rds-env@ecfb1344e4a); no worker count reaches 114/s, because the calls are lost in the load's own stalls (STATUS 11712), so each comparison stays paired (the same achieved rate in both arms).
Build 3b (S1.1-S1.6b, fleet job f1c1f3b55615af21b3259d6a, STATUS 11804), cmux21, OFF/ON/OFF/ON, L1 111.6-115.4 calls/s. Daemon part ms p50/p95/p99/max: OFF 0.10/0.57/1.37/4.59 and 0.10/0.54/1.59/3.83; ON 0.14/0.80/2.23/5.70 and 0.13/0.69/2.30/5.09. Both arms meet A4 (p99 <= 4 ms, max <= 16 ms). In this set ON's tail is about 1 ms longer than OFF's (build 2 had the opposite); the 4 ON keys over 4 ms waited after
daemon.send.written(1.21-3.90 ms to the host's PTY write) or before the daemon's reader (0.33-2.80 ms), not in the dispatcher and not in their own send. The movies were lost in the fetch, so that set has no video verdict.Set s17a (rounds STATUS 12043, 12078, 12112, 12175; round 1 video 12203; verdict 12206: ON meets A4 for the daemon key part, OFF does not), build f8685e3d (S1.7 on S1.6c, without the S1.6d writer QoS), cmux21, OFF/ON/OFF/ON, kit cp/typing-under-load-rds-env@f96caecda8d, arms proven in the daemon's
ps -wwE(CMUX_TUI_RDS_INPUT=0/=1). Daemon part ms:A4 (p99 <= 4 ms, max <= 16 ms) holds in both ON rounds and fails on max in both OFF rounds; the OFF keys over 16 ms waited in their own
send(the shared-lock waits S1 removes). Keys lost 0, merged 0, order ok in all four. Video p95 is 4 refreshes in all four rounds. The video numbers come from the in-app judge with impossible frames (R-HARNESS item), so the L6 gate counts no round; the video max comes from single keys outside the daemon part (the first Ctrl-C at 12-16 refreshes in every round, one key after a Return at 40).Set s17b (build 9d7979eb, S1.7 with S1.6d, cmux21, OFF/ON/OFF/ON; STATUS 12377): ON meets A4 p99 in both rounds (2.34, 1.45 ms) but one ON key took 27.8 ms (max A4 <= 16 ms); OFF failed max in s17b-3 (16.9 ms, in its own send). In all 4 ON rounds of s17a and s17b the echo's way back stalled (host.pty.read -> daemon.output.host gaps over 20 ms, up to 586 ms; the OFF rounds' maximum was 15.2 ms), and in 3 of them the video-max key was on that path. The video max of s17b-4 (key 248) was not; its cause is not known (STATUS 12418).
Set s17d (diagnostic hops, STATUS 12505-12559) found that stall on both arms: each terminal's output reader made durable commits (cwd, progress, program status, command facts, "last output" every 5 s) under the workspace registry lock, which waits up to 1.5 s under L6, and the next frame (the echo) waited behind it. All 16 echo gaps over 20 ms began inside one of those commits. #35765 moves the commits to a per-mux publisher thread.
Set s17fa (build 5cd1899b, with #35765; STATUS 12857, 12866, 12885): 0 echo gaps over 20 ms in all 4 rounds (echo gap max per round 2.33-3.03 ms), video not worse with ON. But ON was worse by both rules below in three places: the daemon part at p95 (+0.17 ms) and p99 (+0.53 ms), key-to-commit p99 (+1.13 ms) and echo gap max (+0.09 ms). The whole difference was in one segment,
daemon.send.written->host.pty.write: the input port's writer-thread wake (ON 0.038-0.046/0.27-0.28/0.67-0.77 ms p50/p95/p99, OFF 0.016-0.019/0.14-0.17/0.38-0.48 ms). #35872 removes that hand-off for a key to an idle port.Set s17g at the final code (HEAD build 2ed758c77964169233240990 at b6b0309: this PR with #35765 and #35872; the PR head differs from it in feat merges, test hooks, comments, a poisoned-lock recovery on the output reader's publication check (#35765 a35e021; no timing effect) and the Chief owner's input lane switch (above)), cmux21, OFF/ON/OFF/ON, arms proven in the daemon's
ps -wwE:Pooled (STATUS 13245), ms p50/p95/p99/max:
Verdict rule (coordinator decision cmux-ad m15, 2026-11-05): a layer is worse with ON only when the difference is larger than OFF's own round-to-round spread at that statistic AND larger than 0.05 ms (1 refresh for video). By that rule no layer is worse: ON meets A4 in both rounds (daemon part p99 1.74 and 1.25 ms, max 2.79 and 3.96 ms; 0 keys over 4 ms), and ON is lower than OFF at daemon part p95, p99 and max and at every key-to-commit statistic. Three statistics are higher in both ON rounds, each under the rule: daemon part p50 +0.016 ms (OFF spread 0.009 ms, under the 0.05 ms floor), echo gap p95 +0.011 ms (spread 0.015 ms) and echo gap p99 +0.124 ms (spread 0.388 ms). Its cause is known: the inline key write's own cost on the key's request thread (write call 0.019-0.020 ms at p50 against 0.005 ms for the old path's direct write; on one boundary for both arms, arrival -> socket, +0.018 ms; STATUS 13302). An earlier call at 00:42Z (STATUS 13254) kept the lane off on that p50 with a rule that had no floor; the coordinator withdrew that rule (STATUS 13355: "The rule that kept S1 OFF was mine, not A4"). The cost is addressed in its own PR, #35988 (one kernel event queue per connection for the macOS room check instead of a poll() per key, no room check on Linux, one sendmsg with no heap copy; red first), which does not gate this one.
The L6 typing pair (typing-load-l6, film and video judge) was held: the 00:42Z call had failed on these builds, and the pair would have held cmux21 for about 2 h. It runs on the next builds (set s17h, with #35988) and is reported here when it ends; it does not gate this PR.
After this change, an A/B's OFF arm must set
CMUX_NEXT_TUI_RDS_INPUT=0, because unset now means ON.Testing
Focused command:
./scripts/verify-cmux-tui-hosted.sh --filter the_input_lane_is_on_unless_the_environment_turns_it_off(Linuxtest (linux)).Red at 9674a79 (
test/rds-s17-red-never-merge), run https://github.com/manaflow-ai/cmux/actions/runs/41795221251:rds_switch::tests::the_input_lane_is_on_unless_the_environment_turns_it_offFAILED ("the input lane is off with no CMUX_TUI_RDS_INPUT").Green at ba4800d, run https://github.com/manaflow-ai/cmux/actions/runs/41795228874: passed.
Prerequisites at fbafa36: run https://github.com/manaflow-ai/cmux/actions/runs/41791397095 (
--filter rds_input_lane), 13 passed.The off spellings (review): red at 5fab0a5 (
test/rds-s17-offvalues-red-never-merge), run https://github.com/manaflow-ai/cmux/actions/runs/41839881020:false_off_and_no_turn_the_input_lane_off_tooFAILED ("CMUX_TUI_RDS_INPUT="false" did not turn the input lane off"); green at 6bfdf88, run https://github.com/manaflow-ai/cmux/actions/runs/41843132208.The old-path tests
a_tab_created_or_closed_elsewhere_keeps_the_key_off_the_session_lockanda_mutation_that_routes_nothing_keeps_the_key_off_the_session_lockfail when many tests share one cargo-test process (the kept terminal's route generations are process-wide slots), on the lane-off base 01af275 too (run https://github.com/manaflow-ai/cmux/actions/runs/41842572836); one process per test (nextest, fleet step 5f4aaea7e0ed2ff3381b4fac) passes all 25input_lanetests at the S1.7 code. S1.8 deletes them with the kept terminal.The full Rust suite on Linux (fleet, cargo nextest, one process per test) at the S1.7 code d62017f, once with the lane on (6bfdf88, hq run https://github.com/manaflow-ai/cmuxterm-hq/actions/runs/41846243904) and once with it off (never-merge commit b0fd51e, an unset switch means off; hq run https://github.com/manaflow-ai/cmuxterm-hq/actions/runs/41846204089): 16930 tests each, 31 failed each, 29 the same (base findings: acpmux, agent-pane policy, cmux-tui cloud help and CLI docs,
link_seed, aterminal_host_recoverycreation test, the panic-ratchet crates). Only with the lane on:title_changed_event_includes_authoritative_surface_title(a test-only check intree_json.rsreads the live title twice while the PTY sets it) andwait_for_off_dispatch_refuses_over_capacity_without_queueing(it also failed on the lane-off base before); each passed alone 3 of 3. Only with the lane off: 2. No test of the key path failed on either arm, so the old path keeps its coverage for the fallback night. An earlier pair of runs at d6d7b68 against its base ca73823 gave the same picture (18 and 20 failed, 16 the same).The registry latency test
a_persons_requests_answer_while_busy_sessions_saturate_the_registry(it failed once in a loaded lane-on run), alone 5 times per arm on fleet Macs at the S1.7 code: lane off failed 2 of 5 (the person's new-conversation-tab p95 232 and 207 ms), lane on passed 5 of 5. The route-snapshot copy the lane adds per route change does not make the person's requests slower there.The full Rust suite on Linux at 91c7515 (this PR with fix(cmux-tui): terminal output never waits for a durable publication (#35413 echo stall) #35765 and perf(cmux-tui): a key to an idle input port is written on its own turn, with no thread hand-off (#35413 A/B) #35872 at their landed heads, the lane on by default; fleet step d9e3172e4d7fb19346e8f71c): 17067 tests, 7 failed. 6 failed again alone, all base failures seen on every branch tonight: 3 acpmux tests (
peer_integration,peer_request_deadlines,session_pool; acpmux does not depend on cmux-tui-core) and the 3server::agent_transfer::tests. 1 failed in the full run and passed alone:new_terminal_first_text a_new_terminals_first_text_and_reply_do_not_wait_for_the_disk(814 ms to the first text; 770 ms of it in the new terminal host's own record write, its file data barrier before Ready in terminal_host_runtime/shared/records.rs, which no PR in this stack changes; it passed in every other full run tonight). Every key-path test passed:surface::tests::hosted39,input_port_inline4,output_publications4,terminal_publisher4,server::tests::rds_input_lane13,server::tests::input_lane7,rds_switch::tests2.The full Rust suite on macOS at 91c7515 (fleet step e7740de8f33233fac49a30a4): 16814 tests, 13 failed. 11 are known base failures from every full macOS run tonight (5 acpmux tests, the 3
server::agent_transfer::tests, the 2 non-UTF-8 secret-name tests, the registry latency load flake). 2 were new tonight and pass alone:pty_mouse_tracking_forwards_click_release_and_wheel_with_shift_override(cmux-tui app; 2 of 2 at afadf63, steps 27ece3f3483747061b4f36da and c0ee9ea9f9d2fbb456d72dec; 2 of 2 on feat cd5abc2) anda_relative_diff_path_reaches_the_app_as_the_callers_folder(ENOENT when it started the binary under load; passes alone at afadf63, step 31cc6666d5a2f00960df2cfe). Every key-path test passed on macOS.The Chief owner's switch (review of afadf63), Swift, focused command
cmux-ci run --class light --script scripts/ci/package-test-lane.sh --arg=suite --arg=Packages/macOS/CmuxNext --arg=DaemonDiagnosticKnobsTests --ref SHA: red at ff22950 (step 2a7624e2ad01d780bf7d4640:theInputLaneSwitchReachesTheChiefOwnerfailed,(environment → ["HOME": "/h"]) == ["HOME": "/h", "CMUX_TUI_RDS_INPUT": "0"]); green at c8900c4 (step c7d3b11c540a61e9eb90a08a, 6 of 6). The delta has no Rust change.Real-app evidence: the s17ga typing journeys above (4 rounds on cmux21, the release app with this PR, all pass, arms proven in the daemon's environment). The title names no P0 item, so no separate display or smoke run applies. The What's New highlight (
release-notes/next/highlights/typing-never-waits-for-busy-sessions.md) passesscripts/whats-new/validate.pyin a digest of this commit.Clippy -D warnings at afadf63: passed (step b8ecf9e713de6c613a7227f4). Input-lane tests at afadf63 on Linux: 23 of 23 pass (step c1cc883a2ce006bcf5ae3c13:
server::tests::rds_input_lane13,server::tests::input_lane7,rds_switch::tests2, one app projection test). Compile: fleet job 2e28e23636f6ea77a63b8b28 at the head f69479f: CmuxNext, app and cmux-tui passed (earlier: dbb0f716613fcf33dffe40ea at c8900c4, a5751411fd1d2f016328b734 at afadf63, 7c5a59d26a66d232a6888af8 at 4fe7618, 2f7c791354ca674647a40f4e at 6bfdf88, 3d25868d6ebd83c5ff8fa03a at 0678016).Observed / Not run
lint (linux)is red only on the base findingacpmux: clippy::indexing_slicing 687 > baseline 686(a crate this PR does not touch).test (windows)was skipped. Video: judged for s17ga (above); the L6 pair is not run yet (above).Changelog
Proof
No capture: a daemon change with no UI. The typing A/B table above is the evidence.