Skip to content

cmux-tui: terminal host accept loop blocks on events (idle host woke 50x/s) - #16864

Merged
lawrencecchen merged 2 commits into
mainfrom
fix-cmux-tui-terminal-host-accept-wakeups
Oct 2, 2026
Merged

lawrencecchen merged 2 commits into
mainfrom
fix-cmux-tui-terminal-host-accept-wakeups

Conversation

@lawrencecchen

@lawrencecchen lawrencecchen commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

An idle terminal host woke about 50 times a second. The VM image lead measured it on a Freestyle VM baked from main (37ee6af9846): strace showed 488 poll(listener, 20 ms) timeouts and 489 accept4 = EAGAIN in 10 s on the warm template host's main thread.

Root cause: the host's accept loop polled its listener with a 20 ms timeout only to re-check dead and the launch-owner deadline, for the life of every terminal.

Fix (the terminal-host part of feat-cmux-next's 51b6863): the loop polls the listener plus an accept waker (a nonblocking socket pair that terminal exit and the last client stream's close write to). The launch-owner deadline is a one-shot timeout used only until it passes. The forced PTY drain also stops polling a hung-up drain waiter.

Test first (red then green): idle_template::parked_and_adopted_template_hosts_do_not_wake_while_idle parks a warm template host the way the bake does, counts its context switches over 10 s, then adopts it with a fresh-identity daemon and counts again. Each window allows 5 switches.

  • Red on main + this test: 498 switches parked, 498 adopted (hosted run 37009612379).
  • Green with the fix on feat-cmux-next: hosted run 37010846549 (Linux and macOS).

Depends on #16862 (main currently fails cargo fmt, which stops the hosted cmux-tui test jobs).

Changelog

Fixed: an idle terminal no longer wakes its terminal host process 50 times a second.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Summary by cubic

Stops an idle terminal from waking its terminal host process about 50 times a second by making the host's accept loop block on events instead of polling.

  • The accept loop polled its listener with a 20 ms timeout only to re-check dead and the launch-owner deadline; it now blocks on the listener plus an accept waker.
  • Terminal exit and the last client stream's close wake the loop through the waker, and the launch-owner deadline is now a one-shot timeout used only until it passes.
  • The forced PTY drain also stops polling a hung-up drain waiter, which stayed readable and busy-looped the rest of the drain window.
  • Adds a test that parks a warm template host, counts its context switches over 10 s, adopts it, then counts again, allowing 5 switches per window.

Written for commit 8fc5e56. Summary will update on new commits.

Review in cubic

Summary by CodeRabbit

  • Bug Fixes
    • Improved terminal host recovery after the daemon restarts, helping adopted terminals return to a running state.
    • Reduced unnecessary background activity while terminal hosts are idle, including after a client closes its connection.
    • Prevented repeated wake-up events from disrupting terminal host shutdown and recovery.

@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@coderabbitai

coderabbitai Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

🧰 Additional context used
📚 Code guidelines (2)
docs/cloud-guest-upgrades.md — configured
.github/review-bot-rules/source-control-artifacts.md — configured

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 08d95ae6-43e6-489f-9764-99c31a305a18

📥 Commits

Reviewing files that changed from the base of the PR and between b9ca453 and 8fc5e56.

📒 Files selected for processing (3)
  • cmux-tui/crates/cmux-tui-core/src/terminal_host_runtime.rs
  • cmux-tui/crates/cmux-tui/tests/terminal_host_recovery.rs
  • cmux-tui/crates/cmux-tui/tests/terminal_host_recovery/idle_template.rs

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 2 remain after this review.


📝 Walkthrough

Walkthrough

The runtime changes PTY forced-drain polling and replaces periodic host-listener polling with listener and lifecycle-waker polling. A recovery test measures context switches for parked and adopted template hosts on Linux and macOS.

Changes

Terminal host runtime

Layer / File(s) Summary
Forced-drain waiter handling
cmux-tui/crates/cmux-tui-core/src/terminal_host_runtime.rs
The drain poller stops polling a closed waiter descriptor during forced drain.
Accept-loop wakeups
cmux-tui/crates/cmux-tui-core/src/terminal_host_runtime.rs
The runtime adds an AcceptWaker and signals it when the host exits or the active client-stream count reaches zero. The accept loop polls the listener and waker, waiting until the launch-owner deadline before resolution and indefinitely afterward.
Parked and adopted host measurement
cmux-tui/crates/cmux-tui/tests/terminal_host_recovery.rs, cmux-tui/crates/cmux-tui/tests/terminal_host_recovery/idle_template.rs
The recovery test measures context-switch counts over 10-second windows for parked and adopted hosts. It requires both counts to be at most five and waits for host-record cleanup after closing the surface.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant HostShared
  participant ActiveClientStream
  participant AcceptWaker
  participant HostAcceptLoop
  HostShared->>AcceptWaker: wake after host exit
  ActiveClientStream->>AcceptWaker: wake when active-stream count reaches zero
  AcceptWaker-->>HostAcceptLoop: reader descriptor becomes ready
  HostAcceptLoop->>AcceptWaker: drain pending bytes
Loading

Suggested reviewers: austinywang

Merge Risk: ⚪ Minimal · up to 8fc5e

The idle-host change is ready to merge after normal checks; no actionable risk remains identified.

Security Architecture Review

Security architecture risk: 🔵 Low · up to 8fc5e

The change is contained within terminal-host waiting and does not establish a new access path or weaken existing authentication. Remaining uncertainty concerns cleanup reliability under exceptional notification failures.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The demonstrated change is scoped to each terminal host's local lifecycle and drain behavior. The new socket pair has no externally bound endpoint and does not establish additional service, tenant, or credential authority.

Trust Boundaries and Controls

  • observed — The existing local listener uses private-directory preparation and mode 0600. Client authentication checks terminal identity and either the owner token or an accepted capability before command dispatch; sensitive terminal operations retain capability-right checks. These controls predate the waiting change.

Resilience and Maintainability Implications

  • inferred — For successful or buffer-full notifications, pending bytes preserve wakeups across the state-check-to-poll interval, and draining precedes the next state evaluation. This supports cleanup under ordinary concurrent exit and final-stream closure without requiring one notification per transition. It does not establish recovery from every discarded write error.

Hardening Proposals

  • proposed — Consider explicitly distinguishing a full notification buffer from interrupted or unexpected wake-write failures, with recovery behavior and focused fault-injection coverage for cleanup. This is additional assurance for indefinite waiting, not an established attacker-triggerable regression.
🚥 Pre-merge checks | ✅ 24 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 16 functions across 3 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (24 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the terminal host accept-loop change and the idle wakeup problem.
Description check ✅ Passed The description clearly explains the problem, root cause, implementation, regression test, test results, dependency, and changelog entry. It omits the template's explicit Demo Video and Checklist sect…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Cmux Cloud Persistent Session And Early Input ✅ Passed PASS. The diff only changes the terminal host's internal PTY drain polling and accept-loop wake mechanism, plus recovery tests. AcceptWaker is one lifecycle wake pair per host, not a new client, aut…
Cmux Swift Actor Isolation ✅ Passed PASS: The authoritative PR diff changes only three Rust files (terminal_host_runtime.rs and two Rust test files). It introduces no Swift production changes, so it cannot introduce or worsen the spec…
Cmux Swift Blocking Runtime ✅ Passed PASS: The authoritative PR diff changes only three Rust files (.rs) and contains no Swift paths or Swift runtime constructs. The Swift-specific blocking-runtime check is therefore not applicable.
Cmux Browser Automation Off-Main ✅ Passed PASS: The authoritative PR diff changes only Rust terminal-host runtime and recovery-test files. It adds AcceptWaker, blocking poll behavior, PTY drain handling, and idle-host tests. It does not c…
Cmux Expensive Synchronous Load ✅ Passed The authoritative PR diff changes only Rust files under cmux-tui; it contains no Swift files or production Swift changes. Therefore the Swift expensive synchronous load check is not applicable.
Cmux Cache Substitution Correctness ✅ Passed PASS: The pull request changes only three Rust files (.rs). It introduces no production Swift, TypeScript, or JavaScript changes, so the cache-substitution correctness check is not applicable.
Cmux No Hacky Sleeps ✅ Passed PASS: The authoritative PR diff changes only three Rust files (.rs). The custom check applies to TypeScript, JavaScript, shell, and non-Swift build/runtime scripts. It does not apply to these Rust r…
Cmux Algorithmic Complexity ✅ Passed The production diff adds a fixed two-descriptor poll set for the listener and AcceptWaker, plus a bounded 64-byte wake drain. It does not add nested scans, repeated sorting/filtering, per-target r…
Cmux Swift Concurrency ✅ Passed PASS: The review-scoped diff changes only Rust files under cmux-tui and adds no Swift code or Swift concurrency patterns. The Swift concurrency check is therefore not applicable.
Cmux Swift @Concurrent ✅ Passed PASS: The reviewed diff changes only three Rust files and contains no Swift files. The Swift @concurrent check is not applicable.
Cmux Swift Package Boundaries ✅ Passed The pull request changes only Rust production code and Rust tests under cmux-tui. It introduces no Swift or SwiftPM files, so the Swift package boundary rule does not apply.
Cmux Swiftpm Lockfiles ✅ Passed The pull request changes only Rust runtime and Rust test files. The authoritative diff contains no Package.swift, Package.resolved, Xcode project/workspace, .gitignore, workflow, or dependency files. …
Cmux Swift Logging ✅ Passed PASS: The review-scoped diff changes only three Rust files (.rs). It adds no Swift files or Swift logging statements, so the Swift logging rule does not apply.
Cmux User-Facing Error Privacy ✅ Passed The changed production code only adds event-driven polling, wake descriptors, and lifecycle handling. It adds no user-facing error, alert, command output, API error body, or recovery copy. The new Fre…
Cmux Full Internationalization ✅ Passed PASS. The authoritative diff changes only Rust terminal-host runtime behavior, internal comments/state, and a recovery test. It adds no Swift UI text, string-catalog or Info.plist entries, web UI/API …
Cmux Swiftui State Layout ✅ Passed PASS: The pull request changes only three Rust files. The authoritative diff contains no Swift or SwiftUI paths and no SwiftUI state/layout constructs. The SwiftUI-specific failure conditions are not …
Cmux Architecture Rethink ✅ Passed PASS. The authoritative PR diff changes only three Rust .rs files and contains no Swift, Objective-C, or SwiftUI/AppKit bridge code. The rule explicitly applies to Swift architecture changes, so its…
Cmux Swift Auxiliary Window Close Shortcuts ✅ Passed PASS: The authoritative pull-request diff changes only three Rust files under cmux-tui. It contains no Swift changes and adds or modifies no NSWindow, NSPanel, NSWindowController, SwiftUI Window, or W…
Cmux Source Artifacts ✅ Passed All three changed paths are intentional Rust source or test files. The new file is a text-only recovery test under cmux-tui/crates/cmux-tui/tests/terminal_host_recovery/ and is wired into the existi…
Cmux No Test Or Debug Seam In Production Source ✅ Passed The pull request changes only Rust files under cmux-tui/crates/.../src and tests. It changes no Swift file under a production **/Sources/** path, so this custom check is not applicable.
  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 2
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR
🛠️ Fix failing CI checks 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

lawrencecchen and others added 2 commits October 2, 2026 07:05
…hile parked or adopted

A Freestyle VM baked from main (cmux-tui 37ee6af) measured the warm
template terminal host (__terminal-host --bootstrap-stdio, started at bake,
adopted by the clone's daemon) at about 50 wakeups per second: its accept
loop called accept4 (EAGAIN) and poll(listener, 20 ms) for the life of the
terminal (strace: 488 polls and 489 accept4 in 10 s). The next commit brings
feat-cmux-next's fix (51b6863) to main. This test fails now: 498 context
switches parked and 498 adopted in 10 s each (hosted run 37009612379).

The existing idle test covers hosts the daemon launches; this one covers the
bake path: park a template host (fenced shutdown, state wiped), count its
context switches over 10 s, adopt it with a fresh-identity daemon, and count
again. Each window allows 5 switches (a 1 Hz wake gives at least 10).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… polling

Root cause (see the previous commit): the host's accept loop polled its
listener with a 20 ms timeout only to re-check `dead` and the launch-owner
deadline, for the whole life of every terminal, so an idle host woke about
50 times a second (a Freestyle VM baked from main: 488 polls in 10 s).

The loop now polls the listener plus an accept waker, a nonblocking socket
pair that terminal exit and the last client stream's close write to; the
launch-owner deadline is a one-shot timeout used only until it passes. The
forced PTY drain also stops polling a hung-up drain waiter, which stayed
readable and busy-looped the rest of the drain window.

This is the terminal-host part of feat-cmux-next's 51b6863 (cherry-picked
from that commit's terminal_host_runtime.rs only; the rest of that commit
depends on code main does not have), so the VM image stops waking.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@lawrencecchen
lawrencecchen force-pushed the fix-cmux-tui-terminal-host-accept-wakeups branch from 3bbf430 to 8fc5e56 Compare October 2, 2026 14:05
@lawrencecchen
lawrencecchen merged commit 9a332ec into main Oct 2, 2026
101 of 103 checks passed
@lawrencecchen
lawrencecchen deleted the fix-cmux-tui-terminal-host-accept-wakeups branch October 2, 2026 14:33
@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Merge receipt for 8fc5e560fc: every check was green at merge (13 verified; 20 skipped by policy). Full suite runs on main after merge.

rustybret pushed a commit to rustybret/bmux that referenced this pull request Oct 2, 2026
67001f1 Update Cloud sidebar tests for the redesign and deferred workspace selection (manaflow-ai#16882)
9a332ec cmux-tui: terminal host accept loop blocks on events (idle host woke 50x/s) (manaflow-ai#16864)
7ba85b7 Merge pull request manaflow-ai#16781 from manaflow-ai/cloud-sidebar-followups
d6a22e7 Merge remote-tracking branch 'origin/cloud-new-machine-top' into repair-pr16781
a3e78a6 refresh hidden cloud detail pools
6cac783 Merge remote-tracking branch 'origin/cloud-new-machine-top' into repair-pr16781
396571d Merge remote-tracking branch 'origin/cloud-new-machine-top' into repair-pr16781
6e324f4 Merge remote-tracking branch 'origin/cloud-new-machine-top' into repair-pr16781
b053083 remove obsolete fleet freshness argument
9f49df6 Merge remote-tracking branch 'origin/cloud-sidebar-followups' into repair-pr16781
08c1463 remove duplicate fork monitor helper
77181d7 fix: remove duplicate fork monitor argument builder
ee62151 use shared fork monitor contract in CLI
bbccaec Merge remote-tracking branch 'origin/cloud-new-machine-top' into repair-pr16781
8cd5139 merge cloud row cleanup
4765a31 Merge remote-tracking branch 'origin/cloud-new-machine-top' into repair-pr16781
d649d29 restore cloud sidebar locale translations
1452e4c Merge remote-tracking branch 'origin/cloud-sidebar-followups' into repair-pr16781
45c2cbd Merge remote-tracking branch 'origin/cloud-new-machine-top' into repair-pr16781
dd52822 fix(localization): avoid stacked merge duplicate locales
09ed71b fix tab density enum conformance
f6595f5 merge cloud machine creation gate
dc7dd14 merge cloud sidebar base and preserve follow-ups
1b83129 fix(ci): restore shared Codex fork monitor helper
a713a66 review fixes: a wide tab floor narrows the other mode bar tabs instead of overflowing, new strings in all 20 locales
969516b refresh cloud machines back to the bottom right of the panel
04331c1 right sidebar min (and opening) width 295, just enough for the machine tabs with counts
c4240cf cloud sidebar follow-ups: displays tab, new workspace on top, refresh row above my devices, tabs never clip
lawrencecchen added a commit that referenced this pull request Oct 2, 2026
…eups) (#17015)

* cmux-tui: failing test, an idle daemon wakes up periodically

Starts a headless daemon with one terminal running cat, keeps an event
subscription and a terminal attach open, and counts context switches and CPU
of the daemon and the terminal host over a quiet 10 s window. The terminal
host part is fixed on main (#16864); the daemon still polls (on a Cloud VM:
main thread 241, journal 121, session journal 61 switches per minute).

(cherry picked from commit dc5cb26 on feat-cmux-next)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* cmux-tui: the daemon blocks on events instead of polling (no idle wakeups)

Port of the daemon part of 51b6863 from feat-cmux-next to main. The
terminal host accept loop part is already on main (#16864).

Root cause: daemon loops waited with a fixed timeout only to re-check a flag,
or retried a persistent error at once. On a Cloud VM an idle daemon's main
thread made 241, its journal thread 121 and its session journal thread 61
voluntary context switches per minute.

Polls removed (each now waits for the event that can change the result):
headless main loop (250 ms recv_timeout; now a condvar woken by signals,
shutdown requests and the remote runtime's end), stream threads (100 ms
recv_timeout; now a StreamInterrupt fired by writer close, stream close and
attach cancel), session event and journal streams (1 s epoch waits), journal
fanout tailer, journal hook dispatcher, journal plugin supervisor (condvar
plus a waitid(WNOWAIT) exit thread), idle-close reaper (sleeps until the next
policy deadline), remote runtime bootstrap and browser proxy parent waits.
Spins removed: accept loops retried EMFILE/ENFILE/ENOBUFS at once (shared
capped jittered Backoff), Kitty budget worker re-ran an identical wave, local
PTY reader slept 1 ms on WouldBlock.

Not ported: the terminal reaper deadline deferral and the connection
scheduler cancel type (their files do not exist on main).

(cherry picked from commit 51b6863 on feat-cmux-next, conflicts resolved)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* cmux-tui: review fixes for the daemon wakeup port

- Session event and session journal streams also stop when their outbound
  closes alone (a victim of a full connection queue); before, the fired
  interrupt made the wait return at once and the loop spun until the next
  journal event.
- Idle-close reaper re-evaluates at once after any due terminal (a failed
  close included), so a failed close is retried after one more period; it
  used to wait for a policy change or a detach.
- AttachTap::try_send drops its queue lock before cancel(), whose interrupt
  wakers lock the same queue.
- The parent-exit watch runs on a detached thread instead of spawn_blocking,
  so dropping the runtime on shutdown no longer waits for the parent to exit.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* cmux-tui: drop a needless path qualification (clippy)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* cmux-tui: rustfmt

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant