Skip to content

cmux-tui headless owner: the terminal reaper and the Chief brain supervisor no longer spin on a past deadline (idle CPU) - #33173

Merged
cmux-bot[bot] merged 12 commits into
feat-cmux-nextfrom
lane/tui-reap-retained-backoff
Nov 3, 2026
Merged

cmux-bot[bot] merged 12 commits into
feat-cmux-nextfrom
lane/tui-reap-retained-backoff

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

The app's headless cmux-tui owner (cmux-tui --headless --session cmux-app-nightly) was seen at about 150% CPU for 40 minutes on an idle Mac, against a budget of no CPU at idle. Two background threads in that owner can run with a zero wait once a deadline is in the past and nothing moves it. Each is fixed here with a test that fails on base.

1. mux-terminal-reap: a reap attempt that leaves the terminal reapable re-runs at once

When a due terminal's reap came back Retained or Reaped, the reaper dropped its deadline (forget). If the next scan still listed the terminal, it re-added the deadline at now + grace - unplaced_for. Since #29476, unplaced_for is the durable time the terminal was first seen unplaced, which is already past the grace, so the new deadline was now.

The thread then waited zero (recv_timeout(0)) and ran the next pass at once, and kept doing so. Every pass costs:

  • a registry prune of keep and resume rows,
  • a SQLite sync_terminal_unplaced write,
  • a screen-archive capture,
  • a guarded close.

When several terminals are stuck this way, their attempts run in parallel on mux-terminal-work.

A terminal ends up stuck this way whenever the scan and the close guard disagree about whether it is placed. The scan matches a view's runtime id or the content's placements. The guard matches any surface with the same host identity.

Fix: ReapSchedule::tried postpones the deadline by grace.max(MIN_RETRY) after every outcome, as Attached and errors already did. A scan that no longer lists the terminal (closed, placed again, or kept) still drops the deadline in observe_unplaced, so terminals that were really reaped or re-placed behave as before.

2. chief-brains: a brain still typing after the idle stop is due leaves a past deadline

next_deadline offered the idle stop (last_activity + 30 min) for every running brain. on_time stops only a brain that is not typing. So a turn that kept its typing on for more than 30 minutes left a deadline in the past that nothing moved. The thread waited zero, did nothing, and ran again, holding one core at 100% until the turn ended.

Fix: a brain has no idle deadline while it is typing, which matches on_time. When the typing ends, handle sets last_activity, and the stop counts from that point.

Loops touched: stop conditions and cleanup

  • mux-terminal-reap:
    • Stop: the existing TerminalReaper::stop / drop (the stop flag plus closing the mailbox), and the thread ends when the Mux is gone.
    • Cleanup: terminal_reaper_events is cleared on exit (unchanged).
    • Waits now: at least MIN_RETRY (1 s) per stuck terminal, normally a full grace.
  • chief-brains supervisor:
    • Stop: Signal::Shutdown, or the inbox disconnecting (unchanged).
    • Waits now: until a signal, a backoff retry, or the idle stop of a brain that is not typing.
  • No new timers, observers or threads.

Tests (red on base)

  • terminal_reap::tests::terminal_reap_schedule_waits_after_an_attempt_that_left_the_terminal_reapable: one terminal unplaced for three graces, with a reap attempt that leaves it reapable. The test simulates the thread's waits for one second and counts the passes.
  • terminal_reap::tests::terminal_reap_schedule_drops_a_settled_terminal_once_it_is_no_longer_reapable
  • chief_brains::supervisor_idle_tests::a_brain_typing_past_the_idle_stop_leaves_no_past_deadline

Performance

metric before after delta harness or run link
reaper passes in 1 s, one terminal left reapable by its reap 1000 (the test's cap; the wait is 0, so unbounded) 1 -99.9% terminal_reap_schedule_waits_after_an_attempt_that_left_the_terminal_reapable
reaper wait after that pass 0 s 30 s (the grace) +30 s same test
chief-brains wait with a brain typing past 30 min idle 0 s (one core spinning) none until its typing ends no spin a_brain_typing_past_the_idle_stop_leaves_no_past_deadline
headless owner CPU at idle on the Mac that showed it ~148% (40 min) to be read after the next NEXT nightly sample after nightly

Related: #33099 fixes the "cmux-tui stopped and could not start again" toast, not this cause. #33117 (open) adds a cap on the reap grace in the same file; the two do not conflict.

Agent session: agent session

🤖 Generated with Claude Code


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Fixes two hot loops that spin the headless cmux-tui owner at ~150% CPU on idle: the terminal reaper and the Chief-brains supervisor can wait zero, doing nothing, when a deadline falls in the past and nothing moves it.

Bug Fixes

  • mux-terminal-reap: a reap attempt that leaves a terminal reapable now keeps a retry floor (the next deadline is the later of the scheduled one and the floor), so the terminal is retried after at least MIN_RETRY instead of being re-added as due at once. A terminal that a scan no longer lists drops its deadline and floor; one placed again and closed again gets a fresh grace from its new unplaced time.
  • chief-brains: a brain that is typing has no idle deadline (matching the idle stop in on_time), and the idle stop counts from when its typing ends, so a past deadline can no longer sit untouched and keep the thread spinning.
  • Adds four tests covering the waits; all fail on base.

Written for commit bb990df. Summary will update on new commits.

View guided diff Turn on auto-fix

teamleaderleo and others added 4 commits November 3, 2026 06:16
…ot make the reaper's next wait zero

Red on base: a due terminal whose reap comes back Retained (the close guard
still sees it placed) or Reaped while still listed loses its deadline
(forget); the next scan re-adds it at now + grace - unplaced_for, and since
#29476 unplaced_for is the durable time already past the grace, so the
deadline is now and the reaper thread waits zero and runs again: a hot loop
of registry prunes, an unplaced-time write, a screen capture and a guarded
close per pass.

The outcome handling moves into ReapSchedule::settle unchanged so the test
drives it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-be6d2cae00b3baee4240
…empt instead of re-running at once

A due terminal's attempt, whatever it came to, now postpones its deadline by
the grace (at least MIN_RETRY) instead of forgetting it. A scan that no
longer lists the terminal (closed, placed again or kept) still drops the
deadline, so a reaped or re-placed terminal behaves as before; one the
attempt left reapable is tried again a grace later, not on a zero wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-be6d2cae00b3baee4240
…e the supervisor a past deadline

Red on base: next_deadline offers the idle stop of every running brain,
but on_time stops only a brain that is not typing. A turn that keeps its
typing on for more than 30 minutes leaves a deadline in the past that
nothing moves: the chief-brains thread waits zero, does nothing and runs
again, one core at 100%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-be6d2cae00b3baee4240
…r a brain that is still typing

Its idle stop matches on_time's: a brain mid-turn is not stopped, so it has
no idle deadline until its typing ends (which sets last_activity).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-be6d2cae00b3baee4240
@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review fa06154: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr33173 -f ref=lane/tui-reap-retained-backoff -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); no finished base run on 4575ff4 to compare; do not merge until green; see lane-rules

Shipped in:

@github-actions github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Nov 3, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6.1-sol high, sweep, via the review router; full diff from merge base eeb28af to this head): SHIP at 9dba1e8

No P1/P2 findings.

@cursor

cursor Bot commented Nov 3, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@cmux-bot
cmux-bot Bot temporarily deployed to ci-routing-agent November 3, 2026 06:53 Inactive
@blacksmith-sh

blacksmith-sh Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Found 2 test failures on Blacksmith runners:

Failures

Test View Logs
test_every_compile_job_names_its_warm_sources (main.ThisTree.test_every_compile_job
_names_its_warm_sources)/
test_every_compile_job_names_its_warm_sources (main.ThisTree.test_every_compile_job
_names_its_warm_sources)
View Logs
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)/
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

teamleaderleo and others added 2 commits November 3, 2026 07:17
…l closed again gets its fresh grace

Review (Sol, BLOCK at 9dba1e8): postponing the deadline of a Retained
terminal let a terminal placed again and closed again before the next
scan keep the attempt's retry deadline, short of a fresh grace from its new
unplaced time. Reaped and Retained now forget the deadline as before and
keep a retry floor while the terminal stays reapable; the next deadline is
the later of the two. Attached and failed attempts postpone as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-be6d2cae00b3baee4240
…ad does

Review (Sol, BLOCK at f498f81): after the retry-floor change a Retained or
Reaped attempt leaves no deadline until the next scan, so the test's
expect on the next wait panicked. The test now scans once a millisecond for
a second, as topology and registry events wake the thread, and counts the
reap attempts: one with the floor, a thousand when each scan re-adds the
terminal at its durable unplaced time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-be6d2cae00b3baee4240
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high, next-review): SHIP at fa06154

SHIP at fa06154
Visual verdict: no images; Leo tries the build

  1. [P2] [follow-up] cmux-tui/crates/cmux-tui-core/src/mux/terminal_reap.rs:165: The capacity-limit path bypasses the new retry floor. With more than 16 reapable terminals, lines 407–409 add the oldest terminals to due regardless of their postponed deadlines. A stuck terminal therefore still incurs archive capture and guarded close on every topology wake. Make capacity-triggered attempts respect the retry floor while allowing an initial attempt before the normal grace expires, and add coverage with more than 16 terminals.

Source review completed; tests could not run because this box has no Rust toolchain.

Follow-ups:

  • Respect retry floors in capacity-triggered reap attempts and test that path.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6.1-sol high, sweep, via the review router; full diff from merge base 76505c9 to this head): SHIP at fa06154

No P1/P2 findings.

cmux-bot[bot]
cmux-bot Bot previously approved these changes Nov 3, 2026

@cmux-bot cmux-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cmux-bot approves fa06154: an independent review passed this exact head (next-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

@cmux-bot
cmux-bot Bot temporarily deployed to ci-routing-agent November 3, 2026 09:46 Inactive
@cmux-bot
cmux-bot Bot dismissed their stale review November 3, 2026 09:54

A newer verdict blocks d6258ef (no independent SHIP at the head).

@cursor

cursor Bot commented Nov 3, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@cmux-bot
cmux-bot Bot temporarily deployed to ci-routing-agent November 3, 2026 13:56 Inactive
@cmux-bot
cmux-bot Bot merged commit 7c41181 into feat-cmux-next Nov 3, 2026
153 of 156 checks passed
@cmux-bot
cmux-bot Bot deleted the lane/tui-reap-retained-backoff branch November 3, 2026 14:08

This branch was previously deployed

3 inactive deployments
artifacts — bb990dfd Deployed Nov 3, 2026 by cmux-bot[bot] via publish tree-addressed cmux-tui #106350
sentry-dsyms — bb990dfd Deployed Nov 3, 2026 by cmux-bot[bot] via upload cmux-tui dSYMs to Sentry #106350
ci-routing-agent — bb990dfd Deployed Nov 3, 2026 by cmux-bot[bot] via cmux-tui tree preflight #106350
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant