Repository navigation
cmux-tui: an app session's owner stops when its app leaves and nothing keeps it (R-LIFECYCLE stage 1, behind a switch) - #33999
Merged
Conversation
…nothing keeps it (red, R-LIFECYCLE stage 1) The test plays the app: a role-main client-hello, then it leaves. With CMUX_TUI_OWNER_KEEPERS=1 the owner must stop at once when no client, live terminal, browser host or app server keeps it, stay while a client or a live terminal does, and stay up for an ensure that no app ever reached. On the base the owner of an app session never stops (STATUS 1370, 8519). Design: handoff/rewrites/R-LIFECYCLE/DESIGN.md section 7.
…othing keeps it (R-LIFECYCLE stage 1, behind CMUX_TUI_OWNER_KEEPERS) `server ensure` starts an app session's owner (`cmux-app`, `cmux-app-*`) detached, so terminals outlive the app; with nothing to keep, that owner ran until reboot after a quit, a crash or a deleted bundle (STATUS 1370: 105 orphans on 8 hosts; STATUS 8519: the cmux20 nightly's owner after its quit). Once the app has connected (a role-main client-hello), the owner is kept only by a connected client, a live terminal (the Reopen hold and `keep` included), its running browser host or a running app server. When none holds, it stops as `server stop` does (stop_orphaned_owner, fenced: a client or terminal that arrives during the decision keeps it). An ensure no app ever reached (a probe, a script) leaves the owner as before. Event-driven, no timer: one thread blocks on the terminal topology events and is woken by client presence (and the role-main hello), browser host starts and stops, and app server starts and stops. It reads the keepers cheap-first (the registry only when nothing else keeps the owner). The switch CMUX_TUI_OWNER_KEEPERS=1 is deleted when the stage passes; that PR also arms the rule for cmux-chief-* and removes dev_orphan_exit.rs. Design: handoff/rewrites/R-LIFECYCLE/DESIGN.md sections 3.5 and 7.
Contributor
|
cmux status (one comment, edited in place; each line is updated by the job that owns it) Review App of this head (app.zip, kept 1 day)
red: cmux-next checks (god files, concurrency, crash safety, l10n); no finished base run on 861e50a to compare; do not merge until green; see lane-rules Shipped in:
|
Contributor
Author
|
Fleet panic ratchet at the head ffa3775 (job 16fb47255d762f9c6cc231cf, Linux): RED on a base finding, not this PR.
|
…ippy unused_qualifications, #33999) Linux clippy job a556fb89 (-D warnings) refused four full paths the modules already import: server/orphan_shutdown.rs:61, :68 and server/client_registry.rs:341, :411.
…te's RecoverPoison (#33999) The three new lock reads in apps.rs spelled out std::sync::PoisonError and std::sync::Mutex; they now use the crate's own RecoverPoison helper and the imported Mutex, as the rest of cmux-tui-core does.
lawrencecchen
temporarily deployed
to
ci-routing-agent
November 3, 2026 16:05 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
Bugbot needs on-demand usage enabledBugbot uses usage-based billing for this team and requires on-demand usage to be enabled. A team admin can enable on-demand usage in the Cursor dashboard. |
lawrencecchen
temporarily deployed
to
artifacts
November 3, 2026 16:16 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
lawrencecchen
temporarily deployed
to
sentry-dsyms
November 3, 2026 16:16 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
lawrencecchen
temporarily deployed
to
artifacts
November 3, 2026 16:29 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
Author
|
Checks at the head 6c65099:
So this PR causes no red of its own. Test proof (red on the base, green on the fix, both platforms) is in the description. |
…ep an app session's owner (red, #33999) Two holes the review of #33999 found in the keepers rule, as regression tests that fail on its head 6c65099: - owner_keepers::a_terminal_whose_host_is_lost_keeps_the_owner_through_its_respawn: the app leaves, then the shell is lost with its host. The respawn waits its backoff (held 6 s here) with the row exited; the owner must stay until the new shell runs, as it did before the rule. - owner_keepers::the_app_probe_stays_open_until_the_long_lived_connection_is_answered (cmux-daemon-client): the client's role-main client-hello probe arms the owner, so it must stay open until the daemon answered the long-lived connection; closed first, the owner had a moment with no client.
…p an app session's owner (#33999) The review of #33999 found two holes in the keepers rule (R-LIFECYCLE stage 1, behind CMUX_TUI_OWNER_KEEPERS): - A terminal whose new shell is on its way kept nothing. A lost host's respawn waits its backoff (0, 2, 10, 30, 60 s) with the row exited and only a Respawning mark; an on_exit shell terminal was marked only after the exit commit released its locks. The owner could stop in between and the shell came back only at the next app launch. Now the terminal keeper counts live rows plus Respawning marks under the registry lock (Mux::kept_terminal_count, one COUNT(*), not a snapshot), and the on_exit shell restart is planned in the exit commit's critical section, as the automatic respawn already was: from the commit on the terminal reads respawning, never neither. - cmux-daemon-client closed its role-main client-hello probe before its long-lived connection was registered, so the armed owner had a moment with no client. The probe now closes after the daemon answered the long-lived connection's first request. Also from the review: the stop is logged through the client log (lifecycle area, with the keepers it read) instead of stderr, which a detached owner sends to /dev/null; the switch is taken out of the process environment at start (no terminal inherits it) and passed only to the owner an ensure spawns; and the apps event sink reads the event name without a JSON index. Red at 51df394, green here: the owner_keepers fleet step (tests above, both platforms).
lawrencecchen
temporarily deployed
to
ci-routing-agent
November 3, 2026 18:07 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
lawrencecchen
temporarily deployed
to
artifacts
November 3, 2026 18:22 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
lawrencecchen
temporarily deployed
to
sentry-dsyms
November 3, 2026 18:22 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
lawrencecchen
temporarily deployed
to
artifacts
November 3, 2026 18:28 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
Author
|
Review (claude-opus-5-5 review subagent): SHIP at cb77d95 Independent review of the whole diff b146190..cb77d95 (read only; no build or test on the reviewer's machine). All six findings of the review of 6c65099 are fixed at this head. No major finding. Questions checked, with file:line evidence in the full report:
Minor findings (none blocks; the switch is off by default and not wired to the app):
Author's plan: these land in the stage's next PR (S1b), red test first for each, before the switch is wired to the app (handoff/rewrites/R-LIFECYCLE/DESIGN.md section 7). The reviewer also asked to check the Swift |
Contributor
Author
|
Checks at the head cb77d95: one non-green check, a cancelled base-infrastructure run, not this PR.
|
lawrencecchen
added
the
ready-to-land
|
Found 12 test failures on Blacksmith runners: Failures
...and 2 more test failures. View all on Blacksmith: run 41710318845, run 41710319049
|
![Fix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/fix-with-codesmith-light-3x.png)
lawrencecchen
added a commit
that referenced
this pull request
Nov 3, 2026
…) into r-lifecycle/s1b-keepers
lawrencecchen
added a commit
that referenced
this pull request
Nov 4, 2026
…n app session's owner (R-LIFECYCLE S1b) (#34188) * test(cmux-tui): an app session's owner stops when its app leaves and nothing keeps it (red, R-LIFECYCLE stage 1) The test plays the app: a role-main client-hello, then it leaves. With CMUX_TUI_OWNER_KEEPERS=1 the owner must stop at once when no client, live terminal, browser host or app server keeps it, stay while a client or a live terminal does, and stay up for an ensure that no app ever reached. On the base the owner of an app session never stops (STATUS 1370, 8519). Design: handoff/rewrites/R-LIFECYCLE/DESIGN.md section 7. * fix(cmux-tui): an app session's owner stops when its app leaves and nothing keeps it (R-LIFECYCLE stage 1, behind CMUX_TUI_OWNER_KEEPERS) `server ensure` starts an app session's owner (`cmux-app`, `cmux-app-*`) detached, so terminals outlive the app; with nothing to keep, that owner ran until reboot after a quit, a crash or a deleted bundle (STATUS 1370: 105 orphans on 8 hosts; STATUS 8519: the cmux20 nightly's owner after its quit). Once the app has connected (a role-main client-hello), the owner is kept only by a connected client, a live terminal (the Reopen hold and `keep` included), its running browser host or a running app server. When none holds, it stops as `server stop` does (stop_orphaned_owner, fenced: a client or terminal that arrives during the decision keeps it). An ensure no app ever reached (a probe, a script) leaves the owner as before. Event-driven, no timer: one thread blocks on the terminal topology events and is woken by client presence (and the role-main hello), browser host starts and stops, and app server starts and stops. It reads the keepers cheap-first (the registry only when nothing else keeps the owner). The switch CMUX_TUI_OWNER_KEEPERS=1 is deleted when the stage passes; that PR also arms the rule for cmux-chief-* and removes dev_orphan_exit.rs. Design: handoff/rewrites/R-LIFECYCLE/DESIGN.md sections 3.5 and 7. * fix(cmux-tui): name AtomicBool and Ordering through their imports (clippy unused_qualifications, #33999) Linux clippy job a556fb89 (-D warnings) refused four full paths the modules already import: server/orphan_shutdown.rs:61, :68 and server/client_registry.rs:341, :411. * fix(cmux-tui): the app server observers read their locks with the crate's RecoverPoison (#33999) The three new lock reads in apps.rs spelled out std::sync::PoisonError and std::sync::Mutex; they now use the crate's own RecoverPoison helper and the imported Mutex, as the rest of cmux-tui-core does. * test(cmux-tui): a respawning terminal and the client's hello probe keep an app session's owner (red, #33999) Two holes the review of #33999 found in the keepers rule, as regression tests that fail on its head 6c65099: - owner_keepers::a_terminal_whose_host_is_lost_keeps_the_owner_through_its_respawn: the app leaves, then the shell is lost with its host. The respawn waits its backoff (held 6 s here) with the row exited; the owner must stay until the new shell runs, as it did before the rule. - owner_keepers::the_app_probe_stays_open_until_the_long_lived_connection_is_answered (cmux-daemon-client): the client's role-main client-hello probe arms the owner, so it must stay open until the daemon answered the long-lived connection; closed first, the owner had a moment with no client. * fix(cmux-tui): a respawning terminal and the client's hello probe keep an app session's owner (#33999) The review of #33999 found two holes in the keepers rule (R-LIFECYCLE stage 1, behind CMUX_TUI_OWNER_KEEPERS): - A terminal whose new shell is on its way kept nothing. A lost host's respawn waits its backoff (0, 2, 10, 30, 60 s) with the row exited and only a Respawning mark; an on_exit shell terminal was marked only after the exit commit released its locks. The owner could stop in between and the shell came back only at the next app launch. Now the terminal keeper counts live rows plus Respawning marks under the registry lock (Mux::kept_terminal_count, one COUNT(*), not a snapshot), and the on_exit shell restart is planned in the exit commit's critical section, as the automatic respawn already was: from the commit on the terminal reads respawning, never neither. - cmux-daemon-client closed its role-main client-hello probe before its long-lived connection was registered, so the armed owner had a moment with no client. The probe now closes after the daemon answered the long-lived connection's first request. Also from the review: the stop is logged through the client log (lifecycle area, with the keepers it read) instead of stderr, which a detached owner sends to /dev/null; the switch is taken out of the process environment at start (no terminal inherits it) and passed only to the owner an ensure spawns; and the apps event sink reads the event name without a JSON index. Red at 51df394, green here: the owner_keepers fleet step (tests above, both platforms). * test(cmux-tui): a kept tab, a crash restart and an abandoned respawn keep an app session's owner (red, R-LIFECYCLE S1b) The re-review of #33999 at cb77d95 (SHIP) found three things the keepers rule (behind CMUX_TUI_OWNER_KEEPERS) does not count yet. Each as a regression test, red at this commit: - owner_keepers::a_kept_tab_of_an_ended_command_keeps_the_owner_until_it_closes: an `on_exit: keep` tab of an ended command lives only in this owner (a new owner detaches it); the owner stops at once today. - apps ...::an_always_server_waiting_for_its_crash_restart_keeps_the_owner: an `always` server in its crash-restart backoff counts as nothing, and a disable during that wait tells no observer. - mux ...::an_abandoned_respawn_wakes_the_terminal_lifecycle_watchers: abandon_terminal_respawn clears the Respawning mark and sends no event. No behavior change outside the tests: the apps server count and its observers move from AppsSlot into the apps Supervisor (still told on each `apps-server` broadcast), so the unit test reaches them. * fix(cmux-tui): a kept tab, a crash restart and an abandoned respawn keep an app session's owner (R-LIFECYCLE S1b) Three keepers the re-review of #33999 (SHIP at cb77d95) found missing, red at 13c217e: - An ended `on_exit` keep or shell terminal whose tabs and final screen only this owner holds: a new owner detaches them at its start (detach_exited_terminal_topology_only). kept_terminal_count now adds these, read under the same registry and state locks with that detach's own checks (a host loss, a store-kept tab and a terminal without tabs are not held here). Closing, restarting or reaping the tab ends it. - An `always` app server that crashed and waits for its restart: the apps Supervisor counts running servers plus pending crash restarts (kept_servers) and tells the owner's observers from emit() whenever that count changes, so a disable during the wait (no apps-server broadcast) also wakes the watch. The browser host in its restart backoff is not a keeper: its pages died with the crash, and it starts again on demand. - abandon_terminal_respawn sends TreeChanged, as the other ends of a respawn do, so the watch wakes when the mark goes. Behind CMUX_TUI_OWNER_KEEPERS, off by default. * test(cmux-tui): a late emit never hides the end of an app server from the owner (red, #34188 review finding 1) The review of #34188 at d06be39 (BLOCK) found that Supervisor::emit reads the kept-server count under the supervisor lock, but compares it with the last count told, and stores it, outside the lock. Two emits can read in one order and store in the other: a stale count stored last makes a later real change look like no change, so the owner's watch is never woken for it and the owner runs on with nothing kept. apps ...::servers::a_late_emit_never_hides_the_end_of_an_app_server_from_the_owner holds one emit after its read (a #[cfg(test)] pause in emit, armed for one thread), starts an on-demand server with a call, lets the held emit go on, then lets the server end by itself. The end must reach the observers; at this commit it does not. No product behavior change: only the test hook and the test. * test(cmux-tui): an old unreadable exited row leaves the kept terminal count readable (red, #34188 review finding 2) The review of #34188 at d06be39 found that the kept-tab part of Mux::kept_terminal_count reads every exited keep row the registry ever kept (no row is deleted), under the registry and state locks, on every wake and inside the stop fence. One row this build cannot read makes the count an error, and an unreadable count keeps an app session's owner forever. mux::pending_terminals::tests::an_old_unreadable_exited_row_leaves_the_kept_terminal_count_readable writes one exited keep row with an unreadable receipt and no runtime in this owner; the count must stay 0. At this commit it is an error. * fix(cmux-tui): the kept-server count is told in lock order, and the kept-tab count reads only this owner's runtimes (#34188 review) Finding 1 (major), red at 8a9a1b8: Supervisor::emit read the kept-server count under the supervisor lock but compared it with the last count told, and stored it, outside the lock. A stale read stored last made a later real change (a server's end) look like no change, so the owner's watch was never woken for it. Now emit compares and stores the told count (Inner::kept_servers_told) in the same locked block that reads the count, so the told count follows the lock order, and tells the observers after the unlock when it changed. The count is now tracked from the supervisor's first emit on, also before the observers are handed over, and AppsSlot wakes the watch once when it created the supervisor, since the `always` servers started before the handover and kept_servers read 0 until the slot held the supervisor. Finding 2 (minor), red at 26eeeab: the kept-tab count read every exited keep row the registry ever kept (no row is deleted), on every wake and inside the stop fence; one unreadable row made the count an error, which keeps the owner forever. Now the held tabs are read only when no live or respawning terminal keeps the owner, and the walk starts from this owner's terminal catalog: only a runtime that lives here can be held, so an earlier owner's rows are never read. * test(cmux-tui): the kept-terminal-count test writes its old row after the owner started; the emit pause has a type name (#34188) Two defects in the review-fix tests, found by the fleet at d3815ba: - an_old_unreadable_exited_row_leaves_the_kept_terminal_count_readable wrote its unreadable row before Mux::from_workspace_registry, which parses every row's receipt, so it failed in the constructor (tests.rs:52, "expected ident") at the red commit and at the fix alike: red for the wrong reason. Renamed the_kept_terminal_count_reads_no_row_this_owner_holds_no_runtime_for: it writes the row after the owner started, so only a read by the count can see it. Red proof on a throwaway branch with the old count code. - clippy -D warnings (type_complexity) refused the test-only emit_pause field type: it is now EmitPause. Test code only; product code is the same as at d3815ba. --------- Co-authored-by: cmux-bot[bot] <348531757+cmux-bot[bot]@users.noreply.github.com>
lawrencecchen
added a commit
that referenced
this pull request
Nov 4, 2026
…rashes or is killed with nothing to keep them (R-LIFECYCLE ORPHANS) (#35218) * test(cmux-tui): an app or Chief owner stops when its app leaves with no switch set; an unsettled signal exit keeps it (failing) R-LIFECYCLE ORPHANS. Fleet repro on FEAT DOGFOOD BUILD 3a43a131 (feat 0043806), cmux19 and cmux21: an app with no terminal quit by Cmd-Q, SIGTERM, SIGKILL or a crash leaves its cmux-app-* owner, its cmux-chief-* owner (each with its journal plugin) and the acpmux pair with parent 1, 30 s later and on. Only Quit and End Sessions ends them. The keepers rule (#33999, #34188) is behind CMUX_TUI_OWNER_KEEPERS=1, which no app sets. The new tests: - an app owner started without the switch stops when its app leaves; - a cmux-chief-* owner stops when its app leaves; - CMUX_TUI_OWNER_KEEPERS=0 leaves the owner running (guard, green on base); - S1c (DESIGN.md section 7): a signal exit within the session shutdown lead keeps the owner until it settles. * test(cmux-next): the app passes the owner keepers switch to its daemon (failing) * journeys: quit-ends-session-daemons: an app with no terminal that quits, crashes or is killed leaves no cmux-tui of its sessions 30 s later (failing) R-LIFECYCLE ORPHANS. Red on FEAT DOGFOOD BUILD 3a43a131 by the fleet repro (rlc-orphans-base-3..6): the cmux-app-* and cmux-chief-* owners and their screen detection plugins run on with parent 1 after Cmd-Q, a crash and SIGKILL. A journey for the R-LIFECYCLE plan, not a CI job. * fix(cmux-tui): an app's daemon and Chief daemon stop when the app quits, crashes or is killed with nothing to keep them R-LIFECYCLE ORPHANS (handoff/rewrites/R-LIFECYCLE/DESIGN.md). Lawrence's Mac had 380 cmux-tui processes with parent 1 from old tagged DEV apps. Fleet repro on FEAT DOGFOOD BUILD 3a43a131 (cmux19, cmux21, mini scheduler rounds rlc-orphans-base-3..6): an app with no terminal, quit by Cmd-Q, SIGTERM, SIGKILL or a crash, left its cmux-app-* owner, its cmux-chief-* owner and both screen detection plugins running 30 s later and on (only their own plugins were connected). Quit and End Sessions ended them. Cause: the keepers rule of R-LIFECYCLE stage 1 (#33999, #34188) stops an app session's owner on the event of its app's last connection closing (a quit, a crash and a kill all close it; no pid, no poll, no timer), but it ran only with CMUX_TUI_OWNER_KEEPERS=1, which no app sets, and it did not cover the Chief session. Fix: - The rule is on by default; CMUX_TUI_OWNER_KEEPERS=0 (the app's CMUX_NEXT_TUI_OWNER_KEEPERS=0, DaemonDiagnosticKnobs) turns it off for a dogfood report. An ensure passes only an off switch to its owner. - It covers cmux-chief-* owners too (the app's Home connects with the same role-main hello; a running brain host is a connected client). - S1c (DESIGN.md section 7): a signal exit whose detach waits out the session shutdown lead keeps the owner until it settles. The settle is scheduled under the registry lock of the exit commit, and kept_terminal_count reads it under the same lock; the settle run wakes the keepers watch. Before, a stop in that wait began a shutdown that made the person's kill a host loss, and the next owner respawned the shell. What stays after a quit, by design (pty-holder.md D1, DESIGN.md 3.5): with Keep Sessions and live terminals, the owner and its terminal hosts; a closed terminal in its Reopen hold keeps the owner until it is reaped; a working Chief brain keeps the Chief owner. The acpmux pair is R-LIFECYCLE S3 (PL-acpmux). Tests (red first: a82dab8, the knob test, the journey): - owner_keepers::an_app_owner_started_as_the_app_starts_it_stops_when_its_app_leaves - owner_keepers::a_chief_owner_stops_when_its_app_leaves - owner_keepers::the_switch_set_to_0_leaves_the_owner_running - owner_keepers::a_signal_exit_within_the_shutdown_lead_keeps_the_owner_until_it_settles - DaemonDiagnosticKnobsTests.theOwnerKeepersSwitchReachesTheDaemon - journey quit-ends-session-daemons (scripts/journeys, P1) * journeys: quit-ends-session-daemons: no first-frame text check (a fresh app shows its import onboarding); the last terminal comes from the script path * test(cmux-next): the owner keepers switch reaches the Chief owner's ensure (failing; review B1 of #35218) * fix(cmux-next, cmux-tui): the keepers off switch reaches the Chief owner; a settle no worker runs keeps no owner (review of #35218) - B1: DaemonLauncher.chiefEnvironment kept only a short list, so CMUX_NEXT_TUI_OWNER_KEEPERS=0 never reached the cmux-chief-* owner. It now passes CMUX_TUI_OWNER_KEEPERS from the app's knob (red test theOwnerKeepersSwitchReachesTheChiefOwner, previous commit). - m1: a settle deadline that no worker will run (owner not bound yet, or the worker failed to start) no longer keeps the owner forever; the next owner's start settles it. - The S1c test names its debug-build seam; the journey says it expects agents off. * journeys: quit-ends-session-daemons also forbids acpmux after a quit with no terminal, and checks a Keep quit with one terminal (only its daemon and host stay) * test(cmux-tui): a running or pending supervised Chief brain keeps the Chief owner (failing; review of #35218) * fix(cmux-tui): a supervised Chief brain that runs or waits to start keeps the Chief owner (review of #35218) The Chief owner (cmux-chief-*) supervises the brains of the Chiefs that are not the default. Its stop SIGTERMs them, which cuts a running turn and drops a waiting start. A brain is a client of the owner only while its link is up, so a reconnect gap, the window before a new brain connects, or a restart backoff after the app left let the owner stop and cut the turn (independent review BLOCK at effca4c). Fix: the supervisor publishes what keeps the owner (Slot::kept): brains with a process or a pending restart, plus signals sent to it and not yet handled (each may start a brain). The owner's keepers read it cheap-first; the supervisor wakes the watch when the count changes (client presence observers). Brains still stop after 30 min idle, so the owner then stops. Signals kept for a supervisor that never starts are uncounted. Red test 22c8a88: owner_keepers::a_supervised_chief_brain_keeps_the_chief_owner_until_it_ends. * journeys: quit-ends-session-daemons: the last terminal comes from the focused socket path (a script's newTab takes no keys in a slot) * journeys: quit-ends-session-daemons: the Keep quit with one terminal first (the P0-proven Ctrl-` path in the fresh app), then End Everything, then Cmd-Q, crash and SIGKILL with no terminal * journeys: split the quit daemons journey in two so each fits one lease: quit-keep-leaves-terminal-daemons and quit-ends-session-daemons * journeys driver: a reopen whose loaded topology has no terminal tab is settled at once (it waited out RELAUNCH_SECONDS, 90 s); the quit journeys' why texts name their own case
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
After the cmux app quits, crashes or is deleted, the owner of its daemon session (
cmux-tui --headless --session cmux-app-*) runs until the machine restarts, even when it holds nothing. The fleet had 105 such leftovers on 8 hosts (STATUS 1370), and the cmux20 nightly's owner stayed after its app quit (STATUS 8519).With
CMUX_TUI_OWNER_KEEPERS=1, the owner of an app session now stops the moment its app has left and nothing else keeps it. The owner is kept by:keep;on_exitshell terminal;The stop is the one
server stopuses (stop_orphaned_owner): it ends no terminal, and it is fenced, so a client or a terminal that arrives during the decision keeps the owner. The rule arms only after the first role-mainclient-hello(the app's connection). A probe's or a script'sensurethat no app ever reached leaves the owner as before. The nextensurestarts a new owner, which restores the session from disk. The stop is written to the owner's client log (lifecyclearea) with the keepers it read.The rule is event driven, with no timer. One thread blocks on the terminal topology events and is woken by client arrivals and departures, the first role-main hello, browser host starts and stops, and app server starts and stops. It reads the cheap keepers first, and reads the registry (one
COUNT(*)of live rows plus the respawn marks, under the registry lock) only when nothing else keeps the owner.Changes from the review of 6c65099 (commits 51df394 red, cb77d95 fix):
on_exitshell restart is now planned in the exit commit's critical section, as the automatic respawn already was, so from the commit on the terminal reads respawning, never neither.cmux-daemon-clientkeeps its role-main hello probe open until the daemon answered the long-lived connection. Before, it closed the probe first, so an armed owner had a moment with no client.ensurespawns.This is stage 1 of R-LIFECYCLE (one supervision tree): handoff/rewrites/R-LIFECYCLE/DESIGN.md, sections 3.5 and 7. The switch is deleted when the stage passes. That later PR also arms the rule for
cmux-chief-*sessions and removesheadless/dev_orphan_exit.rs(its 60 min delay and 10 min re-check).Accepted for this stage, to measure in dogfood before the rule is on by default (DESIGN.md section 7): no grace after a disconnect (an app whose only connection drops, with nothing else kept, gets a new owner on reconnect; state is durable), and a relaunch at once after a quit can meet the old owner still stopping (the app's retry reaches the next owner, as after
server stoptoday).Testing
Red first for each change:
tests/owner_keepers.rs(commit 6c100eb), then the two regression tests from the review (commit 51df394:owner_keepers::a_terminal_whose_host_is_lost_keeps_the_owner_through_its_respawnin cmux-tui,owner_keepers::the_app_probe_stays_open_until_the_long_lived_connection_is_answeredin cmux-daemon-client), each on its own before its fix. Command:cmux-ci run --class isolated --script scripts/ci/cmux-tui-rust-check.sh --arg test --arg owner_keepers --os linux|macos --ref SHA.test owner_keepers, Linuxprobe_open_at_first_request: false(the probe closed before the long-lived connection's first request). Respawn: "the owner stopped while a terminal whose new shell waits its backoff kept it"test terminal_respawn(includes theon_exitshell test), Linuxon_exit_shell_drops_the_tab_into_a_shell_after_the_commandand the manual restart testsclippy, LinuxThe four first Linux runs (5d904a4e, b95c11ba, 48da1103, dcc47e56) ended "cancelled" at GitHub's 1 h job limit with no test result: in the hq
ci-step overflowjob, the step "Require the reviewed script version" runsgit fetch --no-tags --unshallow origin, which fetched every branch of this repo and took 51 min (18:08:16 to 18:59:36Z), so cargo had 8 min. The same step took 18 to 52 min in every overflow run sampled since 16:33Z (an hq workflow defect, not this PR).Review: an independent review subagent gave SHIP at cb77d95 (receipt comment on this PR); its minor findings go to the stage's next PR (S1b).
Head 3ca3323 is next-autoland's merge of feat-cmux-next (cmux-bot, 20:11Z). It changes no line of this PR's diff; feat added 8 lines to 2 of its files (
server.rs: thechief_workmodule and its command;client_hello.rs: one page command name). Proof at that head: macOStest terminal_respawn01e09e4a andtest owner_keepersa31d53ed (running).Observed / Not run
clippy (changed cmux-tui packages),rust workspaces,cargo check (x86_64-pc-windows-gnu), the 4buildtargets andcmux-next crash ratchetpass.action export areas(cmux-next PR autofix) was cancelled insideactions/checkoutby its old 5-minute limit; feat-cmux-next already raised it to 15 minutes (a base finding, see the PR comment).cmux-ci log --full <id>); Linux clippy at 6c65099 passed (2fadd98f); the prepush gate at cb77d95 (no new violations).acpmux: clippy::indexing_slicing 688 > 686(0671f875), fixed by base red: MCP connect reads and writes its card JSON without an index (panic ratchet indexing_slicing 703 > 686) #34032, not merged yet; this PR touches no acpmux file. I run the panic step again after base red: MCP connect reads and writes its card JSON without an index (panic ratchet indexing_slicing 703 > 686) #34032 merges.CMUX_NEXT_TUI_OWNER_KEEPERSis not wired as an app knob in this PR; the stage's next PR turns the rule on for app sessions and runs the quit journey).Changelog
none
Proof
No capture: daemon lifecycle only, behind a runtime switch that is off by default; no UI changes.