Skip to content

cmux-next: a quit in the Swift test helper no longer ends the whole test run - #23658

Merged
teamleaderleo merged 8 commits into
feat-cmux-nextfrom
chief-sidebar-2/test-helper-exit
Oct 22, 2026
Merged

teamleaderleo merged 8 commits into
feat-cmux-nextfrom
chief-sidebar-2/test-helper-exit

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

The CmuxNext Swift test run (cmux-next swift test) has ended with status 0 partway through since at least 2026-10-20 14:35Z. Hundreds of tests are left "started and never finished" (742 in #23599's job 119381263992), with no crash report and no signal. WindowServer logs an ordinary process death. HomeOpenFocusesComposerTests is usually the last suite to report, because its click helper spins the main run loop. About 400 PR swift-test jobs ran since 10-19 12:00Z, and one succeeded. Base pushes skip this job by design (cmux-next.yml:2360), so base never showed it.

Likely cause. Three quits were scheduled on the main run loop as a bare NSApp.terminate(nil):

  • QuitCoordinator.requestQuit
  • UpdateDropBack.relaunchAfterExit
  • DebugUpdater.terminate

In a process whose app never ran (the Swift test helper), AppKit's terminate is exit(0). It fires as soon as any test spins the main run loop.

Fix

  • NSApplication.terminateFromRunLoop (CmuxNextCompat) schedules the quit exactly as before when the app is running. When the app never ran, it refuses, and prints the call site and the stack that asked on stderr (also kept in refusedTerminations). The three sites use it.
  • CmuxNextTestExitTrace is a test-only C target linked into CmuxNextAppTests. Every exit() of the test process prints its stack, so if anything else ends the run, the CI log names the caller.
  • tests/test_cmux_next_deferred_terminate.py (linux-guard, run in cmux-next guards) keeps bare deferred terminates out.

Tests: TestProcessExitTests. A quit asked in the test process is refused and names its call site; before this change, that test ended the whole run. A second test checks that the exit tracer is installed.

The caller that triggers it in the suite will be named here once this PR's swift-test log shows the refusal line.

No capture: no UI change (quit plumbing and test support).

Agent session: cosmic-gray

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Fixes the cmux-next swift test run ending with status 0 partway through since at least 2026-10-20, leaving hundreds of tests started but never finished with no crash report. A bare NSApp.terminate(nil) scheduled on the main run loop became exit(0) in the test helper process, whose app never ran, firing as soon as any test spun the run loop.

Adds NSApplication.terminateFromRunLoop (CmuxNextCompat): it schedules the quit exactly as before when the app is running, but refuses when it is not, printing the call site and the asking stack to stderr and recording it in refusedTerminations. The three quit sites — QuitCoordinator.requestQuit, UpdateDropBack.relaunchAfterExit, DebugUpdater.terminate — now use it. Test-only CmuxNextTestExitTrace prints the stack of every exit() of the test process so any other exit caller is named in CI logs; it installs only when tests actually run, staying quiet during --list-tests/--dump-tests-json. A new linux-guard (test_cmux_next_deferred_terminate.py) keeps bare deferred terminates out. TestProcessExitTests verifies a quit in the test process is refused with its call site named, and that the exit tracer is installed.

No UI change; quit plumbing and test support only.

Written for commit c139c30. Summary will update on new commits.

View guided diff Turn on auto-fix

teamleaderleo and others added 2 commits October 21, 2026 12:18
The CmuxNextAppTests run has ended with status 0 mid-way since at least
2026-10-20: hundreds of tests started and never finished, no crash
report. Three quits were scheduled on the run loop as a bare
NSApp.terminate, which is exit(0) in a process whose app never ran, as
soon as any test spun the main run loop.

NSApplication.terminateFromRunLoop (CmuxNextCompat) schedules the quit
as before, but refuses when the app is not running and prints the call
site and the asking stack. QuitCoordinator.requestQuit,
UpdateDropBack.relaunchAfterExit and DebugUpdater.terminate use it. A
test-only C target prints the stack of every exit() of the test process,
so any other caller is named in the CI log. A guard keeps bare deferred
terminates out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15d18-9c97-76c3-88d7-6d0422d2fadc&host=cosmic-gray
@teamleaderleo teamleaderleo added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 21, 2026
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review c139c30: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr23658 -f ref=chief-sidebar-2/test-helper-exit -f platform=macos -f backend_mode=local

Passes: CI passes on c139c3074d.

CI passes on c139c3074d (run 39716194328 attempt 1).

Written by scripts/ci/classify_failures.py (ci-failure-attribution.yml); signatures are its SIGNATURES table. A machine verdict is the runner's fault, not this PR's; yours means the failing file is one this PR changes, also red on main that main's latest full suite fails the same way, seen on other PRs that it failed on another pull request's run lately.

red: cmux-next checks (god files, concurrency, crash safety, l10n); no finished base run on 118beeb to compare; do not merge until green; see lane-rules

Tour of c139c3074d (run): no media. The tour job failed before the tour ran (build or setup); the run log has the output.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by chief-sidebar-2 on cosmic-gray · session

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Bead cx-j4ymc created for this PR by the cmux coordinator.

@blacksmith-sh

blacksmith-sh Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Found 12 test failures on Blacksmith runners:

Failures

Test View Logs
/tmp/tmp5195xkq7/summary-missing.json (live):/summary-missing.json (live): View Logs
/tmp/tmpk3zry3ph/summary-fail.json (live):/summary-fail.json (live): View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows
View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs
convention violations found (ERROR lines above)./
convention violations found (ERROR lines above).
View Logs
nightly cmux-tui resolve/nightly cmux-tui resolve View Logs
test_actual_workflow_commands_leave_failed_build_receipts_in_upload_directory (main
.SwiftPMBuildMetricsTests.test_actual_workflow_commands_leave_failed_build_receipts_in_
upload_directory)/
test_actual_workflow_commands_leave_failed_build_receipts_in_upload_directory (main
.SwiftPMBuildMetricsTests.test_actual_workflow_commands_leave_failed_build_receipts_in_
upload_directory)
View Logs
test_every_submodule_free_checkout_on_an_owned_runner_resets_submodules (main.Reuse
dWorkspaceSubmodules.test_every_submodule_free_checkout_on_an_owned_runner_resets_submo
dules)/
test_every_submodule_free_checkout_on_an_owned_runner_resets_submodules (main.Reuse
dWorkspaceSubmodules.test_every_submodule_free_checkout_on_an_owned_runner_resets_submo
dules)
View Logs

...and 2 more test failures. View all on Blacksmith: run 39716194200, run 39716194354

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

…ar-2/test-helper-exit

# Conflicts:
#	.github/workflows/cmux-next-guards.yml
#	tests/test-execution.toml
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

cmux-next / cmux-next checks (god files, concurrency, crash safety, l10n) › Tier routing is red on 8 of 8 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

What the tracer named (CI at 81560ef, swift test job 119449955268): the mid-run exit is not NSApp.terminate. The atexit stack is

exit <- swift_task_asyncMainDrainQueue <- main (CmuxNextPackageTests) <- swiftpm-testing-helper

so the async test main's outer CFRunLoopRun() on the main thread returned (the run loop was stopped, or its mode went empty) and the runtime called exit(0), at 20:58:57 with 731 tests unfinished. The only NSApp.terminate refused line in the log is this PR's own TestProcessExitTests.swift:16. The one in-repo CFRunLoopStop(CFRunLoopGetMain()) (MainThreadWatchdogTests) runs in a later, separate step, so it is not it.

This PR still stands as a rail (a test's quit can no longer end the run) and as the tracer. A follow-up from me traces who stops the main run loop (-[NSApplication stop:] and CFRunLoopStop on the main loop) and corrects the comments here that name terminate as the cause.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (subagent, correctness first): SHIP at 81560ef

Release compile green (rerun; the first attempt hit a Blacksmith runner without Xcode 26.6). Reds are not this PR's: checks (Package conventions lint, Tier routing: base), swift test (the mid-run exit this PR traces, base), perf gate ([relaunch] pane_snapshots 4.0 > 1 at PaneController pane mount; same failure on #23638), daemon tests (ClassicUpgradeSimulationTests workspace order; the diff touches no daemon or workspace-order code). Review notes (comments that name terminate as the cause; guard-regex blind spots) go into the follow-up.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (subagent, correctness first): SHIP at 4ec4a78

Same diff as the reviewed 81560ef plus a feat-cmux-next merge. Release compile and the checks job are green. Reds are not this PR's: swift test (the mid-run exit this PR traces), daemon tests (ClassicUpgradeSimulationTests, as before), release-notary (highlight restored-terminals-come-back.md from base) and its Guard status rollup, fleet test-compile (runner setup failed).

@teamleaderleo teamleaderleo added the ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds label Oct 22, 2026
@github-actions github-actions Bot added risk:high Risk tier (docs/ci/risk-tiers.md) and removed risk:normal Risk tier (docs/ci/risk-tiers.md) labels Oct 22, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

cmux-next crash-class rules / cmux-next crash-class rules is red on 10 of 10 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6.1-sol): SHIP at 46d180f

Blocking findings: none.

Non-blocking findings: none.

Checked: merge-base 373ba24 through this exact head; all three deferred quit call sites and the AppKit helper's running/test-process paths; package and test-target wiring; the C exit tracer; the workflow and registry guard; and diff whitespace plus Python syntax. No macOS build or Swift test suite was run.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6.1-sol): SHIP at c139c30

Blocking findings: none.

Non-blocking findings: tests/test_cmux_next_deferred_terminate.py:17-20,31 only searches six physical lines and only matches NSApp/NSApplication.shared. A deferred closure using app.terminate or placing the call later in the closure passes this guard, so it would not catch the regression it claims to prevent.

Checked the exact merge-base diff (118beeb..c139c30), whole-file quit call paths, package target wiring, workflow/registry entries, Python syntax, and diff whitespace; no macOS build or Swift test suite was run.

@teamleaderleo
teamleaderleo merged commit 531771b into feat-cmux-next Oct 22, 2026
@teamleaderleo
teamleaderleo deleted the chief-sidebar-2/test-helper-exit branch October 22, 2026 03:56
teamleaderleo pushed a commit that referenced this pull request Oct 22, 2026
#23658 put CmuxNextTestExitTrace at Tests/Support/CmuxNextTestExitTrace
with an explicit `path:`. The reader assumed Sources/<name> or
Tests/<name>, so test_the_manifest_reader_finds_the_graph_targets failed,
and Tier routing stopped before it reached the build-metrics test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15ba2-4ce2-7e02-8fa2-5cd34f7648b1&host=cosmic-gray
teamleaderleo added a commit that referenced this pull request Oct 22, 2026
… and #24018's reuse skip (base red) (#24196)

* ci(tests): the build-log upload test accepts #24018's reuse skip

The swift-test and generated-files upload steps now read
`(always()) && steps.reuse.outputs.hit != 'true'`: a run that reused a
pass built nothing to upload. The test still pins that a failed build
uploads its measurement and log.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15ba2-4ce2-7e02-8fa2-5cd34f7648b1&host=cosmic-gray

* ci(tests): the route test's manifest reader honours a target's own path:

#23658 put CmuxNextTestExitTrace at Tests/Support/CmuxNextTestExitTrace
with an explicit `path:`. The reader assumed Sources/<name> or
Tests/<name>, so test_the_manifest_reader_finds_the_graph_targets failed,
and Tier routing stopped before it reached the build-metrics test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15ba2-4ce2-7e02-8fa2-5cd34f7648b1&host=cosmic-gray

---------

Co-authored-by: Leo Li <leoli.4u@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
teamleaderleo added a commit that referenced this pull request Oct 22, 2026
…under Tests/ (#24231)

#23658 added CmuxNextTestExitTrace, a regular target at
Tests/Support/CmuxNextTestExitTrace that only test targets link. The
coverage test counted it as a production target CmuxNextApp does not
reach and asked for it in the swift-canary-fleet list.


Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15ba2-4ce2-7e02-8fa2-5cd34f7648b1&host=cosmic-gray

Co-authored-by: Leo Li <leoli.4u@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

This branch was previously deployed

1 inactive deployment
ci-routing-agent — c139c307 Deployed Oct 22, 2026 by teamleaderleo
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant