Skip to content

base red: the test run-loop guard holds only the Swift Testing helper, never xctest (xctest signal 6 after every CmuxNext suite) - #33013

Merged
lawrencecchen merged 6 commits into
feat-cmux-nextfrom
fix/base-red-xctest-contacts
Nov 3, 2026
Merged

lawrencecchen merged 6 commits into
feat-cmux-nextfrom
fix/base-red-xctest-contacts

Conversation

@lawrencecchen

@lawrencecchen lawrencecchen commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Takes back #33023: its -XCTest argv check in CmuxNextTestRunLoopGuard.c is replaced by a stricter gate (the guard installs only in swiftpm-testing-helper), which covers the same xctest leg and also the whole-package swift test XCTest phase that starts xctest <bundle> with no -XCTest (SwiftPM release/6.3 SwiftTestCommand.swift lines 402, 1008-1014); nothing of #33023's behavior is lost.

Update after the feat merge (head 11884af). #33023 (e512575, merged 05:17Z) fixed the same fault on feat in a different way: the guard skips the hold when argv has -XCTest. In the merge conflict I kept one gate: the guard installs nothing unless the process is swiftpm-testing-helper. That covers #33023's case and every other host (an Xcode test host, xctest without -XCTest), and runs_xctest_leg is removed. This PR now adds (1) the stricter gate and (2) the regression test #33023 does not have. At the new base the regression test also passes, because its child xctest has -XCTest in its argv. The red results below are against the old base 362be7b7d362, before #33023.

Base red on feat-cmux-next (STATUS 8187): CmuxNext Swift test steps pass every Swift Testing test, then the XCTest phase of swift test (xctest -XCTest '' …/CmuxNextPackageTests.xctest) hangs for about 10 minutes and ends with "unexpected signal code 6". Seen on aws-m4pro-1, -3, -4, -6, -9 and on Blacksmith macOS 26.

Cause, from the stack of the hung xctest (aws-m4pro-6, feat 305434be55a1, step 3bd61ca1):

_XCTestMain -> -[XCTestDriver _runTests] -> -[XCTFuture value]
  -> +[XCTWaiter _synchronouslyWaitForTimeInterval:] -> CFRunLoopRun -> __CFRunLoopDoBlocks
  -> __cmux_next_test_runloop_guard_install_block_invoke -> guard_hold_the_main_loop -> CFRunLoopRun

The run-loop guard in Tests/Support/CmuxNextTestExitTrace (#25861) queues a block on the main run loop that never returns. That is correct only in swiftpm-testing-helper, whose main is swift_task_asyncMainDrainQueue. xctest loads the same bundle. When XCTest's start-up work is not done at once, XCTest turns the main run loop to wait for it. The guard's block runs in that turn and keeps the main thread, so XCTest never runs its suite. The host-dependent part is only how slow XCTest start-up is. On some hosts a Contacts store lookup in xctest fails for 22 s first: those are the CoreData 134020 / NSXPCConnection lines. On aws-m4pro-1 the same hang came with no CoreData line. No cmux code uses Contacts, AddressBook or CoreData. The CoreData lines are a symptom of a slow start-up, not the abort.

Fix: the guard installs nothing (no observer, no source, no block) in a process that is not swiftpm-testing-helper. In the helper, it works the same as before. Its own regression test (TestProcessExitTests.aStopOfTheOutermostRunLoopDoesNotEndTheTestRun) is unchanged.

Testing

Regression test XCTestPhaseTests.anXCTestWaitInTheXCTestPhaseReturns (red commit 980a603, before the fix): it runs a child xctest of this same bundle with an XCTest probe that waits on an expectation, the same way XCTest's start-up waits. It needs the probe to pass and xctest to exit 0 within 120 s. On the base, the guard keeps the child's main thread on every host, so the test does not depend on a slow start-up.

Focused command (fleet): cmux-ci run --class light --script scripts/ci/package-test-lane.sh --arg suite --arg Packages/macOS/CmuxNext --arg XCTestPhaseTests --arg EmptyLabelMeasureTests --ref SHA --target-worker WORKER

Red: fix/base-red-xctest-red-proof at 69e5a3b7c330 (base 362be7b7d362 plus the head's XCTestPhaseTests.swift, without the one test that calls the new C function; never merge). Green: PR head bc800c77fbe0.

commit worker step XCTestPhaseTests EmptyLabelMeasureTests step exit
red 69e5a3b aws-m4pro-6 7afcf04e failed after 120.3 s (child xctest killed at the limit; its only output is the guard's "holding the main run loop open" line); then xctest signal 6, 787 s xctest signal 6 after 2/2 Swift Testing passed, 659 s 1
red 69e5a3b aws-m4pro-8 7a5152c2 failed after 121.7 s, same three issues CoreData XPC lines, then failed, 660 s 1
green bc800c7 aws-m4pro-6 fb48cb94 2/2 passed (the XCTest wait returns in 0.15 s); XCTest phase "Executed 0 tests", exit 0; 11 s 2/2 passed; XCTest phase exit 0 0
green bc800c7 aws-m4pro-8 1cfd2401 2/2 passed; XCTest phase exit 0; 11 s 2/2 passed; XCTest phase exit 0; 3 s 0

aws-m4pro-6 failed on the base before (steps 3554f3fe, 3bd61ca1), and it fails at the red commit and passes at the head.

Observed / Not run

Observed: the hung xctest stack above (step 3bd61ca1, from the lane watchdog with a 240 s limit); the four steps in the table; scripts/cmux-next/prepush-gate.py on each push: no new violations. The base passed on aws-m4pro-7 and -9 in some runs (steps 7048809b, 65f92591): the base red needs a slow XCTest start-up, but the new test is red at the red commit on every host, because the probe's own wait(for:) turns the run loop.

Not run: the whole-package swift test (the broad pass); Blacksmith macOS. I did not find what in xctest does the Contacts store lookup (a system framework; no cmux code uses Contacts, AddressBook or CoreData). With the fix, that lookup can only make the XCTest phase slower; it cannot hold it.

Changelog

none

Proof

No capture: test-support code only; nothing in the app changes.

swift test runs the bundle a second time in xctest after the Swift
Testing tests. From 2026-11-02 that phase hung for about 10 minutes and
died on signal 6 on CmuxNext suite steps. The hung stack (aws-m4pro-6,
feat 305434b): XCTestDriver waits on its start-up future with a
turn of the main run loop, and the run-loop guard's block, which never
returns, runs in that turn. This runs an XCTest probe that waits the
same way in a child xctest of this bundle and needs it to pass and exit
0 within 120 s. Red on the base on every host; the fix follows.
… helper, never xctest

The guard (#25861) queues a block on the main run loop that never
returns. That is right only in swiftpm-testing-helper, whose main is
swift_task_asyncMainDrainQueue. swift test's XCTest phase loads the
same bundle in xctest, where XCTestDriver waits on its start-up future
with a turn of the main run loop: the guard's block ran in that turn,
XCTest never ran its suite, and the process died on signal 6 about 10
minutes later (stack from aws-m4pro-6 at feat 305434b). Every
CmuxNext suite step was red after its Swift Testing tests passed.

The guard now installs nothing in a process that is not
swiftpm-testing-helper. Its own regression test
(TestProcessExitTests.aStopOfTheOutermostRunLoopDoesNotEndTheTestRun)
runs in the helper and is unchanged.
@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 11884af: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr33013 -f ref=fix/base-red-xctest-contacts -f platform=macos -f backend_mode=local

web and size checks: green

Shipped in:

@github-actions github-actions Bot added risk:low Risk tier (docs/ci/risk-tiers.md) and removed risk:normal Risk tier (docs/ci/risk-tiers.md) labels Nov 3, 2026
@blacksmith-sh

blacksmith-sh Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Found 1 test failure on Blacksmith runners:

Failure

Test View Logs
build_only must be a conjunctive exclusion on the helper, signing, and publication jobs
, and must not gate the unsigned app build/
build_only must be a conjunctive exclusion on the helper, signing, and publication jobs
, and must not gate the unsigned app build
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

…phase

Review of #33013: the child ran through the xcrun shim (SIP-protected,
so the DYLD_* paths SwiftPM gives the phase were dropped) and without
SWIFT_TESTING_ENABLED=0 (so XCTest could run Swift Testing tests in it,
this test included). It now finds xctest with xcrun --find, runs it
directly with the helper's environment and SWIFT_TESTING_ENABLED=0, as
run_swift_test_bundle.py does. Comments in the header and Package.swift
say the guard is in swiftpm-testing-helper only.
…onisolated

This target defaults to the main actor; an XCTestCase subclass there
does not compile (its initializers override nonisolated ones), and the
test's file reads need not run on the main thread.
@github-actions github-actions Bot added risk:normal Risk tier (docs/ci/risk-tiers.md) and removed risk:low Risk tier (docs/ci/risk-tiers.md) labels Nov 3, 2026
@lawrencecchen

Copy link
Copy Markdown
Contributor Author

Non-green checks at bc800c7, classified: all are base findings in files this PR does not touch.

  • cmux-next checks: (1) check-no-process-scan red on cmux-tui/crates/cmux-tui/src/link/machine_update/apply.rs (2 process-scan lines, baseline 1); (2) check-fixture-host-names red on webviews/src/agent-session/acpmux/HarnessStatusCard.gallery.tsx:50.
  • defaults-snapshot: check-menu-actions.py red on Packages/macOS/CmuxNext/Sources/CmuxNextTabs/Playground/TabBarPlaygroundNewTabControl.swift.
    This PR changes only Packages/macOS/CmuxNext/Tests/** and one comment in Packages/macOS/CmuxNext/Package.swift.

@lawrencecchen

Copy link
Copy Markdown
Contributor Author

Review (independent subagent): SHIP at bc800c7

  • Correctness first. The guard now installs only in swiftpm-testing-helper, which swift test uses for Swift Testing on macOS. The direct lane (run-swift-testing-suites.sh, then run_swift_test_bundle.py) and exit-test children use the same helper. Listing runs still return early. No lane runs CmuxNext tests in xcodebuild or in an Xcode test host. xctest has no swift_task_asyncMainDrainQueue exit, so it does not need the guard.
  • The guard's own regression test (TestProcessExitTests.aStopOfTheOutermostRunLoopDoesNotEndTheTestRun, including its CMUX_NEXT_TEST_RUNLOOP_GUARD=0 red proof) runs in the helper and works as before.
  • Earlier HOLD (03c6725) fixed: the child is now xcrun --find xctest run directly, with SWIFT_TESTING_ENABLED=0, as SwiftPM starts that phase. The header and Package.swift comments are up to date. The test types are nonisolated (the target defaults to the main actor).
  • Red proof 69e5a3b is base 362be7b plus the head's test without the new C function. It failed at 120 s on aws-m4pro-6 (7afcf04e) and aws-m4pro-8 (7a5152c2). The lane's own XCTest phase got signal 6 after about 660 s, which is the original base red.
  • Green bc800c7 passed on aws-m4pro-6 (fb48cb94) and aws-m4pro-8 (1cfd2401). Both XCTest phases exit 0 and both steps exit 0.
  • prepush-gate: no new violations.
  • Not shown: the probe inside a whole-package swift test XCTest phase. The child test covers it under the same environment.

@lawrencecchen lawrencecchen added the ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds label Nov 3, 2026
Conflict in CmuxNextTestRunLoopGuard.c with #33023 (e512575),
which stops the hold in a process whose argv has -XCTest. Resolved to
one gate: the guard installs only in swiftpm-testing-helper. That
covers #33023's xctest leg and every other host; runs_xctest_leg and
its crt_externs include are removed, and its note is kept in the
comment.
@lawrencecchen lawrencecchen removed the ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds label Nov 3, 2026
@cmux-bot cmux-bot Bot removed the stack-blocker label Nov 3, 2026
@lawrencecchen

Copy link
Copy Markdown
Contributor Author

Review (independent subagent): SHIP at 11884af

@lawrencecchen lawrencecchen added the ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds label Nov 3, 2026
@cmux-bot
cmux-bot Bot temporarily deployed to ci-routing-agent November 3, 2026 06:53 Inactive
@cmux-bot
cmux-bot Bot temporarily deployed to ci-routing-agent November 3, 2026 06:53 Inactive
@cmux-bot
cmux-bot Bot temporarily deployed to ci-routing-agent November 3, 2026 06:55 Inactive
@lawrencecchen
lawrencecchen merged commit ecfbb44 into feat-cmux-next Nov 3, 2026
180 of 199 checks passed
@lawrencecchen
lawrencecchen deleted the fix/base-red-xctest-contacts branch November 3, 2026 07:28
lawrencecchen added a commit that referenced this pull request Nov 3, 2026
…Test (#33276)

A whole-package swift test starts its XCTest phase as plain
'xctest <bundle>' (swift-package-manager release/6.3,
Sources/Commands/SwiftTestCommand.swift: xctestArgs returns no
arguments with no filter, line 402; args(forTestAt:), lines
1008-1014); -XCTest is added only for a filter (line 938). #33023's
argv check did not cover that start, so the regression test now runs
its child both ways.

Red at feat 145c5a3 (#33023, without #33013) plus this test
(f54406d): the no -XCTest case fails at 120 s on aws-m4pro-6
(step 10ce589a) and aws-m4pro-8 (step 95f2953d).

This branch was previously deployed

1 inactive deployment
ci-routing-agent — f82fe5ef Deployed Nov 3, 2026 by cmux-bot[bot] via wait for same-key tree publisher #104489
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant