[teamleaderleo]

The "cmux-next swift test" broad pass on feat-cmux-next is cut short. One shared test process runs every suite, so a few main-thread-heavy suites (frame benchmarks, real WebKit downloads) and the windows earlier suites leave open starve every main-actor test after them. The pass then hits its time limit with hundreds of tests started and never finished. Run 40705037968 is an example: 466 unfinished, throughput fell from about 2,000 tests a minute to about 100, and 428 windows piled up in one process. The other cut-short cause, a CFRunLoopStop that reaches swift-testing's outermost run loop and ends the process with exit(0), is fixed by #25861's run-loop guard and is not in this PR.

1. Main-thread-heavy suites run in their own processes

• scripts/cmux-next/main-thread-heavy-suites.py derives the list from the test sources on every run. It picks every suite whose file commits frames itself (CATransaction.flush(), displayIfNeeded(), a bench's drawFrame() or uses a bench helper type declared in such a file, plus RemoteVideoPageTests. A new frame-driving suite is covered with no list to edit.
• The broad pass and the changed-suites pass both skip them.
• A step of their own runs each one in a fresh process (hang sampler, 45 min). A hang or a red there is named in that step, and the rest still run. The step covers only the selected targets' heavy suites.
• The job timeout goes from 90 to 120 minutes to fit that step. The per-test stuck watchdog is unchanged.

2. The broad pass runs as several test processes

• scripts/cmux-next/swift-test-shards.py splits the selected targets by target, and a large target by suite-name prefix.
  • Each prefix level ends in a negated class, so every suite lands in exactly one shard. A test checks this against every real suite and against made-up names.
  • CmuxNextAppTests.Cloud* gets a shard of its own: its brokers and daemons run under 30-60 s deadlines.
• It packs the pieces into about twice as many shards as run at once and starts the largest first. Each shard runs through the hang sampler with swift test --skip-build --ignore-lock.
• The number run at once comes from the runner: half its cores, and no more than its memory holds at 4 GB a process. CMUX_NEXT_SWIFT_TEST_WIDTH overrides it.
• Shards start one at a time. The next one starts once the last shard has printed its first test line, has exited, or 180 s have passed. A shard that failed on SwiftPM's "database is locked" before running any test is started again, once.
• Each shard's log prints as one group, a red shard is named, and the step summary gets a table of shard, tests, seconds and exit.

3. The test process catches SIGPIPE, as the app does

A shard died on signal 13 with 19 tests never finished: some test wrote to a pipe whose reader was gone. The app catches SIGPIPE at launch (ChildSignalDefaults.installAppSignalPolicy()), so there that write fails with EPIPE. The test process never ran that step.

• The test bundle now installs the same caught handler when it loads (Tests/Support/CmuxNextTestExitTrace/CmuxNextTestSignalPolicy.c). Exec resets the handler in children.
• The first three catches print the writer's stack on stderr, so the writing test is still named.

Performance

Harness: the cmux-next swift test job's broad pass on a fleet mini, same job and same pass on both sides.

metric	before	after	delta	harness or run link
broad pass wall time	over 1,500 s, cut short at its limit	571 s, every shard with a summary	about -62% or more	before: run 40705037968, job 122875223361; after: run 40723548016 (15-core mini, 14 shards, 7 at once)
tests finished per minute	about 300 average (about 2,000 early, about 100 late)	about 1,000	about 3.3x	same
tests started and never finished	466	0 outside shards with a named crash	-466	same

Tests

• tests/test_ci_swift_test_main_thread_isolation.py (ci-guards) keeps the heavy split and tests the detector. It fails on today's workflow.
• tests/test_ci_swift_test_shards.py (ci-guards) covers:
  • every suite lands in one shard;
  • Cloud gets its own shard;
  • shard width;
  • a fake swift test that holds a lock while it starts: the gate and the retry each fail on the code before them.
• TestProcessSignalTests.aWriteToAClosedPipeFailsWithEPIPEAndTheRunGoesOn: on today's code the write ends the test process.

Risk: test harness and CI only, no product code.

Shard width, heavy-step limits (3727a9c)

• SidebarIncrementalMatchesRebuildTests, SidebarIncrementalMatchesRebuildGapsTests and ClassicUpgradeHugeSimulationTests run in the heavy step (ALWAYS in main-thread-heavy-suites.py). In shared shards they starved the main thread until the stuck watchdog ended shards 7 and 11 (run 40874507839).
• Heavy-step watchdog limits per suite come from main-thread-heavy-suites.py --runs: 600 s run / 300 s with no test finished, or above a suite's own .timeLimit. ClassicUpgradeHugeSimulationTests (15 min limit; 412 s alone, one test open 572 s in a shard) gets 1200 s / 1020 s. The heavy step's timeout is 60 min and the job's is 135 min.
• Broad-pass shards at once: a quarter of the cores (3 on a 14-core mini, was 7), still 15 shards.

metric	before (run 40874507839, 825cea7)	after (run 40896449485, e3c31bf)
shards at once	7	3
shards cut short (exit 143)	3 (4, 7, 11)	0
broad pass wall time	13.0 min	8.1 min
heavy step wall time	9.6 min	14.7 min (3 more suites)
swift test job wall time	35.0 min	31.1 min

Agent session: agent session

🤖 Generated with Claude Code

[teamleaderleo]

Owned by an agent session · session

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review 88f39a8: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr26809 -f ref=swift-cutshort-root -f platform=macos -f backend_mode=local

web and size checks: green

[teamleaderleo]

Bead cx-5immr7 created for this PR by the cmux coordinator.

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[teamleaderleo]

Review (codex gpt-6-astra high, next-review): SHIP at 764e79d

No blocking findings.

[cmux-bot]

cmux-bot approves 764e79d: an independent review passed this exact head (next-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

[teamleaderleo]

Review (next-review patch-id; carried): SHIP at 84121af

The independent SHIP at 764e79d carries to this head: the PR's own change (git diff $(git merge-base feat-cmux-next HEAD) HEAD) has the same git patch-id --verbatim (fdb0e6308040ca2d) at both heads, and no merge between them was resolved by hand. Nothing new to review. A newer BLOCK, or any change to the PR's own diff, ends the carry.

[teamleaderleo]

Review (codex gpt-6-astra high, next-review): SHIP at 4d5237b

No blocking findings.

Reviewed the diff and available surrounding code. In-memory filter and time-limit parsing checks passed; macOS tests were not run.

[cmux-bot]

cmux-bot approves 4d5237b: an independent review passed this exact head (next-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

[teamleaderleo]

Independent review (gpt-6.1-sol high) at 5a39be4: SHIP.

[teamleaderleo]

For awareness: #29490 carries this PR's commits (authorship kept) merged onto current feat-cmux-next, to land the swift test sharding now; the PR diff is identical. This job's swift test at b777afc (job 124552356302) ran all 15 shards with none cut short. This PR stays yours to close.

[teamleaderleo]

Awareness note from an agent lane that does not own this PR; it is not a review or a request to take the PR over.

---

Additional affected PR: #28407 at db65682. Run 41206084049/job 124579567862 passed its changed-suite shard (116 tests/10 suites in 0.742 s), then the separate broad process hit the 1500-second whole-run limit: 8416 finished and 31 unfinished. The 5-second sample shows 1586/2409 main-thread samples in CA flush / NSWindow layout, with GroupHeaderRowView.layout and text measurement hot; process footprint 7.1 GB. Tests continued finishing through the sampling window. This matches the shared-process layout pressure addressed here. The Cloud transition suites were excluded from the broad pass after passing; its gallery was skipped. Please include this run as evidence when landing this harness fix. The cutoff remains unexcused.

[teamleaderleo]

Review (codex gpt-6-astra high, next-review): BLOCK at 88f39a8

BLOCK at 88f39a8

Visual verdict: no images; Leo tries the build

1. [P1] [blocking] Review cannot be completed: review/TARGET:5 reports diff-lines 0, and review/pr.diff is empty. No base checkout or Git history is available to reconstruct the changes. Supply the merge-base-to-head diff so scope and regressions can be verified.

[teamleaderleo]

Superseded by #29490 (merged), the carry of this swift test sharding; this branch now has no diff against the base. Closing.