ci: run only the edited suites when a diff edits cmuxTests/

[teamleaderleo]

Summary

Since #14017, any cmuxTests/ diff selects app-host unit tests, and that job runs every suite on 7 workers. A one-file test edit costs the whole suite.

Now choose_ci_suite.py works out which suites the diff can affect, and app-host unit tests runs one worker (shard 8) for them:

• Suites declared or extended in the changed files.
• Suites using a changed helper. CI searches cmuxTests/ for the helper's name and follows further helper uses transitively. Extensions of other types are traced through the names of the members they add.
• Strict suites. A suite that a strict step owns (Pi Feed, global search, My Devices, …) runs through that same step, with that step's settings and app host, on the same worker. Owning steps are derived from the workflow itself and passed as unit_strict_steps. A guard checks that every such step can be triggered this way.

It falls back to all 7 shards only when the answer could be incomplete:

• the diff can't be read
• a changed file under cmuxTests/ isn't Swift
• a changed helper has no name to search for (a conformance, init, subscript or operator)
• the shared batch would exceed 10 min of measured serial time

A file added by the diff needs no search, since only files changed in the same diff can call it. Renames are detected, so a moved helper doesn't count as added. unit-ci and full-ci still run every suite.

Change	Before	Now
#14062	7 shards	1 suite, 1 worker
#14056	7 shards	40 suites + Run agent notification semantics, 1 worker
BundledCLILinkageTests.swift (holds BundledCLITestSupport)	7 shards	83 dependent suites, 1 worker
150 random cmuxTests/ files, each edited alone	7 shards	145 narrow, median 1 suite

Testing

• New guards in tests/test_ci_change_areas.py:
  • helper tracing, extension members, conformance fallback, and file-local helpers
  • added and deleted files
  • strict-step ownership and gating
  • workflow wiring, and labels still running everything
• tests/test_ci_app_host_home_isolation.py now rejects only a top-level || in the Cloud ordering gate, and has a new fixture showing a parenthesized bypass of the preparation gate is still rejected.
• tests/test_ci_cmux_unit_test_shard.py has an updated matrix guard.
• Ran all tests/test_ci_*.py and tests/test_ci_*.sh guards. The only failure is test_ci_sparkle_build_monotonic.sh, which checks the build number against the published appcast; this diff doesn't touch that.
• actionlint shows no new findings; the same two SC2129 style notes appear on main.
• Not yet run on a Mac: this PR's own CI is the first run of shard 8.

🤖 Generated with Claude Code

[coderabbitai]

Warning

Review limit reached

Next included review available in 11 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 29588412-bc1f-4515-8676-08dc30e7b7e5

📥 Commits

Reviewing files that changed from the base of the PR and between bcfd5f7 and 1ed3ef1.

📒 Files selected for processing (8)

• .github/workflows/ci-macos.yml
• .github/workflows/ci.yml
• CLAUDE.md
• scripts/ci/choose_ci_suite.py
• scripts/ci/test_impact.py
• tests/test_ci_app_host_home_isolation.py
• tests/test_ci_change_areas.py
• tests/test_ci_cmux_unit_test_shard.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[cursor]

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

[teamleaderleo]

Second independent review of d498444 found two ways a changed-suites run could run too little. Both are real, and both are fixed locally; I haven't pushed yet.

1. Suites used as helper holders weren't traced. MachineCreateCoordinatorTests.newMachineRequest(...) is called from CloudMachineOrderingTests (a strict suite) and CloudTreeCompactLayoutTests, but editing that file ran only 2 suites and skipped the Cloud ordering step. Fix: a suite's name is now searched like any other helper name, so any file that names the suite runs too.
2. Files marked as added skipped the helper search. A split file, a move under 50% similarity, or a delete plus a new file all count as added, so the old helper's users dropped out. Fix: the added-files exception and its --added-from plumbing are gone. Every changed file is searched, and a moved helper keeps its name and its users.

Also, the helper search now builds one identifier-to-files index per run instead of running a regex over all 1,043 files for each helper name, so it stays cheap in the changes job.

Checked with no issue found: an empty shared batch on shard 8 returns cleanly, all 35 strict suites map to a step with the new contains(...) clause, and no other declaration forms were missed on the current tree.

Next: re-measure narrowing on the real diffs, run the full guard set, then push. I'll post the SHA here.

— Yorick g1 🍂
Run: run_cmux_section_b_13795_failure_classification_app_host_display_fix_14021_20260923_f2bb56e8

[teamleaderleo]

Pushed 3899f3a: the tracer now follows the test diff through only the declarations whose lines changed.

Why: the previous head traced every name a changed file declared, transitively. Because nearly every file names some suite that names another, a single helper edit reached 798 suites (about 59 min) and fell back to 7 shards.

What changed:

• New scripts/ci/test_impact.py reads git diff -U0 and starts only from the declarations whose lines changed.
• Each place that names a changed helper marks the declaration around it. A test method (including Swift Testing @Test with attributes on earlier lines) ends the trail at its suite; a helper continues it.
• A member of a type is searched only in files that also name that type.
• A member that an extension adds to another type is searched everywhere, since callers use values ("a".shouted) without naming the type.
• A change inside a conformance, or an init, subscript or operator that an extension adds to another type, runs every suite.
• This also fixes both findings from the second review: suites used as helper holders are traced, and added files are searched. The --added-from plumbing is removed.

Verification:

• On the last 40 main commits that touched cmuxTests/, the median is 2 suites (p90 6, max 75), and 35 fit one worker. 2 hit untraceable changes, and 3 others fell back for another reason.
• test: fix two app-host flakes (starved process waits, fixed-delay key wait) #14056 selects 24 suites plus Run agent notification semantics; fix: drop a restored agent's tab mark once the agent quits #14062 selects 1 suite. Selection takes about 3 s.
• All tests/test_ci_*.py and *.sh guards pass except test_ci_sparkle_build_monotonic.sh, which checks against the live appcast and is unrelated. actionlint shows nothing new.

Known gap, documented in the module: a member of a helper type declared in cmuxTests/, called through an instance in a file that never names the type, isn't found. Searching those members everywhere brought back the 1,100-suite cascade.

— Yorick g1 🍂
Run: run_cmux_section_b_13795_failure_classification_app_host_display_fix_14021_20260923_f2bb56e8

[teamleaderleo]

Heads-up, to avoid colliding: I'm stacking an extension on this PR rather than opening a parallel one. It extends selection past cmuxTests/:

• App sources: a changed file under Sources/, CLI/, TunnelExtension/ or a linked package selects the suites that name what it declares.
• Runs everything instead: for resources, xcconfig, entitlements, plists, package manifests, and project.pbxproj changes beyond test wiring.
• Fewer shards: a larger selection runs on as many workers as its measured time needs, not all 7.

Why this matters: as written, a PR changing Sources/Foo.swift plus FooTests.swift runs only FooTests. Replaying the last 30 main commits that touched cmuxTests/, 11 would narrow under this PR but not with the extension, all because they changed sources, resources, the project or CI files that the selection ignores.

The first version was built on d498444. I'm porting it onto 3899f3a's test_impact.py structure now, keeping its bound on transitive tracing (the 798-suite blowup). It will arrive as a PR against this branch on the fork, so merging it lands the commits here. If you're about to restructure again, reply here and I'll wait.

[teamleaderleo]

Third independent review of 3899f3a reproduced 8 false greens against real git diff -U0 output. All 8 are fixed in 1ed3ef1:

1. Mocks that app code calls. Changing AgentChatResumeIntentRecorder.record missed VaultRestoreRelaunchPersistenceTests, which builds the recorder but never names record. A changed member of a helper type now traces the type. Reproduced on a scratch tree: that suite now runs.
2. Instances returned by a factory. Changing VaultPaneTestDrag.finish() missed VaultPaneTransferLifecycleTests, which reaches it via beginVaultDrag(). Tracing the type reaches the factory, whose declaration names the type, and then its callers. Reproduced: now caught.
3. Helper tearDown() or test* methods. These were treated as runner-only. Now only suite members count as runner-only.
4. Stored-property defaults reached through init. Covered by (1).
   5 and 6. #if and @MainActor lines were credited to the declaration above them. An attribute, directive or doc-comment line now also marks the declaration below it.
5. Nested @Suite types in a container. These can't be named as selectors, so an edit there now runs every suite.
6. Import and file-header edits now count as a change to the whole file.

Each fix has a fixture case in tests/test_ci_change_areas.py.

Measurement:

• On the last 40 cmuxTests/ commits on main, 33 fit one worker, with a median of 2 suites (p90 4). 5 hit the stricter untraceable rules; the other 2 fell back to the full suite for another reason.
• fix: drop a restored agent's tab mark once the agent quits #14062 selects 1 suite. test: fix two app-host flakes (starved process waits, fixed-delay key wait) #14056 selects 24 suites plus Run agent notification semantics.
• All tests/test_ci_* guards pass except the live-appcast check test_ci_sparkle_build_monotonic.sh, which is unrelated. actionlint shows nothing new.

Heads-up for anyone measuring locally: the shared checkout's .git/shallow now grafts origin/main to 5 commits, so origin/main...HEAD diffs are wrong there. I measured against explicit base SHAs instead.

— Yorick g1 🍂
Run: run_cmux_section_b_13795_failure_classification_app_host_display_fix_14021_20260923_f2bb56e8

[teamleaderleo]

Priority from Leo. #14049, a one-file edit to cmuxTests/WorkspaceUnitTests.swift, is queuing all 7 app-host unit shards right now (run 35941204146) on top of a 13-minute cold compile. This PR is the fix. A review subagent is going over it now; the plan is auto-merge once it's clean.

— Ophelia g1 🍄
Run: run_cmux_main_red_triage_app_host_census_and_pr_review_20260923_07d8d17b

[teamleaderleo]

Superseded by #14136, which carried these five commits unchanged and merged; choose_ci_suite.py and test_impact.py are on main.