ci: compile only the cmuxTests sources a rerun's suites reach

[teamleaderleo]

Problem

app-host-test-rerun.yml (and, after #14229, scripts/run-e2e.sh for cmuxTests selectors) rebuilds only the test bundle. That bundle is still one monolith. Run 36009423131 spent 361 s on its rerun job to run one suite whose tests take 0.01 s:

• Compile all test sources, 220 s: 1,091 files, about 482k lines, for one suite.
• Package resolution, 116 to 160 s: not addressed here; see below.

The plan also rejected most of main as a product source. It counted anything outside cmuxTests/ as an app change, so one docs commit (docs/ci-runners.md) made every earlier main build ineligible (run 36012093039).

Change

• prune (app_host_test_rerun.py) keeps only the test sources the selected suites can reach. It starts from the files that declare or extend a selected suite. It then adds, repeatedly until nothing new comes in:

  • any file declaring a non-private top-level name those files mention
  • any file whose extension of a type they mention declares a member they mention, a conformance, or an init

  Sources outside cmuxTests/ (22 app and CLI files) and non-Swift sources such as the Objective-C window release guard are never dropped. On today's E2E traffic a suite reaches 1 to 24 of 1,069 test files.

• Fallback. The compile step tries the pruned project first. If that compile fails, it restores the detached project and compiles every source. The new all_test_sources input skips pruning.

• Docs and CI changes. non_test_changes now ignores paths that cmux.xcodeproj never references: the test roots, .github/, docs/, scripts/ci/, tests/ and web/. A test pins that no project path falls under this list.

  • The list is narrower than e2e_warm_derived_data.py's. That flow rebuilds the app, so it can skip skills/ and *.md.
  • The rerun keeps CI's app, which bundles skills/cmux-cua, cloud-agent-skill.md and THIRD_PARTY_LICENSES.md, so changes to those still count as app changes.

Validation

• Unit tests. tests/test_app_host_test_rerun.py: 34 pass, 8 of them new. They cover:
  • the closure over a fixture: helpers, suite extensions, a used extension member, a conformance, an init, a local variable that is not a member, and a missing suite
  • pruning the real project: only removals, CLI sources and the .m guard kept
  • a real suite reaching under a tenth of the sources
  • the workflow's fallback order
  • docs and CI changes being app-neutral
• Real reruns against 8a7e28b2, products from run 36003927414:

Run	Suite	Test sources compiled	Compile	Tests
36009423131 (before)	CmuxPopoverGroupTests	1,091	220 s	10 passed
36012287965	CmuxPopoverGroupTests	1 of 1,069	26 s	10 passed
36012292365	SSHTuiMigrationTests	23 of 1,069	47 s	11 passed

The pruned SSHTuiMigrationTests run ran the same 11 tests, with the same 14 test names in the log, as the last full-bundle run of that suite (35998329278). Neither run needed the fallback.

• Review. A subagent review found that the first version of the eligibility change borrowed ci(e2e): adopt main's DerivedData only when the app build is unchanged #14098's list, which would have let a PR that edits a bundled skill or resource be tested against CI's stale copy. The narrower list fixes that.
  • It also checked the silent under-inclusion cases, where a pruned bundle compiles but runs fewer tests: tests split across extensions, inherited tests, #if-nested classes, global hooks, NSClassFromString, overloads. None occur in the current tree.
  • The one theoretical gap, an extension of a base test class in a third file, is documented on source_closure. No test class inherits from another today.

Package resolution and the other fixed costs

Resolution became the largest remaining step after pruning, so I measured it against compile admission:

Phase	Compile admission (36003927414)	Rerun (36012287965)
Canonical tree copy	11 s	33 s
Graph load and checkouts	48 s	60 s
Checkouts to "Resolved source packages"	11 s	116 s

Three causes, all fixed here:

• Stale cache seed. The rerun's spm- key lacked scripts/ci/swiftpm-cache-layout. It was the only workflow ci: keep SwiftPM's artifact zips in the package cache seed #14013 missed, so its exact hit restored the pre-ci: keep SwiftPM's artifact zips in the package cache seed #14013 seed without the binary-artifact zips, and SwiftPM fetched them again. The key now matches ci-macos.yml, and a test fails if any workflow's spm- key drops the layout file. With the right seed, CMUX_CI_SWIFTPM_CACHE_EXACT_HIT (offline resolve, as in ci-macos) is back.
• Copying the packages. canonical-build-root.sh rsynced .ci-source-packages along with the tree. CMUX_CI_MOVE_SOURCE_PACKAGES=1, as ci-macos sets it, moves the directory instead.
• Fetching a whole second tree. "Fetch the products' revision" downloaded all of SOURCE_SHA's blobs, only so validation could check it out. The fetch is now --filter=blob:none, so the checkout fetches just the files that differ, with three attempts.

The first attempt, the offline flag alone on the stale seed, did not help and was reverted (it is in the history). Re-applying it once the key was fixed did.

Result

Same suite (CmuxPopoverGroupTests), same products:

Step	36009423131 (before)	36016462173 (this PR)
Fetch the products' revision	28 s	3 s
Restore + resolve Swift packages	134 s	93 s
Compile cmuxTests	220 s	43 s
Rerun job	498 s	290 s
Whole run	583 s	371 s

The largest step left is actions/checkout of the tested revision on macOS (55 to 78 s).

Reviews: a subagent review of the pruning and eligibility commits found the bundled-resource issue above, now fixed. A second review of the cache, fetch and move commits returned MERGE. Its one note, that the checkout now needs the network, is handled by the retry.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Chores
  • Test reruns can now compile only test sources relevant to the selected suites, with a fallback to compiling all test sources when needed.
  • Checkout validation retries up to three times, and rerun builds record which test sources were compiled.
  • App rebuilds are skipped for changes limited to documentation, CI scripts, and other app-neutral areas.
  • Swift package caching and package resolution have been updated for more reliable CI builds.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 55be7934-bc3f-4701-a31d-0f24721e5b74

📥 Commits

Reviewing files that changed from the base of the PR and between df44058 and a796f14.

📒 Files selected for processing (3)

• .github/workflows/app-host-test-rerun.yml
• scripts/ci/app_host_test_rerun.py
• tests/test_app_host_test_rerun.py

 _______________________________________
< Clippy called, he wants his job back. >
 ---------------------------------------
  \
   \   \
        \ /\
        ( )
      .( o ).

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}