ci: build E2E products against the nightly compilation-cache seed

[teamleaderleo]

Re-running one E2E test pays a full cold Debug build (#13663). The build job's own e2e-compilation-v1 cache cost about 82 s median to restore across 19 builds and did not make them faster. It could never help: its key hashed the absolute $GITHUB_WORKSPACE path, so it could not share the xcode-compilation-test-* seed that nightly.yml refresh-test-compilation-cache writes and PR compile admission reads.

The E2E build job now compiles the way compile admission does:

• Same paths. DerivedData is $CMUX_CI_CANONICAL_ROOT/derived-data-compile-admission and the CAS is .../compile-admission-cas. The seed's key hashes the DerivedData basename, so it has to be the same name.
• Same entry points. Steps call canonical-resolve, canonical-fingerprint and canonical-build in scripts/ci/compile-app-host-test-product.sh. The key hashes only xcodebuild -version and that basename.
• Same seed, read-only. The job restores through ./.github/actions/cache-restore with backend vars.CI_CACHE_BACKEND || 'r2', using admission's xcode-compilation-test-<os>-<arch>-<fingerprint>- prefix. R2 reads need only CI_CACHE_R2_PUBLIC_URL, now set at workflow level. E2E's save, bound, discard and main-containment steps are removed, and so is scripts/ci/revision_on_main.py, which only gated them. The Swift package cache is restore-only as well: resolution now happens in the canonical copy, so saving from the workspace would store an empty directory.
• Same Xcode on the default runner. CMUX_CI_XCODE_APP is vars.CMUX_CI_XCODE_APP_PR when inputs.runner is auto, which matches the seeder and PR admission, so image-default drift cannot split the key again. With an explicit inputs.runner the job stays unpinned, because a pinned Xcode missing from that image hard-fails scripts/select-ci-xcode.sh. That case falls back to a cold build instead. The stale env comment now explains this.
• Product relocation. A fresh build's manifests name the canonical checkout. So the package step stamps from $CMUX_CI_CANONICAL_ROOT/src, relocates into the workspace, then stamps and seals as before, the same sequence as ci-macos.yml. An adopted product skips the first stamp. The test job already aliases the canonical source through restore-app-host-test-product.sh.
• Cleanup removes only those two canonical paths and refuses any other value. It leaves src in place: the next canonical resolve replaces it with rsync --delete.

Tradeoff. This gives up per-run, workspace-scoped DerivedData. The canonical root is shared per machine, so on a persistent host this job and compile admission use the same two paths. Each clears them on start. On ephemeral Blacksmith VMs and Tart VMs, one machine means one job.

The packaging change sits above the tar line that #14007 edits. git merge-tree against ci/e2e-reuse-parallel-download is clean, and the tests below pass on that merge.

Validation (local, Linux)

• tests/test_ci_e2e_compilation_cache.py now pins the new design and passes (18 tests). The tests run the real prepare, package and cleanup step scripts and the real canonical-fingerprint. They check that the fingerprint does not depend on the workspace and changes with the Xcode version or the basename; that the paths, restore key and Xcode source match admission and the seeder; that nothing in the workflow saves a cache; the stamp order for fresh and adopted products; and that cleanup refuses unowned paths and keeps src. Mutating the basename, the Xcode variable, the canonical stamp or canonical-build in the workflow each makes it fail.
• Every file from grep -rln test-e2e.yml tests passes: test_run_e2e.py, test_ci_e2e_compilation_cache.py, test_ci_self_hosted_guard.sh (its E2E allowlist loses the cache steps), test_ci_app_host_home_isolation.py, test_ci_reusable_workflow_permissions.py, test_ci_app_host_pipe_capture.py, test_reuse_app_host_products.py (68 tests) and test_ci_change_areas.py.
• All 156 run steps of ci-guards.yml workflow-guard-tests, across every matrix group, pass. Four needed their CI context: RUNNER_TEMP for the guard profile and the venv setup, working-directory: agent-chat for the bun test, and python3.9 for the prune check.
• actionlint v1.7.12 on test-e2e.yml is clean.

None of this runs macOS or shows a cache hit. The real proof is an E2E dispatch after the seed run 35884626006 lands (still queued when this PR opened). That run should show r2-cache: restored xcode-compilation-test-... in "Restore test compilation cache" and a shorter "Compiled from source in N s" than the cold baseline. One thing only that run can confirm: the seeder runs on the MACOS_RUNNER_PR pool and E2E on MACOS_RUNNER_TESTS. Their Xcode and paths now match, but whether the compiler entries hit across the two host images has not been measured.

Refs #13663

— Nyan g1 🗝️
Run: run_cmux_main_red_suite_slices_and_errno_fixes_20260923_8053081a

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

---

Summary by cubic

Reworks the E2E build job to compile against the nightly xcode-compilation-test seed that PR compile admission restores, replacing its own e2e-compilation-v1 cache, which never hit because its key hashed the absolute workspace path. The job now resolves, fingerprints and builds through the same canonical entry points at the same DerivedData and CAS paths, and restores the seed read-only via cache-restore.

Changes

• Removes the save, bound, discard and main-containment steps, plus scripts/ci/revision_on_main.py, which only gated them.
• Swift package cache is now restore-only; resolution runs in the canonical copy.
• The default runner pins CMUX_CI_XCODE_APP to vars.CMUX_CI_XCODE_APP_PR to match the seeder; an explicit inputs.runner stays unpinned and falls back to a cold build.
• Fresh products are stamped at the canonical checkout, relocated into the workspace, then stamped and sealed as before.
• Gives up per-run, workspace-scoped DerivedData; on persistent runners the canonical root is shared with compile admission, and each clears the same two paths on start.

Refs #13663

^{Written for commit 445a6ca. Summary will update on new commits.}

[coderabbitai]

Warning

Review limit reached

Next included review available in 26 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: dfa66283-e312-4e8b-9eb9-244d8b7189a9

📥 Commits

Reviewing files that changed from the base of the PR and between 3ca19ad and 445a6ca.

📒 Files selected for processing (4)

• .github/workflows/test-e2e.yml
• scripts/ci/revision_on_main.py
• tests/test_ci_e2e_compilation_cache.py
• tests/test_ci_self_hosted_guard.sh

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[teamleaderleo]

Measured on dispatch 35933554831 from this branch, testing main 72490a9deb. The plumbing works, but the speedup does not show up.

• The restore hits. cache_restore_receipt shows route r2 and result prefix, matching xcode-compilation-test-macOS-ARM64-bac5277611926ed6b40291f1640fed47-ccdbf300c9…. That is the nightly seed from 35884626006, under the same fingerprint. So Xcode 26.6 alignment plus canonical paths do give E2E and compile admission one shared key.
• The compile is not faster.

build	cache	compile step
seed refresh-test-compilation-cache, ccdbf300c9	cold, writes the entry	14.5 min
this E2E build, 72490a9deb	prefix hit on that entry	21.7 min
previous E2E design, 19 runs (#13663)	its own path-keyed cache	~13–19 min

The seed and this build are 17 commits and 55 files under Sources/, Packages/ and cmuxTests/ apart, so most compile units should have been reusable. The next step is the compiler-level hit ratio: Xcode's compilation-cache hit/miss counts in the build step log. Without that, a key hit says nothing about reuse. One sample on a shared pool cannot rule out runner contention either.

Until that is measured, don't merge this PR expecting a speedup. What it does prove today is a restore that costs 22 s instead of about 82 s, with fewer E2E-specific cache steps.

— Nyan g1 🗝️
Run: run_cmux_main_red_suite_slices_and_errno_fixes_20260923_8053081a

[teamleaderleo]

Root cause of the missing speedup, from Xcode's own Cache hit / Cache miss remarks in the build log of 35933554831, counted with scripts/ci/build_metrics.py:

scope	hits	misses	wall time
scheme cmux	881	2852	14.3 min
of which target cmux (the app)	3	2662
scheme cmux-unit	57	44	6.4 min
scheme cmux-numeric-locale	0	0	1.0 min

Between the seed (ccdbf300c9) and this build (72490a9deb), 13 of the app target's 2,659 Swift files changed, and effectively every compile job in the target missed. That is how Swift compilation caching keys jobs. Each frontend job's cache key covers the inputs of the whole module, not just its primary file. So one edit anywhere in a module invalidates every cached job in that module.

cmux is one module with 2,659 files, and main changes some of them within hours. A compile-cache seed from an earlier main commit will almost never hit for the app target, whether in this E2E lane or in PR compile admission. It only hits for packages whose sources did not change, and that is where the 881 hits came from.

So the compile cache cannot be what makes these builds fast. The approaches that fit this code shape are:

1. Exact-revision product reuse (ci: let E2E dispatches adopt a compiled product instead of rebuilding #13958, ci: make cross-run product reuse actually adopt products #14007): with no compile at all when the same revision was already built, the remaining misses don't matter.
2. Incremental builds on persistent DerivedData (the persistent-Mac compile lane): per-file dependency tracking rebuilds only what an edit affects, which the CAS key cannot express.
3. Moving code out of the app target into packages. Each extracted module becomes independently cacheable.

This PR is still correct and simpler than E2E's old cache: it matches the shared key and restores in 22 s instead of about 82 s. But it cannot deliver a build-time win on its own. Merge it for the simplification, not for speed.

— Nyan g1 🗝️
Run: run_cmux_main_red_suite_slices_and_errno_fixes_20260923_8053081a

[teamleaderleo]

Closing: this PR's own 09-24 measurement found 3 cache hits and 2,662 misses on the app target, and the build was no faster. #14161 stopped restoring that cache in compile admission for the same reason: Swift keys compile jobs on the whole module.