ci(e2e): start owned E2E builds from the Mac's kept state, upload after tests

[teamleaderleo]

On an owned Mac (glaeda-*), test-e2e's build spent about 8 minutes on network before it compiled. It downloads at 6–7 MB/s there, against Blacksmith's 35–55 MB/s:

• Swift packages: 229–282 s to restore 1.7 GB.
• DerivedData seed: 212–269 s to adopt 1.6–1.9 GB.
• Product upload: about 130 s for 573 MB before the tests could start.

That covers runs 36169238297 on cmux-austin-mini-1 and 36169200906 on cmux13s. Both hosts' job records say clear, so the cause is network, not heat or contention. The compile itself runs within about 1.1–1.2× of Blacksmith.

What changes

The new steps only run on glaeda- runners. Blacksmith and GitHub-hosted runs are unchanged.

• Start from the Mac's kept state. Compile admission already keeps this per canonical root (owned_build_state.py), and the E2E build uses it after a reuse miss:
  • check clones the Mac's packages, and "Cache Swift packages" is skipped when it did.
  • prefer (with CI_OWNED_PREFER_SEED) can pick a near seed instead.
  • adopt clones the kept DerivedData where the seed would go. A failed adopt is discarded, so the build starts empty rather than from a partial clone.
  • The seed steps take CMUX_SEED_LOCAL_CACHE/CMUX_SEED_EXACT, as admission's do.
  • The helper is fetched at github.workflow_sha, since an older tested revision's copy moved kept state out of the store.
• No record, keep or save. A dispatch builds whatever revision it names and must not become the next pull request's starting point. It does do admission's upkeep: seed-source.json, the size cap, and the local seed cache with CI_OWNED_PREFER_SEED.
• Why the reads are safe: the glaeda runner hook gives this job the root's token for the whole job, and only a job holding root k writes root k's store.
• Upload after the tests on an owned Mac. The tests restore the local archive, not the upload.
  • It runs whatever the tests did, even on cancel, so a failing or superseded run still publishes its product.
  • It runs only when the archive still hashes to what the package step sealed.
  • It uses a second step name, because product_input_identity.py needs unique names. e2e_sibling_build.py and reuse_app_host_products.py accept either name.
  • Consumers testing an older revision only see such a product once the whole build job succeeds.
  • A job timeout still loses it.

Status: draft until an owned-Mac run is measured

The validation dispatch (36179450718) was refused before it started. The live glaeda hook treats this job as an unknown id pinned to /private/tmp/cmux-ci, and root 1 on cmux9s was busy (capacity: the canonical root token is taken (build is compile)). teamleaderleo/glaeda#1259 classes the job properly and lets it take any free root. Once that is deployed, or root 1 is free, the dispatch needs repeating.

Review

An independent read-only review of the first version found two real defects, both fixed here. First, uploading after the tests on every runner lost the product on cancel or timeout, so that now only happens on owned Macs, with a cancel-safe step. Second, a half-cloned DerivedData could survive a failed adopt. It confirmed:

• Blacksmith and GitHub-hosted jobs behave exactly as before.
• Store selection per root and the fingerprints match compile admission.
• Exactly one of the seed or the kept DerivedData is used in every combination.

Verification

• The linux-guard tests touching this change pass: test_ci_e2e_compilation_cache, test_ci_owned_build_state (new E2EWiring), test_e2e_sibling_build, test_reuse_app_host_products, test_ci_change_areas, test_ci_self_hosted_guard.sh.
• The one failure, test_a_missing_manifest_fails_instead_of_silently_compiling, also fails on main on macOS: bash 3.2 reports ONLY_TESTING[@] unbound under set -u.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

---

Summary by cubic

On owned Macs (glaeda-*), the E2E build now starts from the build state compile admission already keeps there, cutting roughly 8 minutes of pre-compile network downloads and uploading the product after the tests instead of before. Blacksmith and GitHub-hosted runs are unchanged.

• The build clones the Mac's kept Swift packages and DerivedData (or a preferred near seed when CI_OWNED_PREFER_SEED is set), skipping the package cache restore and seed adopt on a warm hit.
• A failed DerivedData clone is discarded so the build starts empty rather than from a partial copy.
• The build job never records, keeps, or saves state: a dispatch must not become the next PR's starting point. It does run admission's upkeep (seed-source.json, size cap, local seed cache).
• On owned Macs that test in the build job, the product upload moves to a step running after the tests, even on failure or cancel, and only when the archive still hashes to what was packaged.
• restore-app-host-test-product.sh no longer requires an artifact id, and e2e_sibling_build.py/reuse_app_host_products.py accept either upload step as publication.

Draft status: the validation dispatch was refused because the glaeda runner hook doesn't yet classify this job; it needs repeating once that's deployed or root 1 is free.

^{Written for commit d6ff15e. Summary will update on new commits.}

[coderabbitai]

Warning

Review limit reached

Next included review available in 5 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 12a3cb59-64da-4f88-8463-555e891a4cc3

📥 Commits

Reviewing files that changed from the base of the PR and between ff02854 and d6ff15e.

📒 Files selected for processing (11)

• .github/actions/e2e-run-tests/action.yml
• .github/workflows/test-e2e.yml
• scripts/ci/e2e_sibling_build.py
• scripts/ci/restore-app-host-test-product.sh
• scripts/ci/reuse_app_host_products.py
• tests/test_ci_change_areas.py
• tests/test_ci_e2e_compilation_cache.py
• tests/test_ci_owned_build_state.py
• tests/test_ci_self_hosted_guard.sh
• tests/test_e2e_sibling_build.py
• tests/test_reuse_app_host_products.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[github-actions]

Merge receipt for d6ff15ec5c: every check was green at merge (15 verified; 13 skipped by policy). Full suite runs on main after merge.