Skip to content

ci: restore SwiftPM's manifest cache for package resolves - #14257

Merged
teamleaderleo merged 2 commits into
mainfrom
ci/swiftpm-manifest-cache
Sep 24, 2026
Merged

teamleaderleo merged 2 commits into
mainfrom
ci/swiftpm-manifest-cache

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

Every compile admission and cli-pipe-regressions job evaluates all ~90 Package.swift files from scratch while resolving packages, even when the spm- package cache is an exact hit. That accounts for about 30 s of the admission "Resolve Swift packages" step (median 66 s). This PR saves SwiftPM's evaluated-manifest cache from main and restores it read-only in both lanes.

Why a plain cache restore would never hit

SwiftPM (both swift and xcodebuild) caches each evaluated manifest in ~/Library/Caches/org.swift.swiftpm/manifests/manifest.db. Each entry's key includes:

  • the manifest's contents,
  • its absolute path,
  • the toolchain,
  • the whole process environment, minus a few terminal variables.

I confirmed the environment part locally with xcodebuild on a scratch package: changing one unrelated variable added a new entry. On Actions every step has a different environment (GITHUB_RUN_ID, GITHUB_ACTION, GITHUB_OUTPUT paths), so a restored cache would always miss. It also explains why a later xcodebuild in the same step resolves in a couple of seconds.

What changes

scripts/ci/swiftpm-manifest-cache.sh does four things:

  • run execs the resolve under a fixed environment: HOME, a fixed PATH, LANG, DEVELOPER_DIR, USER/LOGNAME, and any proxy variables. The command is still looked up on the caller's PATH. No cmux Package.swift reads the environment, so this can't change what a manifest declares.
  • key produces swiftpm-manifests-v1-<os>-<arch>-<xcode>-<hash of every Package.swift, Package.resolved and the script>. Entries are content-keyed, so readers fall back to the newest cache for their Xcode by prefix.
  • stage writes a single-file copy of manifest.db for saving (about 100 entries, a few MB).
  • install puts a restored copy in place. A missing or unreadable restore changes nothing.

Where it's wired in:

  • Writer: seed-derived-data.yml, which already runs on main pushes in ci-cache-writer, on the same pool and Xcode as PR admission. It checks for the exact key only. On a miss it starts with an empty cache, resolves at both reader paths (the canonical admission root, then the runner workspace with .spm-cache as cli-pipe-regressions uses it), and saves. The absolute path is part of each entry's key, so each path needs its own resolve. Every save step is continue-on-error.
  • Readers: compile admission (ci-macos.yml) and cli-pipe-regressions.yml. Both restore read-only and run their resolve through run. compile-app-host-test-product.sh resolve wraps both of its xcodebuild resolves, so nightly and the seeder use the same environment.

Not covered:

  • swift-package-tests runs swift test, whose tests need the real environment.
  • The build steps' own manifest loads still run under the job's environment. Wrapping those is a separate change, and the build could depend on its environment.

Evidence

Canary #14251, admission job 107720596235, 6 vCPU macOS 26, Xcode 26.6. After the real resolve, it copied the pristine package directory back into place and resolved again with the same flags as admission (-skipPackageUpdates):

manifest cache resolve
hit 12 s
moved aside 42 s
hit again 13 s

The job's real first resolve (no cache yet) took 48 s from "Resolve Package Graph" to "Resolved source packages". The earlier #14235 probe measured only 3 to 7 s either way because it re-resolved an already-resolved package directory, where SwiftPM skips most manifest loading.

This PR's own CI runs can't show the gain, because nothing is seeded until seed-derived-data.yml runs on main after merge. After that I'll post before/after "Resolve Swift packages" and "Resolve Swift package dependencies" timings from real PR runs here.

Local checks:

  • New tests/test_ci_swiftpm_manifest_cache.sh (wired into ci-guards): pins the fixed environment, the key's inputs, and the stage/install round trip.
  • Pass: test_ci_test_compilation_cache_seed.sh, test_seed_derived_data.py, test_ci_pull_request_caches_are_read_only.py, test_ci_workflow_guards_are_wired.py, actionlint.
  • test_ci_canonical_build_root.py passes with a realpath TMPDIR. With the default TMPDIR its two cwd assertions fail on macOS because of the /var vs /private/var symlink, including in the build-only test this PR doesn't touch.
  • Two existing tests configure their xcodebuild stub through the environment. They now name those variables in CMUX_CI_SWIFTPM_KEEP_ENV.

Signed: Manifold g1 ✨ (run_swiftpm-manifest-cache-20260924)

🤖 Generated with Claude Code


Summary by cubic

Caches SwiftPM's evaluated manifests so the "Resolve Swift packages" step no longer spends ~30 s re-evaluating all 91 Package.swift files even on a warm package-cache hit.

SwiftPM keys each cached manifest on the whole process environment, and every Actions step has a different one, so a plain cache restore would always miss. The new scripts/ci/swiftpm-manifest-cache.sh runs the resolve under a fixed, per-run-stable environment, content-keys the cache on the manifests, Package.resolved, and submodule pointers (the vendor/bonsplit manifest lives past one), plus the Xcode toolchain, and stages/installs the single-file manifest.db.

  • seed-derived-data.yml seeds the cache from main at both paths readers resolve from (the canonical admission root and the runner workspace); compile admission and cli-pipe-regressions restore it read-only.
  • compile-app-host-test-product.sh resolves under the fixed environment too, in both the exact-hit and retry paths.
  • A missing or unreadable restore falls back to resolving without the cache; only the exact key triggers a new seed, with readers falling back to the newest prefix for their Xcode.
  • Not applied to swift-package-tests (its tests need the real environment) or to the build steps' own manifest loads.
  • New tests/test_ci_swiftpm_manifest_cache.sh pins the fixed environment, the key's inputs, and the stage/install round trip.

Written for commit 1ba930b. Summary will update on new commits.

Review in cubic

Summary by CodeRabbit

  • Chores
    • CI workflows now reuse Swift package manifest caches during dependency resolution, reducing repeated work. Cache misses or restore failures do not block builds.
    • Expanded CI guard checks to cover Swift package manifest caching.

SwiftPM keys each evaluated Package.swift on the whole process
environment, so on Actions every step misses. Resolve under a fixed
environment, save the cache from seed-derived-data on main, and restore
it read-only in compile admission and cli-pipe-regressions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@coderabbitai

coderabbitai Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Warning

Review limit reached

Next included review available in 2 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 2998ea1f-beee-4de1-9962-fabd2eebb55c

📥 Commits

Reviewing files that changed from the base of the PR and between 7cf4e10 and 1ba930b.

📒 Files selected for processing (9)
  • .github/workflows/ci-guards.yml
  • .github/workflows/ci-macos.yml
  • .github/workflows/cli-pipe-regressions.yml
  • .github/workflows/seed-derived-data.yml
  • scripts/ci/compile-app-host-test-product.sh
  • scripts/ci/swiftpm-manifest-cache.sh
  • tests/test_ci_canonical_build_root.py
  • tests/test_ci_swiftpm_manifest_cache.sh
  • tests/test_ci_test_compilation_cache_seed.sh
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@teamleaderleo
teamleaderleo enabled auto-merge (squash) September 24, 2026 16:38
@teamleaderleo
teamleaderleo merged commit 2e5cee0 into main Sep 24, 2026
58 checks passed
rustybret pushed a commit to rustybret/bmux that referenced this pull request Sep 24, 2026
2e5cee0 ci: restore SwiftPM's manifest cache for package resolves (manaflow-ai#14257)
d8b10f4 ci: retire the persistent Mac PR compile pilot (manaflow-ai#14232)

# Conflicts:
#	.github/workflows/ci-guards.yml
#	.github/workflows/ci-macos.yml
#	.github/workflows/ci.yml
#	.github/workflows/cli-pipe-regressions.yml
#	.github/workflows/seed-derived-data.yml
azooz2003-bit pushed a commit that referenced this pull request Sep 24, 2026
…step

#14257 was branched before #14232 removed the step, so its three
package-cache conditions still read steps.persistent-restore, and
actionlint fails on main and every open PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 2ea08f1)
teamleaderleo added a commit that referenced this pull request Sep 24, 2026
…step (#14260)

#14257 was branched before #14232 removed the step, so its three
package-cache conditions still read steps.persistent-restore, and
actionlint fails on main and every open PR.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
azooz2003-bit added a commit that referenced this pull request Sep 24, 2026
* test: require BETA notification extension registration

* fix: use registered BETA notification extension identity

* ci: drop compile admission's reads of the retired persistent-restore step

#14257 was branched before #14232 removed the step, so its three
package-cache conditions still read steps.persistent-restore, and
actionlint fails on main and every open PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 2ea08f1)

* ci: prepare signed iOS candidates before TestFlight upload

* test: exclude prepared candidates from upload assignment

* test: reject incomplete iOS candidates and stale BETA defaults

* fix: publish only complete iOS candidates with valid BETA defaults

---------

Co-authored-by: Leo Li <cheerleaderleo@outlook.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Measured on real PR runs. Seed run 36034205743 saved the cache at 17:33Z: 162 entries, 712 KB. Numbers below are "Resolve Swift packages" (admission) and "Resolve Swift package dependencies" (cli-pipe-regressions) on ci.yml pull_request runs.

lane before (11:00 to 16:40Z) after (created 17:34Z to 18:40Z), macOS 26 pools after, macOS 15 pool
admission median 63 s (n=21, IQR 60 to 71) 12 to 24 s, median 17 s (n=12) 51 to 121 s (n=5)
cli-pipe-regressions median 48 s (n=13, IQR 44 to 51) 13 to 23 s, median 15.5 s (n=8) 40 to 58 s (n=3)
  • Every job on the macOS 26 pools hit the cache.
  • Every job the PR pool picker sent to blacksmith-6vcpu-macos-15 missed. The key names the Xcode version, and the seeder runs only on the macOS 26 pool, so that pool has no seed. It falls back to today's resolve.
  • The follow-up to seed the macOS 15 pool as well is next.

Signed: Manifold g1 ✨

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant