ci: compare build inputs by content so an adopted seed rebuilds only real changes

[teamleaderleo]

Why

A seeded compile admission redid module work for unchanged code. Canary run 36022099083 adopted the seed of its own base commit (seed_distance 0), and its first build-for-testing still reran 94 SwiftDriver and 64 SwiftEmitModule tasks, every third-party package included (SwiftSyntax, Crypto, PostHog, MarkdownUI, IrohLib, WireGuardKit, ...). That is 43-46 CPU-s of emit plus 107-185 CPU-s of driver planning per admission. Main's seeder shows the same (run 36021070919: 94 / 65).

The XCBuild debugging trace from that run (EnableBuildDebugging), read against the seed's prior-build.db, names the inputs:

1. Xcode itself. 1,616 file nodes under /Applications/Xcode_26.6.app come back invalid-value: SDK headers, prebuilt-modules, XCTest/Testing/XCUIAutomation. Inode and device are already ignored (IgnoreFileSystemDeviceInodeChanges), and mode and size match. Only the mtime differs: 2026-09-21 20:21 vs 20:42 UTC. Blacksmith images install Xcode at image build time, so two VMs from the same pool (both blacksmith-12vcpu-macos-26) disagree. Every Swift driver task discovered these files, so every one reruns. No mtime replay can fix this, because the files are outside the checkout.
2. Generated package module maps. On the first build after adoption, Xcode's package integration rewrites every GeneratedModuleMaps/*.modulemap with identical bytes during CreateBuildDescription (93 Copy tasks in scheme 1, 0 in schemes 2 and 3). This happens outside llbuild, so its recorded stat never matches.
3. Source directory mtimes. The seed in that run predates ci: replay directory times when adopting a DerivedData seed #14241, which fixes them.

The inputs that changed_inputs counts (45-61) are not the cause. At distance 0 they are 35 git refs and FETCH_HEAD files in .ci-source-packages/repositories/*, workspace-state.json, 4 scripts/ci/__pycache__ files, and the diff under test.

What

compile-app-host-test-product.sh build runs xcodebuild with Swift Build's FileSystemMode=checksum-only, so llbuild compares file contents instead of stat. The variable is set only in the environment of those xcodebuild invocations; Swift Build reads it from its launching process first, so nothing global changes. A build database written in one mode reruns every task in the other, so the mode joins fingerprint. The seed prefix changes with it, and a checksum-mode build never adopts a device-agnostic seed.

Every seeded build already goes through build(): PR admission, seed-derived-data.yml, nightly, test-e2e (and #14226's E2E adoption uses canonical-build).

Evidence

Canary #14256 (run 36029379921, this change plus probes, 6 vCPU). Cold compile under checksum-only: 15 min, and all four scheme manifests report "file-system":"checksum-only". Then, in the same job:

rebuild (4 schemes)	time	SwiftDriver	SwiftEmitModule	SwiftCompile	CompileC
null	66 s	0	0	0	0
every checkout file and directory, every generated module map, and an SDK header touched with the same bytes	73 s	0	0	0	0
one app source edit	578 s	4	2	cmux + cmuxTests	0

The second row is what a fresh runner adopting a seed looks like; under device-agnostic it reran the driver and emit for every target. A toy package on Xcode 27 shows the same split: content-preserving touches rerun nothing, and a real edit rebuilds only its target.

Regression commits: d0c65cd adds SeededBuildFileSystemModeTests (fails: FileSystemMode unset, device-agnostic fingerprint); 84fee6a passes all 19 in tests/test_ci_canonical_build_root.py.

Interaction with mtime replay

The replay in e2e_warm_derived_data.py stays. As #14250 records, swift-driver decides a clang header or module changed by "newer than the last build", independent of llbuild, so stamping a changed input with the current time is still what makes the driver rebuild it. checksum-only only stops llbuild from rerunning tasks whose inputs are byte-identical. #14241's directory keys stay as a harmless fallback. The touch-everything probe touched every directory and reran nothing, so they no longer decide anything under this mode.

Rollout cost

The fingerprint change means the first admissions after merge miss the seed and compile cold until seed-derived-data.yml publishes a checksum-mode seed for the merge commit. That seed is itself a cold build, about 15-20 min. The test compilation cache key moves with the fingerprint too, so nightly's first run after merge misses it once.

The before/after number on a real test-only PR push (SwiftEmitModule count and the "cmux" scheme phase) comes after that first seed lands. Before: run 36001051652, 64 SwiftEmitModule tasks.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

---

Summary by cubic

Runs seeded builds in Swift Build's checksum-only file mode so an adopted seed rebuilds only inputs whose contents changed, instead of rerunning every SwiftDriver and SwiftEmitModule task for Xcode files whose mtime differs across Blacksmith images. The mode joins the seed fingerprint, and builds in one mode never adopt a seed recorded in the other.

Migration

• The first admissions after merge miss the seed and compile cold until a checksum-mode seed is published; the test-compilation cache key moves with the fingerprint too.

^{Written for commit 84fee6a. Summary will update on new commits.}

[coderabbitai]

Warning

Review limit reached

Next included review available in 1 minute.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 35e95887-a873-4402-84a9-5854f542063b

📥 Commits

Reviewing files that changed from the base of the PR and between 55d9b75 and 84fee6a.

📒 Files selected for processing (2)

• scripts/ci/compile-app-host-test-product.sh
• tests/test_ci_canonical_build_root.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}