Skip to content

ci: seed the compilation cache on the runner pull requests restore it from - #13754

Merged
teamleaderleo merged 4 commits into
mainfrom
ci/test-layer-admission
Sep 22, 2026
Merged

teamleaderleo merged 4 commits into
mainfrom
ci/test-layer-admission

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Sep 22, 2026 •

Copy link
Copy Markdown
Collaborator

The Xcode compilation cache nightly seeds from main has never been restorable by a pull request. Every PR that compiles has been compiling cold.

Evidence

ci.yml run 35740499075 — a pull request touching five files, all under cmuxTests/:

Cache not found for input keys:
  xcode-compilation-test-macOS-ARM64-3f6ccf06eca67f0b54759694d11f1c77-f7132b3d5...,
  xcode-compilation-test-macOS-ARM64-3f6ccf06eca67f0b54759694d11f1c77-

The exact key and the prefix fallback both missed. Step timings for that job:

step time
Compile app-host test product 19.9 min
Resolve Swift packages 2.9 min
Package compiled product 1.8 min
Cache Swift packages 1.7 min
Checkout 1.1 min
job total 29.3 min

Cause

compile-app-host-test-product.sh fingerprint hashes xcodebuild -version, $PWD and the DerivedData path, and says why:

Runner pools lay the workspace out differently, and a seed built under another layout cannot hit, so it should be a cache miss and not a download.

The seeder runs on vars.MACOS_RUNNER_15 → warp-macos-15-arm64-6x. Confirmed on the job labels of refresh-test-compilation-cache.

macos-compile-admission selects, for pull_request, vars.MACOS_RUNNER_PR || 'blacksmith-6vcpu-macos-15' — and MACOS_RUNNER_PR is not set, so it lands on blacksmith-6vcpu-macos-15. Confirmed on the job labels of a real admission run.

Different pools, different workspace path, different fingerprint, guaranteed miss. nightly.yml already carries the requirement directly above the job — "Match the admission job's runner and Xcode. The cache key carries the toolchain and the build paths, so a mismatch is a miss, not a wrong hit." The admission job later gained a PR-specific runner and the seeder stayed where it was. Nothing enforced the pairing, and a miss is silent by design.

Change

Point the seeder at the runner the pull request path uses. No cache key, schema, build input or product identity changes — this only puts the seed where the consumer looks.

The guard lands first and fails on main:

FAIL: refresh-test-compilation-cache must run on the same runner pull request admission uses,
      or the seed it writes can never be restored.
  admission:     runs-on: ${{ github.event_name == 'pull_request' && (vars.MACOS_RUNNER_PR || 'blacksmith-6vcpu-macos-15') || ... }}
  seeder:        runs-on: ${{ vars.MACOS_RUNNER_15 || 'blacksmith-6vcpu-macos-15' }}

tests/test_ci_test_compilation_cache_seed.sh already pinned the shared script, the shared build paths and the shared key prefix — every ingredient of the key except the runner that determines $PWD. This adds that.

What this does not claim

The first seed after this lands still has to be written by a scheduled nightly run before any PR can hit it, so the improvement appears on the next 17 */6 * * * cycle, not immediately. I have not measured the warm compile time, because no warm compile has ever happened on this path — the 19.9 min figure is the cold cost this removes the cause of, not a measured saving. Worth watching one PR compile after the first post-merge seed to confirm the hit and get the real number.

Both jobs are on Blacksmith macOS, which is free on this repo, so this is pull-request latency rather than spend.

Full ci-guards.yml sweep locally: 109 commands, all pass except test_check_ghostty_zig_workflows.py (no bashlex) and test_ghostty_zig_version_sync.sh (no ghostty submodule), both of which fail identically on a clean origin/main checkout.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Summary by cubic

Fixes the Xcode compilation cache so pull requests can restore the nightly seed from main instead of compiling cold. The seeder ran on a different runner pool than pull request admission, and since the cache key hashes the workspace path, the seed never matched — the compile step alone took 19.9 minutes per PR.

  • Point the seeder at the runner pull request admission selects (vars.MACOS_RUNNER_PR || 'blacksmith-6vcpu-macos-15'); no cache key or build input changes.
  • Add a guard test that fails if the seeder and the admission job ever use different runners.

The first seed after this lands appears only after the next scheduled nightly run, so PRs benefit starting with the following cycle, and the warm compile time is still unmeasured.

Written for commit a0bb348. Summary will update on new commits.

Review in cubic

Summary by CodeRabbit

  • Bug Fixes

    • Updated nightly cache seeding to use the same macOS runner configuration as pull request validation, improving consistency across CI jobs.
  • Tests

    • Added regression coverage to verify that cache seeding and pull request admission jobs select matching runners.

teamleaderleo and others added 2 commits September 22, 2026 11:20
`compile-app-host-test-product.sh fingerprint` hashes `xcodebuild
-version`, `$PWD` and the DerivedData path, and the script says why:
"Runner pools lay the workspace out differently, and a seed built under
another layout cannot hit, so it should be a cache miss and not a
download." nightly.yml repeats the requirement above the seeder —
"Match the admission job's runner and Xcode."

Nothing enforced it. This test does, and fails on main: the seeder runs
on `vars.MACOS_RUNNER_15` while pull request admission runs on
`vars.MACOS_RUNNER_PR || 'blacksmith-6vcpu-macos-15'`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… from

The Xcode compilation cache that nightly seeds from main has never been
restorable by a pull request. The seeder ran on `vars.MACOS_RUNNER_15`
(`warp-macos-15-arm64-6x`) while `macos-compile-admission` picks
`vars.MACOS_RUNNER_PR || 'blacksmith-6vcpu-macos-15'` for
`pull_request`, and `MACOS_RUNNER_PR` is unset. The cache key hashes the
workspace path, which differs between runner pools, so every pull
request missed and compiled cold.

Observed on ci.yml run 35740499075, a pull request touching only five
files under cmuxTests/:

    Cache not found for input keys:
      xcode-compilation-test-macOS-ARM64-3f6ccf06eca67f0b54759694d11f1c77-f7132b3d5...,
      xcode-compilation-test-macOS-ARM64-3f6ccf06eca67f0b54759694d11f1c77-

Both the exact key and the prefix fallback missed, and "Compile app-host
test product" then took 19.9 of the job's 29.3 minutes.

Point the seeder at the same runner the pull request path uses. This
changes no cache key, no schema and no build input; it only puts the
seed where the consumer looks.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@coderabbitai

coderabbitai Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Warning

Review limit reached

Next included review available in 14 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 25095525-5331-4d00-b62e-0a08912463ce

📥 Commits

Reviewing files that changed from the base of the PR and between bbbb633 and a0bb348.

📒 Files selected for processing (2)
  • .github/workflows/nightly.yml
  • tests/test_ci_test_compilation_cache_seed.sh

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 9151a2ee-7c7c-467c-a6a4-134d69380ec8

📥 Commits

Reviewing files that changed from the base of the PR and between e6b3d6b and bbbb633.

📒 Files selected for processing (2)
  • .github/workflows/nightly.yml
  • tests/test_ci_test_compilation_cache_seed.sh

Included review availability: Your plan provides up to 10 included reviews per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

The nightly compile-cache seeder now uses the pull-request macOS runner. A regression test verifies that the seeder and admission jobs use identical runner expressions.

Changes

Compile-cache runner alignment

Layer / File(s) Summary
Runner selection and validation
.github/workflows/nightly.yml, tests/test_ci_test_compilation_cache_seed.sh
The cache seeder uses vars.MACOS_RUNNER_PR with the existing fallback. The regression test verifies that the seeder matches the macos-compile-admission runner expression.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to bbbb6

No actionable merge-blocking risk remains; the cache seeder now tracks the pull-request admission runner.

🚥 Pre-merge checks | ✅ 25
✅ Passed checks (25 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the main change: aligning compilation-cache seeding with the runner used by pull requests. It is concise and specific, although its wording is slightly awkward.
Description check ✅ Passed The description provides a detailed problem statement, evidence, root cause, implementation summary, testing results, and rollout limitations. It is mostly complete and relevant. The template's review…
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Cmux Cloud Persistent Session And Early Input ✅ Passed The pull request changes only the nightly compilation-cache runner selection and its guard test. The authoritative diff introduces no Cloud terminal creation, cmux-tui transport, manual renderer, read…
Cmux Swift Actor Isolation ✅ Passed The pull request changes only .github/workflows/nightly.yml and tests/test_ci_test_compilation_cache_seed.sh. The authoritative diff contains no Swift or production code changes, so it cannot intr…
Cmux Swift Blocking Runtime ✅ Passed PASS: The PR changes only .github/workflows/nightly.yml and tests/test_ci_test_compilation_cache_seed.sh; the authoritative diff contains no Swift, Objective-C, or Objective-C++ production changes…
Cmux Browser Automation Off-Main ✅ Passed PASS: The pull request changes only the nightly compilation-cache runner expression and its CI regression test. The authoritative diff contains no browser socket automation commands, WebKit/AppKit acc…
Cmux Expensive Synchronous Load ✅ Passed PASS: The pull request changes only .github/workflows/nightly.yml and tests/test_ci_test_compilation_cache_seed.sh. The diff contains no Swift or production code changes and does not add or move a…
Cmux Cache Substitution Correctness ✅ Passed PASS: The pull request changes only a GitHub Actions workflow and a shell CI guard. It introduces no production Swift, TypeScript, or JavaScript change, and it does not alter a persistence, history, u…
Cmux No Hacky Sleeps ✅ Passed PASS: The PR changes only GitHub Actions workflow configuration and a shell regression-test guard. The workflow change is explicitly out of scope under the rule. The added shell code performs determin…
Cmux Algorithmic Complexity ✅ Passed PASS: The pull request changes only a GitHub Actions runner expression and adds assertions to a test script. It does not add or modify production Swift, TypeScript, JavaScript, shell, or runtime algor…
Cmux Swift Concurrency ✅ Passed PASS: The pull request changes only .github/workflows/nightly.yml and tests/test_ci_test_compilation_cache_seed.sh. The authoritative diff contains no Swift files and introduces no Swift concurren…
Cmux Swift @Concurrent ✅ Passed The pull request changes only GitHub Actions YAML and a shell regression test. The authoritative diff contains no Swift files, Swift declarations, async helpers, actor isolation, or @concurrent annota…
Cmux Swift Package Boundaries ✅ Passed PASS: The pull request changes only .github/workflows/nightly.yml and tests/test_ci_test_compilation_cache_seed.sh. It introduces no production Swift changes, so the Swift package boundary rule do…
Cmux Swiftpm Lockfiles ✅ Passed PASS: The PR changes only .github/workflows/nightly.yml runner selection and a regression test. The diff contains no Package.swift, Package.resolved, .gitignore, or Xcode project package-refer…
Cmux Swift Logging ✅ Passed PASS: The pull request changes only .github/workflows/nightly.yml and tests/test_ci_test_compilation_cache_seed.sh. It adds no Swift code and adds no production logging. The shell test's echo st…
Cmux User-Facing Error Privacy ✅ Passed PASS: The diff changes only an internal GitHub Actions runner selection, developer comments, and a CI regression test. The added FAIL/PASS messages are test output, not cmux user-facing text. No c…
Cmux Full Internationalization ✅ Passed The PR changes only .github/workflows/nightly.yml and tests/test_ci_test_compilation_cache_seed.sh. The changes are CI configuration, regression-test logic, and developer-facing comments. They add…
Cmux Swiftui State Layout ✅ Passed PASS: This pull request changes only .github/workflows/nightly.yml and tests/test_ci_test_compilation_cache_seed.sh. The authoritative diff contains no Swift or SwiftUI changes, so the SwiftUI sta…
Cmux Architecture Rethink ✅ Passed The pull request changes only .github/workflows/nightly.yml and a shell regression test. It adds no Swift files or Swift architecture code. The added logic aligns CI runner selection and validates c…
Cmux Swift Auxiliary Window Close Shortcuts ✅ Passed PASS: The PR changes only .github/workflows/nightly.yml and tests/test_ci_test_compilation_cache_seed.sh. It adds CI runner selection and shell checks. It adds no Swift, NSWindow, NSPanel, NSWindo…
Cmux Source Artifacts ✅ Passed The PR changes only .github/workflows/nightly.yml and tests/test_ci_test_compilation_cache_seed.sh. Both are intentional hand-written CI configuration and test source. The diff adds runner configu…
Cmux No Test Or Debug Seam In Production Source ✅ Passed PASS: The pull request changes only .github/workflows/nightly.yml and tests/test_ci_test_compilation_cache_seed.sh. The authoritative diff contains no Swift file under a production Sources/ path…
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Commit to this branch
  • Create a new PR
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@blacksmith-sh

This comment has been minimized.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Reproduced both cache keys from their inputs, so the runner is provably the whole difference rather than the most likely one.

compile-app-host-test-product.sh fingerprint hashes xcodebuild -version, $PWD and the derived-data path. Feeding it each job's observed values:

$ mk(){ printf 'Xcode 26.3\nBuild version 17C529\nworkspace=%s\nderived-data=%s\n' "$1" "$2" | sha256sum | cut -c1-32; }

$ mk /Users/runner/work/cmux/cmux  /Users/runner/work/_temp/cmux-derived-data-compile-admission
c4728a44cf18adc521070ca61d009904

$ mk /Users/runner/_work/cmux/cmux /Users/runner/_work/_temp/cmux-derived-data-compile-admission
3f6ccf06eca67f0b54759694d11f1c77

Those match the two fingerprints in the logs exactly:

  • seeder, run 35767936321 on warp-macos-15-arm64-6x, wrote xcode-compilation-test-macOS-ARM64-c4728a44cf18adc521070ca61d009904-...
  • admission, run 35740499075 on blacksmith-6vcpu-macos-15, asked for xcode-compilation-test-macOS-ARM64-3f6ccf06eca67f0b54759694d11f1c77-...

The only difference between the two inputs is work against _work: Warp lays the workspace out under /Users/runner/work, Blacksmith under /Users/runner/_work. Both runners are on Xcode 26.3 build 17C529, so the toolchain component is identical and contributes nothing to the mismatch.

That rules out the alternative I was worried about. CMUX_CI_XCODE_APP is empty in the admission job, so it takes the runner's default Xcode rather than a pinned one; had those defaults differed, matching the runner pool would not have been sufficient on its own. They do not differ, so it is.

@cursor

cursor Bot commented Sep 22, 2026

Copy link
Copy Markdown

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Rebased onto main (abd614f) after #13723, #13788 and #13797 landed. Both guards in tests/test_ci_test_compilation_cache_seed.sh pass together, and tests/test_ci_canonical_build_root.py still passes.

Worth being explicit about how this now relates to #13723, since that PR fixed the same root cause I was chasing and reached the same evidence independently — down to the same two fingerprints and the one-underscore diagnosis.

#13723 landed the mechanism for a better long-term fix: build from $CMUX_CI_CANONICAL_ROOT/src so the paths stop being pool-specific and the key can drop them. Its own commit message scopes the remainder out: "Nothing calls canonical-build-root.sh yet ... that belongs in the change that converts the seeder and admission together." Until that conversion happens, the seeder is still on vars.MACOS_RUNNER_15 and every pull request still compiles cold.

So this stays useful as the change that makes the seed reachable now, and it has a second benefit I did not claim originally: MACOS_RUNNER_15 is WarpBuild, which is paid, and MACOS_RUNNER_PR resolves to Blacksmith, which is free on this repo. Moving the seeder also moves a ~20 minute job off paid capacity four times a day.

When the canonical conversion lands, the fingerprint no longer depends on the pool, and the guard added here — asserting the seeder matches admission's runner — becomes an unnecessary constraint rather than a wrong one. It should be replaced at that point by a guard that both lanes build from the canonical root. Happy to close this instead if you would rather go straight to the conversion; the tradeoff is that every pull request keeps compiling cold until it lands.

@teamleaderleo
teamleaderleo merged commit 387ec6e into main Sep 22, 2026
47 of 48 checks passed
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

#13754 and #13797 are complementary, not duplicates — please don't close either as a dupe of the other.

They touch exactly the same two files (.github/workflows/nightly.yml, tests/test_ci_test_compilation_cache_seed.sh), which is why an overlap scan flags them as a collision. They fix different bugs:

  • ci: seed the compilation cache on the runner pull requests restore it from #13754 — the seeder ran on vars.MACOS_RUNNER_15 while pull-request admission runs on MACOS_RUNNER_PR. The fingerprint hashes the workspace path and the pools lay it out differently, so the seed existed but was unreachable and every PR compiled cold. Fix: the seeder tracks the pool admission actually restores from.
  • ci: seed the test compilation cache from one clean build #13797 — the seeder restored its own last seed by prefix, so every run stacked another build's objects onto the CAS (3.5 → 5.0 GiB over eight runs), crossed the save bound, and then nothing could save again. Fix: exact key only on the seeder, while deliberately keeping the prefix fallback on admission, since admission's exact key names a base revision no seeder run built.

One makes the seed reachable; the other stops it freezing. Landing only one leaves the other bug.

Verified they compose — merging c8de165 (#13797) into a0bb348 (#13754):

merge: clean, no conflicting files
tests/test_ci_test_compilation_cache_seed.sh: exit 0, 14 PASS
  PASS: the seeder runs on the runner pull request admission restores from   (#13754)
  PASS: the seeder seeds from one clean build                                (#13797)
  PASS: pull requests find the seed by prefix                                (#13797)

Both assertions survive together, so merge order does not matter.

Context: per #13742, MACOS_RUNNER_PR is not currently set as a repository variable, so #13754's vars.MACOS_RUNNER_PR || 'blacksmith-6vcpu-macos-15' resolves to the fallback today — correct either way, but worth knowing the variable is doing nothing until an admin sets it.

teamleaderleo added a commit that referenced this pull request Sep 23, 2026
* docs: tell agent sessions how not to duplicate each other

Several agent sessions work this repo at once and cannot see each other.
Nothing in CLAUDE.md says so, and the resulting waste is now measurable.

On 2026-09-22 a shared observable -- main going red on
test_ci_executes_review_fabric_contracts -- reached every session at once.
Each diagnosed it independently and opened a PR: #13785, #13788, #13800,
#13801 and #13802, five PRs on one test function in twenty-one minutes, two
of them five seconds apart. One landed. The reviewer attention spent on the
other four is the cost this section exists to avoid.

Two failures showed up repeatedly and are written down here because neither
is guessable:

Sessions share one GitHub account, so `author` and `mergedBy` name the
account and never the actor. Three separate claims about which session did
what were made from those fields today, all wrong, and two were relayed to
the user before being retracted.

GitHub keeps serving `mergeable` and `mergeStateStatus` on closed and merged
pull requests, where they are stale. Reading CONFLICTING off an already
merged PR sent a session to resolve a conflict that did not exist, twice.

The last paragraph guards the opposite error. #13754 and #13797 changed
exactly the same two files, fixed different bugs, and both merged, so an
overlap scan keyed on file paths would have proposed closing a good PR.
Composing them locally and running the shared test is what distinguishes
the cases.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs: give sessions a callsign to sign their work with

The section above tells sessions how not to collide. It does not give them a
way to say who they were, and that gap produced its own failures today: three
claims about which session opened, merged or reviewed something, every one of
them read off `author` or `mergedBy`, every one wrong, two relayed to the user
before being retracted.

Those fields name the shared push account. Nothing in the repository answers
"which session did this", so sessions inferred it from timing and were wrong.
A callsign in a commit trailer answers it directly.

Stated as attribution and not authority, deliberately. The Stensibly product
model is explicit that callsigns, names, branches and prior activity never
substitute for current authority evidence, and a self-assigned name two
sessions can pick independently is exactly the kind of identity that must not
gate an action. It records who acted. It grants nothing.

This commit signs itself, which is the whole convention.

Callsign: Teakettle 🫖
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs: correct the callsign section against the live registry

The previous commit invented a convention. There is already a working one, and
checking it showed the invented version wrong in three ways.

`teamleaderleo/stensibly` #454 is a live registrar: a `github-actions[bot]`
workflow that accepts `/callsign reserve`, answers in seconds with a
`callsign-receipt/v0` carrying an accepted generation and a 24h lease, and
releases on request. Its worker quickstart is
`docs/callsign-registry-dogfood.md` in that repo. This section now points there
instead of describing a parallel scheme.

I reserved through it rather than trusting the document, and each correction
below is something the receipt disproved:

The sigil is derived from the callsign by the registrar, not chosen by the
worker. Reserving `Teakettle` returned `💾`, not the emoji the previous commit
had picked for itself and put in its own trailer.

Names are leased. Collision keys are compared without case or separators, so
`Rook`, `rook` and `r-o_o k` are one name. The previous commit said collisions
were expected and tolerable, which is true of the derived sigil and false of
the name.

A generation may be shown only from an accepted receipt, with `pending` or
`unregistered` as the honest fallback. The previous commit had no notion of a
generation at all.

The sign-off format follows the registry's: `— <Callsign> g<generation>
<sigil>`, not a bare name and emoji.

Attribution and not authority is unchanged and now cites its owner:
`teamleaderleo/quarry` #1103 tracks the defect that a callsign in comment text
is marker text rather than an authenticated principal.

Callsign: Teakettle g1 💾
Run: run_cmux_ci_delineation_20260922_01
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs: check local worktrees and recent remote branches

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant