Skip to content

ci: seed the test compilation cache from one clean build - #13797

Merged
teamleaderleo merged 2 commits into
mainfrom
ci/seed-test-compilation-cache-from-one-build
Sep 22, 2026
Merged

teamleaderleo merged 2 commits into
mainfrom
ci/seed-test-compilation-cache-from-one-build

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Sep 22, 2026 •

Copy link
Copy Markdown
Collaborator

What

refresh-test-compilation-cache restored its own last seed by prefix before rebuilding, so every scheduled run stacked another build's objects onto one CAS. Xcode keeps the primary generation and the upstream it faults objects in from, and that pair grew until it crossed the 5 GiB save bound today:

scheduled run pruned CAS size
09-20 07:33 3,542,996 KiB cold, no prefix restore
09-21 06:54 3,907,208 KiB
09-21 12:37 4,998,584 KiB
09-21 18:33 5,008,564 KiB
09-22 01:00 5,020,552 KiB
09-22 06:43 5,019,980 KiB
09-22 12:38 5,027,256 KiB last run that saved
09-22 18:33 5,293,516 KiB over the 5,242,880 KiB bound, save skipped

Past the bound the save step is skipped, so the next run restores the same older entry and lands past the bound again. nightly.yml already documents that state for the Release seed:

Nothing is saved, so the next build restores the same older entry and lands here again: the cache freezes at its last saved entry until the bound (and COMPILATION_CACHE_LIMIT_SIZE) is raised.

The pruner is not at fault. It correctly kept v1.9, v1.10 and removed nothing: with two live generations there is nothing stale to drop.

Change

Keep the exact key, drop the prefix fallback. The exact key still short-circuits a re-run of the same revision, which is the only case it ever hit — a prefix match sets cache-hit to false, so it never skipped a step, it only pre-populated the CAS.

Seeding from one clean build is better on every axis I could measure:

  • Bounded. 3.5 GiB, so the bound stops being reachable instead of being raised.
  • Complete. The build it seeds from is main, so a cold CAS already covers all of main. The stacked objects only covered older revisions of main, which is what a pull request is least likely to match.
  • Faster to produce. The one cold run took 18 min; the warm ones took 19, 19, 21 and 25.
  • Faster to consume. Every pull request restores this entry, so it is ~1.5 GiB less to download on each one.

Verification

  • tests/test_ci_test_compilation_cache_seed.sh fails on the first commit, passes on the second.
  • The new guard checks both directions, because they differ: the seeder must not fall back to a prefix, and admission must, since its exact key names a base revision no seeder run ever built.
  • Full guard sweep: 119 commands. Three fail, all three identically on clean origin/main — test_check_ghostty_zig_workflows.py (no bashlex locally), test_ghostty_zig_version_sync.sh (ghostty submodule not checked out), and test_review_fabric.py, which is red on origin/main at 5a848ea and unrelated to this change.

Relationship to #13754

Independent, and both are needed. #13754 makes the seed reachable by a pull request at all; this makes it keep updating. #13754 also resets the size ratchet as a side effect, because a new runner means a new fingerprint and so the first run finds no prefix to stack on — but without this change the ratchet would just climb back to the bound and freeze again, which the table above shows takes about three runs.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Summary by cubic

Stops the test compilation cache seed from growing past the 5 GiB save bound and freezing at its last saved entry. The seeder restored its own last seed by prefix, so each scheduled run stacked another build's objects onto the CAS until it crossed the bound, at which point saving was skipped and the cache could no longer update.

Seeding from one clean build keeps the cache bounded and current: 3.5 GiB instead of 5.0 GiB, covers all of main, and takes 18 min to produce instead of 19–25.

  • Keeps the exact-key restore, which still short-circuits re-runs of the same revision, and drops the prefix fallback.
  • Adds a guard for both directions: the seeder must not prefix-restore, while the admission path must keep its prefix fallback since its exact key names a base revision no seeder run built.

Written for commit c8de165. Summary will update on new commits.

Review in cubic

teamleaderleo and others added 2 commits September 22, 2026 16:21
The seeder restores its own last seed by prefix, so every run stacks
another build's objects onto one CAS. Xcode keeps the primary generation
and the upstream it faults from, and that pair climbed 3.5 -> 5.0 GiB over
eight scheduled runs and crossed the 5 GiB save bound on 2026-09-22.

Past the bound the save step is skipped, so the next run restores the same
older entry and lands past the bound again: nightly.yml already documents
that state for the Release seed as the cache freezing "at its last saved
entry until the bound (and COMPILATION_CACHE_LIMIT_SIZE) is raised".

Guard both directions, because they differ. The seeder must not fall back
to a prefix; admission must, since its exact key names a base revision no
seeder run ever built.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The seeder restored its own last seed by prefix, so each scheduled run
stacked another build's objects onto one CAS. Xcode keeps the primary
generation plus the upstream it faults from, and that pair grew over the
last eight runs until it crossed the 5 GiB save bound:

  09-20 07:33  3542996 KiB   (cold, no prefix restore)
  09-21 06:54  3907208 KiB
  09-21 12:37  4998584 KiB
  09-21 18:33  5008564 KiB
  09-22 01:00  5020552 KiB
  09-22 06:43  5019980 KiB
  09-22 12:38  5027256 KiB   last run that saved
  09-22 18:33  5293516 KiB   over the bound, save skipped

Past the bound nothing is saved, so the next run restores the same older
entry and lands past it again. nightly.yml already describes that state for
the Release seed: the cache freezes at its last saved entry.

Keep the exact key, which still short-circuits a re-run of one revision,
and drop the prefix fallback. A cold build covers all of main, since the
build it seeds from is main, and it measured both smaller (3.5 GiB against
5.0) and faster (18 min against 19-25) than a stacked one. Every pull
request restores this entry, so it is also 1.5 GiB less to download.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 22, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 20 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 404ce5fc-e282-49c3-9d82-3246cce928e2

📥 Commits

Reviewing files that changed from the base of the PR and between d7f6648 and c8de165.

📒 Files selected for processing (2)
  • .github/workflows/nightly.yml
  • tests/test_ci_test_compilation_cache_seed.sh

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@teamleaderleo
teamleaderleo merged commit 48786b1 into main Sep 22, 2026
48 of 50 checks passed
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

#13754 and #13797 are complementary, not duplicates — please don't close either as a dupe of the other.

They touch exactly the same two files (.github/workflows/nightly.yml, tests/test_ci_test_compilation_cache_seed.sh), which is why an overlap scan flags them as a collision. They fix different bugs:

  • ci: seed the compilation cache on the runner pull requests restore it from #13754 — the seeder ran on vars.MACOS_RUNNER_15 while pull-request admission runs on MACOS_RUNNER_PR. The fingerprint hashes the workspace path and the pools lay it out differently, so the seed existed but was unreachable and every PR compiled cold. Fix: the seeder tracks the pool admission actually restores from.
  • ci: seed the test compilation cache from one clean build #13797 — the seeder restored its own last seed by prefix, so every run stacked another build's objects onto the CAS (3.5 → 5.0 GiB over eight runs), crossed the save bound, and then nothing could save again. Fix: exact key only on the seeder, while deliberately keeping the prefix fallback on admission, since admission's exact key names a base revision no seeder run built.

One makes the seed reachable; the other stops it freezing. Landing only one leaves the other bug.

Verified they compose — merging c8de165 (#13797) into a0bb348 (#13754):

merge: clean, no conflicting files
tests/test_ci_test_compilation_cache_seed.sh: exit 0, 14 PASS
  PASS: the seeder runs on the runner pull request admission restores from   (#13754)
  PASS: the seeder seeds from one clean build                                (#13797)
  PASS: pull requests find the seed by prefix                                (#13797)

Both assertions survive together, so merge order does not matter.

Context: per #13742, MACOS_RUNNER_PR is not currently set as a repository variable, so #13754's vars.MACOS_RUNNER_PR || 'blacksmith-6vcpu-macos-15' resolves to the fallback today — correct either way, but worth knowing the variable is doing nothing until an admin sets it.

teamleaderleo added a commit that referenced this pull request Sep 23, 2026
* docs: tell agent sessions how not to duplicate each other

Several agent sessions work this repo at once and cannot see each other.
Nothing in CLAUDE.md says so, and the resulting waste is now measurable.

On 2026-09-22 a shared observable -- main going red on
test_ci_executes_review_fabric_contracts -- reached every session at once.
Each diagnosed it independently and opened a PR: #13785, #13788, #13800,
#13801 and #13802, five PRs on one test function in twenty-one minutes, two
of them five seconds apart. One landed. The reviewer attention spent on the
other four is the cost this section exists to avoid.

Two failures showed up repeatedly and are written down here because neither
is guessable:

Sessions share one GitHub account, so `author` and `mergedBy` name the
account and never the actor. Three separate claims about which session did
what were made from those fields today, all wrong, and two were relayed to
the user before being retracted.

GitHub keeps serving `mergeable` and `mergeStateStatus` on closed and merged
pull requests, where they are stale. Reading CONFLICTING off an already
merged PR sent a session to resolve a conflict that did not exist, twice.

The last paragraph guards the opposite error. #13754 and #13797 changed
exactly the same two files, fixed different bugs, and both merged, so an
overlap scan keyed on file paths would have proposed closing a good PR.
Composing them locally and running the shared test is what distinguishes
the cases.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs: give sessions a callsign to sign their work with

The section above tells sessions how not to collide. It does not give them a
way to say who they were, and that gap produced its own failures today: three
claims about which session opened, merged or reviewed something, every one of
them read off `author` or `mergedBy`, every one wrong, two relayed to the user
before being retracted.

Those fields name the shared push account. Nothing in the repository answers
"which session did this", so sessions inferred it from timing and were wrong.
A callsign in a commit trailer answers it directly.

Stated as attribution and not authority, deliberately. The Stensibly product
model is explicit that callsigns, names, branches and prior activity never
substitute for current authority evidence, and a self-assigned name two
sessions can pick independently is exactly the kind of identity that must not
gate an action. It records who acted. It grants nothing.

This commit signs itself, which is the whole convention.

Callsign: Teakettle 🫖
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs: correct the callsign section against the live registry

The previous commit invented a convention. There is already a working one, and
checking it showed the invented version wrong in three ways.

`teamleaderleo/stensibly` #454 is a live registrar: a `github-actions[bot]`
workflow that accepts `/callsign reserve`, answers in seconds with a
`callsign-receipt/v0` carrying an accepted generation and a 24h lease, and
releases on request. Its worker quickstart is
`docs/callsign-registry-dogfood.md` in that repo. This section now points there
instead of describing a parallel scheme.

I reserved through it rather than trusting the document, and each correction
below is something the receipt disproved:

The sigil is derived from the callsign by the registrar, not chosen by the
worker. Reserving `Teakettle` returned `💾`, not the emoji the previous commit
had picked for itself and put in its own trailer.

Names are leased. Collision keys are compared without case or separators, so
`Rook`, `rook` and `r-o_o k` are one name. The previous commit said collisions
were expected and tolerable, which is true of the derived sigil and false of
the name.

A generation may be shown only from an accepted receipt, with `pending` or
`unregistered` as the honest fallback. The previous commit had no notion of a
generation at all.

The sign-off format follows the registry's: `— <Callsign> g<generation>
<sigil>`, not a bare name and emoji.

Attribution and not authority is unchanged and now cites its owner:
`teamleaderleo/quarry` #1103 tracks the defect that a callsign in comment text
is marker text rather than an authenticated principal.

Callsign: Teakettle g1 💾
Run: run_cmux_ci_delineation_20260922_01
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs: check local worktrees and recent remote branches

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant