Skip to content

ci: order R2 cache pointers by run ID, so seeder saves take effect - #14174

Merged
teamleaderleo merged 2 commits into
mainfrom
claude/r2-pointer-run-id
Sep 24, 2026
Merged

teamleaderleo merged 2 commits into
mainfrom
claude/r2-pointer-run-id

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The DerivedData seeder has saved a new seed after every main push since 02:41 UTC on 2026-09-24, but none of those saves became the "newest seed". Every seeder run and every pull request has adopted the seed of f2abe643, the commit that introduced seeding.

The cause is how the pointer is ordered. r2-cache.sh stamps each save with GITHUB_RUN_NUMBER and refuses to move a pointer to a lower number. Run numbers are counted per workflow, and two workflows write the admission-derived-data-… prefix:

  • nightly.yml, at run ~5,870. It wrote the current pointer at 02:41.
  • seed-derived-data.yml, at run ~38.

So every seeder save lost the comparison, and publish_pointer returned success without a log line.

Evidence from the last three successful seeder runs (35959372603, 35961043898, 35963073004):

  • Adoption: each restored …-f2abe643… with 436, 439 and 451 changed inputs, a gap that grows with every merge.
  • Builds: each took 845–985 s, which is about a cold build. For a diff of 1–5 app files the seeder has measured 30–130 s.
  • Saves: each saved its own key, with no pointer warning.
  • The pointer: https://ci-cache.cmux.com/v1/macOS-ARM64/latest/admission-derived-data-v1-macOS-ARM64-bac5277611926ed6b40291f1640fed47- still names f2abe643, last-modified 02:41:17 GMT, uncached (cf-cache-status: DYNAMIC).

Fix

  • Order by run ID. The generation is now GITHUB_RUN_ID, which increases across all workflows in the repository in the order runs are created. An older run that finishes late still cannot regress a pointer.
  • Recovery. Every run ID is far above the run numbers stored today, so the first save of each prefix after this lands repairs that pointer. The next seeder run fixes the seed pointer, with no manual R2 write.
  • Logging. A pointer left in place now logs pointer <prefix> names a newer save (generation A > B); left as is.

Testing

  • The first commit adds pointers order saves across workflows to tests/test_ci_r2_cache.sh. It fails on main: a save with a lower run number and a higher run ID does not replace the pointer.
  • With the fix, all 12 cases in tests/test_ci_r2_cache.sh pass. That includes the existing out-of-order and retry cases, which now set GITHUB_RUN_ID.
  • In the full ci-guards.yml sweep, the only local failures are the known ones caused by missing submodules and CI-only variables, plus timeouts from a machine at load ~120.

Not verified yet: a seeder run on main. Expected result: the first seeder run after merge reports saved … and the pointer names its commit. The next one then adopts it with a single merge's worth of changed inputs and builds in minutes rather than about 15.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Summary by cubic

Fixes R2 cache pointer ordering so DerivedData seeder saves can finally replace the pointer, instead of every seeder run and pull request adopting a stale seed from September 24.

  • Swaps pointer generation from per-workflow GITHUB_RUN_NUMBER to GITHUB_RUN_ID, which increases across all workflows in creation order.
  • Logs a message when a pointer is left in place because a newer save exists.
  • Adds a test covering two workflows writing the same prefix.

Migration

  • No action needed: the first save after this change repairs each stuck pointer.

Written for commit 866d2d4. Summary will update on new commits.

Review in cubic

Summary by CodeRabbit

  • Bug Fixes
    • Cache saves are now ordered across workflows using run IDs, so a newer save is preserved even when its run number is lower than another workflow’s.
    • When an existing cache pointer refers to a newer save, it remains unchanged and a message explains why.

teamleaderleo and others added 2 commits September 23, 2026 23:54
Fails on main: the pointer generation is GITHUB_RUN_NUMBER, a per-workflow
counter. nightly.yml (run ~5,870) and seed-derived-data.yml (run ~38)
write the same admission-derived-data prefix, so no seeder save can
replace the pointer nightly wrote, and publish_pointer skips it without a
word.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nightly.yml and seed-derived-data.yml both save admission-derived-data
seeds. The pointer generation was GITHUB_RUN_NUMBER, which each workflow
counts separately, so the pointer nightly wrote at run ~5,870 outranked
every seeder save (run ~38). Since 02:41 UTC on 2026-09-24 every seeder
and pull request has adopted the seed of f2abe64, about 450 changed
inputs behind, and each seeder build took 14-16 minutes.

GITHUB_RUN_ID increases across all workflows in creation order, so it
keeps the rule that an older run finishing late cannot regress a pointer.
Every run ID is far above the run numbers stored today, so the first save
after this lands repairs each stuck pointer. A pointer left in place now
says so in the log.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cursor

cursor Bot commented Sep 24, 2026

Copy link
Copy Markdown

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

@github-actions

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@coderabbitai

coderabbitai Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 775a8a18-4b1a-491a-9480-ae58bc5f097b

📥 Commits

Reviewing files that changed from the base of the PR and between 980f8a2 and 866d2d4.

📒 Files selected for processing (2)
  • scripts/ci/r2-cache.sh
  • tests/test_ci_r2_cache.sh

Included review availability: Your plan provides up to 10 included reviews per hour; 3 remain after this review.


📝 Walkthrough

Walkthrough

The cache save generation now uses GITHUB_RUN_ID instead of GITHUB_RUN_NUMBER. Pointer publication logs when it keeps a pointer with a newer generation. Tests cover save ordering and cross-workflow cases.

Changes

R2 cache ordering

Layer / File(s) Summary
Run-ID generation and pointer ordering
scripts/ci/r2-cache.sh, tests/test_ci_r2_cache.sh
Save generations now use GITHUB_RUN_ID. Pointer publication logs when it keeps a pointer with a newer generation. Tests check out-of-order saves and cross-workflow ordering.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: lawrencecchen

Merge Risk: ⚪ Minimal · up to 866d2

The change aligns cache-save ordering between the seed and nightly workflows. No actionable merge risk remains from the reviewed change.

🚥 Pre-merge checks | ✅ 24 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 2 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (24 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the main change: ordering R2 cache pointers by run ID so seeder saves take effect.
Description check ✅ Passed The description provides a detailed Summary and Testing section, explains the failure mode and recovery behavior, and clearly states that a post-merge seeder run is not yet verified. The template chec…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Cmux Cloud Persistent Session And Early Input ✅ Passed PASS: The pull request changes only scripts/ci/r2-cache.sh and tests/test_ci_r2_cache.sh. The diff changes R2 cache pointer generation from GITHUB_RUN_NUMBER to GITHUB_RUN_ID, adds pointer-ord…
Cmux Swift Actor Isolation ✅ Passed PASS: The pull request changes only scripts/ci/r2-cache.sh and tests/test_ci_r2_cache.sh. The diff contains no Swift files or Swift actor-isolation constructs. The custom check is therefore not ap…
Cmux Swift Blocking Runtime ✅ Passed PASS: The pull request changes only scripts/ci/r2-cache.sh and tests/test_ci_r2_cache.sh. The authoritative diff contains no Swift files or Swift runtime synchronization changes. The custom check …
Cmux Browser Automation Off-Main ✅ Passed The pull request changes only scripts/ci/r2-cache.sh and tests/test_ci_r2_cache.sh. The diff changes R2 cache generation handling from GITHUB_RUN_NUMBER to GITHUB_RUN_ID; it adds no browser so…
Cmux Expensive Synchronous Load ✅ Passed PASS: The pull request changes only scripts/ci/r2-cache.sh and tests/test_ci_r2_cache.sh. The authoritative diff contains no Swift files, agent-history loads, or interactive Swift call sites. The …
Cmux Cache Substitution Correctness ✅ Passed PASS: The pull request changes only scripts/ci/r2-cache.sh and tests/test_ci_r2_cache.sh. Both are shell scripts. The custom check applies only to production Swift, TypeScript, and JavaScript chan…
Cmux No Hacky Sleeps ✅ Passed The production shell diff changes generation ordering and logging only. It adds no sleep, timer, polling loop, fixed backoff, or wall-clock wait. The existing curl --retry-delay 2 and retry logic ar…
Cmux Algorithmic Complexity ✅ Passed PASS: The production diff only changes scalar generation selection and adds constant-time comparisons/logging in scripts/ci/r2-cache.sh:141-145 and :179-183. It introduces no nested collection sca…
Cmux Swift Concurrency ✅ Passed The pull request changes only scripts/ci/r2-cache.sh and tests/test_ci_r2_cache.sh. The authoritative diff contains no Swift files or Swift concurrency code, so the custom check is not applicable.
Cmux Swift @Concurrent ✅ Passed The reviewed range changes only scripts/ci/r2-cache.sh and tests/test_ci_r2_cache.sh. It contains no Swift files or Swift async functions, actor isolation, or @concurrent annotations. The Swift-…
Cmux Swift Package Boundaries ✅ Passed PASS: The pull request changes only scripts/ci/r2-cache.sh and tests/test_ci_r2_cache.sh. The authoritative diff contains no Swift, SwiftPM, or Xcode project files, so it cannot violate the Swift …
Cmux Swiftpm Lockfiles ✅ Passed The pull request changes only scripts/ci/r2-cache.sh and tests/test_ci_r2_cache.sh. The authoritative diff contains no Package.swift, Package.resolved, .gitignore, Xcode project, or workflow…
Cmux Swift Logging ✅ Passed The pull request changes only scripts/ci/r2-cache.sh and tests/test_ci_r2_cache.sh. It adds shell-script CLI messages and test output, not production Swift logging. Therefore the Swift logging fai…
Cmux User-Facing Error Privacy ✅ Passed PASS: The diff changes scripts/ci/r2-cache.sh, an internal CI cache helper, and its test. The changed text is GitHub Actions/cache operator output (pointer ... left as is and invalid run ID), no…
Cmux Full Internationalization ✅ Passed PASS. The PR changes only scripts/ci/r2-cache.sh and tests/test_ci_r2_cache.sh. The changed text is CI cache status/warning output and test assertions. It does not add Swift UI text, app catalog e…
Cmux Swiftui State Layout ✅ Passed PASS: The authoritative diff changes only scripts/ci/r2-cache.sh and tests/test_ci_r2_cache.sh. It contains no SwiftUI or Swift state/layout changes, so the specified SwiftUI failure conditions do…
Cmux Architecture Rethink ✅ Passed PASS: The pull request changes only scripts/ci/r2-cache.sh and tests/test_ci_r2_cache.sh. It does not change Swift code or introduce any listed architectural pattern. The implementation is a small…
Cmux Swift Auxiliary Window Close Shortcuts ✅ Passed PASS: The pull request changes only scripts/ci/r2-cache.sh and tests/test_ci_r2_cache.sh. The authoritative diff contains no Swift or window-related code, so the auxiliary-window close-shortcut ru…
Cmux Source Artifacts ✅ Passed The authoritative diff changes only scripts/ci/r2-cache.sh and tests/test_ci_r2_cache.sh. Both are hand-written CI source/test files. The changes add run-ID pointer logic, logging, and test covera…
Cmux No Test Or Debug Seam In Production Source ✅ Passed The pull request changes only scripts/ci/r2-cache.sh and tests/test_ci_r2_cache.sh. It changes no Swift file under a production Sources/ path, so the no-test-or-debug-seam condition does not app…
  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

An independent review at 866d2d4 found no bugs.

  • Callers: the only save caller is .github/actions/cache-save, which runs only in Actions on main (ci-cache-writer). With GITHUB_RUN_ID unset, the generation is 0, which behaves as before.
  • Other prefixes: each has a single writer, and nothing else reads x-amz-meta-generation, including ci: prune expired R2 cache archives, never one a latest pointer names #14089's prune.
  • Re-runs: a re-run keeps its ID, so a late re-run still cannot regress a pointer.
  • Tests: there are 14 cases, not the 12 the description says. With main's r2-cache.sh swapped in, the cross-workflow case fails.
  • Caveat: GitHub documents run IDs as unique, not as increasing. In practice they increase in creation order across the repository, which is what this relies on.

Auto-merge (squash) is on.

@teamleaderleo
teamleaderleo enabled auto-merge (squash) September 24, 2026 07:13
@teamleaderleo
teamleaderleo merged commit a7ee71d into main Sep 24, 2026
52 of 53 checks passed
teamleaderleo added a commit that referenced this pull request Sep 24, 2026
Required by the fork runner routing guard (#14066). Also correct the
pointer re-read comment: with run-ID generations (#14174) a re-save no
longer moves a pointer backwards; a new prefix's first pointer is the
remaining race.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
teamleaderleo added a commit that referenced this pull request Sep 24, 2026
…#14089)

* ci: prune expired R2 cache archives, never one a latest pointer names

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: keep compilation caches three days, not seven

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: run the R2 prune job in the main-only ci-cache-writer environment

The guard from #14147 requires every job holding the R2 write
credentials to declare it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: route the R2 prune job off Blacksmith outside manaflow-ai

Required by the fork runner routing guard (#14066). Also correct the
pointer re-read comment: with run-ID generations (#14174) a re-save no
longer moves a pointer backwards; a new prefix's first pointer is the
remaining race.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: keep per-commit R2 caches one day, not three

Both families try the pull request's exact base first, then the newest
by prefix. On 2026-09-24, 92 of the 100 most recently updated open pull
requests had a base under a day old, and each day of retention costs
about 35 GiB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Measured after merge (read-only; seeder runs on main).

Changed inputs when the seeder adopts a seed

Window Runs Changed inputs Build step
Before 5 (35959372603 to 35967425494) median 451 (436 to 496); all adopted stale f2abe64 median 845 s
Right after, same 6 vCPU pool 3 221, 129, 64; 35976771125 adopted the newest seed 688, 845, 737 s
09:12Z to 14:00Z, 12 vCPU 16 median 28 median 256 s

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant