ci: share rust cache across branches and restore file mtimes#6246
Merged
universalmind303 merged 2 commits intomainfrom Feb 20, 2026
Merged
ci: share rust cache across branches and restore file mtimes#6246universalmind303 merged 2 commits intomainfrom
universalmind303 merged 2 commits intomainfrom
Conversation
0ae5eb4 to
980246f
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #6246 +/- ##
==========================================
+ Coverage 73.37% 73.43% +0.05%
==========================================
Files 999 1001 +2
Lines 132632 133095 +463
==========================================
+ Hits 97318 97732 +414
- Misses 35314 35363 +49 🚀 New features to boost your workflow:
|
Contributor
Greptile SummaryThis PR implements two complementary optimizations to reduce
Both changes are scoped to the Confidence Score: 5/5
Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Checkout Code] --> B[Restore File Mtimes]
B --> C{Git Log Parser}
C --> D[AWK: Map Files to Commit Timestamps]
D --> E[Touch Files with Timestamps]
E --> F[Setup Rust Cache]
F --> G{Is Main Branch?}
G -->|Yes| H[Read/Write Shared Cache]
G -->|No| I[Read-Only Shared Cache]
H --> J[Cargo Build with Valid Fingerprints]
I --> J
J --> K[Fast Build - No Recompilation]
style B fill:#90EE90
style G fill:#FFD700
style K fill:#90EE90
Last reviewed commit: 0b49db9 |
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
faefdef to
fc256c2
Compare
universalmind303
approved these changes
Feb 20, 2026
desmondcheongzx
added a commit
that referenced
this pull request
Feb 20, 2026
Add `cache-workspace-crates: "true"` to the Swatinem/rust-cache config for the integration-test-build job. `cache-all-crates` only controls registry cleanup (downloaded `.crate` files and extracted sources in `~/.cargo/registry/`). Workspace crate fingerprints, build script outputs, and compiled `.rlib`/`.rmeta` artifacts in `target/` are cleaned by `cleanTargetDir()` during cache save — which builds its keep-list from `getPackagesOutsideWorkspaceRoot()`, explicitly excluding workspace members. This caused all ~70 local crates to recompile from scratch every run (~25 min at `opt-level=3`). `cache-workspace-crates: "true"` adds workspace members to the keep-list so their artifacts survive the save cleanup. ## Results Tested on this PR with temporary `save-if: true` to bootstrap the cache: | Metric | Cold cache | Warm cache | |---|---|---| | Cache restore | 2s | 21s | | Build wheels | **36 min** | **10 sec** | | Total job | ~37 min | ~1 min 41 sec | Combined with the restore-mtime fix from #6246 (source file mtimes older than cached dep-info mtimes), cargo now sees all workspace crates as fresh. See #6244, #6246. --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
desmondcheongzx
added a commit
that referenced
this pull request
Feb 20, 2026
…#6264) Extract the inline restore-mtime script into a reusable composite action (`.github/actions/restore-mtime/`) and apply the full cargo cache treatment to all 10 jobs that use Swatinem/rust-cache: - `fetch-depth: 0` on checkout (full history for mtime restoration) - `restore-mtime` composite action (deterministic source file timestamps for cargo fingerprinting) - `cache-workspace-crates: "true"` (preserve workspace crate artifacts in cache — the key fix from #6261) - `shared-key` by cargo profile (consolidate caches instead of one per job) - `save-if: main-only` (single source of truth, avoids PR cache pollution and LRU eviction) Cache keys are shared by cargo profile: - `Linux-dev-build`: unit-test, rust-tests, doctests, test-imports, style, docgen (build-docs.yml) - `Linux-integration-build`: integration-test-build, profile-daft, property-based-tests - `Linux-dev-bench-build`: benchmark-codspeed (buildjet, separate cache provider) Previously each job maintained its own cache via `prefix-key`, resulting in ~8 separate cache entries for the same profile. Consolidating to 3 shared keys frees significant space against the 10 GB repo cache limit. On the integration-test-build job (#6261), this approach reduced Build wheels from 36 min to 10 sec on warm cache. Expect similar improvements for unit-test (~8 min build) and benchmark-codspeed (~25 min compile + link). See #6244, #6246, #6261. --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
gavin9402
pushed a commit
to gavin9402/Daft
that referenced
this pull request
Apr 7, 2026
…l-Inc#6246) Two changes to fix `integration-test-build` timeouts caused by Rust cache eviction and cargo fingerprint invalidation: **Shared cache key with main-only saves.** Switches Swatinem/rust-cache from `prefix-key` (branch-scoped) to `shared-key` + `save-if: main`. All PR branches read from the same cache entry that only main writes, eliminating per-branch duplication that was filling the 10GB GitHub Actions cache limit and causing evictions. **Restore file mtimes after checkout.** `actions/checkout` sets all file mtimes to the checkout timestamp, which invalidates cargo fingerprints even when Swatinem restores a valid `target/` directory — causing cargo to recompile all ~70 local crates from scratch. A lightweight `git log --raw` script restores each file's mtime to its last-commit timestamp, making fingerprints match across CI runs. The first main build after merge will be a cold build that saves the cache with the new shared key format. Subsequent builds (main and PR) should only recompile changed crates. --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
gavin9402
pushed a commit
to gavin9402/Daft
that referenced
this pull request
Apr 7, 2026
…c#6261) Add `cache-workspace-crates: "true"` to the Swatinem/rust-cache config for the integration-test-build job. `cache-all-crates` only controls registry cleanup (downloaded `.crate` files and extracted sources in `~/.cargo/registry/`). Workspace crate fingerprints, build script outputs, and compiled `.rlib`/`.rmeta` artifacts in `target/` are cleaned by `cleanTargetDir()` during cache save — which builds its keep-list from `getPackagesOutsideWorkspaceRoot()`, explicitly excluding workspace members. This caused all ~70 local crates to recompile from scratch every run (~25 min at `opt-level=3`). `cache-workspace-crates: "true"` adds workspace members to the keep-list so their artifacts survive the save cleanup. ## Results Tested on this PR with temporary `save-if: true` to bootstrap the cache: | Metric | Cold cache | Warm cache | |---|---|---| | Cache restore | 2s | 21s | | Build wheels | **36 min** | **10 sec** | | Total job | ~37 min | ~1 min 41 sec | Combined with the restore-mtime fix from Eventual-Inc#6246 (source file mtimes older than cached dep-info mtimes), cargo now sees all workspace crates as fresh. See Eventual-Inc#6244, Eventual-Inc#6246. --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
gavin9402
pushed a commit
to gavin9402/Daft
that referenced
this pull request
Apr 7, 2026
…Eventual-Inc#6264) Extract the inline restore-mtime script into a reusable composite action (`.github/actions/restore-mtime/`) and apply the full cargo cache treatment to all 10 jobs that use Swatinem/rust-cache: - `fetch-depth: 0` on checkout (full history for mtime restoration) - `restore-mtime` composite action (deterministic source file timestamps for cargo fingerprinting) - `cache-workspace-crates: "true"` (preserve workspace crate artifacts in cache — the key fix from Eventual-Inc#6261) - `shared-key` by cargo profile (consolidate caches instead of one per job) - `save-if: main-only` (single source of truth, avoids PR cache pollution and LRU eviction) Cache keys are shared by cargo profile: - `Linux-dev-build`: unit-test, rust-tests, doctests, test-imports, style, docgen (build-docs.yml) - `Linux-integration-build`: integration-test-build, profile-daft, property-based-tests - `Linux-dev-bench-build`: benchmark-codspeed (buildjet, separate cache provider) Previously each job maintained its own cache via `prefix-key`, resulting in ~8 separate cache entries for the same profile. Consolidating to 3 shared keys frees significant space against the 10 GB repo cache limit. On the integration-test-build job (Eventual-Inc#6261), this approach reduced Build wheels from 36 min to 10 sec on warm cache. Expect similar improvements for unit-test (~8 min build) and benchmark-codspeed (~25 min compile + link). See Eventual-Inc#6244, Eventual-Inc#6246, Eventual-Inc#6261. --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two changes to fix
integration-test-buildtimeouts caused by Rust cache eviction and cargo fingerprint invalidation:Shared cache key with main-only saves. Switches Swatinem/rust-cache from
prefix-key(branch-scoped) toshared-key+save-if: main. All PR branches read from the same cache entry that only main writes, eliminating per-branch duplication that was filling the 10GB GitHub Actions cache limit and causing evictions.Restore file mtimes after checkout.
actions/checkoutsets all file mtimes to the checkout timestamp, which invalidates cargo fingerprints even when Swatinem restores a validtarget/directory — causing cargo to recompile all ~70 local crates from scratch. A lightweightgit log --rawscript restores each file's mtime to its last-commit timestamp, making fingerprints match across CI runs.The first main build after merge will be a cold build that saves the cache with the new shared key format. Subsequent builds (main and PR) should only recompile changed crates.