ci: cache workspace crate artifacts in integration build#6261
Merged
desmondcheongzx merged 3 commits intomainfrom Feb 20, 2026
Merged
ci: cache workspace crate artifacts in integration build#6261desmondcheongzx merged 3 commits intomainfrom
desmondcheongzx merged 3 commits intomainfrom
Conversation
Add `cache-workspace-crates: "true"` to the Swatinem/rust-cache config for the integration-test-build job. Previously, `cache-all-crates` only preserved registry crates (downloaded .crate files and extracted sources), while workspace crate fingerprints, build scripts, and compiled artifacts in target/ were cleaned before cache save — causing all ~70 local crates to recompile from scratch every run (~25 min at opt-level=3). Temporarily sets save-if: true so this PR branch can bootstrap the cache. Must be reverted to main-only before merge. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
Greptile SummaryThis PR adds Key changes:
Expected impact:
Confidence Score: 5/5
Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Checkout with fetch-depth: 0] --> B[Restore file mtimes from git history]
B --> C[Setup Rust with cache: false]
C --> D[Swatinem/rust-cache restore]
D --> E{Cache hit?}
E -->|Yes| F[Restore target/ with workspace artifacts]
E -->|No| G[Cold build all crates]
F --> H[Cargo checks mtimes vs dep-info]
H --> I{Files fresh?}
I -->|Yes| J[Skip recompilation]
I -->|No| K[Recompile changed crates only]
G --> L[Build wheels]
J --> L
K --> L
L --> M{On main branch or save-if: true?}
M -->|Yes| N[Save cache with workspace artifacts]
M -->|No| O[Skip cache save]
N --> P[Upload wheels]
O --> P
Last reviewed commit: 50be6b2 |
Comment on lines
+342
to
+344
| # TODO: revert to main-only before merge: | ||
| # save-if: ${{ github.ref == 'refs/heads/main' }} | ||
| save-if: true |
Contributor
There was a problem hiding this comment.
Revert save-if: true to save-if: ${{ github.ref == 'refs/heads/main' }} before merge
Warm cache test confirmed: Build wheels dropped from 36 min to 10 sec. Revert save-if to only save from main branch. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #6261 +/- ##
==========================================
- Coverage 73.43% 73.43% -0.01%
==========================================
Files 1001 1001
Lines 133185 133096 -89
==========================================
- Hits 97808 97733 -75
+ Misses 35377 35363 -14 🚀 New features to boost your workflow:
|
universalmind303
approved these changes
Feb 20, 2026
desmondcheongzx
added a commit
that referenced
this pull request
Feb 20, 2026
…#6264) Extract the inline restore-mtime script into a reusable composite action (`.github/actions/restore-mtime/`) and apply the full cargo cache treatment to all 10 jobs that use Swatinem/rust-cache: - `fetch-depth: 0` on checkout (full history for mtime restoration) - `restore-mtime` composite action (deterministic source file timestamps for cargo fingerprinting) - `cache-workspace-crates: "true"` (preserve workspace crate artifacts in cache — the key fix from #6261) - `shared-key` by cargo profile (consolidate caches instead of one per job) - `save-if: main-only` (single source of truth, avoids PR cache pollution and LRU eviction) Cache keys are shared by cargo profile: - `Linux-dev-build`: unit-test, rust-tests, doctests, test-imports, style, docgen (build-docs.yml) - `Linux-integration-build`: integration-test-build, profile-daft, property-based-tests - `Linux-dev-bench-build`: benchmark-codspeed (buildjet, separate cache provider) Previously each job maintained its own cache via `prefix-key`, resulting in ~8 separate cache entries for the same profile. Consolidating to 3 shared keys frees significant space against the 10 GB repo cache limit. On the integration-test-build job (#6261), this approach reduced Build wheels from 36 min to 10 sec on warm cache. Expect similar improvements for unit-test (~8 min build) and benchmark-codspeed (~25 min compile + link). See #6244, #6246, #6261. --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
gavin9402
pushed a commit
to gavin9402/Daft
that referenced
this pull request
Apr 7, 2026
…c#6261) Add `cache-workspace-crates: "true"` to the Swatinem/rust-cache config for the integration-test-build job. `cache-all-crates` only controls registry cleanup (downloaded `.crate` files and extracted sources in `~/.cargo/registry/`). Workspace crate fingerprints, build script outputs, and compiled `.rlib`/`.rmeta` artifacts in `target/` are cleaned by `cleanTargetDir()` during cache save — which builds its keep-list from `getPackagesOutsideWorkspaceRoot()`, explicitly excluding workspace members. This caused all ~70 local crates to recompile from scratch every run (~25 min at `opt-level=3`). `cache-workspace-crates: "true"` adds workspace members to the keep-list so their artifacts survive the save cleanup. ## Results Tested on this PR with temporary `save-if: true` to bootstrap the cache: | Metric | Cold cache | Warm cache | |---|---|---| | Cache restore | 2s | 21s | | Build wheels | **36 min** | **10 sec** | | Total job | ~37 min | ~1 min 41 sec | Combined with the restore-mtime fix from Eventual-Inc#6246 (source file mtimes older than cached dep-info mtimes), cargo now sees all workspace crates as fresh. See Eventual-Inc#6244, Eventual-Inc#6246. --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
gavin9402
pushed a commit
to gavin9402/Daft
that referenced
this pull request
Apr 7, 2026
…Eventual-Inc#6264) Extract the inline restore-mtime script into a reusable composite action (`.github/actions/restore-mtime/`) and apply the full cargo cache treatment to all 10 jobs that use Swatinem/rust-cache: - `fetch-depth: 0` on checkout (full history for mtime restoration) - `restore-mtime` composite action (deterministic source file timestamps for cargo fingerprinting) - `cache-workspace-crates: "true"` (preserve workspace crate artifacts in cache — the key fix from Eventual-Inc#6261) - `shared-key` by cargo profile (consolidate caches instead of one per job) - `save-if: main-only` (single source of truth, avoids PR cache pollution and LRU eviction) Cache keys are shared by cargo profile: - `Linux-dev-build`: unit-test, rust-tests, doctests, test-imports, style, docgen (build-docs.yml) - `Linux-integration-build`: integration-test-build, profile-daft, property-based-tests - `Linux-dev-bench-build`: benchmark-codspeed (buildjet, separate cache provider) Previously each job maintained its own cache via `prefix-key`, resulting in ~8 separate cache entries for the same profile. Consolidating to 3 shared keys frees significant space against the 10 GB repo cache limit. On the integration-test-build job (Eventual-Inc#6261), this approach reduced Build wheels from 36 min to 10 sec on warm cache. Expect similar improvements for unit-test (~8 min build) and benchmark-codspeed (~25 min compile + link). See Eventual-Inc#6244, Eventual-Inc#6246, Eventual-Inc#6261. --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add
cache-workspace-crates: "true"to the Swatinem/rust-cache config for the integration-test-build job.cache-all-cratesonly controls registry cleanup (downloaded.cratefiles and extracted sources in~/.cargo/registry/). Workspace crate fingerprints, build script outputs, and compiled.rlib/.rmetaartifacts intarget/are cleaned bycleanTargetDir()during cache save — which builds its keep-list fromgetPackagesOutsideWorkspaceRoot(), explicitly excluding workspace members. This caused all ~70 local crates to recompile from scratch every run (~25 min atopt-level=3).cache-workspace-crates: "true"adds workspace members to the keep-list so their artifacts survive the save cleanup.Results
Tested on this PR with temporary
save-if: trueto bootstrap the cache:Combined with the restore-mtime fix from #6246 (source file mtimes older than cached dep-info mtimes), cargo now sees all workspace crates as fresh.
See #6244, #6246.