Skip to content

ci: share rust cache across branches and restore file mtimes#6246

Merged
universalmind303 merged 2 commits intomainfrom
ci/git-restore-mtime
Feb 20, 2026
Merged

ci: share rust cache across branches and restore file mtimes#6246
universalmind303 merged 2 commits intomainfrom
ci/git-restore-mtime

Conversation

@desmondcheongzx
Copy link
Copy Markdown
Collaborator

@desmondcheongzx desmondcheongzx commented Feb 19, 2026

Two changes to fix integration-test-build timeouts caused by Rust cache eviction and cargo fingerprint invalidation:

Shared cache key with main-only saves. Switches Swatinem/rust-cache from prefix-key (branch-scoped) to shared-key + save-if: main. All PR branches read from the same cache entry that only main writes, eliminating per-branch duplication that was filling the 10GB GitHub Actions cache limit and causing evictions.

Restore file mtimes after checkout. actions/checkout sets all file mtimes to the checkout timestamp, which invalidates cargo fingerprints even when Swatinem restores a valid target/ directory — causing cargo to recompile all ~70 local crates from scratch. A lightweight git log --raw script restores each file's mtime to its last-commit timestamp, making fingerprints match across CI runs.

The first main build after merge will be a cold build that saves the cache with the new shared key format. Subsequent builds (main and PR) should only recompile changed crates.

@codecov
Copy link
Copy Markdown

codecov Bot commented Feb 19, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.43%. Comparing base (ee16fc9) to head (0242fb0).
⚠️ Report is 12 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #6246      +/-   ##
==========================================
+ Coverage   73.37%   73.43%   +0.05%     
==========================================
  Files         999     1001       +2     
  Lines      132632   133095     +463     
==========================================
+ Hits        97318    97732     +414     
- Misses      35314    35363      +49     

see 49 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@desmondcheongzx desmondcheongzx changed the title ci: restore file mtimes after checkout for cargo cache hits ci: share rust cache across branches and restore file mtimes Feb 19, 2026
@desmondcheongzx desmondcheongzx marked this pull request as ready for review February 19, 2026 21:01
@desmondcheongzx desmondcheongzx requested a review from a team as a code owner February 19, 2026 21:01
@desmondcheongzx desmondcheongzx enabled auto-merge (squash) February 19, 2026 21:01
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented Feb 19, 2026

Greptile Summary

This PR implements two complementary optimizations to reduce integration-test-build CI time from ~39 minutes to ~10-15 minutes:

  • Shared cache across branches: Switches from prefix-key to shared-key + save-if to prevent cache duplication across branches. PRs will read from the shared cache without writing, saving ~3.3 GB of the 10 GB cache limit.
  • File mtime restoration: Adds a post-checkout step that restores each file's mtime to its last commit timestamp using git log. This prevents cargo from unnecessarily recompiling all ~70 local crates when the cache fingerprints would otherwise match.

Both changes are scoped to the integration-test-build job as an experiment before potential wider rollout. The implementation is clean and follows CI optimization best practices.

Confidence Score: 5/5

  • This PR is safe to merge with minimal risk
  • Score reflects well-tested CI optimization patterns (mtime restoration is a known cargo caching technique, shared cache keys are standard). Changes are scoped to a single job as an experiment, limiting blast radius. The shell script is deterministic and handles edge cases properly (checks file existence before touching). No functional code changes.
  • No files require special attention

Important Files Changed

Filename Overview
.github/workflows/pr-test-suite.yml Adds mtime restoration after checkout and switches integration-build cache from prefix-key to shared-key with save-if restriction to main branch only

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Checkout Code] --> B[Restore File Mtimes]
    B --> C{Git Log Parser}
    C --> D[AWK: Map Files to Commit Timestamps]
    D --> E[Touch Files with Timestamps]
    E --> F[Setup Rust Cache]
    F --> G{Is Main Branch?}
    G -->|Yes| H[Read/Write Shared Cache]
    G -->|No| I[Read-Only Shared Cache]
    H --> J[Cargo Build with Valid Fingerprints]
    I --> J
    J --> K[Fast Build - No Recompilation]
    
    style B fill:#90EE90
    style G fill:#FFD700
    style K fill:#90EE90
Loading

Last reviewed commit: 0b49db9

Copy link
Copy Markdown
Contributor

@greptile-apps greptile-apps Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 file reviewed, no comments

Edit Code Review Agent Settings | Greptile

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Comment thread .github/workflows/pr-test-suite.yml
@universalmind303 universalmind303 merged commit cd566ab into main Feb 20, 2026
34 checks passed
@universalmind303 universalmind303 deleted the ci/git-restore-mtime branch February 20, 2026 01:31
desmondcheongzx added a commit that referenced this pull request Feb 20, 2026
Add `cache-workspace-crates: "true"` to the Swatinem/rust-cache config
for the integration-test-build job.

`cache-all-crates` only controls registry cleanup (downloaded `.crate`
files and extracted sources in `~/.cargo/registry/`). Workspace crate
fingerprints, build script outputs, and compiled `.rlib`/`.rmeta`
artifacts in `target/` are cleaned by `cleanTargetDir()` during cache
save — which builds its keep-list from
`getPackagesOutsideWorkspaceRoot()`, explicitly excluding workspace
members. This caused all ~70 local crates to recompile from scratch
every run (~25 min at `opt-level=3`).

`cache-workspace-crates: "true"` adds workspace members to the keep-list
so their artifacts survive the save cleanup.

## Results

Tested on this PR with temporary `save-if: true` to bootstrap the cache:

| Metric | Cold cache | Warm cache |
|---|---|---|
| Cache restore | 2s | 21s |
| Build wheels | **36 min** | **10 sec** |
| Total job | ~37 min | ~1 min 41 sec |

Combined with the restore-mtime fix from #6246 (source file mtimes older
than cached dep-info mtimes), cargo now sees all workspace crates as
fresh.

See #6244, #6246.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
desmondcheongzx added a commit that referenced this pull request Feb 20, 2026
…#6264)

Extract the inline restore-mtime script into a reusable composite action
(`.github/actions/restore-mtime/`) and apply the full cargo cache
treatment to all 10 jobs that use Swatinem/rust-cache:

- `fetch-depth: 0` on checkout (full history for mtime restoration)
- `restore-mtime` composite action (deterministic source file timestamps
for cargo fingerprinting)
- `cache-workspace-crates: "true"` (preserve workspace crate artifacts
in cache — the key fix from #6261)
- `shared-key` by cargo profile (consolidate caches instead of one per
job)
- `save-if: main-only` (single source of truth, avoids PR cache
pollution and LRU eviction)

Cache keys are shared by cargo profile:
- `Linux-dev-build`: unit-test, rust-tests, doctests, test-imports,
style, docgen (build-docs.yml)
- `Linux-integration-build`: integration-test-build, profile-daft,
property-based-tests
- `Linux-dev-bench-build`: benchmark-codspeed (buildjet, separate cache
provider)

Previously each job maintained its own cache via `prefix-key`, resulting
in ~8 separate cache entries for the same profile. Consolidating to 3
shared keys frees significant space against the 10 GB repo cache limit.

On the integration-test-build job (#6261), this approach reduced Build
wheels from 36 min to 10 sec on warm cache. Expect similar improvements
for unit-test (~8 min build) and benchmark-codspeed (~25 min compile +
link).

See #6244, #6246, #6261.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
gavin9402 pushed a commit to gavin9402/Daft that referenced this pull request Apr 7, 2026
…l-Inc#6246)

Two changes to fix `integration-test-build` timeouts caused by Rust
cache eviction and cargo fingerprint invalidation:

**Shared cache key with main-only saves.** Switches Swatinem/rust-cache
from `prefix-key` (branch-scoped) to `shared-key` + `save-if: main`. All
PR branches read from the same cache entry that only main writes,
eliminating per-branch duplication that was filling the 10GB GitHub
Actions cache limit and causing evictions.

**Restore file mtimes after checkout.** `actions/checkout` sets all file
mtimes to the checkout timestamp, which invalidates cargo fingerprints
even when Swatinem restores a valid `target/` directory — causing cargo
to recompile all ~70 local crates from scratch. A lightweight `git log
--raw` script restores each file's mtime to its last-commit timestamp,
making fingerprints match across CI runs.

The first main build after merge will be a cold build that saves the
cache with the new shared key format. Subsequent builds (main and PR)
should only recompile changed crates.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
gavin9402 pushed a commit to gavin9402/Daft that referenced this pull request Apr 7, 2026
…c#6261)

Add `cache-workspace-crates: "true"` to the Swatinem/rust-cache config
for the integration-test-build job.

`cache-all-crates` only controls registry cleanup (downloaded `.crate`
files and extracted sources in `~/.cargo/registry/`). Workspace crate
fingerprints, build script outputs, and compiled `.rlib`/`.rmeta`
artifacts in `target/` are cleaned by `cleanTargetDir()` during cache
save — which builds its keep-list from
`getPackagesOutsideWorkspaceRoot()`, explicitly excluding workspace
members. This caused all ~70 local crates to recompile from scratch
every run (~25 min at `opt-level=3`).

`cache-workspace-crates: "true"` adds workspace members to the keep-list
so their artifacts survive the save cleanup.

## Results

Tested on this PR with temporary `save-if: true` to bootstrap the cache:

| Metric | Cold cache | Warm cache |
|---|---|---|
| Cache restore | 2s | 21s |
| Build wheels | **36 min** | **10 sec** |
| Total job | ~37 min | ~1 min 41 sec |

Combined with the restore-mtime fix from Eventual-Inc#6246 (source file mtimes older
than cached dep-info mtimes), cargo now sees all workspace crates as
fresh.

See Eventual-Inc#6244, Eventual-Inc#6246.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
gavin9402 pushed a commit to gavin9402/Daft that referenced this pull request Apr 7, 2026
…Eventual-Inc#6264)

Extract the inline restore-mtime script into a reusable composite action
(`.github/actions/restore-mtime/`) and apply the full cargo cache
treatment to all 10 jobs that use Swatinem/rust-cache:

- `fetch-depth: 0` on checkout (full history for mtime restoration)
- `restore-mtime` composite action (deterministic source file timestamps
for cargo fingerprinting)
- `cache-workspace-crates: "true"` (preserve workspace crate artifacts
in cache — the key fix from Eventual-Inc#6261)
- `shared-key` by cargo profile (consolidate caches instead of one per
job)
- `save-if: main-only` (single source of truth, avoids PR cache
pollution and LRU eviction)

Cache keys are shared by cargo profile:
- `Linux-dev-build`: unit-test, rust-tests, doctests, test-imports,
style, docgen (build-docs.yml)
- `Linux-integration-build`: integration-test-build, profile-daft,
property-based-tests
- `Linux-dev-bench-build`: benchmark-codspeed (buildjet, separate cache
provider)

Previously each job maintained its own cache via `prefix-key`, resulting
in ~8 separate cache entries for the same profile. Consolidating to 3
shared keys frees significant space against the 10 GB repo cache limit.

On the integration-test-build job (Eventual-Inc#6261), this approach reduced Build
wheels from 36 min to 10 sec on warm cache. Expect similar improvements
for unit-test (~8 min build) and benchmark-codspeed (~25 min compile +
link).

See Eventual-Inc#6244, Eventual-Inc#6246, Eventual-Inc#6261.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants