Skip to content

ci: cache workspace crates and share rust caches across all workflows#6264

Merged
desmondcheongzx merged 3 commits intomainfrom
ci/cache-all-workflows
Feb 20, 2026
Merged

ci: cache workspace crates and share rust caches across all workflows#6264
desmondcheongzx merged 3 commits intomainfrom
ci/cache-all-workflows

Conversation

@desmondcheongzx
Copy link
Copy Markdown
Collaborator

Extract the inline restore-mtime script into a reusable composite action (.github/actions/restore-mtime/) and apply the full cargo cache treatment to all 10 jobs that use Swatinem/rust-cache:

  • fetch-depth: 0 on checkout (full history for mtime restoration)
  • restore-mtime composite action (deterministic source file timestamps for cargo fingerprinting)
  • cache-workspace-crates: "true" (preserve workspace crate artifacts in cache — the key fix from ci: cache workspace crate artifacts in integration build #6261)
  • shared-key by cargo profile (consolidate caches instead of one per job)
  • save-if: main-only (single source of truth, avoids PR cache pollution and LRU eviction)

Cache keys are shared by cargo profile:

  • Linux-dev-build: unit-test, rust-tests, doctests, test-imports, style, docgen (build-docs.yml)
  • Linux-integration-build: integration-test-build, profile-daft, property-based-tests
  • Linux-dev-bench-build: benchmark-codspeed (buildjet, separate cache provider)

Previously each job maintained its own cache via prefix-key, resulting in ~8 separate cache entries for the same profile. Consolidating to 3 shared keys frees significant space against the 10 GB repo cache limit.

On the integration-test-build job (#6261), this approach reduced Build wheels from 36 min to 10 sec on warm cache. Expect similar improvements for unit-test (~8 min build) and benchmark-codspeed (~25 min compile + link).

See #6244, #6246, #6261.

Extract the restore-mtime script into a reusable composite action at
.github/actions/restore-mtime/ and apply the full cache treatment to
every job that uses Swatinem/rust-cache:

- fetch-depth: 0 (full history for mtime restoration)
- restore-mtime (deterministic source file timestamps)
- cache-workspace-crates: true (preserve workspace artifacts in cache)
- shared-key (consolidate caches by profile, not per-job)
- save-if: main-only (single source of truth, no PR cache pollution)

Cache keys are shared by cargo profile:
- Linux-dev-build: unit-test, rust-tests, doctests, test-imports, style, docgen
- Linux-integration-build: integration-test-build, profile-daft, property-based-tests
- Linux-dev-bench-build: benchmark-codspeed (buildjet)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@desmondcheongzx desmondcheongzx requested a review from a team as a code owner February 20, 2026 20:46
@github-actions github-actions Bot added the ci label Feb 20, 2026
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented Feb 20, 2026

Greptile Summary

This PR consolidates Rust build caching across all CI workflows by extracting the mtime restoration logic into a reusable composite action and standardizing cache configuration. Previously, each job maintained its own cache via prefix-key, resulting in ~8 separate cache entries for the same cargo profile. This PR consolidates to 3 shared cache keys grouped by cargo profile:

  • Linux-dev-build: unit-test, rust-tests-platform, doctests, test-imports-platform, style, docgen (6 jobs)
  • Linux-integration-build: integration-test-build, profile-daft, property-based-tests (3 jobs)
  • Linux-dev-bench-build: benchmark-codspeed (1 job, separate buildjet cache provider)

Key improvements:

  • Extracted inline mtime restoration script into .github/actions/restore-mtime/ composite action with proper macOS/BSD compatibility
  • Applied uniform cache treatment to all 10 jobs: fetch-depth: 0, restore-mtime action, cache-workspace-crates: "true", shared cache keys by profile, and save-if: main-only
  • Expected significant build time reductions on warm cache (integration-test-build saw 36min → 10sec improvement in ci: cache workspace crate artifacts in integration build #6261)
  • Frees cache space against the 10 GB repo limit by eliminating duplicate cache entries

Confidence Score: 5/5

  • This PR is safe to merge with minimal risk
  • The changes are well-structured CI/CD improvements that follow established patterns. The mtime restoration script has been properly extracted into a reusable composite action with cross-platform compatibility (macOS/BSD detection added). All cache configurations are consistently applied across workflows with appropriate conditionals for Windows. The consolidation strategy is clearly documented and matches the PR description.
  • No files require special attention

Important Files Changed

Filename Overview
.github/actions/restore-mtime/action.yaml New composite action that restores file mtimes from git history with proper macOS/BSD compatibility detection
.github/workflows/build-docs.yml Updated to use shared cache key Linux-dev-build, added fetch-depth: 0, restore-mtime action, and cache-workspace-crates
.github/workflows/daft-profiling.yml Updated to use shared cache key Linux-integration-build, added fetch-depth: 0, restore-mtime action, and cache-workspace-crates
.github/workflows/pr-test-suite.yml Replaced inline mtime restoration script with composite action, updated 7 jobs to use shared cache keys (dev-build, integration-build, dev-bench-build) with workspace crate caching
.github/workflows/property-based-tests.yml Updated to use shared cache key Linux-integration-build, added fetch-depth: 0, restore-mtime action, and cache-workspace-crates

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    subgraph "Cache Key Strategy"
        A[10 CI Jobs] --> B{Cargo Profile}
        
        B -->|dev profile| C[Linux-dev-build]
        B -->|integration profile| D[Linux-integration-build]
        B -->|dev-bench profile| E[Linux-dev-bench-build]
        
        C --> C1[unit-test]
        C --> C2[rust-tests-platform]
        C --> C3[doctests]
        C --> C4[test-imports-platform]
        C --> C5[style]
        C --> C6[docgen]
        
        D --> D1[integration-test-build]
        D --> D2[profile-daft]
        D --> D3[property-based-tests]
        
        E --> E1[benchmark-codspeed]
    end
    
    subgraph "Cache Optimization"
        F[fetch-depth: 0] --> G[restore-mtime action]
        G --> H[Deterministic timestamps]
        H --> I[cache-workspace-crates: true]
        I --> J[shared-key by profile]
        J --> K[save-if: main only]
        K --> L[Reduced cache pollution]
    end
Loading

Last reviewed commit: 64c5ac2

Copy link
Copy Markdown
Contributor

@greptile-apps greptile-apps Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

5 files reviewed, 1 comment

Edit Code Review Agent Settings | Greptile

# @$ts — the @ tells touch to interpret the value as epoch seconds
git log --raw --no-renames --no-merges --pretty=%ct --reverse \
| awk '/^[0-9]+$/{t=$0;next} /^:[0-9]/{f=substr($0,index($0,"\t")+1); c[f]=t} END{for(f in c) printf "%s\t%s\n",c[f],f}' \
| while IFS=$'\t' read -r ts file; do [ -f "$file" ] && touch -d "@$ts" "$file"; done
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

touch -d is GNU-specific and will fail on macOS runners (BSD touch). The unit-test job in pr-test-suite.yml runs on macos-latest for main branch commits.

BSD touch uses different syntax: touch -t [[CC]YY]MMDDhhmm[.SS]

Suggested change
| while IFS=$'\t' read -r ts file; do [ -f "$file" ] && touch -d "@$ts" "$file"; done
| while IFS=$'\t' read -r ts file; do [ -f "$file" ] && touch -d "@$ts" "$file" 2>/dev/null || touch -t "$(date -r "$ts" +%Y%m%d%H%M.%S)" "$file"; done

@desmondcheongzx desmondcheongzx marked this pull request as draft February 20, 2026 20:51
desmondcheongzx and others added 2 commits February 20, 2026 13:15
touch -d "@epoch" is GNU-specific. macOS runners use BSD touch which
doesn't support -d. Detect at runtime and fall back to
touch -t with date -r (BSD date interprets -r as epoch seconds).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
/dev/null is root-owned; utimensat() requires file ownership to change
timestamps, so touch -d @0 /dev/null fails with EPERM on CI runners.
Use mktemp for the GNU vs BSD touch detection instead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@desmondcheongzx desmondcheongzx marked this pull request as ready for review February 20, 2026 21:27
Copy link
Copy Markdown
Contributor

@greptile-apps greptile-apps Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

5 files reviewed, no comments

Edit Code Review Agent Settings | Greptile

@desmondcheongzx desmondcheongzx merged commit 80abf6c into main Feb 20, 2026
34 checks passed
@desmondcheongzx desmondcheongzx deleted the ci/cache-all-workflows branch February 20, 2026 22:47
@codecov
Copy link
Copy Markdown

codecov Bot commented Feb 20, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.45%. Comparing base (f52e4f6) to head (64c5ac2).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #6264      +/-   ##
==========================================
- Coverage   73.45%   73.45%   -0.01%     
==========================================
  Files        1001     1001              
  Lines      133124   133120       -4     
==========================================
- Hits        97792    97778      -14     
- Misses      35332    35342      +10     

see 12 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

desmondcheongzx added a commit that referenced this pull request Feb 21, 2026
…nners (#6268)

GitHub Actions `shell: bash` runs with `set -eo pipefail`. The `[ -f
"$file" ] && touch_epoch` pattern returns exit code 1 when the file
doesn't exist on disk (deleted in a later commit), and if this happens
on the last iteration of the while loop, `pipefail` propagates that exit
code and kills the step.

Switch to `if/then/fi` so a non-existent file is a no-op with exit code
0.

Verified with a temporary smoke-test job on both `ubuntu-latest` (11s)
and `macos-latest` (54s) that confirmed mtimes are correctly restored on
both platforms.

Fixes the restore-mtime failures seen on macOS runners in #6264.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
gavin9402 pushed a commit to gavin9402/Daft that referenced this pull request Apr 7, 2026
…Eventual-Inc#6264)

Extract the inline restore-mtime script into a reusable composite action
(`.github/actions/restore-mtime/`) and apply the full cargo cache
treatment to all 10 jobs that use Swatinem/rust-cache:

- `fetch-depth: 0` on checkout (full history for mtime restoration)
- `restore-mtime` composite action (deterministic source file timestamps
for cargo fingerprinting)
- `cache-workspace-crates: "true"` (preserve workspace crate artifacts
in cache — the key fix from Eventual-Inc#6261)
- `shared-key` by cargo profile (consolidate caches instead of one per
job)
- `save-if: main-only` (single source of truth, avoids PR cache
pollution and LRU eviction)

Cache keys are shared by cargo profile:
- `Linux-dev-build`: unit-test, rust-tests, doctests, test-imports,
style, docgen (build-docs.yml)
- `Linux-integration-build`: integration-test-build, profile-daft,
property-based-tests
- `Linux-dev-bench-build`: benchmark-codspeed (buildjet, separate cache
provider)

Previously each job maintained its own cache via `prefix-key`, resulting
in ~8 separate cache entries for the same profile. Consolidating to 3
shared keys frees significant space against the 10 GB repo cache limit.

On the integration-test-build job (Eventual-Inc#6261), this approach reduced Build
wheels from 36 min to 10 sec on warm cache. Expect similar improvements
for unit-test (~8 min build) and benchmark-codspeed (~25 min compile +
link).

See Eventual-Inc#6244, Eventual-Inc#6246, Eventual-Inc#6261.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
gavin9402 pushed a commit to gavin9402/Daft that referenced this pull request Apr 7, 2026
…nners (Eventual-Inc#6268)

GitHub Actions `shell: bash` runs with `set -eo pipefail`. The `[ -f
"$file" ] && touch_epoch` pattern returns exit code 1 when the file
doesn't exist on disk (deleted in a later commit), and if this happens
on the last iteration of the while loop, `pipefail` propagates that exit
code and kills the step.

Switch to `if/then/fi` so a non-existent file is a no-op with exit code
0.

Verified with a temporary smoke-test job on both `ubuntu-latest` (11s)
and `macos-latest` (54s) that confirmed mtimes are correctly restored on
both platforms.

Fixes the restore-mtime failures seen on macOS runners in Eventual-Inc#6264.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants