ci: cache workspace crates and share rust caches across all workflows#6264
ci: cache workspace crates and share rust caches across all workflows#6264desmondcheongzx merged 3 commits intomainfrom
Conversation
Extract the restore-mtime script into a reusable composite action at .github/actions/restore-mtime/ and apply the full cache treatment to every job that uses Swatinem/rust-cache: - fetch-depth: 0 (full history for mtime restoration) - restore-mtime (deterministic source file timestamps) - cache-workspace-crates: true (preserve workspace artifacts in cache) - shared-key (consolidate caches by profile, not per-job) - save-if: main-only (single source of truth, no PR cache pollution) Cache keys are shared by cargo profile: - Linux-dev-build: unit-test, rust-tests, doctests, test-imports, style, docgen - Linux-integration-build: integration-test-build, profile-daft, property-based-tests - Linux-dev-bench-build: benchmark-codspeed (buildjet) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Greptile SummaryThis PR consolidates Rust build caching across all CI workflows by extracting the mtime restoration logic into a reusable composite action and standardizing cache configuration. Previously, each job maintained its own cache via
Key improvements:
Confidence Score: 5/5
Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
subgraph "Cache Key Strategy"
A[10 CI Jobs] --> B{Cargo Profile}
B -->|dev profile| C[Linux-dev-build]
B -->|integration profile| D[Linux-integration-build]
B -->|dev-bench profile| E[Linux-dev-bench-build]
C --> C1[unit-test]
C --> C2[rust-tests-platform]
C --> C3[doctests]
C --> C4[test-imports-platform]
C --> C5[style]
C --> C6[docgen]
D --> D1[integration-test-build]
D --> D2[profile-daft]
D --> D3[property-based-tests]
E --> E1[benchmark-codspeed]
end
subgraph "Cache Optimization"
F[fetch-depth: 0] --> G[restore-mtime action]
G --> H[Deterministic timestamps]
H --> I[cache-workspace-crates: true]
I --> J[shared-key by profile]
J --> K[save-if: main only]
K --> L[Reduced cache pollution]
end
Last reviewed commit: 64c5ac2 |
| # @$ts — the @ tells touch to interpret the value as epoch seconds | ||
| git log --raw --no-renames --no-merges --pretty=%ct --reverse \ | ||
| | awk '/^[0-9]+$/{t=$0;next} /^:[0-9]/{f=substr($0,index($0,"\t")+1); c[f]=t} END{for(f in c) printf "%s\t%s\n",c[f],f}' \ | ||
| | while IFS=$'\t' read -r ts file; do [ -f "$file" ] && touch -d "@$ts" "$file"; done |
There was a problem hiding this comment.
touch -d is GNU-specific and will fail on macOS runners (BSD touch). The unit-test job in pr-test-suite.yml runs on macos-latest for main branch commits.
BSD touch uses different syntax: touch -t [[CC]YY]MMDDhhmm[.SS]
| | while IFS=$'\t' read -r ts file; do [ -f "$file" ] && touch -d "@$ts" "$file"; done | |
| | while IFS=$'\t' read -r ts file; do [ -f "$file" ] && touch -d "@$ts" "$file" 2>/dev/null || touch -t "$(date -r "$ts" +%Y%m%d%H%M.%S)" "$file"; done |
touch -d "@epoch" is GNU-specific. macOS runners use BSD touch which doesn't support -d. Detect at runtime and fall back to touch -t with date -r (BSD date interprets -r as epoch seconds). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
/dev/null is root-owned; utimensat() requires file ownership to change timestamps, so touch -d @0 /dev/null fails with EPERM on CI runners. Use mktemp for the GNU vs BSD touch detection instead. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #6264 +/- ##
==========================================
- Coverage 73.45% 73.45% -0.01%
==========================================
Files 1001 1001
Lines 133124 133120 -4
==========================================
- Hits 97792 97778 -14
- Misses 35332 35342 +10 🚀 New features to boost your workflow:
|
…nners (#6268) GitHub Actions `shell: bash` runs with `set -eo pipefail`. The `[ -f "$file" ] && touch_epoch` pattern returns exit code 1 when the file doesn't exist on disk (deleted in a later commit), and if this happens on the last iteration of the while loop, `pipefail` propagates that exit code and kills the step. Switch to `if/then/fi` so a non-existent file is a no-op with exit code 0. Verified with a temporary smoke-test job on both `ubuntu-latest` (11s) and `macos-latest` (54s) that confirmed mtimes are correctly restored on both platforms. Fixes the restore-mtime failures seen on macOS runners in #6264. --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…Eventual-Inc#6264) Extract the inline restore-mtime script into a reusable composite action (`.github/actions/restore-mtime/`) and apply the full cargo cache treatment to all 10 jobs that use Swatinem/rust-cache: - `fetch-depth: 0` on checkout (full history for mtime restoration) - `restore-mtime` composite action (deterministic source file timestamps for cargo fingerprinting) - `cache-workspace-crates: "true"` (preserve workspace crate artifacts in cache — the key fix from Eventual-Inc#6261) - `shared-key` by cargo profile (consolidate caches instead of one per job) - `save-if: main-only` (single source of truth, avoids PR cache pollution and LRU eviction) Cache keys are shared by cargo profile: - `Linux-dev-build`: unit-test, rust-tests, doctests, test-imports, style, docgen (build-docs.yml) - `Linux-integration-build`: integration-test-build, profile-daft, property-based-tests - `Linux-dev-bench-build`: benchmark-codspeed (buildjet, separate cache provider) Previously each job maintained its own cache via `prefix-key`, resulting in ~8 separate cache entries for the same profile. Consolidating to 3 shared keys frees significant space against the 10 GB repo cache limit. On the integration-test-build job (Eventual-Inc#6261), this approach reduced Build wheels from 36 min to 10 sec on warm cache. Expect similar improvements for unit-test (~8 min build) and benchmark-codspeed (~25 min compile + link). See Eventual-Inc#6244, Eventual-Inc#6246, Eventual-Inc#6261. --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…nners (Eventual-Inc#6268) GitHub Actions `shell: bash` runs with `set -eo pipefail`. The `[ -f "$file" ] && touch_epoch` pattern returns exit code 1 when the file doesn't exist on disk (deleted in a later commit), and if this happens on the last iteration of the while loop, `pipefail` propagates that exit code and kills the step. Switch to `if/then/fi` so a non-existent file is a no-op with exit code 0. Verified with a temporary smoke-test job on both `ubuntu-latest` (11s) and `macos-latest` (54s) that confirmed mtimes are correctly restored on both platforms. Fixes the restore-mtime failures seen on macOS runners in Eventual-Inc#6264. --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Extract the inline restore-mtime script into a reusable composite action (
.github/actions/restore-mtime/) and apply the full cargo cache treatment to all 10 jobs that use Swatinem/rust-cache:fetch-depth: 0on checkout (full history for mtime restoration)restore-mtimecomposite action (deterministic source file timestamps for cargo fingerprinting)cache-workspace-crates: "true"(preserve workspace crate artifacts in cache — the key fix from ci: cache workspace crate artifacts in integration build #6261)shared-keyby cargo profile (consolidate caches instead of one per job)save-if: main-only(single source of truth, avoids PR cache pollution and LRU eviction)Cache keys are shared by cargo profile:
Linux-dev-build: unit-test, rust-tests, doctests, test-imports, style, docgen (build-docs.yml)Linux-integration-build: integration-test-build, profile-daft, property-based-testsLinux-dev-bench-build: benchmark-codspeed (buildjet, separate cache provider)Previously each job maintained its own cache via
prefix-key, resulting in ~8 separate cache entries for the same profile. Consolidating to 3 shared keys frees significant space against the 10 GB repo cache limit.On the integration-test-build job (#6261), this approach reduced Build wheels from 36 min to 10 sec on warm cache. Expect similar improvements for unit-test (~8 min build) and benchmark-codspeed (~25 min compile + link).
See #6244, #6246, #6261.