Skip to content

refactor(ci): fix caching and improve [ci-] tag handling#633

Merged
RichardAH merged 8 commits intodevfrom
nd-use-github-actions-cache-with-bug-fixes-from-s3-implementation-2025-11-24
Nov 25, 2025
Merged

refactor(ci): fix caching and improve [ci-] tag handling#633
RichardAH merged 8 commits intodevfrom
nd-use-github-actions-cache-with-bug-fixes-from-s3-implementation-2025-11-24

Conversation

@sublimator
Copy link
Copy Markdown
Collaborator

@sublimator sublimator commented Nov 24, 2025

Summary

Fixes ccache and Conan caching issues in GitHub Actions CI by addressing configuration ordering bugs and improving cache management.

Fixes #620 (ccache caches were empty)
Fixes #618 (Conan not caching on feature branches)

Root Causes

  1. ccache was being overridden by Conan's toolchain - The -DCMAKE_C_COMPILER_LAUNCHER=ccache flags were being overwritten when Conan's toolchain was loaded
  2. Configuration applied before cache restore - Cached stale configs would override fresh settings
  3. Buggy branch comparison for Conan saves - The should-save-conan-cache logic was ineffective since cache keys don't include branch names

Changes

New Actions

  • xahau-ga-cache-restore - Generic cache restore with integrated cache clearing via GitHub API
  • xahau-ga-get-commit-message - Captures commit message for [ci-ga-*] tag detection

Deleted Actions

  • xahau-configure-ccache - Replaced with inline configuration in build action

Bug Fixes

  • Wrapper toolchain - Creates a wrapper that includes Conan's toolchain first, then overlays ccache with FORCE flag
  • Configuration ordering - ccache and Conan configs now applied AFTER cache restore
  • Removed buggy save logic - Simplified to just check cache-hit != 'true'
  • Fixed cache key - Dropped conanfile.txt (we don't have conanfile.txt)
  • Matrix-setup commit message detection - For PRs, now fetches commit message via GitHub API using PR head SHA (previously head_commit.message was empty for PR events)

Cache Clearing via Commit Tags

  • [ci-ga-clear-cache] - Clear all caches for this job
  • [ci-ga-clear-cache:ccache] - Clear only if key contains "ccache"
  • [ci-ga-clear-cache:conan] - Clear only if key contains "conan"

Other

  • Upgraded macOS runner to macos-15-xlarge (M2, more cores, fixes: "The template is not valid" )
  • Bumped CACHE_VERSION to 3 (invalidates old broken caches)
  • Changed save conditions from always() to success() (don't cache failed builds)

Implements enhanced caching strategy for GitHub Actions CI workflows
with efficient cache management and commit-message-driven controls.

Dual-cache system:
- Main branch: uses ~/.ccache-main (read-write)
- Feature branches: use ~/.ccache-current (read-write) + ~/.ccache-main
  as secondary_storage (read-only fallback)
- Feature branches only save new compilation artifacts, inheriting
  everything else from main branch cache
- Eliminates cache duplication across branches

Cache clearing via GitHub API:
- New xahau-ga-clear-cache reusable action
- [ci-ga-clear-cache] - Clear all caches for this job
- [ci-ga-clear-cache:ccache] - Clear only if key contains "ccache"
- [ci-ga-clear-cache:conan] - Clear only if key contains "conan"
- [ci-ga-clear-cache:gcc Debug] - Clear if key contains both (AND logic)
- Uses gh CLI with github.token (no additional credentials)
- Cache deletion happens before restore step

ccache improvements:
- Wrapper toolchain prevents Conan from overriding ccache settings
- Single cache directory approach with proper secondary_storage config
- Configuration applied AFTER cache restore (prevents stale config)
- Verbose build flag: [ci-ga-cmake-verbose]

Conan improvements:
- Configuration applied AFTER cache restore (prevents stale profile)
- Removed ineffective branch comparison logic for cache saves
- Fixed cache keys to use conanfile.py (not conanfile.txt)

Breaking changes:
- Cache version bumped to v3 (invalidates old caches)

Fixes #620 (ccache caches were empty)
Fixes #618 (conan not caching on feature branches)
@sublimator sublimator changed the title feat: improve ci caching with dual-cache system and api-based clearing fix: fix caching Nov 24, 2025
Consolidates cache clearing + restore logic into a single reusable
action to eliminate key synthesis duplication.

Changes:
- New xahau-ga-cache-restore action (wraps clear + restore)
- Integrates cache clearing directly into restore operation
- Eliminates separate cache clearing steps
- Cache keys built once in restore action (not duplicated)

Usage:
- ccache: two restore calls (main + current branch dual-cache)
- Conan: one restore call (single cache)
- Saves use actions/cache/save@v4 directly (no wrapper needed)

Benefits:
- DRY: cache keys no longer synthesized in multiple places
- Cleaner: cache clearing automatic during restore
- Simpler: one action call instead of two steps per cache
Consolidates cache clearing and restore into a single self-contained
action, eliminating the need for a separate clear-cache action.

Changes:
- Inline cache clearing logic directly into xahau-ga-cache-restore
- Delete xahau-ga-clear-cache action (no longer needed)
- Add additional-clear-keys input for clearing multiple keys
- Remove nd-experiment-overlayfs branch from workflow triggers

Benefits:
- Single action does everything (clear + restore)
- Simpler architecture (one action instead of two)
- Self-contained and easier to maintain
Upgrades macOS CI runner from macos-15 to macos-15-xlarge for faster
builds with more CPU cores.

Changes:
- runs-on: macos-15-xlarge (was: macos-15)

Benefits:
- M2 processor with more cores
- Faster compilation times
- Better ccache performance
@sublimator sublimator changed the title fix: fix caching fix: ci ga caching, closes #620, #618 Nov 24, 2025
@sublimator sublimator changed the title fix: ci ga caching, closes #620, #618 fix: improve ci caching with proper ccache integration Nov 25, 2025
…full-matrix]

The matrix-setup job was using github.event.head_commit.message which
is empty for PR events. Now fetches via API using the PR head SHA.
@sublimator sublimator changed the title fix: improve ci caching with proper ccache integration refactor(ci): fix caching and improve [ci-] tag handling Nov 25, 2025
@RichardAH RichardAH merged commit 8efc02b into dev Nov 25, 2025
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fail to hash files Nix CI ccache caches are empty Nix CI not caching conan on feature branches where conanfile hasn't changed yet main is empty

2 participants