Skip to content

Build: Speed up Windows jobs on CircleCI by optimizing cache management #33134

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
valentinpalkovic wants to merge 8 commits into from
Closed

Build: Speed up Windows jobs on CircleCI by optimizing cache management #33134

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
5174882
Build: Speed up Windows jobs on CircleCI by optimizing cache management
valentinpalkovic Nov 24, 2025
7389482
Fix get-template parallelism validation
valentinpalkovic Nov 24, 2025
a4c64a9
Fix cache key checksum
valentinpalkovic Nov 24, 2025
353958f
Fix cache key for sandbox folders
valentinpalkovic Nov 24, 2025
5f08899
Fix cache key to use revision instead of build number for sandbox jobs
valentinpalkovic Nov 24, 2025
c6975ee
Refactor getTemplate function to streamline template key retrieval an…
valentinpalkovic Nov 24, 2025
80195fa
Remove unused imports and functions to streamline getTemplate script
valentinpalkovic Nov 24, 2025
d06a7e3
Merge branch 'next' into valentin/speed-up-windows-jobs-on-ci
valentinpalkovic Nov 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
214 changes: 204 additions & 10 deletions .circleci/config.yml
View file
Open in desktop

Large diffs are not rendered by default.

6 changes: 6 additions & 0 deletions .circleci/src/jobs/bench-packages.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,12 @@ steps:
clone_options: '--depth 1 --verbose'
- attach_workspace:
at: .
- restore_cache:
keys:
- code-node-modules-{{ checksum "code/yarn.lock" }}
- restore_cache:
keys:
- scripts-node-modules-{{ checksum "scripts/yarn.lock" }}
Comment on lines +10 to +15

@coderabbitai coderabbitai Bot Nov 24, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

Verify that corresponding save_cache steps exist and job platform targeting is correct.

The added restore_cache steps look syntactically correct and appropriately keyed by yarn.lock checksums. However, the PR description mentions switching to "saving node_modules in the CircleCI cache with distinct cache keys via a save_cache step," but no save_cache steps are visible in this file.

Several clarifications needed:

  1. Missing save_cache steps: Are there corresponding save_cache steps defined elsewhere (e.g., in a shared template or different file)? If not, these restore_cache steps won't find cached entries on the first run or after yarn.lock changes.

  2. Windows job confirmation: The PR rationale targets "Windows jobs" specifically, but the executor definition here (sb_node_22_classic) doesn't indicate platform targeting. Confirm whether this bench-packages job runs exclusively on Windows, or if platform-conditional logic exists elsewhere in your CircleCI config.

  3. Cache path verification: Verify that the restore paths (code/node_modules, scripts/node_modules) match the actual locations where node_modules will be installed after yarn install steps.

To verify that save_cache steps exist elsewhere in the codebase, run:

🏁 Script executed:

#!/bin/bash
# Search for save_cache steps targeting code-node-modules or scripts-node-modules
rg -n "save_cache" --type yaml --type yml -A 5 | rg -A 5 "code-node-modules|scripts-node-modules"

Length of output: 27

I'll fix the script syntax error and re-verify the codebase for corresponding save_cache steps.

🏁 Script executed:

#!/bin/bash
# Search for save_cache steps with relevant cache keys
rg -n "save_cache" --type yaml -A 10 | rg -B 2 -A 8 "code-node-modules|scripts-node-modules"

Length of output: 0

🏁 Script executed:

#!/bin/bash
# Also search for any save_cache definitions to see what caches are being saved
rg -n "save_cache:" --type yaml -A 8

Length of output: 0

Let me search more broadly to understand the caching strategy in this repository:





---

🏁 Script executed:

```shell
#!/bin/bash
# Search for the cache key names anywhere in YAML files
rg -n "code-node-modules|scripts-node-modules" --type yaml -B 2 -A 2

Length of output: 0

🏁 Script executed:

#!/bin/bash
# Check the structure of the bench-packages.yml file
cat ".circleci/src/jobs/bench-packages.yml"

Length of output: 2233

🏁 Script executed:

#!/bin/bash
# Search for all restore_cache steps to understand the overall caching pattern
rg -n "restore_cache:" --type yaml -A 3

Length of output: 0

Let me expand the search to find save_cache steps and understand the overall workflow context:

🏁 Script executed:

#!/bin/bash
# Search for save_cache in ALL files (not limited to yaml)
rg -n "save_cache" -A 5

Length of output: 353

🏁 Script executed:

#!/bin/bash
# Search for yarn install to understand where node_modules gets created
rg -n "yarn install" -B 2 -A 2

Length of output: 9484

🏁 Script executed:

#!/bin/bash
# Find workflow files that reference bench-packages
rg -n "bench-packages" --type yaml -B 3 -A 3

Length of output: 0

🏁 Script executed:

#!/bin/bash
# Look for any orb definitions or job templates that might contain cache logic
find .circleci -name "*.yml" -o -name "*.yaml" | head -20

Length of output: 793

# if there is a base branch AND a PR number in parameters, benchmark packages against those
# this happens when run against a PR
- when:
Expand Down
6 changes: 6 additions & 0 deletions .circleci/src/jobs/bench-sandboxes.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,12 @@ steps:
clone_options: '--depth 1 --verbose'
- attach_workspace:
at: .
- restore_cache:
keys:
- code-node-modules-{{ checksum "code/yarn.lock" }}
- restore_cache:
keys:
- scripts-node-modules-{{ checksum "scripts/yarn.lock" }}
- run:
name: Install sandbox dependencies
command: |
Expand Down
34 changes: 17 additions & 17 deletions .circleci/src/jobs/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,25 +21,25 @@ steps:
cd code
yarn local-registry --publish
- report-workflow-on-failure
- store_artifacts:
path: code/bench/esbuild-metafiles
- save_cache:
name: Save Yarn cache
key: build-yarn-2-cache-v5--{{ checksum "code/yarn.lock" }}--{{ checksum "scripts/yarn.lock" }}
key: code-node-modules-{{ checksum "code/yarn.lock" }}
paths:
- ~/.yarn/berry/cache
- code/node_modules
- save_cache:
key: scripts-node-modules-{{ checksum "scripts/yarn.lock" }}
paths:
- scripts/node_modules
# Persist source/artifacts only (exclude node_modules)
- persist_to_workspace:
root: .
paths:
- code/node_modules
- code/addons
- scripts/node_modules
- code/bench
- code/examples
- code/frameworks
- code/lib
- code/core
- code/builders
- code/renderers
- code/presets
- .verdaccio-cache
- code/addons
- code/bench
- code/examples
- code/frameworks
- code/lib
- code/core
- code/builders
- code/renderers
- code/presets
- .verdaccio-cache
14 changes: 14 additions & 0 deletions .circleci/src/jobs/check-sandboxes.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,20 @@ steps:
clone_options: '--depth 1 --verbose'
- attach_workspace:
at: .
- restore_cache:
keys:
- code-node-modules-{{ checksum "code/yarn.lock" }}
- restore_cache:
keys:
- scripts-node-modules-{{ checksum "scripts/yarn.lock" }}
- run:
name: Compute sandbox cache key
command: |
TEMPLATE=$(yarn get-template --cadence << pipeline.parameters.workflow >> --task check-sandbox)
echo "sandbox/$TEMPLATE/package.json" > .sandbox-package-path.txt
echo "Computed sandbox template: $TEMPLATE"
- restore_cache:
key: sandbox-{{ .Revision }}-{{ checksum ".sandbox-package-path.txt" }}
- run:
name: Install sandbox dependencies
command: |
Expand Down
6 changes: 6 additions & 0 deletions .circleci/src/jobs/chromatic-internal-storybook.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,12 @@ steps:
- checkout
- attach_workspace:
at: .
- restore_cache:
keys:
- code-node-modules-{{ checksum "code/yarn.lock" }}
- restore_cache:
keys:
- scripts-node-modules-{{ checksum "scripts/yarn.lock" }}
- run:
name: Running Chromatic
command: yarn storybook:ui:chromatic
Expand Down
14 changes: 14 additions & 0 deletions .circleci/src/jobs/chromatic-sandboxes.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,20 @@ steps:
- checkout
- attach_workspace:
at: .
- restore_cache:
keys:
- code-node-modules-{{ checksum "code/yarn.lock" }}
- restore_cache:
keys:
- scripts-node-modules-{{ checksum "scripts/yarn.lock" }}
- run:
name: Compute sandbox cache key
command: |
TEMPLATE=$(yarn get-template --cadence << pipeline.parameters.workflow >> --task chromatic)
echo "sandbox/$TEMPLATE/package.json" > .sandbox-package-path.txt
echo "Computed sandbox template: $TEMPLATE"
- restore_cache:
key: sandbox-{{ .Revision }}-{{ checksum ".sandbox-package-path.txt" }}
- run:
name: Install sandbox dependencies
command: |
Expand Down
19 changes: 16 additions & 3 deletions .circleci/src/jobs/create-sandboxes.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,12 @@ steps:
clone_options: '--depth 1 --verbose'
- attach_workspace:
at: .
- restore_cache:
keys:
- code-node-modules-{{ checksum "code/yarn.lock" }}
- restore_cache:
keys:
- scripts-node-modules-{{ checksum "scripts/yarn.lock" }}
- run:
name: Setup Corepack
command: |
Expand All @@ -31,15 +37,22 @@ steps:
if [[ $TEMPLATE != bench/* ]]; then
yarn --cwd scripts jiti ./event-log-checker.ts build $TEMPLATE
fi
# Clean node_modules before caching sandbox to keep cache light
cd sandbox/$(yarn get-sandbox-dir --template $TEMPLATE) && rm -rf node_modules
environment:
STORYBOOK_TELEMETRY_DEBUG: 1
STORYBOOK_TELEMETRY_URL: 'http://localhost:6007/event-log'
- report-workflow-on-failure:
template: $(yarn get-template --cadence << pipeline.parameters.workflow >> --task sandbox)
- persist_to_workspace:
root: .
- run:
name: Compute sandbox cache key
command: |
TEMPLATE=$(yarn get-template --cadence << pipeline.parameters.workflow >> --task sandbox)
echo "sandbox/$TEMPLATE/package.json" > .sandbox-package-path.txt
echo "Computed sandbox template: $TEMPLATE"
- save_cache:
key: sandbox-{{ .Revision }}-{{ checksum ".sandbox-package-path.txt" }}
paths:
- sandbox/**
- sandbox
- store_test_results:
path: test-results
14 changes: 14 additions & 0 deletions .circleci/src/jobs/e2e-dev.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,20 @@ steps:
clone_options: '--depth 1 --verbose'
- attach_workspace:
at: .
- restore_cache:
keys:
- code-node-modules-{{ checksum "code/yarn.lock" }}
- restore_cache:
keys:
- scripts-node-modules-{{ checksum "scripts/yarn.lock" }}
- run:
name: Compute sandbox cache key
command: |
TEMPLATE=$(yarn get-template --cadence << pipeline.parameters.workflow >> --task e2e-tests-dev)
echo "sandbox/$TEMPLATE/package.json" > .sandbox-package-path.txt
echo "Computed sandbox template: $TEMPLATE"
- restore_cache:
key: sandbox-{{ .Revision }}-{{ checksum ".sandbox-package-path.txt" }}
- run:
name: Install sandbox dependencies
command: |
Expand Down
14 changes: 14 additions & 0 deletions .circleci/src/jobs/e2e-production.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,20 @@ steps:
clone_options: '--depth 1 --verbose'
- attach_workspace:
at: .
- restore_cache:
keys:
- code-node-modules-{{ checksum "code/yarn.lock" }}
- restore_cache:
keys:
- scripts-node-modules-{{ checksum "scripts/yarn.lock" }}
- run:
name: Compute sandbox cache key
command: |
TEMPLATE=$(yarn get-template --cadence << pipeline.parameters.workflow >> --task e2e-tests)
echo "sandbox/$TEMPLATE/package.json" > .sandbox-package-path.txt
echo "Computed sandbox template: $TEMPLATE"
- restore_cache:
key: sandbox-{{ .Revision }}-{{ checksum ".sandbox-package-path.txt" }}
- run:
name: Install sandbox dependencies
command: |
Expand Down
6 changes: 6 additions & 0 deletions .circleci/src/jobs/e2e-ui-vitest-3.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,12 @@ steps:
clone_options: '--depth 1 --verbose'
- attach_workspace:
at: .
- restore_cache:
keys:
- code-node-modules-{{ checksum "code/yarn.lock" }}
- restore_cache:
keys:
- scripts-node-modules-{{ checksum "scripts/yarn.lock" }}
- run:
name: Install dependencies
command: yarn install --no-immutable
Expand Down
6 changes: 6 additions & 0 deletions .circleci/src/jobs/e2e-ui.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,12 @@ steps:
clone_options: '--depth 1 --verbose'
- attach_workspace:
at: .
- restore_cache:
keys:
- code-node-modules-{{ checksum "code/yarn.lock" }}
- restore_cache:
keys:
- scripts-node-modules-{{ checksum "scripts/yarn.lock" }}
- run:
name: Install dependencies
command: yarn install --no-immutable
Expand Down
6 changes: 6 additions & 0 deletions .circleci/src/jobs/knip.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,12 @@ steps:
clone_options: '--depth 1 --verbose'
- attach_workspace:
at: .
- restore_cache:
keys:
- code-node-modules-{{ checksum "code/yarn.lock" }}
- restore_cache:
keys:
- scripts-node-modules-{{ checksum "scripts/yarn.lock" }}
- run:
name: Knip
command: |
Expand Down
6 changes: 6 additions & 0 deletions .circleci/src/jobs/lint.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,12 @@ steps:
clone_options: '--depth 1 --verbose'
- attach_workspace:
at: .
- restore_cache:
keys:
- code-node-modules-{{ checksum "code/yarn.lock" }}
- restore_cache:
keys:
- scripts-node-modules-{{ checksum "scripts/yarn.lock" }}
- run:
name: Lint
command: |
Expand Down
6 changes: 6 additions & 0 deletions .circleci/src/jobs/script-checks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,12 @@ steps:
clone_options: '--depth 1 --verbose'
- attach_workspace:
at: .
- restore_cache:
keys:
- code-node-modules-{{ checksum "code/yarn.lock" }}
- restore_cache:
keys:
- scripts-node-modules-{{ checksum "scripts/yarn.lock" }}
- run:
name: Check parallelism count
command: |
Expand Down
6 changes: 6 additions & 0 deletions .circleci/src/jobs/stories-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,12 @@ steps:
clone_options: --depth 1 --verbose
- attach_workspace:
at: .
- restore_cache:
keys:
- code-node-modules-{{ checksum "code/yarn.lock" }}
- restore_cache:
keys:
- scripts-node-modules-{{ checksum "scripts/yarn.lock" }}
- run:
command: |
cd code
Expand Down
6 changes: 6 additions & 0 deletions .circleci/src/jobs/test-init-empty.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,12 @@ steps:
clone_options: '--depth 1 --verbose'
- attach_workspace:
at: .
- restore_cache:
keys:
- code-node-modules-{{ checksum "code/yarn.lock" }}
- restore_cache:
keys:
- scripts-node-modules-{{ checksum "scripts/yarn.lock" }}
- when:
condition:
equal: ['npm', << parameters.packageManager >>]
Expand Down
6 changes: 6 additions & 0 deletions .circleci/src/jobs/test-init-features.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,12 @@ steps:
clone_options: '--depth 1 --verbose'
- attach_workspace:
at: .
- restore_cache:
keys:
- code-node-modules-{{ checksum "code/yarn.lock" }}
- restore_cache:
keys:
- scripts-node-modules-{{ checksum "scripts/yarn.lock" }}
- run:
name: Verdaccio
background: true
Expand Down
6 changes: 6 additions & 0 deletions .circleci/src/jobs/test-portable-stories.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,12 @@ steps:
clone_options: '--depth 1 --verbose'
- attach_workspace:
at: .
- restore_cache:
keys:
- code-node-modules-{{ checksum "code/yarn.lock" }}
- restore_cache:
keys:
- scripts-node-modules-{{ checksum "scripts/yarn.lock" }}
- run:
name: Install dependencies
command: yarn install --no-immutable
Expand Down
14 changes: 14 additions & 0 deletions .circleci/src/jobs/test-runner-dev.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,20 @@ steps:
clone_options: '--depth 1 --verbose'
- attach_workspace:
at: .
- restore_cache:
keys:
- code-node-modules-{{ checksum "code/yarn.lock" }}
- restore_cache:
keys:
- scripts-node-modules-{{ checksum "scripts/yarn.lock" }}
- run:
name: Compute sandbox cache key
command: |
TEMPLATE=$(yarn get-template --cadence << pipeline.parameters.workflow >> --task test-runner-dev)
echo "sandbox/$TEMPLATE/package.json" > .sandbox-package-path.txt
echo "Computed sandbox template: $TEMPLATE"
- restore_cache:
key: sandbox-{{ .Revision }}-{{ checksum ".sandbox-package-path.txt" }}
- run:
name: Running Test Runner in Dev mode
command: yarn task --task test-runner-dev --template $(yarn get-template --cadence << pipeline.parameters.workflow >> --task test-runner-dev) --no-link --start-from=never --junit
Expand Down
14 changes: 14 additions & 0 deletions .circleci/src/jobs/test-runner-production.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,20 @@ steps:
clone_options: '--depth 1 --verbose'
- attach_workspace:
at: .
- restore_cache:
keys:
- code-node-modules-{{ checksum "code/yarn.lock" }}
- restore_cache:
keys:
- scripts-node-modules-{{ checksum "scripts/yarn.lock" }}
- run:
name: Compute sandbox cache key
command: |
TEMPLATE=$(yarn get-template --cadence << pipeline.parameters.workflow >> --task test-runner)
echo "sandbox/$TEMPLATE/package.json" > .sandbox-package-path.txt
echo "Computed sandbox template: $TEMPLATE"
- restore_cache:
key: sandbox-{{ .Revision }}-{{ checksum ".sandbox-package-path.txt" }}
- run:
name: Install sandbox dependencies
command: |
Expand Down
6 changes: 6 additions & 0 deletions .circleci/src/jobs/test-yarn-pnp.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,12 @@ steps:
clone_options: '--depth 1 --verbose'
- attach_workspace:
at: .
- restore_cache:
keys:
- code-node-modules-{{ checksum "code/yarn.lock" }}
- restore_cache:
keys:
- scripts-node-modules-{{ checksum "scripts/yarn.lock" }}
- run:
name: Install dependencies
command: yarn install --no-immutable
Expand Down
6 changes: 6 additions & 0 deletions .circleci/src/jobs/unit-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,12 @@ steps:
clone_options: '--depth 1 --verbose'
- attach_workspace:
at: .
- restore_cache:
keys:
- code-node-modules-{{ checksum "code/yarn.lock" }}
- restore_cache:
keys:
- scripts-node-modules-{{ checksum "scripts/yarn.lock" }}
- run:
name: Run tests
# TEST_FILES should include all relevant test files, including story files, but not e2e-tests
Expand Down
Loading
Loading