Build: NX Cloud vs CircleCI evaluation experiment#34282
Draft
kasperpeulen wants to merge 114 commits into
Draft
Conversation
kasperpeulen
added
documentation
ci:docs
|
View your CI Pipeline Execution ↗ for commit 245a5b4
☁️ Nx Cloud last updated this comment at |
The test target on the code project was using "default" inputs which only includes files owned by the code project itself, not child projects. This caused stale cache hits when test files in child projects (e.g. addon-a11y) were modified.
kasperpeulen
added
ci:daily
Package BenchmarksCommit: The following packages have significant changes to their size or dependencies:
|
| Before | After | Difference | |
|---|---|---|---|
| Dependency count | 50 | 50 | 0 |
| Self size | 20.47 MB | 20.46 MB | 🎉 -12 KB 🎉 |
| Dependency size | 16.55 MB | 16.55 MB | 0 B |
| Bundle Size Analyzer | Link | Link |
@storybook/nextjs-vite
| Before | After | Difference | |
|---|---|---|---|
| Dependency count | 92 | 92 | 0 |
| Self size | 1.12 MB | 1.12 MB | 0 B |
| Dependency size | 22.76 MB | 22.73 MB | 🎉 -33 KB 🎉 |
| Bundle Size Analyzer | Link | Link |
@storybook/react-native-web-vite
| Before | After | Difference | |
|---|---|---|---|
| Dependency count | 121 | 121 | 0 |
| Self size | 30 KB | 30 KB | 🚨 +18 B 🚨 |
| Dependency size | 23.83 MB | 23.80 MB | 🎉 -32 KB 🎉 |
| Bundle Size Analyzer | Link | Link |
@storybook/react-vite
| Before | After | Difference | |
|---|---|---|---|
| Dependency count | 82 | 82 | 0 |
| Self size | 35 KB | 35 KB | 0 B |
| Dependency size | 20.54 MB | 20.51 MB | 🎉 -33 KB 🎉 |
| Bundle Size Analyzer | Link | Link |
@storybook/cli
| Before | After | Difference | |
|---|---|---|---|
| Dependency count | 184 | 184 | 0 |
| Self size | 780 KB | 780 KB | 🎉 -27 B 🎉 |
| Dependency size | 67.69 MB | 67.68 MB | 🎉 -11 KB 🎉 |
| Bundle Size Analyzer | Link | Link |
@storybook/codemod
| Before | After | Difference | |
|---|---|---|---|
| Dependency count | 177 | 177 | 0 |
| Self size | 32 KB | 32 KB | 🚨 +24 B 🚨 |
| Dependency size | 66.22 MB | 66.20 MB | 🎉 -12 KB 🎉 |
| Bundle Size Analyzer | Link | Link |
create-storybook
| Before | After | Difference | |
|---|---|---|---|
| Dependency count | 51 | 51 | 0 |
| Self size | 1.04 MB | 1.04 MB | 0 B |
| Dependency size | 37.03 MB | 37.01 MB | 🎉 -12 KB 🎉 |
| Bundle Size Analyzer | node | node |
kasperpeulen
force-pushed
the
kasper/nx-ai
branch
from
8c65eab to
ec04fcf
Compare
- Uncomment NX Linux job in nx.yml, comment out NX Windows - Add NX_EXPERIMENT flag to CircleCI config to disable chromatic, benchmark, and Windows jobs for fair comparison - Add evaluate-ci.ts script with exact credits from both APIs - Merge next branch
kasperpeulen
added
ci:normal
and removed
ci:daily
- Fix react-vitest-3 tag: ci:normal → ci:merged (match CircleCI) - Fix yarn-pnp tag: ci:normal → ci:daily (match CircleCI) - Disable init-empty/init-features in both systems via NX_EXPERIMENT - Remove init-empty/init-features from NX ALL_TASKS
- Add concurrency group with cancel-in-progress: false to nx.yml - Fix ts-expect-error for ini module in evaluate-ci.ts
Keep only experiment-related changes on this branch: - NX_EXPERIMENT flag in CircleCI config - medium+ agent downgrade - concurrency: cancel-in-progress: false - evaluate-ci.ts dashboard script - tag fixes for react-vitest-3 and yarn-pnp The NX porting work (sandbox project.json files, nx.json target defaults, agents.yaml, Windows CI, codemod fix, etc) will be moved to a separate child PR.
kasperpeulen
changed the title
8 tasks
Prepares the evaluation tooling to compare measured NX runs across many workflow variants (next:*, *:prs wild-branches, base nx-ai) without re-fetching the CircleCI/NX APIs on every report: - scripts/evaluate-ci.ts: SQLite cache (ci-eval.db), incremental run sync, --since / --days / --report-only / --flaky-range flags, new EVAL_BRANCHES keys (base (nx-ai), next:merged, next:daily, normal:prs, merged:prs, daily:prs) with MEDIUM_PLUS_BRANCHES filter for wild-branch workflows. - scripts/ci-eval.db.README.md: schema + example queries. - scripts/generate-canvas-data.ts: regenerates the inline data constants in nx-vs-circleci-findings.canvas.tsx from the DB. - scripts/investigate-nx-cache.ts, scripts/run-backfill-only.ts: one-off helpers for cache field discovery and idempotent backfills.
Removes CircleCI entirely from kasper/nx-ai so every child eval branch inherits a clean NX-only setup: - Delete .circleci/config.yml. With no config file present, CircleCI webhooks arrive at an empty pipeline and do nothing (no billable compute, no confusing "canceled" entries in insights). - Drop NX_EXPERIMENT flag from scripts/ci/main.ts and inline the !NX_EXPERIMENT branches. Windows, chromatic, benchmark and init-empty jobs are now unconditional in the generated config. This file becomes dead code (no config.yml to trigger it) but stays self-consistent for readability. The scripts/ci/* generator code is intentionally left in place — widening the scope to delete it belongs in a follow-up.
Verifies that after deleting .circleci/config.yml CircleCI no longer runs on this branch — only NX Cloud should react to this push.
Deletes .github/workflows/trigger-circle-ci-workflow.yml so pushes to kasper/nx-ai (and the rc-sweep child branches) no longer POST to CircleCI's /pipeline endpoint. Previously that call timed out at 5s and left a "failure" check on every commit, plus a red CircleCI Pipeline commit status — all noise, since .circleci/config.yml was already removed in the prior commit. Only NX Cloud runs on these branches from now on.
This was referenced Apr 17, 2026
The --with-deps flag forces apt-get update which frequently races against Ubuntu mirror sync (1622846 vs 1622826 byte mismatch on Packages.gz) and fails the CIPE. The NX base image ubuntu22.04-node20.19-v2 already has enough X/audio/font libraries for Chromium to launch, so --with-deps adds mostly redundant work. If Chromium starts failing with missing .so at runtime, revert or switch to a pre-baked image (mcr.microsoft.com/playwright:*-jammy).
Adds 12 workflow keys to EVAL_BRANCHES / WORKFLOW_NAMES / NX_TAG_MAP:
rc:{xlarge-plus,xlarge,large-plus,large,medium-plus,medium}
rc2:{xlarge-plus,xlarge,large-plus,large,medium-plus,medium}
All rc:* PRs carry the ci:normal label, so they map to ci:normal in
NX_TAG_MAP. Each workflow filters to its single dedicated branch.
Also adds RC_BRANCH_LINUX_BROWSERS_CPM: a hard-coded map from rc branch
name to the linux-browsers-js credit multiplier that branch's agents.yaml
sets. When the dashboard /cipes/{id}/analysis endpoint returns an empty
computeCreditUsages (observed ~50% of the time for the cheaper classes),
the fallback credit computation uses this per-branch rate instead of the
local worktree's agents.yaml — otherwise cross-branch syncs would always
bill linux-browsers-js at whatever class kasper/nx-ai is on.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #
What I did
Run a controlled experiment comparing NX Cloud vs CircleCI on identical workloads to evaluate flakiness, speed, and cost.
Changes in this PR:
NX_EXPERIMENTflag in scripts/ci/main.ts — disables chromatic, benchmark, Windows, and init-empty jobs in CircleCI so both systems run the exact same workloadlinux-browsers-jsagents fromextra_large+(60 credits/min) tomedium+(15 credits/min) — 4x cost reduction per agent-minuteconcurrency: cancel-in-progress: falseto the NX GitHub Actions workflow so rapid pushes queue instead of cancelreact-vitest-3→ci:merged(wasci:normal),yarn-pnp→ci:daily(wasci:normal)Depends on:
Evaluation PRs (target this branch):
ci:normalci:mergedci:dailyfix sidebar - content float issue #1ci:dailyAdd support for custom webpack config #2Early results from
nextbranch (100 merged runs):NX Cloud with
medium+agents would be 36% cheaper, 4x faster, and 10x less flaky.Checklist for Contributors
Testing
The changes in this PR are covered in the following automated tests:
Manual testing
Run the evaluation script to verify data collection:
source ~/.config/secrets.shyarn jiti scripts/evaluate-ci.ts --workflow "next:merged" --limit 10 --show-runsDocumentation
MIGRATION.MD
Checklist for Maintainers
When this PR is ready for testing, make sure to add
ci:normal,ci:mergedorci:dailyGH label to it to run a specific set of sandboxes. The particular set of sandboxes can be found incode/lib/cli-storybook/src/sandbox-templates.tsMake sure this PR contains one of the labels below:
Available labels
build: Internal-facing build tooling & test updates. Will not show up in release changelog.🦋 Canary release
This PR does not have a canary release associated. You can request a canary release of this pull request by mentioning the
@storybookjs/coreteam here.core team members can create a canary release here or locally with
gh workflow run --repo storybookjs/storybook publish.yml --field pr=<PR_NUMBER>