Build: NX Cloud vs CircleCI evaluation experiment

[kasperpeulen]

Closes #

What I did

Run a controlled experiment comparing NX Cloud vs CircleCI on identical workloads to evaluate flakiness, speed, and cost.

Key changes (on kasper/nx-ai):

• Uncomment NX Cloud Linux CI workflow, disable Windows (not part of comparison)
• Add NX_EXPERIMENT flag to CircleCI config — disables chromatic, benchmark, Windows, and init-empty jobs that aren't ported to NX
• Downgrade NX Cloud linux-browsers-js agents from extra_large+ to medium+ (4x cost reduction)
• Fix tag cadence: react-vitest-3 → ci:merged, yarn-pnp → ci:daily
• Add cancel-in-progress: false to GH Actions concurrency

This PR adds (on top of kasper/nx-ai):

• evaluate-ci.ts — comparison dashboard script with SQLite caching
  • Pulls exact credits from CircleCI Insights API and NX Cloud dashboard API
  • Computes flake rates, duration percentiles, cost per run
  • Predicted cost with medium+ agents for next branch reference data
  • --report-only for instant cached reports, --show-runs for per-run detail
• retrigger-ci.ts — staggered cache-busting script for collecting 20+ runs per workflow type across 4 eval PRs

Evaluation PRs (targeting kasper/nx-ai):

• Build: NX evaluation — ci:normal #34562 — ci:normal
• Build: NX evaluation — ci:merged #34563 — ci:merged
• Build: NX evaluation — ci:daily #1 #34564 — ci:daily (fix sidebar - content float issue #1)
• Build: NX evaluation — ci:daily #2 #34565 — ci:daily (Add support for custom webpack config #2)

Early results from next branch (100 merged runs):

Metric	CircleCI	NX Cloud	NX if medium+
Flake rate	22.0%	2.2%
Avg duration	19m	4m18s
P50 duration	19m	1m19s
Avg cost/run	$2.99	$6.05	$1.92

NX Cloud with medium+ agents would be 36% cheaper, 4x faster, and 10x less flaky.

Checklist for Contributors

Testing

The changes in this PR are covered in the following automated tests:

• stories
• unit tests
• integration tests
• end-to-end tests

Manual testing

Run the evaluation script to verify data collection:

1. source ~/.config/secrets.sh
2. yarn jiti scripts/evaluate-ci.ts --workflow "next:merged" --limit 10 --show-runs
3. Second run should show "X cached, 0 new" and complete in <3s

Documentation

• Add or update documentation reflecting your changes
• If you are deprecating/removing a feature, make sure to update
  MIGRATION.MD

Checklist for Maintainers

• When this PR is ready for testing, make sure to add ci:normal, ci:merged or ci:daily GH label to it to run a specific set of sandboxes. The particular set of sandboxes can be found in code/lib/cli-storybook/src/sandbox-templates.ts

• Make sure this PR contains one of the labels below:

  Available labels

  • build: Internal-facing build tooling & test updates. Will not show up in release changelog.

🦋 Canary release

This PR does not have a canary release associated. You can request a canary release of this pull request by mentioning the @storybookjs/core team here.

core team members can create a canary release here or locally with gh workflow run --repo storybookjs/storybook publish.yml --field pr=<PR_NUMBER>

[nx-cloud]

View your CI Pipeline Execution ↗ for commit eb2f81d

Command	Status	Duration	Result
nx run-many -t compile,check,knip,test,lint,fmt...	✅ Succeeded	10m 43s	View ↗

---

☁️ Nx Cloud last updated this comment at 2026-04-16 16:31:39 UTC

[kasperpeulen]

Superseded by #34282 (experiment lives on kasper/nx-ai directly now, with porting work split off to #34568)