Eval: Add worktree prep and PR artifacts#34421
Merged
yannbf merged 53 commits intoApr 15, 2026
Merged
Conversation
kasperpeulen
added
build
|
View your CI Pipeline Execution ↗ for commit 0045954
☁️ Nx Cloud last updated this comment at |
|
View your CI Pipeline Execution ↗ for commit 940e76d ☁️ Nx Cloud last updated this comment at |
Package BenchmarksCommit: The following packages have significant changes to their size or dependencies:
|
| Before | After | Difference | |
|---|---|---|---|
| Dependency count | 50 | 50 | 0 |
| Self size | 20.56 MB | 20.52 MB | 🎉 -37 KB 🎉 |
| Dependency size | 16.56 MB | 16.56 MB | 🎉 -93 B 🎉 |
| Bundle Size Analyzer | Link | Link |
@storybook/cli
| Before | After | Difference | |
|---|---|---|---|
| Dependency count | 184 | 184 | 0 |
| Self size | 819 KB | 818 KB | 🎉 -1 KB 🎉 |
| Dependency size | 68.20 MB | 68.17 MB | 🎉 -36 KB 🎉 |
| Bundle Size Analyzer | Link | Link |
@storybook/codemod
| Before | After | Difference | |
|---|---|---|---|
| Dependency count | 177 | 177 | 0 |
| Self size | 32 KB | 32 KB | 🎉 -36 B 🎉 |
| Dependency size | 66.72 MB | 66.68 MB | 🎉 -37 KB 🎉 |
| Bundle Size Analyzer | Link | Link |
create-storybook
| Before | After | Difference | |
|---|---|---|---|
| Dependency count | 51 | 51 | 0 |
| Self size | 1.05 MB | 1.05 MB | 🚨 +985 B 🚨 |
| Dependency size | 37.11 MB | 37.08 MB | 🎉 -37 KB 🎉 |
| Bundle Size Analyzer | node | node |
Sidnioulz
force-pushed
the
project/sb-agentic-setup
branch
from
cc92a8c to
11bc8f0
Compare
kasperpeulen
commented
Apr 14, 2026
kasperpeulen
commented
Apr 14, 2026
kasperpeulen
commented
Apr 14, 2026
kasperpeulen
commented
Apr 14, 2026
kasperpeulen
commented
Apr 14, 2026
kasperpeulen
commented
Apr 14, 2026
Co-authored-by: Codex <noreply@openai.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
scripts/eval/lib/publish-trial.test.ts (2)
133-217: Extract a shared trial-data factory to reduce drift across tests.The schema payload is repeated three times with minor overrides. A
createTrialData(overrides)helper would make scenarios clearer and reduce maintenance risk when schema fields change.Also applies to: 311-373, 430-489
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@scripts/eval/lib/publish-trial.test.ts` around lines 133 - 217, Tests repeat the same trial payload across multiple publishTrialBranch calls; extract a createTrialData(overrides) factory that returns the full default schema object and merges in any overrides, then replace direct inline payloads (e.g., the data argument passed to publishTrialBranch) with data: createTrialData({...}) so each test only specifies the differing fields; update all other occurrences (the other publishTrialBranch/test cases referenced) to use the same factory to keep a single source of truth for the schema.
1-3: Consider mocking FS dependencies in this unit test suite.This file heavily uses real
node:fsside effects for setup/assertions. For unit-test isolation and consistency with repo rules, prefer mocking filesystem access (or keep a dedicated integration-style test and mock the rest).As per coding guidelines, “Mock external dependencies like file system access and loggers in tests”.
Also applies to: 38-66, 130-131, 302-307, 427-427
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@scripts/eval/lib/publish-trial.test.ts` around lines 1 - 3, The test file currently uses real filesystem functions (existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync) which causes side effects; replace these real fs calls by mocking node:fs (e.g., with jest.mock('node:fs') or an in-memory fs like memfs) so setup/assertions run against the mock, or alternatively convert the file to an explicit integration test and isolate it; update tests that call mkdtempSync/mkdirSync/readFileSync/writeFileSync/existsSync/rmSync to use the mock API, ensure any cleanup uses the mock too, and adjust assertions to read from the mocked filesystem rather than the real disk.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@scripts/eval/lib/publish-trial.test.ts`:
- Around line 102-122: Replace the inline vi.doMock usage for 'tinyexec' with a
top-level vi.mock('tinyexec', ... , { spy: true }) and move all per-test
implementations into beforeEach using vi.mocked(...) to get a typed spy;
specifically, create the mock factory that exposes x as vi.fn() (so tests can
inspect calls array and use createExecResult), then in beforeEach set
vi.mocked(x).mockImplementation(...) to return the different createExecResult
scenarios (empty, exit 1, PR URL, default) and reset mocks between tests; update
tests that currently call vi.doMock('tinyexec') to assume the top-level mock and
only configure behavior in beforeEach and use vi.mocked(x) when asserting call
arguments.
---
Nitpick comments:
In `@scripts/eval/lib/publish-trial.test.ts`:
- Around line 133-217: Tests repeat the same trial payload across multiple
publishTrialBranch calls; extract a createTrialData(overrides) factory that
returns the full default schema object and merges in any overrides, then replace
direct inline payloads (e.g., the data argument passed to publishTrialBranch)
with data: createTrialData({...}) so each test only specifies the differing
fields; update all other occurrences (the other publishTrialBranch/test cases
referenced) to use the same factory to keep a single source of truth for the
schema.
- Around line 1-3: The test file currently uses real filesystem functions
(existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync) which
causes side effects; replace these real fs calls by mocking node:fs (e.g., with
jest.mock('node:fs') or an in-memory fs like memfs) so setup/assertions run
against the mock, or alternatively convert the file to an explicit integration
test and isolate it; update tests that call
mkdtempSync/mkdirSync/readFileSync/writeFileSync/existsSync/rmSync to use the
mock API, ensure any cleanup uses the mock too, and adjust assertions to read
from the mocked filesystem rather than the real disk.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 47f875fe-3040-4c28-b902-2e099dcfd202
📒 Files selected for processing (2)
scripts/eval/lib/publish-trial.test.tsscripts/eval/lib/publish-trial.ts
✅ Files skipped from review due to trivial changes (1)
- scripts/eval/lib/publish-trial.ts
Move sync-baselines to repo-local string templates, simplify the CLI, and add excalidraw to the benchmark project list. Co-authored-by: Codex <noreply@openai.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (5)
scripts/eval/sync-baselines.ts (2)
218-226: Consider usingdirname()for clarity.Using
dirname(targetPath)is semantically clearer thanjoin(targetPath, '..')for getting the parent directory.Proposed fix
+import { dirname, join, relative, resolve } from 'node:path'; + for (const [name, contents] of sourceFiles) { const targetPath = join(targetDir, name); - await mkdir(join(targetPath, '..'), { recursive: true }); + await mkdir(dirname(targetPath), { recursive: true }); const rewritten = contents.replace(Note:
dirnameis already imported at line 3, so this is just a code clarity improvement.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@scripts/eval/sync-baselines.ts` around lines 218 - 226, Replace the use of join(targetPath, '..') with dirname(targetPath) for clarity when creating the parent directory: in the loop over sourceFiles where targetPath is computed, call mkdir(dirname(targetPath), { recursive: true }) instead of mkdir(join(targetPath, '..'), { recursive: true }); keep the rest (rewritten replacement and writeFile(targetPath, rewritten)) unchanged — dirname is already imported so no new imports are required.
303-305: Redundant regex pattern innormalizeRepoPath.The second replace pattern
/^\.(?=\/)/will never match because the first pattern/^\.\//already removes any leading./. The second pattern attempts to match a dot followed by a slash without consuming the slash, but after the first replace, no such pattern can exist.Proposed fix
function normalizeRepoPath(value: string) { - return value.replace(/^\.\//, '').replace(/^\.(?=\/)/, ''); + return value.replace(/^\.\//, ''); }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@scripts/eval/sync-baselines.ts` around lines 303 - 305, The normalizeRepoPath function contains a redundant second replace; simplify it by replacing both patterns with a single regex that removes a leading "." or "./". Update normalizeRepoPath to return value.replace(/^\.\/?/, '') so you only strip a leading dot or dot+slash in one operation (look for the normalizeRepoPath function to apply this change).scripts/eval/lib/baseline-template-files.ts (2)
83-84: Remove internal development path reference from template.The comment references a local filesystem path (
~/code/github/mcp/eval/templates/result-docs/) which is an internal development location and not useful to end users or other contributors.Proposed fix
const EVAL_SUPPORT_TRANSCRIPT_MDX = template(String.raw` -{/* Transcript renderer copied directly from ~/code/github/mcp/eval/templates/result-docs/transcript.tsx and transcript.types.ts */} +{/* Transcript renderer for eval results visualization */} import data from '../eval-results/data.json';🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@scripts/eval/lib/baseline-template-files.ts` around lines 83 - 84, The template string assigned to EVAL_SUPPORT_TRANSCRIPT_MDX contains a comment with an internal dev path; remove or replace that path-only reference and use a generic, public-facing description instead (e.g., "Transcript renderer copied from templates/result-docs/transcript.tsx and transcript.types.ts" or simply "Transcript renderer"). Update the comment inside the template literal for EVAL_SUPPORT_TRANSCRIPT_MDX to avoid any local filesystem paths while keeping the pointer to the original module names.
551-570: Consider guarding cleanup against already-removed elements.The cleanup function calls
removeChildunconditionally. If any element was already removed (e.g., by page navigation or Storybook hot reload), this could throw. Using optional checks would make it more robust.Proposed fix
return () => { - document.head.removeChild(script); - document.head.removeChild(style); - document.head.removeChild(codeStyle); + if (script.parentNode) document.head.removeChild(script); + if (style.parentNode) document.head.removeChild(style); + if (codeStyle.parentNode) document.head.removeChild(codeStyle); };🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@scripts/eval/lib/baseline-template-files.ts` around lines 551 - 570, The cleanup function returned from useEffect unconditionally calls document.head.removeChild(script|style|codeStyle) which can throw if elements were already removed; update the cleanup in the useEffect to safely remove those nodes (e.g., check parentNode or use element.remove with optional chaining) for the three variables script, style, and codeStyle so removal is guarded against already-removed elements.scripts/eval/sync-baselines.test.ts (1)
462-464: Prefer Node.js built-inmkdirSyncover shell command.Using Node's
fs.mkdirSyncwithrecursive: trueis more portable and consistent with the other fs operations in the file. The shellmkdircommand may behave differently on Windows CI runners.Proposed fix
+import { mkdirSync } from 'node:fs'; + function mkdirSyncRecursive(path: string) { - execFileSync('mkdir', ['-p', path]); + mkdirSync(path, { recursive: true }); }Note:
mkdirSyncis already imported at line 2 as part of the destructured imports fromnode:fs.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@scripts/eval/sync-baselines.test.ts` around lines 462 - 464, The helper mkdirSyncRecursive uses a shell call; replace its body to call the Node fs API instead: use the existing mkdirSync function (already imported) with the option { recursive: true } inside mkdirSyncRecursive so it creates the directory tree in a cross-platform, portable way (keep the function name mkdirSyncRecursive).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@scripts/eval/lib/baseline-template-files.ts`:
- Around line 687-693: The percentage prop calculation on the Turn component can
divide by zero when totalMessageTokens is 0; update the expression where
percentage is passed (the Turn element rendering User Prompt) to guard against
zero by computing percentage as zero (or a safe fallback) when
totalMessageTokens is falsy, e.g., use a conditional or Math.max fallback around
(promptTokenCount / totalMessageTokens) * 100 so you never pass Infinity to the
percentage prop.
---
Nitpick comments:
In `@scripts/eval/lib/baseline-template-files.ts`:
- Around line 83-84: The template string assigned to EVAL_SUPPORT_TRANSCRIPT_MDX
contains a comment with an internal dev path; remove or replace that path-only
reference and use a generic, public-facing description instead (e.g.,
"Transcript renderer copied from templates/result-docs/transcript.tsx and
transcript.types.ts" or simply "Transcript renderer"). Update the comment inside
the template literal for EVAL_SUPPORT_TRANSCRIPT_MDX to avoid any local
filesystem paths while keeping the pointer to the original module names.
- Around line 551-570: The cleanup function returned from useEffect
unconditionally calls document.head.removeChild(script|style|codeStyle) which
can throw if elements were already removed; update the cleanup in the useEffect
to safely remove those nodes (e.g., check parentNode or use element.remove with
optional chaining) for the three variables script, style, and codeStyle so
removal is guarded against already-removed elements.
In `@scripts/eval/sync-baselines.test.ts`:
- Around line 462-464: The helper mkdirSyncRecursive uses a shell call; replace
its body to call the Node fs API instead: use the existing mkdirSync function
(already imported) with the option { recursive: true } inside mkdirSyncRecursive
so it creates the directory tree in a cross-platform, portable way (keep the
function name mkdirSyncRecursive).
In `@scripts/eval/sync-baselines.ts`:
- Around line 218-226: Replace the use of join(targetPath, '..') with
dirname(targetPath) for clarity when creating the parent directory: in the loop
over sourceFiles where targetPath is computed, call mkdir(dirname(targetPath), {
recursive: true }) instead of mkdir(join(targetPath, '..'), { recursive: true
}); keep the rest (rewritten replacement and writeFile(targetPath, rewritten))
unchanged — dirname is already imported so no new imports are required.
- Around line 303-305: The normalizeRepoPath function contains a redundant
second replace; simplify it by replacing both patterns with a single regex that
removes a leading "." or "./". Update normalizeRepoPath to return
value.replace(/^\.\/?/, '') so you only strip a leading dot or dot+slash in one
operation (look for the normalizeRepoPath function to apply this change).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 915b8a56-ca00-4fad-88a8-f13e7c2d29bf
📒 Files selected for processing (6)
scripts/eval/lib/baseline-template-files.tsscripts/eval/lib/projects.tsscripts/eval/lib/result-docs.tsscripts/eval/lib/transcript-types.tsscripts/eval/sync-baselines.test.tsscripts/eval/sync-baselines.ts
✅ Files skipped from review due to trivial changes (1)
- scripts/eval/lib/transcript-types.ts
🚧 Files skipped from review as they are similar to previous changes (2)
- scripts/eval/lib/projects.ts
- scripts/eval/lib/result-docs.ts
Keep the string-based baseline template source byte-exact while switching it back to readable backtick literals. Co-authored-by: Codex <noreply@openai.com>
Contributor
There was a problem hiding this comment.
♻️ Duplicate comments (1)
scripts/eval/lib/baseline-template-files.ts (1)
674-680:⚠️ Potential issue | 🟡 MinorGuard against division by zero when calculating percentage.
If
totalMessageTokensis zero (e.g., empty messages array or all zero token counts), the percentage calculation will produceInfinity. Other percentage calculations in this file (lines 807, 842-843) already have this guard.Proposed fix
<Turn type="prompt" title="User Prompt" tokenCount={\`\${promptTokenCount.toLocaleString()} tokens\`} - percentage={(promptTokenCount / totalMessageTokens) * 100} + percentage={totalMessageTokens > 0 ? (promptTokenCount / totalMessageTokens) * 100 : 0} >🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@scripts/eval/lib/baseline-template-files.ts` around lines 674 - 680, The percentage expression for the Turn component can divide by zero when totalMessageTokens is 0; update the JSX where percentage is computed (the Turn element rendering with props tokenCount and percentage) to guard against division by zero by checking totalMessageTokens before computing (e.g., use a conditional or fallback to 0) so percentage uses promptTokenCount and totalMessageTokens safely.
🧹 Nitpick comments (2)
scripts/eval/lib/baseline-template-files.ts (2)
83-85: Remove local development path from comment.Same issue as the MDX template - this local path (
~/code/github/mcp/eval/templates/result-docs/transcript.tsx) isn't meaningful outside the original development environment.Suggested fix
const EVAL_SUPPORT_TRANSCRIPT_TSX = `/* - * Keep the baseline copies exact to ~/code/github/mcp/eval/templates/result-docs/transcript.tsx. - * This repo-local template keeps only the minimum lint shims required by Storybook's monorepo. + * Transcript component for rendering eval results. + * This baseline template keeps only the minimum lint shims required by Storybook's monorepo. */🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@scripts/eval/lib/baseline-template-files.ts` around lines 83 - 85, Update the comment in baseline-template-files.ts to remove the hard-coded local development path; replace the string "~/code/github/mcp/eval/templates/result-docs/transcript.tsx" with a neutral, repo-relative or generic path (for example "templates/result-docs/transcript.tsx" or a description like "the repo's transcript.tsx template") so the comment remains meaningful outside the original developer's environment.
73-73: Remove local development path from comment.The comment references a local filesystem path (
~/code/github/mcp/eval/templates/result-docs/transcript.tsx) which exposes internal development structure and isn't meaningful in the committed codebase.Suggested fix
-const EVAL_SUPPORT_TRANSCRIPT_MDX = `{/* Transcript renderer copied directly from ~/code/github/mcp/eval/templates/result-docs/transcript.tsx and transcript.types.ts */} +const EVAL_SUPPORT_TRANSCRIPT_MDX = `{/* Transcript renderer for eval results */}🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@scripts/eval/lib/baseline-template-files.ts` at line 73, The string constant EVAL_SUPPORT_TRANSCRIPT_MDX contains a comment that exposes a local development path; update the comment inside EVAL_SUPPORT_TRANSCRIPT_MDX to remove the filesystem path and replace it with a neutral description (e.g., "Transcript renderer copied from templates/result-docs/transcript") or delete the path entirely so no local directory structure is committed, ensuring the comment still documents origin without leaking local paths.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Duplicate comments:
In `@scripts/eval/lib/baseline-template-files.ts`:
- Around line 674-680: The percentage expression for the Turn component can
divide by zero when totalMessageTokens is 0; update the JSX where percentage is
computed (the Turn element rendering with props tokenCount and percentage) to
guard against division by zero by checking totalMessageTokens before computing
(e.g., use a conditional or fallback to 0) so percentage uses promptTokenCount
and totalMessageTokens safely.
---
Nitpick comments:
In `@scripts/eval/lib/baseline-template-files.ts`:
- Around line 83-85: Update the comment in baseline-template-files.ts to remove
the hard-coded local development path; replace the string
"~/code/github/mcp/eval/templates/result-docs/transcript.tsx" with a neutral,
repo-relative or generic path (for example
"templates/result-docs/transcript.tsx" or a description like "the repo's
transcript.tsx template") so the comment remains meaningful outside the original
developer's environment.
- Line 73: The string constant EVAL_SUPPORT_TRANSCRIPT_MDX contains a comment
that exposes a local development path; update the comment inside
EVAL_SUPPORT_TRANSCRIPT_MDX to remove the filesystem path and replace it with a
neutral description (e.g., "Transcript renderer copied from
templates/result-docs/transcript") or delete the path entirely so no local
directory structure is committed, ensuring the comment still documents origin
without leaking local paths.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: ca5fb2f4-cf24-4324-902a-b709289a3378
📒 Files selected for processing (2)
scripts/eval/lib/baseline-template-files.tsscripts/eval/lib/transcript-types.ts
✅ Files skipped from review due to trivial changes (1)
- scripts/eval/lib/transcript-types.ts
Avoid top-level const declarations in the synced MDX files so Storybook can index them, while keeping empty data.json handling in the consumers. Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Codex <noreply@openai.com>
- sync-baselines.test.ts: remove unused resolve import and type BASELINE_STORYBOOK_FILES as Record to allow dynamic key lookup - run-batch.test.ts: include excalidraw in expected project list
kasperpeulen
force-pushed
the
kasper/eval-worktrees-chromatic-pr
branch
from
81de85f to
8929a4c
Compare
…n file check The script checked for .storybook/main.* before git pull and failed on fresh clones. Removed that unnecessary check (syncStorybookDir recreates the directory from scratch) and reuse ensureSourceClone to auto-clone repos that haven't been cloned yet.
Sidnioulz
approved these changes
Apr 15, 2026
Contributor
Sidnioulz
left a comment
Sidnioulz
left a comment
There was a problem hiding this comment.
The PR is big so I can't wrap my head around it without taking more time to review. The only thing I spotted is a potential logic issue in the gain compute function.
yannbf
approved these changes
Apr 15, 2026
…high effort - Add formatScorePercent for CLI, PR body, and eval summary (data.json stays 0-1) - Default run-batch: both agents, opus + gpt-5.4, high effort; update tests - Refresh scripts/eval README
This was referenced Apr 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #
What I did
Expanded the eval harness into a fuller end-to-end trial pipeline.
This PR now focuses on the
scripts/evalworkflow and related artifacts:.storybookbaseline sync tooling for benchmark repospattern-copy-playprompt as the setup-eval prompt pathDetailed walkthrough
Pipeline architecture
Two entry points (
eval.tsfor single trials,run-batch.tsfor batch) feed the core pipeline: prepare a worktree → run the agent → grade the output → compute a score → publish a draft PR. Supporting modules supply configuration and templates; post-pipeline tools collect data into SQLite or sync baselines to benchmark repos.Worktree isolation strategy
Instead of cloning fresh every time, the pipeline maintains one persistent source clone per project and spawns lightweight git worktrees for each trial:
The worktree is created with a single git command:
Seven benchmark repos cover different tech stacks:
Trial lifecycle (8 steps)
run-trial.tsorchestrates the full sequence. The critical detail is the double-write ofdata.json— a provisional version before grading (so MDX imports don't crash the build), then the final version after:The agent execution uses a pluggable driver interface:
Grading: four-dimensional evaluation
Build and TypeScript checks run in parallel:
The story-render evaluation runs Vitest on the agent's new story files:
The baseline comparison temporarily swaps preview config back to the original, reruns the same tests, then restores the agent's files:
The final score is a normalized preview gain — how much of the remaining room-for-improvement did the agent capture?
End-to-end data flow
The system forms a cycle:
sync-baselines.tspushes canonical.storybookconfigs to 7 benchmark reposeval.ts/run-batch.tscreates worktrees and runs agent trialseval,project:X,agent:Y,model:Z)collect-pr-data.tsscrapes those PRs into SQLite for analysisPublishing commits the trial, pushes the branch, and opens a draft PR:
Batch runner fans out trials with concurrency control and round-robin interleaving:
Checklist for Contributors
Testing
The changes in this PR are covered in the following automated tests:
Manual testing
Caution
This section is mandatory for all contributions. If you believe no manual test is necessary, please state so explicitly. Thanks!
No manual end-to-end benchmark run was performed.
Validated with:
yarn vitest run scripts/eval/collect-pr-data.test.ts scripts/eval/lib/prepare-trial.test.ts scripts/eval/lib/run-trial.test.ts scripts/eval/lib/publish-trial.test.ts scripts/eval/lib/story-render.test.tsyarn vitest run scripts/eval/lib/utils.test.ts scripts/eval/run-batch.test.ts scripts/eval/sync-baselines.test.tsyarn vitest run code/addons/vitest/src/vitest-plugin/viewports.test.ts code/core/src/csf-tools/vitest-plugin/transformer.test.tsDocumentation
MIGRATION.MD
Checklist for Maintainers
When this PR is ready for testing, make sure to add
ci:normal,ci:mergedorci:dailyGH label to it to run a specific set of sandboxes. The particular set of sandboxes can be found incode/lib/cli-storybook/src/sandbox-templates.tsMake sure this PR contains one of the labels below:
Available labels
bug: Internal changes that fixes incorrect behavior.maintenance: User-facing maintenance tasks.dependencies: Upgrading (sometimes downgrading) dependencies.build: Internal-facing build tooling & test updates. Will not show up in release changelog.cleanup: Minor cleanup style change. Will not show up in release changelog.documentation: Documentation only changes. Will not show up in release changelog.feature request: Introducing a new feature.BREAKING CHANGE: Changes that break compatibility in some way with current major version.other: Changes that don't fit in the above categories.🦋 Canary release
This pull request has been released as version
0.0.0-pr-34421-sha-8babe7bc. Try it out in a new sandbox by runningnpx storybook@0.0.0-pr-34421-sha-8babe7bc sandboxor in an existing project withnpx storybook@0.0.0-pr-34421-sha-8babe7bc upgrade.More information
0.0.0-pr-34421-sha-8babe7bckasper/eval-worktrees-chromatic-pr8babe7bc1775206099)To request a new release of this pull request, mention the
@storybookjs/coreteam.core team members can create a new canary release here or locally with
gh workflow run --repo storybookjs/storybook publish.yml --field pr=34421Summary by CodeRabbit
New Features
Tests
Documentation