diff --git a/agent-eval/README.md b/agent-eval/README.md index aba3257ae814..3346db868746 100644 --- a/agent-eval/README.md +++ b/agent-eval/README.md @@ -37,7 +37,7 @@ yarn eval:dry ### Run Experiments -Run all configured experiments: +Run the default experiments: ```bash yarn eval @@ -65,10 +65,10 @@ EVAL_ALL=1 yarn eval EVAL_ONLY=803-edit-component yarn eval ``` -A full `EVAL_ALL=1` run (13 workflow evals × 4 experiments + 4 -lifecycle evals × 2 plugin experiments) costs roughly **$30–45** in agent -tokens at current per-run averages ($0.30–0.80 per workflow eval, $1–2 per -lifecycle eval). The budget guardrail is **$75 per full run** — check the +A full `EVAL_ALL=1` run (14 workflow evals × 4 experiments + 4 +lifecycle evals × 2 plugin experiments) costs roughly **$6** in agent +tokens at the averages measured in October 2026 (under $0.01 per eval on Haiku 5.5 low, and about +$0.05–0.17 per eval on GPT-6.1-Sol low and the medium MCP experiments). The budget guardrail is **$75 per full run** — check the usage metadata in the results playground before growing the eval set past it (see [storybookjs/mcp#324](https://github.com/storybookjs/mcp/pull/324)). @@ -129,9 +129,16 @@ via `ANTHROPIC_API_KEY`; Codex experiments use the direct Codex API via `OPENAI_API_KEY`): - `cc-mcp-opus-5.5-medium`: Claude Code (Opus 5.5 at medium effort) with project-local Storybook MCP config in `.mcp.json`. -- `cc-plugin-opus-5.5-medium`: Claude Code (Opus 5.5 at medium effort) with Storybook plugin skills copied to `.claude/skills`. +- `cc-plugin-haiku-5.5-low`: Claude Code (Haiku 5.5 at low effort) with Storybook plugin skills copied to `.claude/skills`. - `codex-mcp-gpt-6.1-sol-medium`: Codex (gpt-6.1-sol at medium reasoning effort) with project-local Storybook MCP config in `.codex/config.toml` and the Storybook MCP server instructions in `AGENTS.md` (none with `EVAL_STORYBOOK_LATEST=1`). -- `codex-plugin-gpt-6.1-sol-medium`: Codex (gpt-6.1-sol at medium reasoning effort) with Storybook plugin skills copied to `.agents/skills`. +- `codex-plugin-gpt-6.1-sol-low`: Codex (gpt-6.1-sol at low reasoning effort) with Storybook plugin skills copied to `.agents/skills`. + +These four are the default set: a bare `yarn eval`, which is what CI runs, runs them and they have to stay green. Four more plugin experiments run only when you name them by their exact file name (a glob does not select them), for example `yarn exec agent-eval codex-plugin-gpt-6-luna-low`: + +- `cc-plugin-sonnet-5.5-low`: Claude Code (Sonnet 5.5 at low effort). +- `cc-plugin-opus-5.5-medium`: Claude Code (Opus 5.5 at medium effort). +- `codex-plugin-gpt-6.1-sol-medium`: Codex (gpt-6.1-sol at medium reasoning effort). +- `codex-plugin-gpt-6-luna-low`: Codex (gpt-6-luna at low reasoning effort). Its Codex system prompt says not to run tests unless asked, so evals do not require it to run the story tests itself. ## Running evals in CI diff --git a/agent-eval/evals/801-create-accessible-component/EVAL.ts b/agent-eval/evals/801-create-accessible-component/EVAL.ts index ad148e3a444a..4a0507be3c03 100644 --- a/agent-eval/evals/801-create-accessible-component/EVAL.ts +++ b/agent-eval/evals/801-create-accessible-component/EVAL.ts @@ -9,13 +9,17 @@ import { expectStoryTestsRanAndPassed, expectWorkflowCalls, getEvalContext, + modelRunsTestsOnlyWhenAsked, } from '#test-utils'; import { transcript } from '@vercel/agent-eval/eval'; import { describe, expect, test } from 'vitest'; describe('creating an accessible ToggleSwitch', () => { test('runs story tests after the change and finishes with them passing', async () => { - await expectStoryTestsRanAndPassed({ covering: ['toggleswitch'] }); + await expectStoryTestsRanAndPassed({ + requireAgentRun: !modelRunsTestsOnlyWhenAsked(), + covering: ['toggleswitch'], + }); }); test('uses Storybook story instructions and publishes a display review', () => { diff --git a/agent-eval/evals/802-create-component/EVAL.ts b/agent-eval/evals/802-create-component/EVAL.ts index 5ed0038a61c8..94da69f563f5 100644 --- a/agent-eval/evals/802-create-component/EVAL.ts +++ b/agent-eval/evals/802-create-component/EVAL.ts @@ -8,12 +8,16 @@ import { expectStoryTestsRanAndPassed, expectWorkflowCalls, getEvalContext, + modelRunsTestsOnlyWhenAsked, } from '#test-utils'; import { describe, test } from 'vitest'; describe('creating a ProfileCard component', () => { test('runs story tests after the change and finishes with them passing', async () => { - await expectStoryTestsRanAndPassed({ covering: ['profilecard'] }); + await expectStoryTestsRanAndPassed({ + requireAgentRun: !modelRunsTestsOnlyWhenAsked(), + covering: ['profilecard'], + }); }); test('uses Storybook story instructions and publishes a display review', () => { diff --git a/agent-eval/evals/803-edit-component/EVAL.ts b/agent-eval/evals/803-edit-component/EVAL.ts index 17bd97f2594d..54c64f84b899 100644 --- a/agent-eval/evals/803-edit-component/EVAL.ts +++ b/agent-eval/evals/803-edit-component/EVAL.ts @@ -8,12 +8,16 @@ import { expectStoryTestsRanAndPassed, expectWorkflowCalls, getEvalContext, + modelRunsTestsOnlyWhenAsked, } from '#test-utils'; import { describe, test } from 'vitest'; describe('editing ReviewCard to add date and optional onReport', () => { test('runs story tests after the change and finishes with them passing', async () => { - await expectStoryTestsRanAndPassed({ covering: ['reviewcard'] }); + await expectStoryTestsRanAndPassed({ + requireAgentRun: !modelRunsTestsOnlyWhenAsked(), + covering: ['reviewcard'], + }); }); test('uses Storybook story instructions and publishes a display review', () => { diff --git a/agent-eval/evals/804-write-story-for-existing-component/EVAL.ts b/agent-eval/evals/804-write-story-for-existing-component/EVAL.ts index b80e496a8eed..8307d4dedbfb 100644 --- a/agent-eval/evals/804-write-story-for-existing-component/EVAL.ts +++ b/agent-eval/evals/804-write-story-for-existing-component/EVAL.ts @@ -9,11 +9,15 @@ import { expectStoryIdsInDisplayReview, expectStoryTestsRanAndPassed, expectWorkflowCalls, + modelRunsTestsOnlyWhenAsked, } from '#test-utils'; describe('writing stories for an existing AlertBanner', () => { test('runs story tests after the change and finishes with them passing', async () => { - await expectStoryTestsRanAndPassed({ covering: ['alertbanner'] }); + await expectStoryTestsRanAndPassed({ + requireAgentRun: !modelRunsTestsOnlyWhenAsked(), + covering: ['alertbanner'], + }); }); test('uses Storybook story instructions and publishes a display review', () => { diff --git a/agent-eval/evals/808-shared-infra-fallback/EVAL.ts b/agent-eval/evals/808-shared-infra-fallback/EVAL.ts index d065c71b23b5..3a3435088f03 100644 --- a/agent-eval/evals/808-shared-infra-fallback/EVAL.ts +++ b/agent-eval/evals/808-shared-infra-fallback/EVAL.ts @@ -9,6 +9,7 @@ import { getEvalContext, getWorkflowCalls, getWorkflowToolResults, + modelRunsTestsOnlyWhenAsked, } from '#test-utils'; import { readFileSync } from 'node:fs'; import { describe, expect, test } from 'vitest'; @@ -36,7 +37,10 @@ describe('changing a shared accent token and surfacing consumer stories', () => test.skipIf(codexMcpGap)( 'runs story tests after the change and finishes with them passing', async () => { - await expectStoryTestsRanAndPassed({ covering: ['badge', 'statuspill'] }); + await expectStoryTestsRanAndPassed({ + requireAgentRun: !modelRunsTestsOnlyWhenAsked(), + covering: ['badge', 'statuspill'], + }); } ); diff --git a/agent-eval/evals/812-first-story-empty-project/EVAL.ts b/agent-eval/evals/812-first-story-empty-project/EVAL.ts index 46b5839d7ff1..c13d905721d1 100644 --- a/agent-eval/evals/812-first-story-empty-project/EVAL.ts +++ b/agent-eval/evals/812-first-story-empty-project/EVAL.ts @@ -10,11 +10,15 @@ import { expectStoryIdsInDisplayReview, expectStoryTestsRanAndPassed, expectWorkflowCalls, + modelRunsTestsOnlyWhenAsked, } from '#test-utils'; describe('writing the first Button stories in an empty Storybook', () => { test('runs story tests after the change and finishes with them passing', async () => { - await expectStoryTestsRanAndPassed({ covering: ['button'] }); + await expectStoryTestsRanAndPassed({ + requireAgentRun: !modelRunsTestsOnlyWhenAsked(), + covering: ['button'], + }); }); test('uses Storybook story instructions and publishes a display review', () => { diff --git a/agent-eval/evals/813-monorepo-leaf-create-component/EVAL.ts b/agent-eval/evals/813-monorepo-leaf-create-component/EVAL.ts index f3a99e830a9e..5c745d257b49 100644 --- a/agent-eval/evals/813-monorepo-leaf-create-component/EVAL.ts +++ b/agent-eval/evals/813-monorepo-leaf-create-component/EVAL.ts @@ -10,6 +10,7 @@ import { expectStoryIdsInDisplayReview, expectStoryTestsRanAndPassed, expectWorkflowCalls, + modelRunsTestsOnlyWhenAsked, } from '#test-utils'; describe('creating a Callout in a monorepo UI package', () => { @@ -26,7 +27,11 @@ describe('creating a Callout in a monorepo UI package', () => { }); test('runs story tests after the change and finishes with them passing', async () => { - await expectStoryTestsRanAndPassed({ covering: ['callout'], cwd: 'packages/ui' }); + await expectStoryTestsRanAndPassed({ + requireAgentRun: !modelRunsTestsOnlyWhenAsked(), + covering: ['callout'], + cwd: 'packages/ui', + }); }); test('uses Storybook story instructions and publishes a display review', () => { diff --git a/agent-eval/evals/814-write-stories-for-fetching-component/EVAL.ts b/agent-eval/evals/814-write-stories-for-fetching-component/EVAL.ts index 5fbe8761e21a..1d7bdb91ee44 100644 --- a/agent-eval/evals/814-write-stories-for-fetching-component/EVAL.ts +++ b/agent-eval/evals/814-write-stories-for-fetching-component/EVAL.ts @@ -10,13 +10,17 @@ import { expectStoryTestsRanAndPassed, expectWorkflowCalls, getEvalContext, + modelRunsTestsOnlyWhenAsked, } from '#test-utils'; const STORIES_PATH = 'stories/OrderHistory.stories.tsx'; describe('writing stories for an existing OrderHistory that fetches its data', () => { test('runs story tests after the change and finishes with them passing', async () => { - await expectStoryTestsRanAndPassed({ covering: ['orderhistory'] }); + await expectStoryTestsRanAndPassed({ + requireAgentRun: !modelRunsTestsOnlyWhenAsked(), + covering: ['orderhistory'], + }); }); test('mocks the request instead of changing the component', () => { diff --git a/agent-eval/evals/815-write-stories-for-fetching-component-without-msw/.storybook/main.ts b/agent-eval/evals/815-write-stories-for-fetching-component-without-msw/.storybook/main.ts new file mode 100644 index 000000000000..14702fcbc4e4 --- /dev/null +++ b/agent-eval/evals/815-write-stories-for-fetching-component-without-msw/.storybook/main.ts @@ -0,0 +1,8 @@ +import type { StorybookConfig } from '@storybook/react-vite'; + +const config: StorybookConfig = { + stories: ['../@(stories|src)/**/*.stories.@(js|jsx|mjs|ts|tsx)'], + addons: ['@storybook/addon-a11y', '@storybook/addon-vitest', '@storybook/addon-docs'], + framework: '@storybook/react-vite', +}; +export default config; diff --git a/agent-eval/evals/815-write-stories-for-fetching-component-without-msw/.storybook/preview.tsx b/agent-eval/evals/815-write-stories-for-fetching-component-without-msw/.storybook/preview.tsx new file mode 100644 index 000000000000..46602baafb56 --- /dev/null +++ b/agent-eval/evals/815-write-stories-for-fetching-component-without-msw/.storybook/preview.tsx @@ -0,0 +1,7 @@ +import type { Preview } from '@storybook/react-vite'; + +const preview: Preview = { + parameters: {}, +}; + +export default preview; diff --git a/agent-eval/evals/815-write-stories-for-fetching-component-without-msw/EVAL.ts b/agent-eval/evals/815-write-stories-for-fetching-component-without-msw/EVAL.ts new file mode 100644 index 000000000000..3ac0382aed36 --- /dev/null +++ b/agent-eval/evals/815-write-stories-for-fetching-component-without-msw/EVAL.ts @@ -0,0 +1,87 @@ +import { existsSync, readFileSync, readdirSync } from 'node:fs'; +import { describe, expect, test } from 'vitest'; +import { + expectDisplayReviewForVisualChange, + expectReviewOpenedInBrowser, + expectSkillInvoked, + expectStoryDiscoveryBeforeReview, + expectStoryIdsInDisplayReview, + expectStoryTestsRanAndPassed, + expectWorkflowCalls, + getEvalContext, + modelRunsTestsOnlyWhenAsked, +} from '#test-utils'; + +const STORIES_PATH = 'src/components/OrderHistory.stories.tsx'; + +describe('writing stories for a fetching OrderHistory in a project without MSW', () => { + test('runs story tests after the change and finishes with them passing', async () => { + await expectStoryTestsRanAndPassed({ + requireAgentRun: !modelRunsTestsOnlyWhenAsked(), + covering: ['orderhistory'], + }); + }); + + test('installs MSW and its Storybook addon', () => { + const { dependencies = {}, devDependencies = {} } = JSON.parse( + readFileSync('package.json', 'utf8') + ); + const installed = { ...dependencies, ...devDependencies }; + expect(installed, 'Expected msw to be installed').toHaveProperty('msw'); + expect(installed, 'Expected msw-storybook-addon to be installed').toHaveProperty( + 'msw-storybook-addon' + ); + expect( + existsSync('public/mockServiceWorker.js'), + 'Expected the worker script generated by `msw init`' + ).toBe(true); + const preview = readdirSync('.storybook').find((file) => file.startsWith('preview.')); + expect( + readFileSync(`.storybook/${preview}`, 'utf8'), + 'Expected the preview file to set up msw-storybook-addon' + ).toMatch(/msw-storybook-addon/); + }); + + test('mocks the request with MSW instead of a module mock or a stubbed fetch', () => { + const component = readFileSync('src/components/OrderHistory.tsx', 'utf8'); + expect(component, 'Expected OrderHistory to keep taking no props').toMatch( + /export default function OrderHistory\s*\(\s*\)/ + ); + expect(component, 'Expected OrderHistory to keep fetching /api/orders').toMatch( + /fetch\(\s*['"`]\/api\/orders/ + ); + + expect(existsSync(STORIES_PATH), `Expected the stories in ${STORIES_PATH}`).toBe(true); + const stories = readFileSync(STORIES_PATH, 'utf8'); + expect(stories, 'Expected handlers registered with msw.use').toMatch(/\bmsw\.use\(/); + expect(stories, 'Expected the handlers to mock /api/orders').toMatch(/\/api\/orders/); + expect(stories, 'Expected no module mock or stubbed fetch').not.toMatch( + /\bsb\.mock\(|\bvi\.mock\(|\bmocked\(|\bspyOn\(|\b(globalThis|window)\.fetch\s*=/ + ); + expect( + stories.match(/^export const \w+/gm)?.length ?? 0, + 'Expected loaded, empty, and error stories' + ).toBeGreaterThanOrEqual(3); + }); + + test('uses Storybook story instructions and publishes a display review', () => { + expectWorkflowCalls(['get-storybook-story-instructions', 'review-create']); + expectDisplayReviewForVisualChange(); + }); + + test('opens the review in the in-app browser', () => { + expectReviewOpenedInBrowser(); + }); + + test('the review covers the new OrderHistory stories', () => { + expectStoryIdsInDisplayReview(['orderhistory']); + }); + + test('discovers stories through the workflow tools before publishing the review', () => { + expectStoryDiscoveryBeforeReview(); + }); + + test.skipIf(getEvalContext().integration === 'mcp')('invokes the stories skill', () => { + expectSkillInvoked('stories'); + }); +}); diff --git a/agent-eval/evals/815-write-stories-for-fetching-component-without-msw/PROMPT.md b/agent-eval/evals/815-write-stories-for-fetching-component-without-msw/PROMPT.md new file mode 100644 index 000000000000..0cfee5a96a0f --- /dev/null +++ b/agent-eval/evals/815-write-stories-for-fetching-component-without-msw/PROMPT.md @@ -0,0 +1 @@ +Add stories in `src/components/OrderHistory.stories.tsx` for the existing `OrderHistory` component, covering the loaded, empty, and error states. diff --git a/agent-eval/evals/815-write-stories-for-fetching-component-without-msw/package.json b/agent-eval/evals/815-write-stories-for-fetching-component-without-msw/package.json new file mode 100644 index 000000000000..e9a8abee60e2 --- /dev/null +++ b/agent-eval/evals/815-write-stories-for-fetching-component-without-msw/package.json @@ -0,0 +1,23 @@ +{ + "name": "815-write-stories-for-fetching-component-without-msw", + "type": "module", + "scripts": { + "build-storybook": "storybook build", + "postinstall": "playwright install chromium && node ./scripts/start-storybook-mcp.mjs", + "storybook": "storybook dev", + "test:stories": "vitest run --config ./vitest.storybook.config.ts --project storybook" + }, + "devDependencies": { + "@storybook/addon-a11y": "workspace:*", + "@storybook/addon-docs": "workspace:*", + "@storybook/addon-vitest": "workspace:*", + "@storybook/react": "workspace:*", + "@storybook/react-vite": "workspace:*", + "@vitest/browser-playwright": "4.0.6", + "playwright": "1.56.1", + "storybook": "workspace:*" + }, + "evals": { + "template": "vite-app" + } +} diff --git a/agent-eval/evals/815-write-stories-for-fetching-component-without-msw/src/components/OrderHistory.tsx b/agent-eval/evals/815-write-stories-for-fetching-component-without-msw/src/components/OrderHistory.tsx new file mode 100644 index 000000000000..8e2a8744bfdb --- /dev/null +++ b/agent-eval/evals/815-write-stories-for-fetching-component-without-msw/src/components/OrderHistory.tsx @@ -0,0 +1,40 @@ +import { useEffect, useState } from 'react'; + +type Order = { + id: string; + placedAt: string; + total: number; +}; + +type State = { status: 'loading' } | { status: 'error' } | { status: 'loaded'; orders: Order[] }; + +export default function OrderHistory() { + const [state, setState] = useState({ status: 'loading' }); + + useEffect(() => { + fetch('/api/orders') + .then((response) => { + if (!response.ok) throw new Error(`Request failed with ${response.status}`); + return response.json() as Promise; + }) + .then((orders) => setState({ status: 'loaded', orders })) + .catch(() => setState({ status: 'error' })); + }, []); + + if (state.status === 'loading') return

Loading orders…

; + + if (state.status === 'error') return

Could not load your orders.

; + + if (state.orders.length === 0) return

You have not placed any orders yet.

; + + return ( + + ); +} diff --git a/agent-eval/evals/815-write-stories-for-fetching-component-without-msw/vitest.config.ts b/agent-eval/evals/815-write-stories-for-fetching-component-without-msw/vitest.config.ts new file mode 100644 index 000000000000..86e885122ec6 --- /dev/null +++ b/agent-eval/evals/815-write-stories-for-fetching-component-without-msw/vitest.config.ts @@ -0,0 +1,5 @@ +// addon-vitest's test-run starts Vitest from the project root, so the +// storybook test project must be reachable from the default config. The eval +// runner overwrites this file before executing EVAL.ts; it only matters while +// the agent runs. +export { default } from './vitest.storybook.config.ts'; diff --git a/agent-eval/evals/815-write-stories-for-fetching-component-without-msw/vitest.storybook.config.ts b/agent-eval/evals/815-write-stories-for-fetching-component-without-msw/vitest.storybook.config.ts new file mode 100644 index 000000000000..e9c1264ea5bf --- /dev/null +++ b/agent-eval/evals/815-write-stories-for-fetching-component-without-msw/vitest.storybook.config.ts @@ -0,0 +1,35 @@ +import * as path from 'node:path'; +import { fileURLToPath } from 'node:url'; + +import { defineConfig } from 'vitest/config'; + +import { storybookTest } from '@storybook/addon-vitest/vitest-plugin'; + +import { playwright } from '@vitest/browser-playwright'; + +const dirname = + typeof __dirname !== 'undefined' ? __dirname : path.dirname(fileURLToPath(import.meta.url)); + +// More info at: https://storybook.js.org/docs/next/writing-tests/integrations/vitest-addon +export default defineConfig({ + optimizeDeps: { + include: ['react/jsx-dev-runtime'], + }, + test: { + projects: [ + { + extends: true, + plugins: [storybookTest({ configDir: path.join(dirname, '.storybook') })], + test: { + name: 'storybook', + browser: { + enabled: true, + headless: true, + provider: playwright({}), + instances: [{ browser: 'chromium' }], + }, + }, + }, + ], + }, +}); diff --git a/agent-eval/experiments/cc-mcp-opus-5.5-medium.ts b/agent-eval/experiments/cc-mcp-opus-5.5-medium.ts index eaf9a953aad3..5a263c0b167a 100644 --- a/agent-eval/experiments/cc-mcp-opus-5.5-medium.ts +++ b/agent-eval/experiments/cc-mcp-opus-5.5-medium.ts @@ -6,14 +6,16 @@ import { writeClaudeMcpConfig, } from '../lib/templates.ts'; +const model = 'claude-opus-5-5'; + export default { ...DEFAULT_EXPERIMENT_CONFIG, agent: 'claude-code', // direct Anthropic API, requires ANTHROPIC_API_KEY - model: 'claude-opus-5-5', + model, agentOptions: { effort: 'medium' }, evals: WORKFLOW_STORYBOOK_EVALS, setup: async (sandbox) => { - await setupSandbox(sandbox, { agent: 'claude-code', integration: 'mcp' }); + await setupSandbox(sandbox, { agent: 'claude-code', integration: 'mcp', model }); await writeClaudeMcpConfig(sandbox); await writeClaudeInAppBrowserMock(sandbox); }, diff --git a/agent-eval/experiments/cc-plugin-haiku-5.5-low.ts b/agent-eval/experiments/cc-plugin-haiku-5.5-low.ts new file mode 100644 index 000000000000..c3acc4557fd8 --- /dev/null +++ b/agent-eval/experiments/cc-plugin-haiku-5.5-low.ts @@ -0,0 +1,23 @@ +import type { ExperimentConfig } from '@vercel/agent-eval'; +import { DEFAULT_EXPERIMENT_CONFIG, PLUGIN_STORYBOOK_EVALS } from '../lib/experiment.ts'; +import { + setupSandbox, + writeClaudeInAppBrowserMock, + writeClaudePluginSkills, +} from '../lib/templates.ts'; + +const model = 'claude-haiku-5-5'; + +export default { + ...DEFAULT_EXPERIMENT_CONFIG, + agent: 'claude-code', // direct Anthropic API, requires ANTHROPIC_API_KEY + model, + agentOptions: { effort: 'low' }, + // Skipped under EVAL_STORYBOOK_LATEST=1; see PLUGIN_STORYBOOK_EVALS. + evals: PLUGIN_STORYBOOK_EVALS, + setup: async (sandbox) => { + await setupSandbox(sandbox, { agent: 'claude-code', integration: 'plugin', model }); + await writeClaudePluginSkills(sandbox); + await writeClaudeInAppBrowserMock(sandbox); + }, +} satisfies ExperimentConfig; diff --git a/agent-eval/experiments/cc-plugin-opus-5.5-medium.ts b/agent-eval/experiments/cc-plugin-opus-5.5-medium.ts index 005ef58ad7d0..10908546f4ae 100644 --- a/agent-eval/experiments/cc-plugin-opus-5.5-medium.ts +++ b/agent-eval/experiments/cc-plugin-opus-5.5-medium.ts @@ -1,20 +1,26 @@ import type { ExperimentConfig } from '@vercel/agent-eval'; -import { DEFAULT_EXPERIMENT_CONFIG, PLUGIN_STORYBOOK_EVALS } from '../lib/experiment.ts'; +import { + DEFAULT_EXPERIMENT_CONFIG, + onlyWhenNamed, + PLUGIN_STORYBOOK_EVALS, +} from '../lib/experiment.ts'; import { setupSandbox, writeClaudeInAppBrowserMock, writeClaudePluginSkills, } from '../lib/templates.ts'; +const model = 'claude-opus-5-5'; + export default { ...DEFAULT_EXPERIMENT_CONFIG, agent: 'claude-code', // direct Anthropic API, requires ANTHROPIC_API_KEY - model: 'claude-opus-5-5', + model, agentOptions: { effort: 'medium' }, - // Skipped under EVAL_STORYBOOK_LATEST=1; see PLUGIN_STORYBOOK_EVALS. - evals: PLUGIN_STORYBOOK_EVALS, + // Not in the default set: runs only when named. Also skipped under EVAL_STORYBOOK_LATEST=1. + evals: onlyWhenNamed(import.meta.url, PLUGIN_STORYBOOK_EVALS), setup: async (sandbox) => { - await setupSandbox(sandbox, { agent: 'claude-code', integration: 'plugin' }); + await setupSandbox(sandbox, { agent: 'claude-code', integration: 'plugin', model }); await writeClaudePluginSkills(sandbox); await writeClaudeInAppBrowserMock(sandbox); }, diff --git a/agent-eval/experiments/cc-plugin-sonnet-5.5-low.ts b/agent-eval/experiments/cc-plugin-sonnet-5.5-low.ts new file mode 100644 index 000000000000..1b761f516d2e --- /dev/null +++ b/agent-eval/experiments/cc-plugin-sonnet-5.5-low.ts @@ -0,0 +1,27 @@ +import type { ExperimentConfig } from '@vercel/agent-eval'; +import { + DEFAULT_EXPERIMENT_CONFIG, + onlyWhenNamed, + PLUGIN_STORYBOOK_EVALS, +} from '../lib/experiment.ts'; +import { + setupSandbox, + writeClaudeInAppBrowserMock, + writeClaudePluginSkills, +} from '../lib/templates.ts'; + +const model = 'claude-sonnet-5-5'; + +export default { + ...DEFAULT_EXPERIMENT_CONFIG, + agent: 'claude-code', // direct Anthropic API, requires ANTHROPIC_API_KEY + model, + agentOptions: { effort: 'low' }, + // Not in the default set: runs only when named. Also skipped under EVAL_STORYBOOK_LATEST=1. + evals: onlyWhenNamed(import.meta.url, PLUGIN_STORYBOOK_EVALS), + setup: async (sandbox) => { + await setupSandbox(sandbox, { agent: 'claude-code', integration: 'plugin', model }); + await writeClaudePluginSkills(sandbox); + await writeClaudeInAppBrowserMock(sandbox); + }, +} satisfies ExperimentConfig; diff --git a/agent-eval/experiments/codex-mcp-gpt-6.1-sol-medium.ts b/agent-eval/experiments/codex-mcp-gpt-6.1-sol-medium.ts index b5722002a992..7bcb8d513b3d 100644 --- a/agent-eval/experiments/codex-mcp-gpt-6.1-sol-medium.ts +++ b/agent-eval/experiments/codex-mcp-gpt-6.1-sol-medium.ts @@ -7,16 +7,18 @@ import { writeCodexMcpConfig, } from '../lib/templates.ts'; +const model = 'gpt-6.1-sol?reasoningEffort=medium'; + export default { ...DEFAULT_EXPERIMENT_CONFIG, // Use direct Codex for MCP evals. The AI Gateway Codex path does not reliably // handle Codex's Responses namespace tool shape yet: // https://github.com/openai/codex/issues/26234 agent: 'codex', - model: 'gpt-6.1-sol?reasoningEffort=medium', + model, evals: WORKFLOW_STORYBOOK_EVALS, setup: async (sandbox) => { - await setupSandbox(sandbox, { agent: 'codex', integration: 'mcp' }); + await setupSandbox(sandbox, { agent: 'codex', integration: 'mcp', model }); await writeCodexMcpConfig(sandbox); await writeCodexAgentsMd(sandbox); await writeCodexInAppBrowserMock(sandbox); diff --git a/agent-eval/experiments/codex-plugin-gpt-6-luna-low.ts b/agent-eval/experiments/codex-plugin-gpt-6-luna-low.ts new file mode 100644 index 000000000000..a67d79468392 --- /dev/null +++ b/agent-eval/experiments/codex-plugin-gpt-6-luna-low.ts @@ -0,0 +1,26 @@ +import type { ExperimentConfig } from '@vercel/agent-eval'; +import { + DEFAULT_EXPERIMENT_CONFIG, + onlyWhenNamed, + PLUGIN_STORYBOOK_EVALS, +} from '../lib/experiment.ts'; +import { + setupSandbox, + writeCodexInAppBrowserMock, + writeCodexPluginSkills, +} from '../lib/templates.ts'; + +const model = 'gpt-6-luna?reasoningEffort=low'; + +export default { + ...DEFAULT_EXPERIMENT_CONFIG, + agent: 'codex', + model, + // Not in the default set: runs only when named. Also skipped under EVAL_STORYBOOK_LATEST=1. + evals: onlyWhenNamed(import.meta.url, PLUGIN_STORYBOOK_EVALS), + setup: async (sandbox) => { + await setupSandbox(sandbox, { agent: 'codex', integration: 'plugin', model }); + await writeCodexPluginSkills(sandbox); + await writeCodexInAppBrowserMock(sandbox); + }, +} satisfies ExperimentConfig; diff --git a/agent-eval/experiments/codex-plugin-gpt-6.1-sol-low.ts b/agent-eval/experiments/codex-plugin-gpt-6.1-sol-low.ts new file mode 100644 index 000000000000..95fccca4bce1 --- /dev/null +++ b/agent-eval/experiments/codex-plugin-gpt-6.1-sol-low.ts @@ -0,0 +1,25 @@ +import type { ExperimentConfig } from '@vercel/agent-eval'; +import { DEFAULT_EXPERIMENT_CONFIG, PLUGIN_STORYBOOK_EVALS } from '../lib/experiment.ts'; +import { + setupSandbox, + writeCodexInAppBrowserMock, + writeCodexPluginSkills, +} from '../lib/templates.ts'; + +const model = 'gpt-6.1-sol?reasoningEffort=low'; + +export default { + ...DEFAULT_EXPERIMENT_CONFIG, + // Keep Codex plugin and MCP experiments on the same direct Codex runner. + // The MCP variant cannot use the AI Gateway path yet: + // https://github.com/openai/codex/issues/26234 + agent: 'codex', + model, + // Skipped under EVAL_STORYBOOK_LATEST=1; see PLUGIN_STORYBOOK_EVALS. + evals: PLUGIN_STORYBOOK_EVALS, + setup: async (sandbox) => { + await setupSandbox(sandbox, { agent: 'codex', integration: 'plugin', model }); + await writeCodexPluginSkills(sandbox); + await writeCodexInAppBrowserMock(sandbox); + }, +} satisfies ExperimentConfig; diff --git a/agent-eval/experiments/codex-plugin-gpt-6.1-sol-medium.ts b/agent-eval/experiments/codex-plugin-gpt-6.1-sol-medium.ts index 6768399aa557..ca25d368844f 100644 --- a/agent-eval/experiments/codex-plugin-gpt-6.1-sol-medium.ts +++ b/agent-eval/experiments/codex-plugin-gpt-6.1-sol-medium.ts @@ -1,22 +1,28 @@ import type { ExperimentConfig } from '@vercel/agent-eval'; -import { DEFAULT_EXPERIMENT_CONFIG, PLUGIN_STORYBOOK_EVALS } from '../lib/experiment.ts'; +import { + DEFAULT_EXPERIMENT_CONFIG, + onlyWhenNamed, + PLUGIN_STORYBOOK_EVALS, +} from '../lib/experiment.ts'; import { setupSandbox, writeCodexInAppBrowserMock, writeCodexPluginSkills, } from '../lib/templates.ts'; +const model = 'gpt-6.1-sol?reasoningEffort=medium'; + export default { ...DEFAULT_EXPERIMENT_CONFIG, // Keep Codex plugin and MCP experiments on the same direct Codex runner. // The MCP variant cannot use the AI Gateway path yet: // https://github.com/openai/codex/issues/26234 agent: 'codex', - model: 'gpt-6.1-sol?reasoningEffort=medium', - // Skipped under EVAL_STORYBOOK_LATEST=1; see PLUGIN_STORYBOOK_EVALS. - evals: PLUGIN_STORYBOOK_EVALS, + model, + // Not in the default set: runs only when named. Also skipped under EVAL_STORYBOOK_LATEST=1. + evals: onlyWhenNamed(import.meta.url, PLUGIN_STORYBOOK_EVALS), setup: async (sandbox) => { - await setupSandbox(sandbox, { agent: 'codex', integration: 'plugin' }); + await setupSandbox(sandbox, { agent: 'codex', integration: 'plugin', model }); await writeCodexPluginSkills(sandbox); await writeCodexInAppBrowserMock(sandbox); }, diff --git a/agent-eval/lib/experiment.test.ts b/agent-eval/lib/experiment.test.ts index 2581e1a00132..b94979e86cbc 100644 --- a/agent-eval/lib/experiment.test.ts +++ b/agent-eval/lib/experiment.test.ts @@ -2,7 +2,7 @@ import { describe, expect, it } from 'vitest'; import type { RunCompleteContext } from '@vercel/agent-eval'; -import { DEFAULT_EXPERIMENT_CONFIG } from './experiment.ts'; +import { DEFAULT_EXPERIMENT_CONFIG, onlyWhenNamed } from './experiment.ts'; describe('the run-complete hook', () => { it('records the checkout commit next to the existing metadata', async () => { @@ -18,3 +18,24 @@ describe('the run-complete hook', () => { }); }); }); + +describe('an experiment outside the default set', () => { + const experimentUrl = 'file:///repo/agent-eval/experiments/codex-plugin-gpt-6-luna-low.ts'; + const evals = ['801-create-accessible-component' as const]; + + it('runs no evals when the command line does not name it', () => { + expect(onlyWhenNamed(experimentUrl, evals, ['node', 'agent-eval'])).toEqual([]); + }); + + it('runs its evals when the command line names it', () => { + expect( + onlyWhenNamed(experimentUrl, evals, ['node', 'agent-eval', 'codex-plugin-gpt-6-luna-low']) + ).toEqual(evals); + }); + + it('does not run for another experiment whose name starts the same', () => { + expect( + onlyWhenNamed(experimentUrl, evals, ['node', 'agent-eval', 'codex-plugin-gpt-6-luna-low-x3']) + ).toEqual([]); + }); +}); diff --git a/agent-eval/lib/experiment.ts b/agent-eval/lib/experiment.ts index 7cf5c413b5e1..8758289ae9a1 100644 --- a/agent-eval/lib/experiment.ts +++ b/agent-eval/lib/experiment.ts @@ -22,6 +22,7 @@ const CORE_STORYBOOK_EVALS = [ '812-first-story-empty-project', '813-monorepo-leaf-create-component', '814-write-stories-for-fetching-component', + '815-write-stories-for-fetching-component-without-msw', ] as const; // The 82x block: lifecycle-skill evals (storybook-init / storybook-upgrade). @@ -136,6 +137,18 @@ export const PLUGIN_STORYBOOK_EVALS: EvalName[] = STORYBOOK_LATEST ? [] : [...ACTIVE_EVALS.core, ...ACTIVE_EVALS.lifecycle]; +// For an experiment outside the default set: its evals run only when the experiment is named on +// the command line (`agent-eval codex-plugin-gpt-6-luna-low`), so a bare `agent-eval`, which is +// what CI runs, skips it. +export function onlyWhenNamed( + experimentUrl: string, + evals: EvalName[], + argv: string[] = process.argv +): EvalName[] { + const name = path.basename(fileURLToPath(experimentUrl), '.ts'); + return argv.some((arg) => path.basename(arg, '.ts') === name) ? evals : []; +} + const REPO_ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..', '..'); const execFileAsync = promisify(execFile); let checkoutRevision: Promise<{ commit: string; dirty: boolean }> | undefined; diff --git a/agent-eval/lib/templates.ts b/agent-eval/lib/templates.ts index aad227eaa16d..63a1fe8de6f7 100644 --- a/agent-eval/lib/templates.ts +++ b/agent-eval/lib/templates.ts @@ -141,7 +141,7 @@ const CODEX_PLUGIN_SKILLS_DIR = path.join( export async function setupSandbox( sandbox: Sandbox, - options: { agent: EvalAgent; integration: EvalIntegration } + options: { agent: EvalAgent; integration: EvalIntegration; model?: string } ): Promise { await writeEvalSupportFiles(sandbox, options); @@ -214,14 +214,14 @@ export async function setupSandbox( async function writeEvalSupportFiles( sandbox: Sandbox, - options: { agent: EvalAgent; integration: EvalIntegration } + options: { agent: EvalAgent; integration: EvalIntegration; model?: string } ): Promise { await sandbox.writeFiles({ [TRANSCRIPT_HELPER_SANDBOX_PATH]: await fs.readFile(TRANSCRIPT_HELPER_SOURCE_PATH, 'utf8'), [SHELL_PARSE_SANDBOX_PATH]: await fs.readFile(SHELL_PARSE_SOURCE_PATH, 'utf8'), [TYPE_UTIL_SANDBOX_PATH]: await fs.readFile(TYPE_UTIL_SOURCE_PATH, 'utf8'), [AGENT_CONTEXT_SANDBOX_PATH]: JSON.stringify( - { agent: options.agent, integration: options.integration }, + { agent: options.agent, model: options.model, integration: options.integration }, null, 2 ).concat('\n'), diff --git a/agent-eval/lib/test-utils.test.ts b/agent-eval/lib/test-utils.test.ts index 661d60d469b8..17f8a5ccc212 100644 --- a/agent-eval/lib/test-utils.test.ts +++ b/agent-eval/lib/test-utils.test.ts @@ -525,6 +525,18 @@ describe('expectStoryTestsRanAndPassed', () => { await expect(expectStoryTestsRanAndPassed()).rejects.toThrow(/Expected test-run to be called/); }); + + test('lets the agent skip the tests with requireAgentRun false, but still runs them in the sandbox', async () => { + givenRun({ + commands: ['npm run typecheck'], + stdout: testRunDocument({ 'example-badge--accent': 'status-value:success' }), + }); + const { expectStoryTestsRanAndPassed } = await loadTestUtils(); + + await expectStoryTestsRanAndPassed({ requireAgentRun: false, covering: ['badge'] }); + + expect(execFile).toHaveBeenCalled(); + }); }); describe('parseCodexBrowserNavigations', () => { diff --git a/agent-eval/lib/test-utils.ts b/agent-eval/lib/test-utils.ts index d73e3affa898..9cd453b00d6d 100644 --- a/agent-eval/lib/test-utils.ts +++ b/agent-eval/lib/test-utils.ts @@ -29,11 +29,14 @@ const TRANSCRIPT_PATH = '__agent_eval__/transcript.txt'; type AgentContext = { agent?: unknown; + model?: unknown; integration?: unknown; }; type EvalContext = { agent: string; + // Such as `gpt-6-luna?reasoningEffort=low`; absent when the experiment passes none. + model?: string; // 'none' is the agentic-reference bare control (no Storybook tooling flavor); // see lib/templates.ts. Plugin-only helpers below treat it as "not plugin". integration: 'mcp' | 'plugin' | 'none'; @@ -57,7 +60,7 @@ export type StoryInputExpectation = { export function getEvalContext(): EvalContext { const agentContext = readAgentContext(); - const { agent, integration } = agentContext; + const { agent, model, integration } = agentContext; if (typeof agent !== 'string') { throw new Error( @@ -74,7 +77,13 @@ export function getEvalContext(): EvalContext { ); } - return { agent, integration }; + return { agent, ...(typeof model === 'string' && { model }), integration }; +} + +// Codex's system prompt for GPT-6 Luna alone says "Do not add or run tests unless the user asks +// you to test or verify implementation." +export function modelRunsTestsOnlyWhenAsked(): boolean { + return getEvalContext().model?.startsWith('gpt-6-luna') === true; } export function getTranscript(agent = getEvalContext().agent): Transcript { @@ -392,11 +401,16 @@ export type WorkflowToolResult = { // // `covering` requires one of the given substrings in the passing story ids. // `cwd` is the Storybook project, for fixtures where that is not the root. +// `requireAgentRun: false` lets the agent skip the tests, for a model that runs +// them only when asked; a run it did make must still not end red. export async function expectStoryTestsRanAndPassed(options?: { covering?: string[]; cwd?: string; + requireAgentRun?: boolean; }): Promise { - expectWorkflowCalls(['test-run']); + if (options?.requireAgentRun !== false) { + expectWorkflowCalls(['test-run']); + } expect( getWorkflowToolResults('test-run').at(-1)?.output ?? '', "The agent's last test run must not report failing stories or unhandled errors" diff --git a/agent-eval/lib/usage.ts b/agent-eval/lib/usage.ts index 435c34c25d1a..c71479305d97 100644 --- a/agent-eval/lib/usage.ts +++ b/agent-eval/lib/usage.ts @@ -12,6 +12,10 @@ const MODEL_PRICING: Record { expect(instructions).not.toContain('get-storybook-story-instructions'); }); - it('points at the story instructions in the same document when they are inline', () => { - const instructions = buildServerInstructions({ - transport: 'cli', - devEnabled: true, - testSupported: false, - docsEnabled: false, - moduleGraphSupported: true, - storyInstructionsInline: true, - }); - - expect(instructions).toContain( - '- Before creating or editing components or stories, read **Writing User Interfaces** below; it is the source of truth for imports, story patterns, and testing conventions.' - ); - expect(instructions).not.toContain('npx storybook skills write-story'); - }); - it('renders review, preview, and discovery references as CLI commands', () => { const instructions = buildServerInstructions({ transport: 'cli', diff --git a/code/core/src/cli/skills/content/build-server-instructions.ts b/code/core/src/cli/skills/content/build-server-instructions.ts index 0eb8f712812d..86d35c20658d 100644 --- a/code/core/src/cli/skills/content/build-server-instructions.ts +++ b/code/core/src/cli/skills/content/build-server-instructions.ts @@ -12,13 +12,8 @@ export type ServerInstructionsInputs = { docsEnabled: boolean; /** `stories-find-by-component` is registered, so the workflow can point at it. */ moduleGraphSupported: boolean; - /** The story instructions follow in the same document, so the workflow points at them there. */ - storyInstructionsInline?: boolean; }; -const INLINE_STORY_INSTRUCTIONS_STEP = - 'Before creating or editing components or stories, read **Writing User Interfaces** below; it is the source of truth for imports, story patterns, and testing conventions.'; - /** * The full rule for how the agent should present links in its final * user-facing response, delivered through the @@ -66,9 +61,7 @@ export function buildServerInstructions({ devInstructions .replace( '{{STORY_INSTRUCTIONS_STEP}}', - options.storyInstructionsInline - ? INLINE_STORY_INSTRUCTIONS_STEP - : `Before creating or editing components or stories, call **${skillRef('write-story')}**; its output is the source of truth for imports, story patterns, and testing conventions.` + `Before creating or editing components or stories, call **${skillRef('write-story')}**; its output is the source of truth for imports, story patterns, and testing conventions.` ) .replace('{{PREVIEW_STORIES_STEP}}', previewStoriesStep) .replace('{{FINAL_LINKS_STEP}}', finalLinksStep) diff --git a/code/core/src/cli/skills/content/build-stories-skill.test.ts b/code/core/src/cli/skills/content/build-stories-skill.test.ts new file mode 100644 index 000000000000..2780d711ca21 --- /dev/null +++ b/code/core/src/cli/skills/content/build-stories-skill.test.ts @@ -0,0 +1,117 @@ +import { describe, expect, it } from 'vitest'; + +import { buildStoriesSkill, type StoriesSkillInputs } from './build-stories-skill.ts'; + +const everything: StoriesSkillInputs = { + framework: '@storybook/react-vite', + csfFactories: false, + previewFile: '.storybook/preview.ts', + typescript: true, + docsEnabled: true, + testSupported: true, + a11yEnabled: true, + moduleGraphSupported: true, +}; + +describe('buildStoriesSkill', () => { + it('shows every command with its arguments when every feature is on', () => { + const text = buildStoriesSkill(everything); + + expect(text).toContain('npx storybook tools docs show --id '); + expect(text).toContain(`npx storybook tools test run --stories '[{"storyId":""}]'`); + expect(text).toContain('npx storybook tools stories changed'); + expect(text).toContain(`npx storybook tools stories find-by-component --componentPaths '["`); + expect(text).toContain(`npx storybook tools review create --input '{`); + expect(text).toContain('accessibility violations'); + }); + + it('leaves out testing when the project cannot run story tests', () => { + const text = buildStoriesSkill({ ...everything, testSupported: false, a11yEnabled: false }); + + expect(text).not.toContain('test run'); + expect(text).not.toContain('accessibility'); + }); + + it('requires the commands that find affected stories before a review, not the full story list', () => { + const text = buildStoriesSkill(everything); + + expect(text).toContain( + 'Run `stories changed` or `find-by-component` before every review, also when you already know the ids' + ); + expect(text).not.toContain('Run one of these'); + }); + + it('takes story ids from the story list when the builder has no module graph', () => { + const text = buildStoriesSkill({ ...everything, moduleGraphSupported: false }); + + expect(text).toContain('docs list --withStoryIds true'); + expect(text).toContain('Story ids come only from this command.'); + expect(text).toContain('When it does not find a story'); + expect(text).not.toContain('stories changed'); + expect(text).not.toContain('find-by-component'); + }); + + it('selects stories by file and export when no command lists story ids', () => { + const text = buildStoriesSkill({ + ...everything, + docsEnabled: false, + moduleGraphSupported: false, + }); + + expect(text).toContain('"absoluteStoryPath"'); + expect(text).not.toContain('stories changed'); + }); + + it('teaches CSF Factories instead of Meta and StoryObj when the preview uses definePreview', () => { + const text = buildStoriesSkill({ + ...everything, + csfFactories: true, + previewFile: '.storybook/preview.tsx', + }); + + expect(text).toContain('preview.meta({'); + expect(text).toContain('Do not import `Meta` or `StoryObj`'); + expect(text).not.toContain('Import `Meta` and `StoryObj`'); + }); + + it('teaches Meta and StoryObj from the framework package otherwise', () => { + const text = buildStoriesSkill({ ...everything, framework: '@storybook/vue3-vite' }); + + expect(text).toContain('Import `Meta` and `StoryObj` from `@storybook/vue3-vite`'); + expect(text).not.toContain('preview.meta({'); + }); + + it('leaves out the Meta and StoryObj types in a JavaScript project', () => { + const text = buildStoriesSkill({ + ...everything, + previewFile: '.storybook/preview.js', + typescript: false, + }); + + expect(text).not.toContain('StoryObj'); + expect(text).toContain('- Import `fn`, `expect`, `mocked` and `sb` from `storybook/test`.'); + expect(text).toContain("sb.mock(import('../src/api.js')"); + }); + + it('puts MSW before module mocks and names the preview file of the project', () => { + const text = buildStoriesSkill({ ...everything, previewFile: 'config/preview.tsx' }); + + expect(text.indexOf('msw.use(')).toBeLessThan(text.indexOf('sb.mock(')); + expect(text).toContain('register it in `config/preview.tsx`'); + }); + + it('tells how to set up MSW in the format of the project', () => { + const csf3 = buildStoriesSkill({ ...everything, previewFile: 'config/preview.tsx' }); + expect(csf3).toContain('never with a module mock or a replaced `fetch`'); + expect(csf3).toContain( + "`loaders: [mswLoader()]` (`import { mswLoader } from 'msw-storybook-addon/csf3'`" + ); + expect(csf3).toContain('to `config/preview.tsx`'); + + const factories = buildStoriesSkill({ ...everything, csfFactories: true }); + expect(factories).toContain( + "`addons: [addonMsw()]` (`import addonMsw from 'msw-storybook-addon'`" + ); + expect(factories).not.toContain('mswLoader'); + }); +}); diff --git a/code/core/src/cli/skills/content/build-stories-skill.ts b/code/core/src/cli/skills/content/build-stories-skill.ts new file mode 100644 index 000000000000..7f5ff33323e2 --- /dev/null +++ b/code/core/src/cli/skills/content/build-stories-skill.ts @@ -0,0 +1,147 @@ +import { getToolName } from '../../../shared/open-service/toolset-names.ts'; + +export type StoriesSkillInputs = { + framework: string; + /** The preview file uses `definePreview`, so story files use CSF Factories. */ + csfFactories: boolean; + /** Path of the preview file, relative to the working directory. */ + previewFile: string; + typescript: boolean; + docsEnabled: boolean; + testSupported: boolean; + a11yEnabled: boolean; + moduleGraphSupported: boolean; +}; + +const ref = getToolName({ transport: 'cli' }); + +function docsSection(): string { + return `## Look up components first + +\`\`\`sh +${ref('docs.list')} # every component and docs page, with its id +${ref('docs.show')} --id # props and usage examples of one entry +${ref('docs.showStory')} --storyId # the code of a story that docs show only lists +\`\`\` + +Do this before you edit any file: run \`docs list\` once, then \`docs show\` for every listed component your change uses, even one you only add such as a button, and for each one you are asked about. Reuse what exists instead of building a duplicate. Use only the props that \`docs show\` documents, never one you assume from a name or another library, and answer props, API and usage questions from these commands. Read source files or \`node_modules\` only when the commands return nothing relevant. When \`docs list\` groups its entries under sources (\`id: acme\`), pass the source of the entry to both commands that show it: \`--storybookId acme\`.`; +} + +function writeSection({ + framework, + csfFactories, + previewFile, + typescript, +}: StoriesSkillInputs): string { + const format = csfFactories + ? `This project writes stories with CSF Factories, because \`${previewFile}\` uses \`definePreview\`: \`import preview from '../.storybook/preview'\` (the path to that file), then \`const meta = preview.meta({ component: Button })\` and \`export const Primary = meta.story({ args: { ... } })\`. Do not import \`Meta\` or \`StoryObj\`. Import` + : typescript + ? `Import \`Meta\` and \`StoryObj\` from \`${framework}\`, and` + : 'Import'; + const mswSetup = csfFactories + ? "`addons: [addonMsw()]` (`import addonMsw from 'msw-storybook-addon'`, without an `initialize()` call)" + : "`loaders: [mswLoader()]` (`import { mswLoader } from 'msw-storybook-addon/csf3'`, without an `initialize()` call)"; + return `## Write the component and its stories + +Every component you create or change gets stories: one per distinct state it can reach (variants, loading, empty, error, disabled), with realistic props. Never export a story under the name of a global such as \`Error\`: export \`ErrorState\` and set \`name: 'Error'\`. An interactive component also gets a \`play\` function that drives it and asserts the visible result; for a callback passed as \`fn()\`, assert whether it was called, as that state expects. The rules below are for the stories you write; do not rewrite existing stories only to match them. + +- ${format} \`fn\`, \`expect\`, \`mocked\` and \`sb\` from \`storybook/test\`. +- \`play: async ({ canvas, userEvent }) => { ... }\`: query \`canvas\` directly, by role or label. Never wrap it in \`within()\`. \`userEvent.click(element)\` takes no options. +- A story never calls a real service. Mock network requests with MSW, never with a module mock or a replaced \`fetch\`: give the story \`beforeEach({ msw }) { msw.use(http.get('/api/users', () => HttpResponse.json([...]))) }\`, and leave the code that calls \`fetch\` as it is. When the project has no \`msw-storybook-addon\` yet, set it up first: install \`msw\` and \`msw-storybook-addon\` as dev dependencies, run \`npx msw init ./public --save\` (\`../public\` has to be in \`staticDirs\` of the main config), and add ${mswSetup} to \`${previewFile}\`. Read the addon's version in \`package.json\`: only before version 3 do stories have no \`msw\` in \`beforeEach\` and take \`parameters: { msw: { handlers: [...] } }\` instead. From version 3 on, never write \`parameters.msw\`. +- Mock a module only for a dependency that is not a network call: register it in \`${previewFile}\` with \`sb.mock(import('../src/api.${typescript ? 'ts' : 'js'}'), { spy: true })\` (path relative to that file, with its extension), then set the result per story in \`beforeEach\` with \`mocked(getUser).mockResolvedValue(...)\`.`; +} + +function testSection(a11yEnabled: boolean): string { + const a11y = a11yEnabled + ? '\n\nThe run lists accessibility violations under `## Accessibility Violations`. Fix semantic ones yourself (roles, labels, alt text, keyboard access). For visual ones such as color contrast, do not change the design: describe the problem, offer two or three options and ask the user.' + : ''; + return `## Test + +\`\`\`sh +${ref('test.run')} --stories '[{"storyId":""}]' # ids: see "Find the stories"; leave out --stories to run every story +\`\`\` + +Run this after every change. It is the only way to run story tests: never a \`package.json\` test script (such as \`npm run test:stories\`) or \`vitest\` directly, also when the user asks you to run the tests. Use focused runs while iterating and one full run before you finish. Fix failures and rerun; never finish with failing tests.${a11y}`; +} + +function discoverSection(inputs: StoriesSkillInputs): string { + const commands = [ + inputs.moduleGraphSupported && + `${ref('stories.changed')} # stories affected by your uncommitted changes`, + inputs.moduleGraphSupported && + `${ref('stories.findByComponent')} --componentPaths '["/abs/path/Button.tsx"]' # stories that render these files`, + inputs.docsEnabled && `${ref('docs.list')} --withStoryIds true # every story id`, + ].filter(Boolean); + if (commands.length === 0) { + return `## Find the stories + +This project has no command that lists story ids. Select a story by its file and export instead: \`--stories '[{"absoluteStoryPath":"/abs/path/Button.stories.tsx","exportName":"Primary"}]'\`.`; + } + const several = commands.length > 1; + const guidance = [ + inputs.moduleGraphSupported && + 'Run \`stories changed\` or \`find-by-component\` before every review, also when you already know the ids of the stories you wrote: they add the stories of other components that your change affects.', + `Story ids come only from ${several ? 'these commands' : 'this command'}. Never build one from a file name, a title or memory.`, + inputs.moduleGraphSupported && + 'When \`stories changed\` leaves out a file you touched, pass that file to \`find-by-component\`; for a shared file (design token, theme, util, hook), which has no stories of its own, pass the components that use it. When \`find-by-component\` reports stories hidden by \`maxDistance\`, rerun it with a higher \`--maxDistance\`.', + `When ${several ? 'none of them finds' : 'it does not find'} a story for a component, it has no stories yet: say so, or write them.`, + ] + .filter(Boolean) + .join(' '); + return `## Find the stories + +\`\`\`sh +${commands.join('\n')} +\`\`\` + +${guidance}`; +} + +function reviewSection({ moduleGraphSupported }: StoriesSkillInputs): string { + const discoverFirst = moduleGraphSupported + ? ' When you changed files, run \`stories changed\` or \`find-by-component\` right before it and take the story ids from there, also when you only wrote stories, not from the output of the tests.' + : ''; + return `## Finish with a review + +\`\`\`sh +${ref('review.create')} --input '{ + "title": "Rounder buttons", + "description": "The **Button** corner radius goes from 4px to 8px.", + "collections": [ + { "title": "Button variants", "rationale": "The component that changed.", "storyIds": ["button--primary", "button--disabled"] }, + { "title": "Forms that use it", "rationale": "Where the change shows up in context.", "storyIds": ["login-form--default"] } + ], + "changedFiles": ["src/Button.tsx"] +}' +\`\`\` + +Publish a review after every change the user can see, and again after each later change. It needs a running Storybook.${discoverFirst} Group the stories into two to five collections, from the changed component up to the pages that show it (one is enough when a single component is affected), and include every story you created. List every file you created or changed in \`changedFiles\`, story files too. When the user only asks to see or browse components or stories and you changed no files, publish the same review with \`"changedFiles": []\`. Skip the review only when nothing visible changed and the user did not ask to see anything, and say that instead. + +Then do both things the command prints, every time: open the review in the in-app browser with a browser tool (unless you have none), and end your answer with the review section it gives you. Do not list separate story links next to it.`; +} + +function previewSection(): string { + return `## Link to one story + +\`\`\`sh +${ref('stories.preview')} --stories '[{"storyId":""}]' # direct links to stories +\`\`\` + +Only for a look at one story while you iterate, or when the user asks for a direct link. It does not replace the review, unless \`review create\` keeps failing after you fixed what it reported: then share these links and say why.`; +} + +export function buildStoriesSkill(inputs: StoriesSkillInputs): string { + return [ + `# Storybook workflow + +Follow this for every change to how the UI looks (components, stories, styles, themes, tokens) and for requests to show components or stories. Some commands print what to do next; follow that too. Add \`--help\` to a command only when a call fails.`, + inputs.docsEnabled && docsSection(), + writeSection(inputs), + inputs.testSupported && testSection(inputs.a11yEnabled), + discoverSection(inputs), + reviewSection(inputs), + previewSection(), + ] + .filter(Boolean) + .join('\n\n'); +} diff --git a/code/core/src/cli/skills/content/skills.ts b/code/core/src/cli/skills/content/skills.ts index 831dedc5d808..6be486e2f793 100644 --- a/code/core/src/cli/skills/content/skills.ts +++ b/code/core/src/cli/skills/content/skills.ts @@ -11,7 +11,7 @@ export type SkillId = (typeof SKILL_IDS)[number]; export const SKILLS: Record = { stories: { blurb: - 'The mandatory, ordered workflow for UI changes: discover affected stories, test, and present results. Includes `write-story`.', + 'The workflow for UI changes, with every command it needs: look up components, write stories, test, and publish a review.', }, 'write-story': { blurb: diff --git a/code/core/src/cli/skills/inputs.test.ts b/code/core/src/cli/skills/inputs.test.ts index d820a4b6fe3d..1822767ce357 100644 --- a/code/core/src/cli/skills/inputs.test.ts +++ b/code/core/src/cli/skills/inputs.test.ts @@ -1,15 +1,33 @@ -import { describe, expect, it, vi } from 'vitest'; +import { existsSync } from 'node:fs'; +import { readFile } from 'node:fs/promises'; + +import { beforeEach, describe, expect, it, vi } from 'vitest'; + +import { fs as memfs, vol } from 'memfs'; import type { Options } from '../../types/index.ts'; import { resolveSkillInputs } from './inputs.ts'; +vi.mock('node:fs', { spy: true }); +vi.mock('node:fs/promises', { spy: true }); + +beforeEach(() => { + vol.reset(); + vi.mocked(existsSync).mockImplementation(memfs.existsSync); + vi.mocked(readFile).mockImplementation(memfs.promises.readFile as typeof readFile); + vi.spyOn(process, 'cwd').mockReturnValue('/project'); +}); + function createMockOptions({ framework = '@storybook/react-vite', + configDir, }: { framework?: string | { name: string }; + configDir?: string; } = {}): Options { return { + configDir, presets: { apply: vi.fn(async (key: string, defaultValue?: unknown) => { if (key === 'framework') { @@ -71,4 +89,50 @@ describe('resolveSkillInputs', () => { expect(inputs.moduleGraphSupported).toBe(false); }); + + it('detects CSF Factories from a preview file that imports definePreview', async () => { + vol.fromNestedJSON({ + '/project/.storybook/preview.tsx': `import { definePreview } from '@storybook/react-vite';\nexport default definePreview({});`, + }); + + const inputs = await resolveSkillInputs( + createMockOptions({ configDir: '/project/.storybook' }) + ); + + expect(inputs).toMatchObject({ + csfFactories: true, + previewFile: '.storybook/preview.tsx', + typescript: true, + }); + }); + + it('reads a plain JavaScript preview as CSF 3', async () => { + vol.fromNestedJSON({ + '/project/.storybook/preview.js': 'export default { parameters: {} };', + }); + + const inputs = await resolveSkillInputs( + createMockOptions({ configDir: '/project/.storybook' }) + ); + + expect(inputs).toMatchObject({ + csfFactories: false, + previewFile: '.storybook/preview.js', + typescript: false, + }); + }); + + it('names the preview file after the main config when the project has none', async () => { + vol.fromNestedJSON({ '/project/.storybook/main.ts': 'export default {};' }); + + const inputs = await resolveSkillInputs( + createMockOptions({ configDir: '/project/.storybook' }) + ); + + expect(inputs).toMatchObject({ + csfFactories: false, + previewFile: '.storybook/preview.ts', + typescript: true, + }); + }); }); diff --git a/code/core/src/cli/skills/inputs.ts b/code/core/src/cli/skills/inputs.ts index 8f9c226eaa62..8384a6b80ae0 100644 --- a/code/core/src/cli/skills/inputs.ts +++ b/code/core/src/cli/skills/inputs.ts @@ -1,10 +1,42 @@ +import { relative } from 'node:path'; + import type { Options } from '../../types/index.ts'; import { extractFrameworkPackageName } from '../../common/utils/get-framework-name.ts'; +import { findConfigFile } from '../../common/utils/get-storybook-info.ts'; +import { isCsfFactoryPreview, readConfig } from '../../csf-tools/ConfigFile.ts'; import { getToolAvailability, type ToolAvailability } from './availability.ts'; import { frameworkToRendererMap } from './content/framework-renderer.ts'; -export type SkillInputs = ToolAvailability & { framework: string; renderer?: string }; +export type SkillInputs = ToolAvailability & { + framework: string; + renderer?: string; + /** The preview file imports `definePreview`, the entry point of CSF Factories. */ + csfFactories: boolean; + /** Path of the preview file relative to the working directory; a default name when there is none. */ + previewFile: string; + /** The preview file, or `main` when there is none, is TypeScript. */ + typescript: boolean; +}; + +const isTypeScriptFile = (path: string) => /\.[cm]?tsx?$/.test(path); + +async function resolvePreview(configDir = '.storybook') { + const previewPath = findConfigFile('preview', configDir); + const typescript = isTypeScriptFile(previewPath ?? findConfigFile('main', configDir) ?? ''); + const path = previewPath ?? `${configDir}/preview.${typescript ? 'ts' : 'js'}`; + let csfFactories = false; + if (previewPath) { + try { + csfFactories = isCsfFactoryPreview(await readConfig(previewPath)); + } catch {} + } + return { + csfFactories, + typescript, + previewFile: relative(process.cwd(), path).replaceAll('\\', '/') || path, + }; +} /** * The one probing path for skill-content assembly: everything the pure builders need, resolved @@ -12,12 +44,13 @@ export type SkillInputs = ToolAvailability & { framework: string; renderer?: str * this, so the two channels cannot drift. */ export async function resolveSkillInputs(options: Options): Promise { - const [availability, frameworkPreset] = await Promise.all([ + const [availability, frameworkPreset, preview] = await Promise.all([ getToolAvailability(options), options.presets.apply('framework'), + resolvePreview(options.configDir), ]); const framework = extractFrameworkPackageName( typeof frameworkPreset === 'string' ? frameworkPreset : (frameworkPreset?.name ?? '') ); - return { ...availability, framework, renderer: frameworkToRendererMap[framework] }; + return { ...availability, ...preview, framework, renderer: frameworkToRendererMap[framework] }; } diff --git a/code/core/src/cli/skills/run.test.ts b/code/core/src/cli/skills/run.test.ts index 36b512465d54..602ecb363cf4 100644 --- a/code/core/src/cli/skills/run.test.ts +++ b/code/core/src/cli/skills/run.test.ts @@ -29,6 +29,9 @@ const deps = () => ({ resolveSkillInputs: vi.fn().mockResolvedValue({ framework: '@storybook/react-vite', renderer: '@storybook/react', + csfFactories: false, + previewFile: '.storybook/preview.ts', + typescript: true, moduleGraphSupported: true, docsEnabled: false, docsEnabledForCli: false, @@ -108,7 +111,7 @@ describe('runSkillsCommand', () => { expect(d.loadStorybook).not.toHaveBeenCalled(); }); - it('stories assembles CLI-transport server instructions using the CLI review gate', async () => { + it('stories prints the workflow with CLI commands', async () => { const d = deps(); const result = await runSkillsCommand({ tokens: ['stories'], target: {} }, d); expect(result.exitCode).toBe(0); @@ -125,7 +128,7 @@ describe('runSkillsCommand', () => { }); const stories = await runSkillsCommand({ tokens: ['stories'], target: {} }, d); - expect(stories.output).toContain('Documentation Workflow'); + expect(stories.output).toContain('npx storybook tools docs show --id '); const writeStory = await runSkillsCommand({ tokens: ['write-story'], target: {} }, d); expect(writeStory.output).toContain('npx storybook tools docs list'); @@ -134,7 +137,7 @@ describe('runSkillsCommand', () => { it('omits the docs workflow when the CLI docs gate is off', async () => { const d = deps(); const stories = await runSkillsCommand({ tokens: ['stories'], target: {} }, d); - expect(stories.output).not.toContain('Documentation Workflow'); + expect(stories.output).not.toContain('npx storybook tools docs'); }); it('write-story assembles CLI-transport story instructions', async () => { @@ -145,25 +148,27 @@ describe('runSkillsCommand', () => { expect(result.output).toContain('npx storybook tools stories changed'); }); - it('stories carries the write-story text and ends with a reference of the tools it names', async () => { + it('stories teaches each command inline, without a command reference', async () => { const d = deps(); + d.resolveSkillInputs.mockResolvedValue({ + ...(await d.resolveSkillInputs()), + docsEnabledForCli: true, + a11yEnabled: true, + }); const stories = await runSkillsCommand({ tokens: ['stories'], target: {} }, d); - const writeStory = await runSkillsCommand({ tokens: ['write-story'], target: {} }, d); - expect(stories.output).toContain(writeStory.output.split('# Command reference')[0]); - expect(describedTools(stories.output)).toEqual([ - 'stories preview', - 'stories changed', - 'stories find-by-component', - 'review create', - 'test run', - ]); + expect(describedTools(stories.output)).toEqual([]); + expect(stories.output).toContain("npx storybook tools test run --stories '[{"); + expect(stories.output).toContain("npx storybook tools review create --input '{"); + expect(stories.output.length).toBeLessThan(10000); }); - it('--all prints the write-story text once', async () => { + it('--all prints each skill once', async () => { const result = await runSkillsCommand({ tokens: [], all: true, target: {} }, deps()); + expect(result.output.split('# Storybook workflow')).toHaveLength(2); expect(result.output.split('# Writing User Interfaces')).toHaveLength(2); + expect(result.output.split('# Storybook Setup')).toHaveLength(2); }); it('setup emits the setup markdown from the lightweight probe, without loading config', async () => { diff --git a/code/core/src/cli/skills/run.ts b/code/core/src/cli/skills/run.ts index ef6aff02338f..7efce816507d 100644 --- a/code/core/src/cli/skills/run.ts +++ b/code/core/src/cli/skills/run.ts @@ -5,7 +5,7 @@ import type { Options } from '../../types/index.ts'; import { resolveStorybookConfigDir } from '../tools/config-dir.ts'; import type { ToolsetCatalogEntry } from '../tools/sdk/types.ts'; import { renderCommandReference } from './command-reference.ts'; -import { buildServerInstructions } from './content/build-server-instructions.ts'; +import { buildStoriesSkill } from './content/build-stories-skill.ts'; import { buildStoryInstructions } from './content/build-story-instructions.ts'; import type { getSetupMarkdownOutput } from './content/setup-prompts/index.ts'; import { SKILLS, SKILL_IDS, isSkillId, type SkillId } from './content/skills.ts'; @@ -107,9 +107,7 @@ export async function runSkillsCommand( return { output: '', errorOutput: intent.message, exitCode: 1 }; } try { - // `stories` carries the `write-story` text, so `--all` does not print it a second time. - const ids = - intent.kind === 'all' ? SKILL_IDS.filter((id) => id !== 'write-story') : [intent.id]; + const ids = intent.kind === 'all' ? SKILL_IDS : [intent.id]; const { docs, setupRun } = await serveSkills( ids, resolveStorybookConfigDir(input.target), @@ -217,21 +215,25 @@ function withCommandReference( toolsets: ToolsetCatalogEntry[] ): string { const text = assemble(id, inputs); + if (id === 'stories') { + return text; + } const reference = renderCommandReference(text, toolsets); return reference ? `${text.trimEnd()}\n\n${reference}` : text; } function assemble(id: Exclude, inputs: SkillInputs): string { if (id === 'stories') { - const workflow = buildServerInstructions({ - transport: 'cli', - devEnabled: true, - testSupported: inputs.testSupported, + return buildStoriesSkill({ + framework: inputs.framework, + csfFactories: inputs.csfFactories, + previewFile: inputs.previewFile, + typescript: inputs.typescript, docsEnabled: inputs.docsEnabledForCli, + testSupported: inputs.testSupported, + a11yEnabled: inputs.a11yEnabled, moduleGraphSupported: inputs.moduleGraphSupported, - storyInstructionsInline: true, }); - return `${workflow}\n\n${assemble('write-story', inputs)}`; } return buildStoryInstructions({ transport: 'cli', diff --git a/code/lib/claude-plugin/skills/stories/SKILL.md b/code/lib/claude-plugin/skills/stories/SKILL.md index e9b979035094..0c8177896a3c 100644 --- a/code/lib/claude-plugin/skills/stories/SKILL.md +++ b/code/lib/claude-plugin/skills/stories/SKILL.md @@ -1,6 +1,6 @@ --- name: stories -description: Invoke FIRST, before creating, editing, or deleting components, stories, styles, CSS, themes, colors, or design tokens — anything that changes how the UI looks, no exceptions. Also for starting or previewing Storybook to verify UI, requests to show, browse, or list components, stories, or UI states, and docs, props, or usage lookups. +description: Invoke FIRST, before creating, editing, or deleting components, stories, styles, CSS, themes, colors, or design tokens — anything that changes how the UI looks, no exceptions. Also before running story tests, answering a question about a component's props, API, or usage, starting Storybook, or showing or listing components, stories, or UI states. --- Prerequisites: diff --git a/code/lib/claude-plugin/skills/storybook-init/SKILL.md b/code/lib/claude-plugin/skills/storybook-init/SKILL.md index 3e270a63232a..f07b7ba88ee6 100644 --- a/code/lib/claude-plugin/skills/storybook-init/SKILL.md +++ b/code/lib/claude-plugin/skills/storybook-init/SKILL.md @@ -3,5 +3,7 @@ name: storybook-init description: Use when adding Storybook to a project that does not have Storybook configured yet. --- +If Storybook is already installed (`package.json`, `.storybook/`), do not install it again: switch to `/storybook-setup`. + 1. Run `npm create storybook@latest` inside your project's root directory to install the latest version of Storybook (if the latest stable release is below 11.0, run `npm create storybook@next` instead to get the 11.0 alpha). Use the matching package-manager command when appropriate, such as `pnpm create storybook@latest` or `yarn create storybook`. 2. Invoke the `/storybook-setup` skill to help the user set up project-specific Storybook configuration, such as the `.storybook/preview.ts` file. diff --git a/code/lib/claude-plugin/skills/storybook-setup/SKILL.md b/code/lib/claude-plugin/skills/storybook-setup/SKILL.md index 97f24d738cc9..6a5d791d0afb 100644 --- a/code/lib/claude-plugin/skills/storybook-setup/SKILL.md +++ b/code/lib/claude-plugin/skills/storybook-setup/SKILL.md @@ -6,7 +6,7 @@ description: Use this skill when Storybook is already installed and the user wan Prerequisites: 1. Confirm Storybook exists (`package.json`, `.storybook/`). If not, switch to `/storybook-init`. -2. Storybook must be 11.0 or later. An 11.0 prerelease (`11.0.0-alpha.x`) or a canary build (`0.0.0-pr-*`) also counts. If it is older, or upgrade/repair is needed first, switch to `/storybook-upgrade`, but only if the user explicitly approved a Storybook upgrade. +2. Storybook must be 11.0 or later. An 11.0 prerelease (`11.0.0-alpha.x`) or a canary build (`0.0.0-pr-*`) also counts. If it is older, or upgrade/repair is needed first, switch to `/storybook-upgrade` and come back here afterwards. Treat a request to set up or install Storybook as approval to perform any required Storybook upgrade. For other requests, switch only after the user explicitly approves an upgrade. Run `npx storybook skills setup` from the project root (or the Storybook package in a monorepo). diff --git a/code/lib/codex-plugin/plugins/storybook/skills/init/SKILL.md b/code/lib/codex-plugin/plugins/storybook/skills/init/SKILL.md index 6b06b1749c91..3407e293db0f 100644 --- a/code/lib/codex-plugin/plugins/storybook/skills/init/SKILL.md +++ b/code/lib/codex-plugin/plugins/storybook/skills/init/SKILL.md @@ -3,5 +3,7 @@ name: init description: Use when adding Storybook to a project that does not have Storybook configured yet. --- +If Storybook is already installed (`package.json`, `.storybook/`), do not install it again: switch to `$storybook:setup`. + 1. Run `npm create storybook@latest` inside your project's root directory to install the latest version of Storybook (if the latest stable release is below 11.0, run `npm create storybook@next` instead to get the 11.0 alpha). Use the matching package-manager command when appropriate, such as `pnpm create storybook@latest` or `yarn create storybook`. 2. Invoke the `$storybook:setup` skill to help the user set up project-specific Storybook configuration, such as the `.storybook/preview.ts` file. diff --git a/code/lib/codex-plugin/plugins/storybook/skills/setup/SKILL.md b/code/lib/codex-plugin/plugins/storybook/skills/setup/SKILL.md index 7197e5ee2a4b..54247ea6772d 100644 --- a/code/lib/codex-plugin/plugins/storybook/skills/setup/SKILL.md +++ b/code/lib/codex-plugin/plugins/storybook/skills/setup/SKILL.md @@ -6,7 +6,7 @@ description: Use this skill when Storybook is already installed and the user wan Prerequisites: 1. Confirm Storybook exists (`package.json`, `.storybook/`). If not, switch to `$storybook:init`. -2. Storybook must be 11.0 or later. An 11.0 prerelease (`11.0.0-alpha.x`) or a canary build (`0.0.0-pr-*`) also counts. If it is older, or upgrade/repair is needed first, switch to `$storybook:upgrade`, but only if the user explicitly approved a Storybook upgrade. +2. Storybook must be 11.0 or later. An 11.0 prerelease (`11.0.0-alpha.x`) or a canary build (`0.0.0-pr-*`) also counts. If it is older, or upgrade/repair is needed first, switch to `$storybook:upgrade` and come back here afterwards. Treat a request to set up or install Storybook as approval to perform any required Storybook upgrade. For other requests, switch only after the user explicitly approves an upgrade. Run `npx storybook skills setup` from the project root (or the Storybook package in a monorepo). diff --git a/code/lib/codex-plugin/plugins/storybook/skills/stories/SKILL.md b/code/lib/codex-plugin/plugins/storybook/skills/stories/SKILL.md index a46b5c0e9b47..17e2c7276605 100644 --- a/code/lib/codex-plugin/plugins/storybook/skills/stories/SKILL.md +++ b/code/lib/codex-plugin/plugins/storybook/skills/stories/SKILL.md @@ -1,6 +1,6 @@ --- name: stories -description: Invoke FIRST, before creating, editing, or deleting components, stories, styles, CSS, themes, colors, or design tokens — anything that changes how the UI looks, no exceptions. Also for starting or previewing Storybook to verify UI, requests to show, browse, or list components, stories, or UI states, and docs, props, or usage lookups. +description: Invoke FIRST, before creating, editing, or deleting components, stories, styles, CSS, themes, colors, or design tokens — anything that changes how the UI looks, no exceptions. Also before running story tests, answering a question about a component's props, API, or usage, starting Storybook, or showing or listing components, stories, or UI states. --- Prerequisites: