[ndelangen]

Closes #35260
Related: #34824

What I did

Fixes an out-of-memory crash in the new docgen-server mode (STORYBOOK_EXPERIMENTAL_DOCGEN_SERVER=true).
On a large Storybook, editing a widely-imported component (e.g. Button.tsx) can crash the dev server with:

Reached heap limit Allocation failed - JavaScript heap out of memory

It's not a leak

I first checked whether this was a memory leak. It isn't: post-GC (garbage collection) retained memory stays flat across saves. What's actually happening is a spike problem, not a growth problem.

• All docgen extraction runs through one shared TypeScript program. As you edit and save files, this program's type-resolution cache grows and shrinks in a sawtooth pattern (normal behavior), and it comes back down on its own (due to GC).
• Every time the dev server extracts docgen info from that program (which happens per edit or per request), it adds its own memory spike on top.
• On a large project, the program's normal working set already sits close to the memory ceiling (in our internal Storybook, ~3.3–3.9 GB against a 4096 MB default cap).
  So, during a file save, the TS server is allocating memory, as well as the part trying to extract; there isn't enough headroom left, and the process runs out of memory.

This is why the crash felt random: it's not that memory keeps climbing, it's that two independent sawtooths occasionally stack high enough to blow the ceiling.

The fix

We need to create more memory headroom by running the GC.

I added a mechanism that lets the shared TypeScript program drop its internal cache once extract is complete, freeing up the memory it was holding. Concretely: the docgen server now checks memory usage, and once it crosses 70% of the available heap, it disposes the cached program (and lazily rebuilds it on the next request). This is on by default.

A couple of things worth calling out about the approach:

• Forcing garbage collection was not an option here; GC can't reclaim memory that's still referenced. Disposing of the program first is what makes the memory collectible.
• It's triggered by actual heap pressure (a % of the real V8 heap limit at runtime), not a fixed memory threshold, or number of edits/saves. It is self-adjusting to whatever --max-old-space-size is set, and so for small projects it should never trigger.
• The extracted docs themselves (the actual output we need to keep) are stored separately and aren't affected by disposing of the program.

I verified that at a heap cap where the bug reproduces reliably (--max-old-space-size=3072), the server without this fix crashes by the 7th edit. With the fix, it survives 20 edits in a row, recycling the cache twice along the way and settling back down each time instead of climbing to the cap.

Trade-offs

• Each time the cache is recycled, the next docs panel load takes an extra ~1.8s (depending on the project size) to rebuild. This is the same cold-start cost the server already pays once at boot.
• If a project's normal working set sits right at the 70% threshold, it could recycle more often than ideal, reducing performance. But that's better than crashing with an OOM error.
• This bounds memory between extractions — if a single extraction call itself is bigger than the heap cap (a much larger Storybook than we tested against), this fix won't save it.

Testing

Since the original bug report was on our internal Storybook, I didn't want to test against that directly, since it changes over time as stories/code/features are added/removed. A fix verified against it today could look "fixed" or "broken" later for unrelated reasons.
Instead, I built a synthetic generator that produces a large, TypeScript-heavy fixture project (many components + stories, no real UI) specifically for reproducing this memory profile on demand.

I added this as a daily CI check (yarn bench:docgen-memory) that asserts two things:

1. With the fix disabled, the server OOMs at a given heap cap; with it enabled, it survives. Asserting both directions (not just "it survives") ensures the check actually exercises the fix.
2. Steady-state memory stays flat, and per-save memory use stays under budget, to catch any regression that reintroduces the original problem.

To run it locally:

cd scripts && yarn bench:docgen-memory

---

Checklist for Contributors

Testing

The changes in this PR are covered in the following automated tests:

• stories
• unit tests
• integration tests
• end-to-end tests

A daily CI job (yarn bench:docgen-memory) runs a synthetic in-process gate asserting:

1. Crash → survive negative control (live mode, cap 1536 MB): recycle off must OOM, recycle
   on must survive. Asserting the flip — not just survival — guards the fix itself.
2. Steady-state churn/leak-freeness: post-GC retained heap stays flat and per-save transient stays
   under budget — catches a regression that reintroduces whole-index re-extraction.

Manual testing

1. Compile: yarn nx compile react --skip-nx-cache
2. Run the synthetic gate (~32 s, deterministic):

   cd scripts && yarn bench:docgen-memory

3. Optionally verify the flip by hand:

   # Must OOM:
   NODE_OPTIONS=--max_old_space_size=1536 node --expose-gc --import jiti/register \
     scripts/bench/docgen-memory/memory-harness.ts \
     --components 800 --props 10 --saves 1 --mode live --heavy --no-force-gc --recycle off
   # Must survive:
   NODE_OPTIONS=--max_old_space_size=1536 node --expose-gc --import jiti/register \
     scripts/bench/docgen-memory/memory-harness.ts \
     --components 800 --props 10 --saves 1 --mode live --heavy --no-force-gc --recycle on

Real-environment proof (not in repo — ad hoc script to reproduce against the real internal Storybook)

Boots code/ in docgen-server mode, edits Button.tsx in a loop, samples process-tree RSS.
Not committed — tied to internal Storybook structure — but preserved here for reference.

// Save as dev-server-repro.ts and run from repo root:
//   node --import jiti/register dev-server-repro.ts --edits 20 --cap 3072
import { type ChildProcess, spawn, spawnSync } from 'node:child_process';
import * as fs from 'node:fs';
import * as path from 'node:path';

const CODE_DIRECTORY = path.resolve('code');
const SANDBOX_DIRECTORY = path.resolve('../storybook-sandboxes');

interface Options { edits: number; capMb: number; port: number; editIntervalMs: number; targetFile: string; }

function parseArgs(argv: string[]): Options {
  const get = (flag: string, fallback: string) => { const idx = argv.indexOf(flag); return idx >= 0 && argv[idx + 1] ? argv[idx + 1] : fallback; };
  return { edits: Number(get('--edits', '30')), capMb: Number(get('--cap', '4096')), port: Number(get('--port', '6006')), editIntervalMs: Number(get('--interval', '2500')), targetFile: get('--target', path.join(CODE_DIRECTORY, 'core/src/components/components/Button/Button.tsx')) };
}

const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));

function processTreeRssMb(rootPid: number): number {
  const out = spawnSync('ps', ['-A', '-o', 'pid=,ppid=,rss='], { encoding: 'utf8' }).stdout ?? '';
  const children = new Map<number, number[]>(); const rssByPid = new Map<number, number>();
  for (const line of out.trim().split('\n')) { const [pid, ppid, rss] = line.trim().split(/\s+/).map(Number); if (!Number.isFinite(pid)) continue; rssByPid.set(pid, rss); const s = children.get(ppid) ?? []; s.push(pid); children.set(ppid, s); }
  let totalKb = 0; const stack = [rootPid]; const seen = new Set<number>();
  while (stack.length) { const pid = stack.pop()!; if (seen.has(pid)) continue; seen.add(pid); totalKb += rssByPid.get(pid) ?? 0; for (const c of children.get(pid) ?? []) stack.push(c); }
  return totalKb / 1024;
}

async function waitForReady(port: number, timeoutMs: number): Promise<boolean> {
  const deadline = Date.now() + timeoutMs;
  while (Date.now() < deadline) { try { const res = await fetch(`http://localhost:${port}/iframe.html`); if (res.ok || res.status === 404) return true; } catch {} await sleep(1000); }
  return false;
}

async function main() {
  const options = parseArgs(process.argv.slice(2));
  console.log(`cap=${options.capMb}MB edits=${options.edits}`);
  const originalSource = fs.readFileSync(options.targetFile, 'utf8');
  let crashed = false;
  const server: ChildProcess = spawn(process.execPath, ['core/dist/bin/dispatcher.js', 'dev', '--port', String(options.port), '--config-dir', './.storybook'], { cwd: CODE_DIRECTORY, env: { ...process.env, STORYBOOK_EXPERIMENTAL_DOCGEN_SERVER: 'true', STORYBOOK_DISABLE_TELEMETRY: 'true', NODE_OPTIONS: `--max_old_space_size=${options.capMb}`, BROWSER: 'none' }, stdio: ['ignore', 'pipe', 'pipe'] });
  const logPath = path.join(SANDBOX_DIRECTORY, 'docgen-dev-repro.log');
  const logStream = fs.createWriteStream(logPath); server.stdout?.pipe(logStream); server.stderr?.pipe(logStream);
  let lastErr = ''; server.stderr?.on('data', (c) => { lastErr = String(c); });
  server.on('exit', (code) => { crashed = true; console.log(`\n*** server exited (code=${code}) ***`); if (lastErr) console.log(lastErr.split('\n').slice(-6).join('\n')); });
  try {
    console.log('booting (~60s)…');
    if (!await waitForReady(options.port, 180_000) || crashed) throw new Error(`not ready (see ${logPath})`);
    await fetch(`http://localhost:${options.port}/manifests/components.html`).catch(() => {});
    const baselineRss = processTreeRssMb(server.pid!);
    console.log(`baseline: ${baselineRss.toFixed(0)}MB\n`);
    const samples: number[] = [];
    for (let edit = 1; edit <= options.edits && !crashed; edit++) {
      fs.appendFileSync(options.targetFile, `\n// probe-${edit}\n`);
      await sleep(options.editIntervalMs);
      if (crashed) { console.log(`edit ${edit}: CRASHED`); break; }
      const rss = processTreeRssMb(server.pid!); samples.push(rss);
      console.log(`edit ${String(edit).padStart(2)}: rss=${rss.toFixed(0).padStart(5)}MB`);
    }
    console.log(`\npeak: ${Math.max(...samples).toFixed(0)}MB  final: ${samples.at(-1)!.toFixed(0)}MB`);
    console.log(crashed ? 'result: OOM CRASH ✗' : 'result: survived ✓');
  } finally { fs.writeFileSync(options.targetFile, originalSource); if (!crashed) server.kill('SIGKILL'); logStream.end(); }
}

main().catch((err) => { console.error(err); process.exitCode = 1; });

Documentation

• Add or update documentation reflecting your changes
• If you are deprecating/removing a feature, make sure to update
  MIGRATION.MD

Checklist for Maintainers

• When this PR is ready for testing, make sure to add ci:normal, ci:merged or ci:daily GH label to it to run a specific set of sandboxes. The particular set of sandboxes can be found in code/lib/cli-storybook/src/sandbox-templates.ts

• Declare whether manual QA will be needed for this PR during the next release, through qa:needed or qa:skip

• Make sure this PR contains one of the labels below:

  Available labels

  • bug: Internal changes that fixes incorrect behavior.
  • maintenance: User-facing maintenance tasks.
  • dependencies: Upgrading (sometimes downgrading) dependencies.
  • build: Internal-facing build tooling & test updates. Will not show up in release changelog.
  • cleanup: Minor cleanup style change. Will not show up in release changelog.
  • documentation: Documentation only changes. Will not show up in release changelog.
  • feature request: Introducing a new feature.
  • BREAKING CHANGE: Changes that break compatibility in some way with current major version.
  • other: Changes that don't fit in the above categories.

🦋 Canary release

This PR does not have a canary release associated. You can request a canary release of this pull request by mentioning the @storybookjs/core team here.

core team members can create a canary release here or locally with gh workflow run --repo storybookjs/storybook publish.yml --field pr=<PR_NUMBER>

Summary by CodeRabbit

• New Features

  • Added a memory-aware safeguard for React docgen processing to reduce out-of-memory crashes on large projects.
  • Added a new CI benchmark and regression check for docgen memory behavior.

• Bug Fixes

  • Improved handling of heavy TypeScript projects by recycling program state when memory usage gets too high.
  • Added clearer warning messages when memory limits are being approached.

• Tests

  • Expanded coverage for program recycling behavior and memory-limit scenarios.

[storybook-app-bot]

Package Benchmarks

^{Commit: 994214b, ran on 1 July 2026 at 13:58:18 UTC}

No significant changes detected, all good. 👏

[ndelangen]

@JReinhold, what do you think the impact of the perf regression would be for the mcp-server?

[Copilot]

Pull request overview

This PR addresses Node.js OOM crashes in the experimental docgen-server workflow by adding heap-pressure-based recycling of the shared TypeScript program used for react-component-meta extraction, and by introducing a CI regression gate to ensure the crash→survive behavior remains stable over time.

Changes:

• Add heap-pressure-triggered disposal + lazy rebuild of ComponentMetaManager’s shared TS projects to reclaim TypeScript type-resolution caches under memory pressure.
• Add a deterministic benchmark harness + CI gate (bench:docgen-memory) to assert both steady-state behavior and a crash→survive negative control.
• Wire the new gate into the daily CircleCI workflow.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
scripts/package.json	Adds bench:docgen-memory script to run the new regression gate.
scripts/ci/main.ts	Includes the new docgen memory gate job in the daily workflow.
scripts/ci/common-jobs.ts	Defines the docgenMemoryGate CircleCI job to execute the gate.
scripts/bench/docgen-memory/memory-harness.ts	Implements an in-process harness to reproduce/measure docgen-server memory pressure deterministically.
scripts/bench/docgen-memory/generate-project.ts	Generates a Storybook-shaped React project used by the harness/gate.
scripts/bench/docgen-memory/gate.ts	Runs the harness in child processes and enforces regression assertions (metrics + crash→survive pair).
code/renderers/react/src/componentManifest/componentMeta/ComponentMetaManager.ts	Implements heap-pressure recycling of TS projects after extraction to prevent OOM in repeated extraction scenarios.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds heap-pressure-based recycling of TypeScript projects in ComponentMetaManager with a new memory-limit error class, plus new benchmarking scripts (project generator, memory harness, CI gate) and CircleCI wiring to run the gate on daily+ workflows.

Changes

Docgen memory fix and benchmark tooling

Layer / File(s)	Summary
Heap-pressure recycling in ComponentMetaManager code/renderers/react/src/componentManifest/componentMeta/ComponentMetaManager.ts, ComponentMetaManager.test.ts, code/core/src/server-errors.ts	Computes a heap threshold from v8.getHeapStatistics() and a configurable ratio, disposes cached TS projects and warns once when batchExtract detects heap pressure, adds ReactDocgenServerMemoryLimitError, and adds tests for recycle/no-recycle behavior and the warning message.
Synthetic project generator scripts/bench/docgen-memory/generate-project.ts	Generates a configurable React/TypeScript project with components, stories, optional heavy type blocks, base64 payloads, and minimal node_modules for benchmarking.
Memory harness scripts/bench/docgen-memory/memory-harness.ts	Loads ComponentMetaManager, drives refresh/live extraction cycles against the generated project, samples RSS/heap, and classifies retained-growth vs transient-pressure behavior with optional JSON output.
CI gate script and wiring scripts/bench/docgen-memory/gate.ts, scripts/package.json, scripts/ci/common-jobs.ts, scripts/ci/main.ts	Runs the harness as child processes against metric-budget and negative-control scenarios, adds the bench:docgen-memory script, and wires a new docgenMemoryGate CircleCI job into daily+ workflows.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
    participant CI as CircleCI (docgenMemoryGate)
    participant Gate as gate.ts
    participant Harness as memory-harness.ts
    participant Manager as ComponentMetaManager

    CI->>Gate: yarn bench:docgen-memory
    loop for each CONFIGS scenario
        Gate->>Harness: spawn child process (mode, recycle, heap cap)
        Harness->>Manager: batchExtract() across save/wave cycles
        Manager->>Manager: recycleProjectsIfHeapPressured()
        Manager-->>Harness: extraction results
        Harness->>Harness: sample RSS/heap, compute slopes
        Harness-->>Gate: exit code + result.json
    end
    Gate->>Gate: checkMetricConfig / checkNegativeControl
    Gate-->>CI: exit 0 (pass) or exit 1 (failures)

Loading

Possibly related issues

• Dev server OOM: react-component-meta re-extracts the entire story index on every file change (docgen-server path) #35260: Adds heap-pressure recycling and gate/tests targeting the docgen-server OOM-on-file-change problem described in this issue.

Possibly related PRs

• storybookjs/storybook#35324: Introduces a worker-thread docgen extraction flow that relies on ComponentMetaManager, directly affected by this PR's recycling changes to the same class.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/bench/docgen-memory/generate-project.ts`:
- Around line 15-17: The direct-run example in generate-project.ts points to the
wrong script name, so update the usage text to reference generate-project.ts
instead of generate.ts. Keep the example aligned with the actual entrypoint
shown in the file comment, and verify the command under the run-directly section
matches the script’s real filename.
- Around line 295-303: The self-exec guard in generateProject’s CLI entrypoint
is using a raw file URL string that breaks on Windows. Update the
import.meta.url comparison to build the process.argv[1] value with
pathToFileURL() so the direct-run check works cross-platform, and keep the logic
in the existing generateProject/parseArgs entrypoint block unchanged otherwise.

In `@scripts/bench/docgen-memory/memory-harness.ts`:
- Around line 333-346: The save-and-reextract loop in memory-harness.ts updates
component files on disk but never notifies ComponentMetaManager that files
changed. After fs.writeFileSync in the mutation path, call
manager.onFilesChanged() before manager.batchExtract(toExtract) so project
versions are bumped and refresh-mode measures the post-save state. Use the
existing manager and batchExtract flow as the insertion point.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: b202105c-b95e-4768-82be-acf55ffa9580

📥 Commits

Reviewing files that changed from the base of the PR and between 91b9c34 and e4903ff.

📒 Files selected for processing (7)

• code/renderers/react/src/componentManifest/componentMeta/ComponentMetaManager.ts
• scripts/bench/docgen-memory/gate.ts
• scripts/bench/docgen-memory/generate-project.ts
• scripts/bench/docgen-memory/memory-harness.ts
• scripts/ci/common-jobs.ts
• scripts/ci/main.ts
• scripts/package.json

[ndelangen]

@shilman, shall we add telemetry so we can know how often this happens? That would help prioritize, maybe investigating an alternate fix?