Bench: Add the docgen perf engine framework and the compodoc engine by valentinpalkovic · Pull Request #35636 · storybookjs/storybook

valentinpalkovic · 2026-07-28T21:05:03Z

What I did

This is the base of the new per-engine docgen performance suite, plus its first engine.

The framework: metric types, the measurement profiles, the aggregation that turns N repetitions into medians, the child-process spawn helper, and the BenchEngine base class every engine implements.

The engine: Compodoc, which is what Storybook shells out to for Angular docgen. It runs as a fresh CLI process, so cold extraction and whole-project scan are the same measurement, and peak memory is sampled from outside the child.

Heads-up: there is no CLI entry point yet. registry.ts, cli.ts and run.ts all land later, so you cannot run this end to end here.

Checklist for Contributors

Testing

The changes in this PR are covered in the following automated tests:

stories
unit tests
integration tests
end-to-end tests

aggregate.test.ts and compodoc-doc.test.ts, on top of part 2's tests — 4 files, 43 tests.

Manual testing

yarn install          # pulls in @compodoc/compodoc
yarn dedupe --check
cd scripts && yarn check && yarn lint && yarn test bench/docgen

Expect 4 test files and 43 tests passing.
compodoc-doc.test.ts counts against the real Compodoc captures already committed in code/lib/docgen-harness, so it checks the counter against output the tool actually produced, not against a hand-written fixture.

There is no runnable benchmark at this point in the stack — that first becomes possible in part 4.

Documentation

Add or update documentation reflecting your changes

Checklist for Maintainers

When this PR is ready for testing, make sure to add ci:normal, ci:merged or ci:daily GH label to it to run a specific set of sandboxes. The particular set of sandboxes can be found in code/lib/cli-storybook/src/sandbox-templates.ts
Declare whether manual QA will be needed for this PR during the next release, through qa:needed or qa:skip
Make sure this PR contains one of the labels below: build

Summary by CodeRabbit

New Features
- Added a benchmark suite for comparing documentation-generation performance, latency, memory usage, and output quality.
- Added support for benchmarking Compodoc, including cold and warm runs, member counts, and unresolved-type reporting.
- Added deterministic Angular project generation for repeatable benchmark scenarios.
- Added standardized benchmark metrics, scenario results, engine status, and comparison reporting.
Tests
- Added coverage for metric aggregation, repetition selection, validation, documentation counting, and edge cases.

github-actions · 2026-07-28T21:05:40Z

	Warnings
⚠️	This PR targets `valentin/bench-2-shared`. The default branch for contributions is `next`. Please make sure you are targeting the correct branch.

Generated by 🚫 dangerJS against 3892cd0

coderabbitai · 2026-07-29T09:21:11Z

📝 Walkthrough

Walkthrough

The change introduces a typed docgen performance suite with configurable repetition profiles, reusable benchmark engines, synthetic Angular project generation, Compodoc execution and documentation metrics, strict child-process handling, aggregation utilities, tests, and the Compodoc dependency.

Changes

Compodoc benchmark suite

Layer / File(s)	Summary
Metrics contracts and aggregation `scripts/bench/docgen-perf/types.ts`, `scripts/bench/docgen-perf/config.ts`, `scripts/bench/docgen-perf/aggregate.ts`, `scripts/bench/docgen-shared/engine-ids.ts`, `scripts/bench/docgen-perf/aggregate.test.ts`	Defines benchmark result types, fixed profiles, the `compodoc` engine identifier, repetition aggregation, and validation tests.
Engine orchestration and child execution `scripts/bench/docgen-perf/engine.ts`, `scripts/bench/docgen-perf/spawn.ts`	Adds reusable engine lifecycle hooks, series-child configuration, child process execution, and strict JSON result validation.
Synthetic Angular project generation `scripts/bench/docgen-perf/generators/angular.ts`	Generates deterministic Angular component projects, local Angular type declarations, compiler configuration, and CLI options.
Compodoc measurement and documentation metrics `scripts/bench/docgen-perf/engines/compodoc.ts`, `scripts/bench/docgen-perf/engines/compodoc-doc.ts`, `scripts/bench/docgen-perf/engines/compodoc-doc.test.ts`, `scripts/package.json`	Runs cold and warm Compodoc measurements, samples RSS, counts documented and opaque members, validates fixture cases, and adds the Compodoc package.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CompodocEngine
  participant AngularGenerator
  participant CompodocCLI
  participant DocumentationCounter

  CompodocEngine->>AngularGenerator: generateAngularProject(scenario)
  CompodocEngine->>CompodocCLI: run cold documentation generation
  CompodocCLI-->>CompodocEngine: documentation.json and RSS metrics
  CompodocEngine->>CompodocEngine: modify one component and run warm measurement
  CompodocEngine->>DocumentationCounter: countDocumentation(documentation.json)
  DocumentationCounter-->>CompodocEngine: members and opaqueTypes

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/bench/docgen-perf/engines/compodoc.ts`:
- Around line 57-70: Update the RSS polling logic around the setInterval
callback and peakRssKb to track whether any valid RSS sample was collected and
whether polling errors persist. Before reporting the benchmark result, detect an
all-error or all-zero measurement and surface it as unavailable or unreliable
instead of returning peakRssMb: 0 as a valid metric, while preserving normal
reporting when valid samples exist.
- Around line 40-94: Update runCompodocOnce to enforce a timeout for the spawned
Compodoc child: terminate the process when the limit is exceeded, clear the
polling interval and timeout, and reject the promise with a descriptive timeout
error. Ensure normal close and spawn-error paths clean up the timeout without
changing their existing results.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: bba9c5d6-6a5a-4516-a09f-4d7f94da38a7

📥 Commits

Reviewing files that changed from the base of the PR and between 4663f29 and fd4e8c8.

⛔ Files ignored due to path filters (1)

yarn.lock is excluded by !**/yarn.lock, !**/*.lock

📒 Files selected for processing (12)

scripts/bench/docgen-perf/aggregate.test.ts
scripts/bench/docgen-perf/aggregate.ts
scripts/bench/docgen-perf/config.ts
scripts/bench/docgen-perf/engine.ts
scripts/bench/docgen-perf/engines/compodoc-doc.test.ts
scripts/bench/docgen-perf/engines/compodoc-doc.ts
scripts/bench/docgen-perf/engines/compodoc.ts
scripts/bench/docgen-perf/generators/angular.ts
scripts/bench/docgen-perf/spawn.ts
scripts/bench/docgen-perf/types.ts
scripts/bench/docgen-shared/engine-ids.ts
scripts/package.json

The orchestrator settles on the child's close event, so a compodoc that hangs - which it can, on some inputs - stalled the whole suite with no way out. The run is now killed after ten minutes and reported as a timeout, so the engine fails on its own rather than taking the suite with it. Reported by CodeRabbit on #35636.

storybook-app-bot · 2026-07-31T09:41:21Z

Package Benchmarks

^{Commit: 3892cd0, ran on 31 July 2026 at 14:12:24 UTC}

No significant changes detected, all good. 👏

This is the base of the per-engine perf suite: the metric types, the fixed measurement profiles, the aggregation that turns repetitions into medians, the child-process spawn helper, and the BenchEngine base class every engine implements. It comes with the first engine, compodoc, which measures Angular. Compodoc runs as a fresh CLI process, so cold extraction and whole-project scan are the same measurement and peak memory is sampled from outside the child. The engine also reads the documentation.json compodoc writes, and reports two counts: how many members it documented, and how many of those carry a type it never resolved. The second count matters because compodoc never expands a named type. Chain a type through twenty files and it documents every member at the same speed, having looked through none of them, so a member count alone would read as identical to an engine that expanded all of them. @compodoc/compodoc is pinned at exactly 2.0.0, the version the Angular docgen baselines capture against. Its cost moves between versions. The profile and the engine id list carry Angular only. React and Vue arrive with their engines in the next PR. There is no CLI entry point yet. registry.ts, cli.ts and run.ts land in the PR stacked on top, so this cannot be run end to end; the two test files cover the aggregation and the documentation.json parsing.

Collapse coldMembers/warmMembers/coldOpaqueTypes into one members() hook returning MemberCounts, and give BenchEngine an assemble() that turns a scenario's repetitions into its ScenarioResult. Both engines implemented the three count hooks as pass-through one-liners, and the orchestrator had to call all three plus aggregate() and pick the designated repetition itself - which meant naming a sample shape it cannot know, so the generic parameter was erased and re-asserted with a cast. assemble() keeps Sample bound inside the class, so the cast has nowhere to live, and picking the designated repetition happens in one place instead of two. Sample is now constrained to { coldMs: number }, which is what designatedRep has always required and what the erased generic hid. Pin N at 6 rather than 5. The two sides of a control pair alternate which runs first by repetition parity, so an odd N hands one side the first slot once more than the other, and the cold median then lands on a repetition from the majority slot - turning the ordering effect the alternation exists to cancel into a systematic bias on the headline ratio. Report compodoc's peak RSS as n/a when nothing was sampled. `ps` is POSIX-only and a run can finish inside one poll interval, and the initial 0 was reported as a measured peak on either path. oneShotMetrics now requires every repetition to carry a reading before it reports a mean. Also: read the compodoc RSS poll interval from its constant rather than back out of the params bag, and say in the "retained metrics missing" error that a series shorter than two saves reaches it too.

The orchestrator settles on the child's close event, so a compodoc that hangs - which it can, on some inputs - stalled the whole suite with no way out. The run is now killed after ten minutes and reported as a timeout, so the engine fails on its own rather than taking the suite with it. Reported by CodeRabbit on #35636.

The generator was a sequence of writes, so the only way to see what compodoc is pointed at was to run it and look at a directory. angularProjectFiles returns the project as path -> contents instead, and generateAngularProject writes that map. The bookkeeping of which paths are components stops being tracked separately from the writes that create them. The new test snapshots that map: the tree, the fake @angular/core that keeps it hermetic, the tsconfig include that decides the measured set, and a component at zero and four extra props. Generated output is byte-identical.

The orchestrator settles on the child's close event, so a compodoc that hangs - which it can, on some inputs - stalled the whole suite with no way out. The run is now killed after ten minutes and reported as a timeout, so the engine fails on its own rather than taking the suite with it. Reported by CodeRabbit on #35636.

The orchestrator settles on the child's close event, so a compodoc that hangs - which it can, on some inputs - stalled the whole suite with no way out. The run is now killed after ten minutes and reported as a timeout, so the engine fails on its own rather than taking the suite with it. Reported by CodeRabbit on storybookjs#35636.

valentinpalkovic added the build Internal-facing build tooling & test updates label Jul 28, 2026

valentinpalkovic mentioned this pull request Jul 28, 2026

Bench: Add the docgen perf methodology note, the per-engine harness and the shared bench plumbing #35573

Closed

5 of 9 tasks

valentinpalkovic force-pushed the valentin/bench-4-framework branch from 2cf56c8 to 25afc98 Compare July 28, 2026 21:18

valentinpalkovic force-pushed the valentin/bench-4-framework branch 4 times, most recently from 4a2eeb4 to fd4e8c8 Compare July 29, 2026 09:03

valentinpalkovic self-assigned this Jul 29, 2026

valentinpalkovic added the ci:normal Run our default set of CI jobs (choose this for most PRs). label Jul 29, 2026

valentinpalkovic marked this pull request as ready for review July 29, 2026 09:08

valentinpalkovic requested a review from a team July 29, 2026 09:08

coderabbitai Bot reviewed Jul 29, 2026

View reviewed changes

Comment thread scripts/bench/docgen-perf/engines/compodoc.ts

Comment thread scripts/bench/docgen-perf/engines/compodoc.ts

valentinpalkovic mentioned this pull request Jul 29, 2026

Bench: Make the docgen perf suite runnable #35634

Merged

2 of 8 tasks

valentinpalkovic force-pushed the valentin/bench-4-framework branch from fd4e8c8 to dc0af52 Compare July 30, 2026 19:38

valentinpalkovic added the qa:skip Pull Requests that do not need any QA. (e.g. documentation) label Jul 30, 2026

coderabbitai Bot mentioned this pull request Jul 30, 2026

Bench: Compare an engine against another release of itself #35675

Merged

3 of 8 tasks

valentinpalkovic force-pushed the valentin/bench-4-framework branch from dc0af52 to c11bb76 Compare July 30, 2026 19:47

coderabbitai Bot mentioned this pull request Jul 30, 2026

Bench: Add the React and Vue docgen perf engines #35651

Merged

2 of 8 tasks

valentinpalkovic force-pushed the valentin/bench-4-framework branch from c11bb76 to 1322ce3 Compare July 30, 2026 19:58

valentinpalkovic added ci:normal Run our default set of CI jobs (choose this for most PRs). and removed ci:normal Run our default set of CI jobs (choose this for most PRs). labels Jul 31, 2026

valentinpalkovic requested a review from huang-julien July 31, 2026 08:58

huang-julien approved these changes Jul 31, 2026

View reviewed changes

valentinpalkovic force-pushed the valentin/bench-4-framework branch from f5cb06a to 6992914 Compare July 31, 2026 13:39

valentinpalkovic mentioned this pull request Jul 31, 2026

Bench: Add the shared docgen bench plumbing and move the memory harness onto it #35631

Merged

2 of 8 tasks

valentinpalkovic and others added 4 commits July 31, 2026 15:55

valentinpalkovic force-pushed the valentin/bench-4-framework branch from 6992914 to 3892cd0 Compare July 31, 2026 13:59

valentinpalkovic merged commit af289bb into next Jul 31, 2026
136 checks passed

valentinpalkovic deleted the valentin/bench-4-framework branch July 31, 2026 17:26

coderabbitai Bot mentioned this pull request Aug 3, 2026

Angular: Record two compodoc signal gaps as harness fixtures #35628

Closed

3 of 8 tasks

coderabbitai Bot mentioned this pull request Aug 8, 2026

Angular: Add an in-process docgen analyzer, replacing Compodoc under the flag #35805

Merged

4 of 9 tasks

Navigation Menu

Repository navigation

Bench: Add the docgen perf engine framework and the compodoc engine - #35636

Bench: Add the docgen perf engine framework and the compodoc engine#35636
valentinpalkovic merged 4 commits into
valentin/bench-2-sharedfrom
valentin/bench-4-framework

valentinpalkovic commented Jul 28, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

github-actions Bot commented Jul 28, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jul 29, 2026

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

storybook-app-bot Bot commented Jul 31, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

valentinpalkovic commented Jul 28, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What I did

Checklist for Contributors

Testing

The changes in this PR are covered in the following automated tests:

Manual testing

Documentation

Checklist for Maintainers

Summary by CodeRabbit

Uh oh!

github-actions Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Jul 29, 2026

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

storybook-app-bot Bot commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Package Benchmarks

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

valentinpalkovic commented Jul 28, 2026 •

edited by coderabbitai Bot

Loading

github-actions Bot commented Jul 28, 2026 •

edited

Loading

storybook-app-bot Bot commented Jul 31, 2026 •

edited

Loading