From 6934f8cad0c1e9dc7eba9cd67e7ac7d71e34e076 Mon Sep 17 00:00:00 2001 From: Valentin Palkovic Date: Thu, 23 Jul 2026 23:19:21 +0200 Subject: [PATCH 01/14] Bench: Add the docgen perf methodology note Fixes the metric set, the determinism method, the budget shape, and the CI tiers for the per-engine docgen perf suite, and adds the budgets table skeleton. The three user-perceived metrics join as report-only extensions of the sandbox bench task; the gated suite stays Node-only. --- scripts/bench/PERF-METHODOLOGY.md | 108 ++++++++++++++++++++++++++++++ 1 file changed, 108 insertions(+) create mode 100644 scripts/bench/PERF-METHODOLOGY.md diff --git a/scripts/bench/PERF-METHODOLOGY.md b/scripts/bench/PERF-METHODOLOGY.md new file mode 100644 index 000000000000..f6dc8e9d3092 --- /dev/null +++ b/scripts/bench/PERF-METHODOLOGY.md @@ -0,0 +1,108 @@ +# Docgen performance methodology + +This note is the measurement contract for the per-engine docgen performance suite. +It fixes the metric set, the determinism method, the budget shape, and the CI tiers that the harness and its budgets implement. +The suite measures; it never optimizes. + +## Scope + +The gated suite runs in plain Node: no browser, no dev server, no Vite. +Each engine is measured through a directly callable entry point: + +- React, new engine: `ComponentMetaManager.batchExtract` (`code/renderers/react/src/componentManifest/componentMeta/ComponentMetaManager.ts`) or `buildDocgenPayload` (`code/renderers/react/src/docgen/buildDocgen.ts`). +- React, legacy engines: `getReactDocgen` (`code/renderers/react/src/componentManifest/reactDocgen.ts`) and `parseWithReactDocgenTypescript` (`code/renderers/react/src/componentManifest/reactDocgenTypescript.ts`). + The live legacy path is a Vite transform hook and cannot run standalone; these wrappers share its extraction logic but are not byte-identical to it, so legacy numbers describe the engines, not the Vite pipeline. +- Angular: a standalone Compodoc CLI run. +- Vue, Svelte, and web components: their extraction engines are plain functions callable from Node; exact entry points get pinned when each engine's harness lands. + +## Metrics + +Five metrics per engine, all candidates for gating: + +- **Cold extraction**: time for the first full extraction in a fresh process. + For engines that build a TypeScript program, this includes building it. +- **Warm extraction**: time to re-extract one changed component after a simulated save, once the process is warm. +- **Whole-project scan**: time for one batch pass over the entire project. + Only Compodoc works this way, so this metric applies to Compodoc alone; per-component engines record it as n/a rather than a faked equivalent. +- **Peak memory (transient)**: the allocation spike a save causes above the retained baseline, sampled around forced GC. +- **Leak detection**: retained-heap growth after a long save series, plus the least-squares slope of retained heap over that series. + +The memory and leak half exists today in `scripts/bench/docgen-memory/` (`yarn bench:docgen-memory` from `scripts/`). +The latency half is new: the current harness measures per-save duration but throws it away, so recording cold, warm, and scan timings is work for the per-engine harness. + +## Determinism method + +- **Fixed synthetic projects.** + Inputs come from a generator with fixed parameters (component count, props per component, type-heaviness levers), never from real-world checkouts. + The existing generator is `scripts/bench/docgen-memory/generate-project.ts`. +- **Warmup.** + Every run starts with one full cold pass. + That pass is the cold-extraction sample and is excluded from all warm samples. + This exists today in `scripts/bench/docgen-memory/memory-harness.ts`. +- **Median-of-N for latency.** + Each latency metric runs N times and records the median; a single-run number is noise. + This does not exist today - the current gate runs each configuration exactly once - and is a requirement on the per-engine harness. +- **Slope for leak metrics.** + Retained growth and retained slope keep their least-squares fit over the save series instead of median-of-N; the series itself is the statistic. + This exists today. +- **Series mean for transient memory.** + Peak memory (transient) is the mean of the per-save spikes across one run's save series - not repeated and medianed like latency, not slope-fitted like the leak metrics. + This exists today in `scripts/bench/docgen-memory/memory-harness.ts`. +- **Fresh child per configuration.** + Each configuration runs in a freshly spawned child process so every measurement starts from a clean heap. + This exists today in `scripts/bench/docgen-memory/gate.ts`. +- **Relative comparison on the same machine.** + Perf questions are answered by ratios between runs executed sequentially inside a single CI job on one executor, or on one local machine. + "Same machine" means exactly that; cross-job, cross-run, and cross-PR comparisons are non-comparisons because CI executors are ephemeral and not guaranteed identical. + The two standing comparisons are docgen-server flag on vs off, and new engine vs legacy engine, both measured in the same run. + +## Budget shape + +Timing budgets are ratios or slopes, never absolute milliseconds; absolute wall-clock on shared CI executors is too noisy to gate. +The calibration reference for React is the legacy-vs-new-engine ratio measured in the same run on the same machine. +Memory budgets stay absolute megabytes with generous headroom: budgets sit well above observed values so the gate is not flaky, while still failing hard on a real regression. +Every engine must also carry its own negative control - a configuration that must fail, proving the gate can catch the regression class it exists for. +Budgets and controls are derived per engine from that engine's own baseline runs; nothing is ported between engines. + +## CI tiers + +- **Per PR: report-only.** + The seam is the unconditional job block of the generated CircleCI config (`scripts/ci/main.ts`), where the package benchmark job already runs on every build without gating. + Perf jobs there report numbers and never fail the build. +- **Daily: the only gating tier.** + "Daily" names the CircleCI `workflow=daily` pipeline parameter (tier order `normal < merged < daily`, `scripts/ci/utils/types.ts`); daily-only jobs attach in `scripts/ci/main.ts`, where the docgen memory gate already runs. +- **The daily cadence is unproven.** + The only trigger for the daily tier in version control is the `ci:daily` PR label (`.github/workflows/trigger-circle-ci-workflow.yml`); no scheduled trigger exists in the repo, and a CircleCI-side schedule cannot be verified from here. + Until that is resolved, gating on the daily tier means gating when someone applies the label. + The baseline-and-budget work must settle the cadence before this gate can be described as nightly protection. + +## User-perceived metrics + +Three user-perceived metrics join the suite as report-only extensions of the sandbox bench task; none joins the gated floor. +They need a real dev server and Chromium, which makes them variance-heavy; they exist to show user-visible impact, not to gate. +All three run on existing tooling with no new dependencies: + +- **Dev-server startup delta.** + Run the sandbox bench task (`scripts/tasks/bench.ts` driving `scripts/bench/browse.ts` over Playwright/Chromium, with dev-server readiness timings from `scripts/tasks/dev.ts`) twice, docgen-server feature on and off, and report the delta. + Today only the internal Storybook config reads the on/off switch (`code/.storybook/main.ts`); generated sandboxes do not, so the harness work must inject the feature toggle into the sandbox config before the delta can be measured there. + No first-class on/off comparison mode exists yet; today this is two runs diffed by hand. +- **Time-to-Controls-populated.** + Adapt the existing Playwright flow that selects a story, opens the Controls panel, and waits for an args-table cell (`code/e2e-internal/docgen-hot-update.spec.ts`) into a timing probe, using the elapsed-time pattern from `browse.ts`. + Today that flow is a pass/fail assertion and records no elapsed time. +- **Docs-page props-table render.** + `benchAutodocs` in `scripts/bench/browse.ts` already times the docs page but waits on the component description text. + The probe swaps that wait target for a props-table element; the swap does not exist yet. + +## Budgets table skeleton + +Values land when per-engine baselines are recorded; until then every cell is a placeholder. +Each engine's negative control is re-derived from that engine's own baseline, never ported. + +| Engine | Cold extraction | Warm extraction | Whole-project scan | Peak memory (transient) | Retained growth | Retained slope | Negative control | Tier | +| ------------------------------------ | --------------- | --------------- | ------------------ | ----------------------- | --------------- | -------------- | ---------------- | ---------- | +| react-legacy (react-docgen, control) | TBD (1.12) | TBD (1.12) | n/a | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | +| react-osa (react-component-meta, control) | TBD (1.12) | TBD (1.12) | n/a | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | +| vue-component-meta | TBD (1.12) | TBD (1.12) | n/a | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | +| compodoc | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | +| svelte (stretch) | TBD (1.12) | TBD (1.12) | n/a | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | +| cem (stretch) | TBD (1.12) | TBD (1.12) | n/a | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | From 46368a6955507fe06a9416cb9eefa4613144ca15 Mon Sep 17 00:00:00 2001 From: Valentin Palkovic Date: Thu, 23 Jul 2026 23:52:55 +0200 Subject: [PATCH 02/14] Bench: Tighten the perf methodology note per cross-model review Resolves the ambiguities a Sol + Fable review pass confirmed: how median-of-N samples map onto fresh processes per metric, what the timing ratio divides, what Compodoc's per-save metrics mean for a one-shot CLI, and what a negative control must name. Corrects two claims the code contradicted: retained growth is a last-minus-baseline delta, not a least-squares fit, and benchAutodocs has no real wait on the description text. Drops a duplicated sentence, plain-words the CI tier wording, and realigns the table. --- scripts/bench/PERF-METHODOLOGY.md | 71 +++++++++++++++++++------------ 1 file changed, 43 insertions(+), 28 deletions(-) diff --git a/scripts/bench/PERF-METHODOLOGY.md b/scripts/bench/PERF-METHODOLOGY.md index f6dc8e9d3092..03aaefd77175 100644 --- a/scripts/bench/PERF-METHODOLOGY.md +++ b/scripts/bench/PERF-METHODOLOGY.md @@ -9,9 +9,12 @@ The suite measures; it never optimizes. The gated suite runs in plain Node: no browser, no dev server, no Vite. Each engine is measured through a directly callable entry point: -- React, new engine: `ComponentMetaManager.batchExtract` (`code/renderers/react/src/componentManifest/componentMeta/ComponentMetaManager.ts`) or `buildDocgenPayload` (`code/renderers/react/src/docgen/buildDocgen.ts`). +- React, new engine: `ComponentMetaManager.batchExtract` (`code/renderers/react/src/componentManifest/componentMeta/ComponentMetaManager.ts`). + `buildDocgenPayload` (`code/renderers/react/src/docgen/buildDocgen.ts`) wraps it with story-file resolution and argTypes conversion, so it measures more work than the bare legacy parse calls; `batchExtract` is the metric of record. - React, legacy engines: `getReactDocgen` (`code/renderers/react/src/componentManifest/reactDocgen.ts`) and `parseWithReactDocgenTypescript` (`code/renderers/react/src/componentManifest/reactDocgenTypescript.ts`). - The live legacy path is a Vite transform hook and cannot run standalone; these wrappers share its extraction logic but are not byte-identical to it, so legacy numbers describe the engines, not the Vite pipeline. + The live legacy paths are builder plugins - a Vite plugin or a webpack loader, some third-party - and cannot run standalone; these wrappers share their extraction logic without being byte-identical to any of them, so legacy numbers describe the engines, not the builder pipelines. + Both wrappers cache per file for the life of the process and expose only global invalidation; a simulated save must invalidate before re-parsing or the warm sample is a cache hit. + `react-docgen` is the budgeted legacy control; `react-docgen-typescript` is measurable through the same wrappers but carries no budget row. - Angular: a standalone Compodoc CLI run. - Vue, Svelte, and web components: their extraction engines are plain functions callable from Node; exact entry points get pinned when each engine's harness lands. @@ -25,7 +28,10 @@ Five metrics per engine, all candidates for gating: - **Whole-project scan**: time for one batch pass over the entire project. Only Compodoc works this way, so this metric applies to Compodoc alone; per-component engines record it as n/a rather than a faked equivalent. - **Peak memory (transient)**: the allocation spike a save causes above the retained baseline, sampled around forced GC. -- **Leak detection**: retained-heap growth after a long save series, plus the least-squares slope of retained heap over that series. +- **Leak detection**: retained-heap growth after the save series, plus the least-squares slope of retained heap over that series. + +Compodoc maps onto these differently because it is a fresh CLI process per run: cold extraction and whole-project scan are the same full-project measurement, warm extraction is a second full run after touching one file, and peak memory is the child's peak RSS sampled from outside the process. +Whether the retained-series metrics mean anything for a fresh-process engine is settled when its baselines are recorded; until then its leak cells stay placeholders. The memory and leak half exists today in `scripts/bench/docgen-memory/` (`yarn bench:docgen-memory` from `scripts/`). The latency half is new: the current harness measures per-save duration but throws it away, so recording cold, warm, and scan timings is work for the per-engine harness. @@ -38,41 +44,51 @@ The latency half is new: the current harness measures per-save duration but thro - **Warmup.** Every run starts with one full cold pass. That pass is the cold-extraction sample and is excluded from all warm samples. - This exists today in `scripts/bench/docgen-memory/memory-harness.ts`. + This exists today in the refresh mode of `scripts/bench/docgen-memory/memory-harness.ts`; live mode skips it by design. - **Median-of-N for latency.** - Each latency metric runs N times and records the median; a single-run number is noise. - This does not exist today - the current gate runs each configuration exactly once - and is a requirement on the per-engine harness. -- **Slope for leak metrics.** - Retained growth and retained slope keep their least-squares fit over the save series instead of median-of-N; the series itself is the statistic. + Each latency metric records a median, never a single-run number. + Cold extraction and whole-project scan yield one sample per fresh process, so their N samples come from N spawns. + Warm extraction records the median of the per-save durations inside a single run's save series. + The harness pins one N for all engines and records it with the results; numbers taken at different N are not comparable. + None of this exists today - the current gate runs each configuration exactly once. +- **Series statistics for leak metrics.** + Retained slope is a least-squares fit over the save series; retained growth is the delta between the final retained sample and the pre-run baseline. + Both read one run's series instead of repeated runs. This exists today. - **Series mean for transient memory.** - Peak memory (transient) is the mean of the per-save spikes across one run's save series - not repeated and medianed like latency, not slope-fitted like the leak metrics. + Peak memory (transient) is the mean of the per-save spikes across the save series - not repeated and medianed like cold latency, not slope-fitted like retained slope. + Warm latency, transient memory, and the leak metrics all read the same fixed-length series from the same run. This exists today in `scripts/bench/docgen-memory/memory-harness.ts`. -- **Fresh child per configuration.** - Each configuration runs in a freshly spawned child process so every measurement starts from a clean heap. +- **Fresh process per measurement.** + Every measured process is spawned fresh so it starts from a clean heap. This exists today in `scripts/bench/docgen-memory/gate.ts`. - **Relative comparison on the same machine.** Perf questions are answered by ratios between runs executed sequentially inside a single CI job on one executor, or on one local machine. "Same machine" means exactly that; cross-job, cross-run, and cross-PR comparisons are non-comparisons because CI executors are ephemeral and not guaranteed identical. The two standing comparisons are docgen-server flag on vs off, and new engine vs legacy engine, both measured in the same run. + Paired runs alternate their order across repetitions so cache warming and thermal drift do not consistently favor one side. ## Budget shape Timing budgets are ratios or slopes, never absolute milliseconds; absolute wall-clock on shared CI executors is too noisy to gate. +A timing ratio divides the median of one side by the median of the other, both measured in the same job. The calibration reference for React is the legacy-vs-new-engine ratio measured in the same run on the same machine. +Engines without a second implementation in the same job get their timing reference picked when their baselines are recorded; until then their timing budgets stay placeholders. Memory budgets stay absolute megabytes with generous headroom: budgets sit well above observed values so the gate is not flaky, while still failing hard on a real regression. Every engine must also carry its own negative control - a configuration that must fail, proving the gate can catch the regression class it exists for. +A control names its lever and the one metric it must trip; it does not need to trip every metric, and where no credible lever exists the gap is recorded instead of inventing a hollow control. +The only worked example today is the React memory gate's out-of-memory control. Budgets and controls are derived per engine from that engine's own baseline runs; nothing is ported between engines. ## CI tiers - **Per PR: report-only.** - The seam is the unconditional job block of the generated CircleCI config (`scripts/ci/main.ts`), where the package benchmark job already runs on every build without gating. - Perf jobs there report numbers and never fail the build. + Per-PR perf jobs attach to the unconditional job block of the generated CircleCI config (`scripts/ci/main.ts`), where the package benchmark job already runs on every build without gating. + They report numbers and never fail the build. - **Daily: the only gating tier.** "Daily" names the CircleCI `workflow=daily` pipeline parameter (tier order `normal < merged < daily`, `scripts/ci/utils/types.ts`); daily-only jobs attach in `scripts/ci/main.ts`, where the docgen memory gate already runs. - **The daily cadence is unproven.** - The only trigger for the daily tier in version control is the `ci:daily` PR label (`.github/workflows/trigger-circle-ci-workflow.yml`); no scheduled trigger exists in the repo, and a CircleCI-side schedule cannot be verified from here. + The only trigger for the daily tier in version control is the `ci:daily` PR label (`.github/workflows/trigger-circle-ci-workflow.yml`); no scheduled trigger exists in the repo, and no CircleCI-side schedule is confirmed to exist. Until that is resolved, gating on the daily tier means gating when someone applies the label. The baseline-and-budget work must settle the cadence before this gate can be described as nightly protection. @@ -80,7 +96,7 @@ Budgets and controls are derived per engine from that engine's own baseline runs Three user-perceived metrics join the suite as report-only extensions of the sandbox bench task; none joins the gated floor. They need a real dev server and Chromium, which makes them variance-heavy; they exist to show user-visible impact, not to gate. -All three run on existing tooling with no new dependencies: +All three build on existing tooling and add no new dependencies: - **Dev-server startup delta.** Run the sandbox bench task (`scripts/tasks/bench.ts` driving `scripts/bench/browse.ts` over Playwright/Chromium, with dev-server readiness timings from `scripts/tasks/dev.ts`) twice, docgen-server feature on and off, and report the delta. @@ -90,19 +106,18 @@ All three run on existing tooling with no new dependencies: Adapt the existing Playwright flow that selects a story, opens the Controls panel, and waits for an args-table cell (`code/e2e-internal/docgen-hot-update.spec.ts`) into a timing probe, using the elapsed-time pattern from `browse.ts`. Today that flow is a pass/fail assertion and records no elapsed time. - **Docs-page props-table render.** - `benchAutodocs` in `scripts/bench/browse.ts` already times the docs page but waits on the component description text. - The probe swaps that wait target for a props-table element; the swap does not exist yet. + `benchAutodocs` in `scripts/bench/browse.ts` times the docs page today, but its wait on the component description text is a locator construction with no real wait condition, so the recorded number mostly reflects page load. + The probe waits on a props-table element with a real wait condition; it does not exist yet. ## Budgets table skeleton -Values land when per-engine baselines are recorded; until then every cell is a placeholder. -Each engine's negative control is re-derived from that engine's own baseline, never ported. - -| Engine | Cold extraction | Warm extraction | Whole-project scan | Peak memory (transient) | Retained growth | Retained slope | Negative control | Tier | -| ------------------------------------ | --------------- | --------------- | ------------------ | ----------------------- | --------------- | -------------- | ---------------- | ---------- | -| react-legacy (react-docgen, control) | TBD (1.12) | TBD (1.12) | n/a | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | -| react-osa (react-component-meta, control) | TBD (1.12) | TBD (1.12) | n/a | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | -| vue-component-meta | TBD (1.12) | TBD (1.12) | n/a | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | -| compodoc | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | -| svelte (stretch) | TBD (1.12) | TBD (1.12) | n/a | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | -| cem (stretch) | TBD (1.12) | TBD (1.12) | n/a | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | +The baseline work fills in the values; until then every cell is a placeholder. + +| Engine | Cold extraction | Warm extraction | Whole-project scan | Peak memory (transient) | Retained growth | Retained slope | Negative control | Tier | +| ----------------------------------------- | --------------- | --------------- | ------------------ | ----------------------- | --------------- | -------------- | ---------------- | ---------- | +| react-legacy (react-docgen, control) | TBD (1.12) | TBD (1.12) | n/a | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | +| react-osa (react-component-meta, control) | TBD (1.12) | TBD (1.12) | n/a | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | +| vue-component-meta | TBD (1.12) | TBD (1.12) | n/a | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | +| compodoc | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | +| svelte (stretch) | TBD (1.12) | TBD (1.12) | n/a | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | +| cem (stretch) | TBD (1.12) | TBD (1.12) | n/a | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | From bab7ca578fe41aabcee2fc51b0ebadd860009dac Mon Sep 17 00:00:00 2001 From: Valentin Palkovic Date: Fri, 24 Jul 2026 14:49:33 +0200 Subject: [PATCH 03/14] Bench: Add the per-engine docgen perf harness with React as control Implements the harness half of the perf methodology note. A new orchestrator (yarn bench:docgen-perf) measures the five floor metrics per engine in plain Node: one fresh child process per measurement, cold medians from N spawns with the pinned N recorded in the results, warm as the median of the first repetition's save series. react-docgen and ComponentMetaManager form the calibration pair: both run in one invocation, their spawn order alternates per repetition, and the results carry the legacy-vs-osa ratio of medians. react-docgen-typescript is measurable behind a flag but carries no budget row. vue-component-meta runs over generated monorepo-shaped projects: workspace packages with per-package tsconfigs and paths aliases, cross-package prop-type chains, a fake heavy .d.ts library in the generated tree, and a scenario that touches a widely-imported base type to measure warm extraction under wide checker invalidation. Compodoc runs as the external CLI it is, with peak RSS polled from outside; it skips with a message when no binary is installed, since compodoc is a user-project dependency. The memory harness now persists the cold and per-save durations it already computed, and the gate reads its budget values from the shared budgets.ts. Gate behavior is unchanged (verified on Node 22.22.3; its hand-tuned OOM pair is margin-sensitive on other Node majors with or without this change). The new entry points run on native Node type stripping because react-docgen's browserslist data require breaks under the jiti loader. --- scripts/bench/docgen-memory/gate.ts | 14 +- scripts/bench/docgen-memory/memory-harness.ts | 8 +- scripts/bench/docgen-perf/budgets.ts | 23 + scripts/bench/docgen-perf/config.ts | 118 +++++ scripts/bench/docgen-perf/engines/compodoc.ts | 126 +++++ .../bench/docgen-perf/engines/react-legacy.ts | 243 ++++++++++ .../docgen-perf/engines/vue-component-meta.ts | 220 +++++++++ .../bench/docgen-perf/generators/angular.ts | 139 ++++++ scripts/bench/docgen-perf/generators/vue.ts | 293 +++++++++++ scripts/bench/docgen-perf/paths.ts | 13 + scripts/bench/docgen-perf/run.ts | 457 ++++++++++++++++++ scripts/bench/docgen-perf/sampling.ts | 29 ++ scripts/bench/docgen-perf/stats.test.ts | 56 +++ scripts/bench/docgen-perf/stats.ts | 36 ++ scripts/bench/docgen-perf/types.ts | 99 ++++ scripts/package.json | 2 + yarn.lock | 1 + 17 files changed, 1863 insertions(+), 14 deletions(-) create mode 100644 scripts/bench/docgen-perf/budgets.ts create mode 100644 scripts/bench/docgen-perf/config.ts create mode 100644 scripts/bench/docgen-perf/engines/compodoc.ts create mode 100644 scripts/bench/docgen-perf/engines/react-legacy.ts create mode 100644 scripts/bench/docgen-perf/engines/vue-component-meta.ts create mode 100644 scripts/bench/docgen-perf/generators/angular.ts create mode 100644 scripts/bench/docgen-perf/generators/vue.ts create mode 100644 scripts/bench/docgen-perf/paths.ts create mode 100644 scripts/bench/docgen-perf/run.ts create mode 100644 scripts/bench/docgen-perf/sampling.ts create mode 100644 scripts/bench/docgen-perf/stats.test.ts create mode 100644 scripts/bench/docgen-perf/stats.ts create mode 100644 scripts/bench/docgen-perf/types.ts diff --git a/scripts/bench/docgen-memory/gate.ts b/scripts/bench/docgen-memory/gate.ts index e70c0f71ff96..97ca2d41bc69 100644 --- a/scripts/bench/docgen-memory/gate.ts +++ b/scripts/bench/docgen-memory/gate.ts @@ -23,6 +23,7 @@ import { spawnSync } from 'node:child_process'; import * as fs from 'node:fs'; import * as path from 'node:path'; +import { MEMORY_BUDGETS, type MemoryBudgets } from '../docgen-perf/budgets.ts'; import { SANDBOX_DIRECTORY } from '../../utils/constants.ts'; interface HarnessResult { @@ -33,15 +34,6 @@ interface HarnessResult { avgTransient?: number; } -interface MetricBudgets { - /** Max allowed post-GC retained growth (MB) across the run. */ - maxRetainedGrowthMb: number; - /** Max allowed average transient working set added per save (MB). */ - maxTransientMb: number; - /** Max allowed post-GC retained-heap slope (MB/save). */ - maxRetainedSlopeMb: number; -} - interface GateConfig { name: string; args: string[]; @@ -55,7 +47,7 @@ interface GateConfig { */ expectOom?: boolean; /** Post-GC metric budgets, asserted from `result.json` (metric configs only). */ - budgets?: MetricBudgets; + budgets?: MemoryBudgets; } const HARNESS = path.join(import.meta.dirname, 'memory-harness.ts'); @@ -78,7 +70,7 @@ const CONFIGS: GateConfig[] = [ '--scope', 'changed', ], outSubdir: 'changed-scope', - budgets: { maxRetainedGrowthMb: 60, maxTransientMb: 90, maxRetainedSlopeMb: 3 }, + budgets: MEMORY_BUDGETS['react-osa'], }, { // Positive control: the live per-edit workload survives under a tight cap BECAUSE the shared diff --git a/scripts/bench/docgen-memory/memory-harness.ts b/scripts/bench/docgen-memory/memory-harness.ts index 7797d12abaf4..928ab7782312 100644 --- a/scripts/bench/docgen-memory/memory-harness.ts +++ b/scripts/bench/docgen-memory/memory-harness.ts @@ -112,6 +112,7 @@ interface HarnessOptions { interface Sample { save: number; + durMs: number; rssMb: number; heapUsedMb: number; retainedHeapMb?: number; @@ -312,7 +313,8 @@ async function main() { const initialStart = Date.now(); console.log(` full extraction over ${entries.length} components (cold pass == refresh-all save)…`); manager.batchExtract(entries); - console.log(` initial batchExtract: ${Date.now() - initialStart}ms`); + const coldMs = Date.now() - initialStart; + console.log(` initial batchExtract: ${coldMs}ms`); const baseline = sampleMemory(options.forceGc && gcAvailable); console.log( @@ -350,7 +352,7 @@ async function main() { const durMs = Date.now() - saveStart; const mem = sampleMemory(options.forceGc && gcAvailable); - samples.push({ save, ...mem }); + samples.push({ save, durMs, ...mem }); console.log( ` save ${String(save).padStart(3)}: ${String(durMs).padStart(5)}ms ` + `rss=${mem.rssMb.toFixed(0).padStart(5)}MB heapUsed=${mem.heapUsedMb.toFixed(0).padStart(5)}MB` + @@ -404,7 +406,7 @@ async function main() { fs.writeFileSync( options.jsonOut, JSON.stringify( - { options, baseline, samples, peakRss, finalRss, rssSlope, retainedSlope, retainedGrowth, avgTransient }, + { options, coldMs, baseline, samples, peakRss, finalRss, rssSlope, retainedSlope, retainedGrowth, avgTransient }, null, 2 ) diff --git a/scripts/bench/docgen-perf/budgets.ts b/scripts/bench/docgen-perf/budgets.ts new file mode 100644 index 000000000000..cdcd5a4adf9e --- /dev/null +++ b/scripts/bench/docgen-perf/budgets.ts @@ -0,0 +1,23 @@ +/** + * Budget values shared between the docgen-memory gate and the per-engine perf suite. Budgets are + * derived per engine from that engine's own baseline runs and are never ported between engines; + * missing engines get their rows when their baselines are recorded. + */ +import type { EngineId } from './types.ts'; + +export interface MemoryBudgets { + /** Max allowed post-GC retained growth (MB) across the run. */ + maxRetainedGrowthMb: number; + /** Max allowed average transient working set added per save (MB). */ + maxTransientMb: number; + /** Max allowed post-GC retained-heap slope (MB/save). */ + maxRetainedSlopeMb: number; +} + +/** + * Budgets sit well above observed values so the gate is not flaky, while still failing hard on a + * real regression. + */ +export const MEMORY_BUDGETS: Partial> = { + 'react-osa': { maxRetainedGrowthMb: 60, maxTransientMb: 90, maxRetainedSlopeMb: 3 }, +}; diff --git a/scripts/bench/docgen-perf/config.ts b/scripts/bench/docgen-perf/config.ts new file mode 100644 index 000000000000..b124ad19edd3 --- /dev/null +++ b/scripts/bench/docgen-perf/config.ts @@ -0,0 +1,118 @@ +/** + * Fixed measurement parameters for the per-engine docgen performance suite. + * + * N is pinned here for every engine and recorded with the results; numbers taken at different N are + * not comparable. The --quick profile exists for smoke runs only and marks its results + * non-comparable. + */ + +/** Fresh-process spawns per cold/scan median. One value for all engines. */ +export const PINNED_N = 5; + +/** Spawns for --quick smoke runs. Never comparable with PINNED_N results. */ +export const QUICK_N = 2; + +/** Sampling interval for the compodoc child's externally-polled peak RSS. */ +export const RSS_POLL_INTERVAL_MS = 100; + +export interface ReactScenarioConfig { + components: number; + variants: number; + props: number; + saves: number; +} + +export interface VueScenarioConfig { + name: 'flat' | 'workspace' | 'base-type-touch'; + packages: number; + componentsPerPackage: number; + chainDepth: number; + fanOut: number; + heavyLib: boolean; + saves: number; +} + +export interface AngularScenarioConfig { + components: number; + props: number; +} + +export interface SuiteProfile { + n: number; + comparable: boolean; + react: ReactScenarioConfig; + vue: VueScenarioConfig[]; + angular: AngularScenarioConfig; +} + +export const DEFAULT_PROFILE: SuiteProfile = { + n: PINNED_N, + comparable: true, + react: { components: 300, variants: 4, props: 10, saves: 20 }, + vue: [ + { + name: 'flat', + packages: 1, + componentsPerPackage: 20, + chainDepth: 1, + fanOut: 4, + heavyLib: false, + saves: 15, + }, + { + name: 'workspace', + packages: 4, + componentsPerPackage: 10, + chainDepth: 3, + fanOut: 4, + heavyLib: true, + saves: 15, + }, + { + name: 'base-type-touch', + packages: 4, + componentsPerPackage: 10, + chainDepth: 3, + fanOut: 4, + heavyLib: true, + saves: 10, + }, + ], + angular: { components: 100, props: 8 }, +}; + +export const QUICK_PROFILE: SuiteProfile = { + n: QUICK_N, + comparable: false, + react: { components: 20, variants: 2, props: 4, saves: 4 }, + vue: [ + { + name: 'flat', + packages: 1, + componentsPerPackage: 5, + chainDepth: 1, + fanOut: 2, + heavyLib: false, + saves: 3, + }, + { + name: 'workspace', + packages: 2, + componentsPerPackage: 3, + chainDepth: 2, + fanOut: 2, + heavyLib: true, + saves: 3, + }, + { + name: 'base-type-touch', + packages: 2, + componentsPerPackage: 3, + chainDepth: 2, + fanOut: 2, + heavyLib: true, + saves: 2, + }, + ], + angular: { components: 10, props: 4 }, +}; diff --git a/scripts/bench/docgen-perf/engines/compodoc.ts b/scripts/bench/docgen-perf/engines/compodoc.ts new file mode 100644 index 000000000000..5c11b5478ac8 --- /dev/null +++ b/scripts/bench/docgen-perf/engines/compodoc.ts @@ -0,0 +1,126 @@ +/** + * Adapter for the Compodoc engine. Compodoc is a one-shot CLI - a fresh process per run - so this + * adapter runs inside the orchestrator and spawns the compodoc CLI itself as the measured child: + * cold extraction and whole-project scan are the same full-project run, warm extraction is a second + * full run after touching one component file, and peak memory is the child's peak RSS sampled from + * outside the process. + * + * The compodoc binary is resolved at runtime: it is a user-project dependency in this repo, not a + * workspace package, so the engine skips with an explicit message when no binary is found. Version + * pinning is baseline work. + */ +import { spawn, spawnSync } from 'node:child_process'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +import { angularComponentSource, generateAngularProject } from '../generators/angular.ts'; +import type { AngularScenarioConfig } from '../config.ts'; + +/** Locations probed for a compodoc binary, in order: workspace .bin dirs, then PATH. */ +export function resolveCompodocBinary(): string | undefined { + const candidates = [ + path.resolve(import.meta.dirname, '../../../node_modules/.bin/compodoc'), + path.resolve(import.meta.dirname, '../../../../node_modules/.bin/compodoc'), + ]; + for (const candidate of candidates) { + if (fs.existsSync(candidate)) { + return candidate; + } + } + const which = spawnSync('which', ['compodoc'], { encoding: 'utf8' }); + const fromPath = which.status === 0 ? which.stdout.trim() : ''; + return fromPath || undefined; +} + +interface CompodocRun { + durMs: number; + peakRssMb: number; +} + +/** Run compodoc once over the project, polling the child's RSS from outside. */ +function runCompodocOnce( + binary: string, + projectDir: string, + docsOutDir: string, + pollIntervalMs: number +): Promise { + fs.rmSync(docsOutDir, { recursive: true, force: true }); + const args = ['-p', 'tsconfig.json', '-e', 'json', '-d', docsOutDir, '--silent']; + + return new Promise((resolve, reject) => { + const start = Date.now(); + const child = spawn(binary, args, { cwd: projectDir }); + let peakRssKb = 0; + let output = ''; + child.stdout.on('data', (chunk: Buffer) => (output += chunk.toString())); + child.stderr.on('data', (chunk: Buffer) => (output += chunk.toString())); + + const poll = setInterval(() => { + if (child.pid === undefined) { + return; + } + const ps = spawnSync('ps', ['-o', 'rss=', '-p', String(child.pid)], { encoding: 'utf8' }); + const rssKb = Number(ps.stdout.trim()); + if (Number.isFinite(rssKb) && rssKb > peakRssKb) { + peakRssKb = rssKb; + } + }, pollIntervalMs); + + child.on('error', (err) => { + clearInterval(poll); + reject(err); + }); + child.on('close', (status) => { + clearInterval(poll); + const durMs = Date.now() - start; + if (status !== 0) { + const tail = output.trim().split('\n').slice(-8).join('\n'); + reject(new Error(`compodoc exited with status ${status}:\n${tail}`)); + return; + } + if (!fs.existsSync(path.join(docsOutDir, 'documentation.json'))) { + reject(new Error('compodoc run produced no documentation.json')); + return; + } + // The polled peak misses spikes shorter than the interval; the recorded value is a floor. + resolve({ durMs, peakRssMb: peakRssKb / 1024 }); + }); + }); +} + +export interface CompodocRepetition { + coldMs: number; + warmMs: number; + peakRssMb: number; +} + +/** + * One repetition: fresh project, cold full run, touch one component, warm full run. Both runs are + * fresh compodoc processes, matching the one-sample-per-fresh-process topology. + */ +export async function runCompodocRepetition( + binary: string, + scenario: AngularScenarioConfig, + workDir: string, + pollIntervalMs: number +): Promise { + const projectDir = path.join(workDir, 'project'); + const docsOutDir = path.join(workDir, 'docs'); + const project = generateAngularProject({ + outDir: projectDir, + components: scenario.components, + props: scenario.props, + }); + + const cold = await runCompodocOnce(binary, project.outDir, docsOutDir, pollIntervalMs); + + // Touch one component so the warm run sees a genuinely changed file. + fs.writeFileSync(project.componentPaths[0], angularComponentSource(0, scenario.props + 1)); + const warm = await runCompodocOnce(binary, project.outDir, docsOutDir, pollIntervalMs); + + return { + coldMs: cold.durMs, + warmMs: warm.durMs, + peakRssMb: Math.max(cold.peakRssMb, warm.peakRssMb), + }; +} diff --git a/scripts/bench/docgen-perf/engines/react-legacy.ts b/scripts/bench/docgen-perf/engines/react-legacy.ts new file mode 100644 index 000000000000..a228747f8517 --- /dev/null +++ b/scripts/bench/docgen-perf/engines/react-legacy.ts @@ -0,0 +1,243 @@ +/** + * Series harness for the legacy React docgen engines: `react-docgen` (the budgeted legacy control) + * and `react-docgen-typescript` (measurable, no budget row). Same shape as the docgen-memory + * harness: generate a project, one timed cold pass, then per-save touch + invalidate + re-extract + * with memory sampling around forced GC. + * + * Both engines cache per file for the life of the process and expose only global invalidation, so + * every simulated save calls the global invalidation first - skipping it would measure a cache hit. + * The react-docgen-typescript parser additionally keys its TS program off `process.cwd()`, so this + * harness chdirs into the generated project before the first parse. + * + * Runs on native Node type stripping - NOT under jiti: react-docgen's browserslist dependency + * fails on its JSON data require under the jiti loader ("jsReleases.map is not a function"). + * + * Run: + * node --expose-gc scripts/bench/docgen-perf/engines/react-legacy.ts \ + * --parser react-docgen --components 300 --saves 20 --json /tmp/result.json + */ +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +import { SANDBOX_DIRECTORY } from '../paths.ts'; +import { componentSource, generateProject } from '../../docgen-memory/generate-project.ts'; +import { gcAvailable, sampleMemory } from '../sampling.ts'; +import { leastSquaresSlope, mean } from '../stats.ts'; +import type { SaveSample } from '../types.ts'; + +/** + * Minimal structural views of the legacy engine modules, kept local so the `scripts` typecheck does + * not descend into `code/renderers` internals; the real types live in + * `code/renderers/react/src/componentManifest`. + */ +interface ComponentRefLike { + componentName: string; + importName: string; + localImportName: string; + importId: string; + isPackage: boolean; + path: string; +} + +interface ReactDocgenModule { + getReactDocgen( + path: string, + component: ComponentRefLike + ): { type: 'success' } | { type: 'error'; error: { name: string; message: string } }; +} + +interface UtilsModule { + invalidateCache(): void; +} + +interface ReactDocgenTypescriptModule { + parseWithReactDocgenTypescript(filePath: string): Promise>; + invalidateParser(): void; +} + +/** + * Load a `code/renderers/react` module at runtime. The specifier is built with `new URL` so the + * static `scripts` typecheck does not pull `code/renderers` source into its program. + */ +async function loadRendererModule(relativePath: string): Promise { + const url = new URL( + `../../../../code/renderers/react/src/componentManifest/${relativePath}`, + import.meta.url + ).href; + return (await import(url)) as T; +} + +type Parser = 'react-docgen' | 'react-docgen-typescript'; + +interface HarnessOptions { + parser: Parser; + components: number; + variants: number; + props: number; + saves: number; + outDir: string; + jsonOut?: string; +} + +function parseArgs(argv: string[]): HarnessOptions { + const get = (flag: string, fallback: string) => { + const idx = argv.indexOf(flag); + return idx >= 0 && argv[idx + 1] ? argv[idx + 1] : fallback; + }; + const parser = get('--parser', 'react-docgen'); + if (parser !== 'react-docgen' && parser !== 'react-docgen-typescript') { + throw new Error(`--parser must be "react-docgen" or "react-docgen-typescript", got "${parser}"`); + } + return { + parser, + components: Number(get('--components', '300')), + variants: Number(get('--variants', '4')), + props: Number(get('--props', '10')), + saves: Number(get('--saves', '20')), + outDir: get('--out', path.join(SANDBOX_DIRECTORY, 'docgen-perf', 'react-legacy', 'project')), + jsonOut: argv.indexOf('--json') >= 0 ? get('--json', '') : undefined, + }; +} + +function componentRef(i: number, componentPath: string): ComponentRefLike { + return { + componentName: `Comp${i}`, + importName: `Comp${i}`, + localImportName: `Comp${i}`, + importId: `./Comp${i}`, + isPackage: false, + path: componentPath, + }; +} + +async function main() { + const options = parseArgs(process.argv.slice(2)); + + console.log(`react-legacy harness (${options.parser})`); + console.log( + ` components=${options.components} variants=${options.variants} props=${options.props} saves=${options.saves}` + ); + if (!gcAvailable()) { + console.log(' (run with `node --expose-gc` to measure retained heap; continuing without it)'); + } + + const genStart = Date.now(); + const project = generateProject({ + outDir: options.outDir, + components: options.components, + variants: options.variants, + props: options.props, + heavyTypes: false, + heavyFactor: 1, + base64Kb: 0, + withNodeModules: true, + }); + console.log(` generated project in ${Date.now() - genStart}ms at ${project.outDir}`); + + const { invalidateCache } = await loadRendererModule('utils.ts'); + + let extractOne: (i: number) => Promise | void; + let invalidate: () => void; + + if (options.parser === 'react-docgen') { + const { getReactDocgen } = await loadRendererModule('reactDocgen.ts'); + extractOne = (i) => { + const result = getReactDocgen(project.componentPaths[i], componentRef(i, project.componentPaths[i])); + if (result.type === 'error') { + throw new Error(`react-docgen failed on Comp${i}: ${result.error.name} ${result.error.message}`); + } + }; + invalidate = invalidateCache; + } else { + const { parseWithReactDocgenTypescript, invalidateParser } = + await loadRendererModule('reactDocgenTypescript.ts'); + // The parser resolves its tsconfig from process.cwd(); point it at the generated project. + process.chdir(project.outDir); + extractOne = async (i) => { + const docs = await parseWithReactDocgenTypescript(project.componentPaths[i]); + if (docs.length === 0) { + throw new Error(`react-docgen-typescript returned no docs for Comp${i}`); + } + }; + invalidate = () => { + invalidateParser(); + invalidateCache(); + }; + } + + console.log(` full extraction over ${options.components} components (cold pass)…`); + const coldStart = Date.now(); + for (let i = 0; i < options.components; i++) { + await extractOne(i); + } + const coldMs = Date.now() - coldStart; + console.log(` cold pass: ${coldMs}ms`); + + const baseline = sampleMemory(true); + + const samples: SaveSample[] = []; + const extraByComponent = new Array(options.components).fill(options.props); + + for (let save = 1; save <= options.saves; save++) { + const i = (save - 1) % options.components; + extraByComponent[i] += 1; + fs.writeFileSync(project.componentPaths[i], componentSource(i, extraByComponent[i])); + // Global invalidation is the only surface these engines expose; without it the re-extraction + // below is a cache hit. + invalidate(); + + const saveStart = Date.now(); + await extractOne(i); + const durMs = Date.now() - saveStart; + + const mem = sampleMemory(true); + samples.push({ save, durMs, ...mem }); + console.log( + ` save ${String(save).padStart(3)}: ${String(durMs).padStart(5)}ms ` + + `rss=${mem.rssMb.toFixed(0).padStart(5)}MB heapUsed=${mem.heapUsedMb.toFixed(0).padStart(5)}MB` + + (mem.retainedHeapMb !== undefined + ? ` retained=${mem.retainedHeapMb.toFixed(0).padStart(5)}MB` + : '') + ); + } + + const retainedValues = samples + .map((s) => s.retainedHeapMb) + .filter((v): v is number => v !== undefined); + const retainedSlope = retainedValues.length ? leastSquaresSlope(retainedValues) : undefined; + const retainedGrowth = + retainedValues.length && baseline.retainedHeapMb !== undefined + ? (retainedValues.at(-1) as number) - baseline.retainedHeapMb + : undefined; + const transients = samples + .map((s) => (s.retainedHeapMb !== undefined ? s.heapUsedMb - s.retainedHeapMb : undefined)) + .filter((v): v is number => v !== undefined); + const avgTransient = transients.length ? mean(transients) : undefined; + + console.log('\nsummary'); + console.log(` cold pass: ${coldMs}ms`); + if (avgTransient !== undefined) { + console.log(` avg transient/save: ${avgTransient.toFixed(0)}MB`); + } + if (retainedSlope !== undefined && retainedGrowth !== undefined) { + console.log(` retained slope: ${retainedSlope.toFixed(2)}MB/save`); + console.log(` retained growth: ${retainedGrowth.toFixed(0)}MB over ${options.saves} saves`); + } + + if (options.jsonOut) { + fs.writeFileSync( + options.jsonOut, + JSON.stringify( + { options, coldMs, baseline, samples, retainedSlope, retainedGrowth, avgTransient }, + null, + 2 + ) + ); + console.log(` wrote ${options.jsonOut}`); + } +} + +main().catch((err) => { + console.error(err); + process.exitCode = 1; +}); diff --git a/scripts/bench/docgen-perf/engines/vue-component-meta.ts b/scripts/bench/docgen-perf/engines/vue-component-meta.ts new file mode 100644 index 000000000000..79d031fe48b0 --- /dev/null +++ b/scripts/bench/docgen-perf/engines/vue-component-meta.ts @@ -0,0 +1,220 @@ +/** + * Series harness for the vue-component-meta engine. Same shape as the docgen-memory harness: + * generate a project, one timed cold pass (checker creation included - it is part of the engine's + * first-extraction cost), then per-save touch + invalidate + re-extract with memory sampling. + * + * The checker never re-stats files on its own; a disk rewrite must be followed by + * `checker.updateFile(path, content)` (the surface the production Vite plugin drives from HMR) or + * the re-extraction measures a stale-cache no-op. + * + * Scenarios: + * flat single package, checker over the project root (`createCheckerByJson`, the + * production fallback for tsconfig-less projects); saves touch components + * round-robin. + * workspace packages/* layout; the checker is driven at the deepest package's tsconfig + * (`createChecker`); saves touch that package's components round-robin. + * base-type-touch workspace layout; every save touches the widely-imported base type instead, + * then re-extracts one fixed dependent component, so each sample pays the wide + * checker invalidation. + * + * Run: + * node --expose-gc scripts/bench/docgen-perf/engines/vue-component-meta.ts \ + * --scenario workspace --packages 4 --components-per-package 10 --heavy-lib --json /tmp/result.json + */ +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +import { + type MetaCheckerOptions, + createChecker, + createCheckerByJson, +} from 'vue-component-meta'; + +import { SANDBOX_DIRECTORY } from '../paths.ts'; +import { baseTypesSource, generateVueProject, vueComponentSource } from '../generators/vue.ts'; +import { gcAvailable, sampleMemory } from '../sampling.ts'; +import { leastSquaresSlope, mean } from '../stats.ts'; +import type { SaveSample } from '../types.ts'; + +type Scenario = 'flat' | 'workspace' | 'base-type-touch'; + +interface HarnessOptions { + scenario: Scenario; + packages: number; + componentsPerPackage: number; + chainDepth: number; + fanOut: number; + heavyLib: boolean; + saves: number; + outDir: string; + jsonOut?: string; +} + +/** Mirrors the production Vite plugin's checker options. */ +const CHECKER_OPTIONS: MetaCheckerOptions = { + forceUseTs: true, + noDeclarations: true, + printer: { newLine: 1 }, +}; + +function parseArgs(argv: string[]): HarnessOptions { + const get = (flag: string, fallback: string) => { + const idx = argv.indexOf(flag); + return idx >= 0 && argv[idx + 1] ? argv[idx + 1] : fallback; + }; + const scenario = get('--scenario', 'workspace'); + if (scenario !== 'flat' && scenario !== 'workspace' && scenario !== 'base-type-touch') { + throw new Error(`--scenario must be "flat", "workspace" or "base-type-touch", got "${scenario}"`); + } + return { + scenario, + packages: Number(get('--packages', '4')), + componentsPerPackage: Number(get('--components-per-package', '10')), + chainDepth: Number(get('--chain-depth', '3')), + fanOut: Number(get('--fan-out', '4')), + heavyLib: argv.includes('--heavy-lib'), + saves: Number(get('--saves', '15')), + outDir: get('--out', path.join(SANDBOX_DIRECTORY, 'docgen-perf', 'vue-component-meta', 'project')), + jsonOut: argv.indexOf('--json') >= 0 ? get('--json', '') : undefined, + }; +} + +async function main() { + const options = parseArgs(process.argv.slice(2)); + const flat = options.scenario === 'flat'; + const packages = flat ? 1 : options.packages; + + console.log(`vue-component-meta harness (${options.scenario})`); + console.log( + ` packages=${packages} componentsPerPackage=${options.componentsPerPackage} ` + + `chainDepth=${options.chainDepth} fanOut=${options.fanOut} heavyLib=${options.heavyLib} saves=${options.saves}` + ); + if (!gcAvailable()) { + console.log(' (run with `node --expose-gc` to measure retained heap; continuing without it)'); + } + + const genStart = Date.now(); + const project = generateVueProject({ + outDir: options.outDir, + packages, + componentsPerPackage: options.componentsPerPackage, + chainDepth: options.chainDepth, + fanOut: options.fanOut, + heavyLib: options.heavyLib, + }); + console.log(` generated project in ${Date.now() - genStart}ms at ${project.outDir}`); + + // The measured set: every component in flat layout, the deepest package's components otherwise + // (their prop types pull the whole cross-package chain). + const targetPackage = packages - 1; + const targetPaths = flat + ? project.componentPaths + : project.componentPaths.slice(-options.componentsPerPackage); + + const extractOne = (checker: ReturnType, sfcPath: string) => { + const exportNames = checker.getExportNames(sfcPath); + if (!exportNames.includes('default')) { + throw new Error(`no default export found in ${sfcPath} (got: ${exportNames.join(', ') || 'none'})`); + } + const meta = checker.getComponentMeta(sfcPath, 'default'); + // Consume the surfaces production consumes so their evaluation cost is inside the measurement. + const propCount = meta.props.length + meta.events.length + meta.slots.length + meta.exposed.length; + if (meta.props.length === 0) { + throw new Error(`vue-component-meta returned zero props for ${sfcPath}`); + } + return propCount; + }; + + // Cold pass: checker creation is part of the engine's first-extraction cost. + console.log(` full extraction over ${targetPaths.length} components (cold pass, checker included)…`); + const coldStart = Date.now(); + const checker = flat + ? createCheckerByJson(project.outDir, { include: ['**/*'] }, CHECKER_OPTIONS) + : createChecker(project.packageConfigPaths[targetPackage], CHECKER_OPTIONS); + for (const sfcPath of targetPaths) { + extractOne(checker, sfcPath); + } + const coldMs = Date.now() - coldStart; + console.log(` cold pass: ${coldMs}ms`); + + const baseline = sampleMemory(true); + + const samples: SaveSample[] = []; + const extraByTarget = new Array(targetPaths.length).fill(0); + let extraBaseProps = 0; + + for (let save = 1; save <= options.saves; save++) { + let measuredPath: string; + if (options.scenario === 'base-type-touch') { + // Touch the widely-imported base type; every dependent package's props type changes. + extraBaseProps += 1; + const content = baseTypesSource(options.fanOut, extraBaseProps); + fs.writeFileSync(project.baseTypesPath, content); + checker.updateFile(project.baseTypesPath, content); + // Re-extract one fixed dependent - the first component of the deepest package - so each + // sample pays the wide invalidation. + measuredPath = targetPaths[0]; + } else { + const t = (save - 1) % targetPaths.length; + extraByTarget[t] += 1; + const content = vueComponentSource(targetPackage, t, extraByTarget[t]); + fs.writeFileSync(targetPaths[t], content); + checker.updateFile(targetPaths[t], content); + measuredPath = targetPaths[t]; + } + + const saveStart = Date.now(); + extractOne(checker, measuredPath); + const durMs = Date.now() - saveStart; + + const mem = sampleMemory(true); + samples.push({ save, durMs, ...mem }); + console.log( + ` save ${String(save).padStart(3)}: ${String(durMs).padStart(5)}ms ` + + `rss=${mem.rssMb.toFixed(0).padStart(5)}MB heapUsed=${mem.heapUsedMb.toFixed(0).padStart(5)}MB` + + (mem.retainedHeapMb !== undefined + ? ` retained=${mem.retainedHeapMb.toFixed(0).padStart(5)}MB` + : '') + ); + } + + const retainedValues = samples + .map((s) => s.retainedHeapMb) + .filter((v): v is number => v !== undefined); + const retainedSlope = retainedValues.length ? leastSquaresSlope(retainedValues) : undefined; + const retainedGrowth = + retainedValues.length && baseline.retainedHeapMb !== undefined + ? (retainedValues.at(-1) as number) - baseline.retainedHeapMb + : undefined; + const transients = samples + .map((s) => (s.retainedHeapMb !== undefined ? s.heapUsedMb - s.retainedHeapMb : undefined)) + .filter((v): v is number => v !== undefined); + const avgTransient = transients.length ? mean(transients) : undefined; + + console.log('\nsummary'); + console.log(` cold pass: ${coldMs}ms`); + if (avgTransient !== undefined) { + console.log(` avg transient/save: ${avgTransient.toFixed(0)}MB`); + } + if (retainedSlope !== undefined && retainedGrowth !== undefined) { + console.log(` retained slope: ${retainedSlope.toFixed(2)}MB/save`); + console.log(` retained growth: ${retainedGrowth.toFixed(0)}MB over ${options.saves} saves`); + } + + if (options.jsonOut) { + fs.writeFileSync( + options.jsonOut, + JSON.stringify( + { options, coldMs, baseline, samples, retainedSlope, retainedGrowth, avgTransient }, + null, + 2 + ) + ); + console.log(` wrote ${options.jsonOut}`); + } +} + +main().catch((err) => { + console.error(err); + process.exitCode = 1; +}); diff --git a/scripts/bench/docgen-perf/generators/angular.ts b/scripts/bench/docgen-perf/generators/angular.ts new file mode 100644 index 000000000000..eb8b5fdab106 --- /dev/null +++ b/scripts/bench/docgen-perf/generators/angular.ts @@ -0,0 +1,139 @@ +/** + * Generator for a synthetic Angular project consumable by a standalone Compodoc CLI run. + * + * Compodoc documents inputs/outputs from the TypeScript AST, so the generated tree ships a minimal + * fake `(at)angular/core` type surface inside its own node_modules (hermetic - no npm install of the + * real framework) just so type resolution stays clean. + * + * Run directly: + * node --import jiti/register scripts/bench/docgen-perf/generators/angular.ts --out ../storybook-sandboxes/docgen-perf-angular --components 100 + */ +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { pathToFileURL } from 'node:url'; + +export interface AngularGenerateOptions { + outDir: string; + components: number; + /** Extra `@Input()` members per component on top of the fixed baseline set. */ + props: number; +} + +export interface GeneratedAngularProject { + outDir: string; + configPath: string; + /** Absolute component file paths, in component order. */ + componentPaths: string[]; +} + +const FAKE_ANGULAR_CORE = `export declare function Component(metadata: { + selector?: string; + template?: string; + standalone?: boolean; +}): ClassDecorator; +export declare function Input(bindingPropertyName?: string): PropertyDecorator; +export declare function Output(bindingPropertyName?: string): PropertyDecorator; +export declare class EventEmitter { + emit(value?: T): void; + subscribe(next: (value: T) => void): { unsubscribe(): void }; +} +`; + +function emitFakeAngularCore(projectDir: string): void { + const dir = path.join(projectDir, 'node_modules', '@angular', 'core'); + fs.mkdirSync(dir, { recursive: true }); + fs.writeFileSync( + path.join(dir, 'package.json'), + JSON.stringify({ name: '@angular/core', version: '0.0.0-bench', types: 'index.d.ts' }, null, 2) + ); + fs.writeFileSync(path.join(dir, 'index.d.ts'), FAKE_ANGULAR_CORE); +} + +/** + * A component's source. `extraProps` grows the input surface by one per warm-run touch, so the + * second Compodoc run sees a genuinely changed file. + */ +export function angularComponentSource(i: number, extraProps: number): string { + const extras = Array.from( + { length: extraProps }, + (_, p) => ` /** Extra input ${p} for component ${i}. */ + @Input() extra${p}?: ${p % 3 === 0 ? `'a' | 'b' | 'c'` : p % 3 === 1 ? 'number' : 'string'};` + ).join('\n'); + + return `import { Component, EventEmitter, Input, Output } from '@angular/core'; + +/** + * Comp${i} - generated bench component. + */ +@Component({ + selector: 'bench-comp${i}', + template: '
{{ label }}
', +}) +export class Comp${i}Component { + /** Primary label shown to the user. */ + @Input() label = ''; + /** Numeric size token. */ + @Input() size?: number; + /** Visual variant. */ + @Input() variant?: 'primary' | 'secondary' | 'tertiary'; + /** Disable interaction. */ + @Input() disabled = false; +${extras ? `${extras}\n` : ''} /** Emits when the user acts on the component. */ + @Output() action = new EventEmitter<{ id: string; value: number }>(); +} +`; +} + +const TSCONFIG = { + compilerOptions: { + target: 'ES2020', + module: 'ESNext', + moduleResolution: 'Bundler', + strict: true, + skipLibCheck: true, + experimentalDecorators: true, + emitDecoratorMetadata: false, + }, + include: ['src/**/*.ts'], +}; + +export function generateAngularProject(options: AngularGenerateOptions): GeneratedAngularProject { + const outDir = path.resolve(options.outDir); + fs.rmSync(outDir, { recursive: true, force: true }); + fs.mkdirSync(path.join(outDir, 'src', 'app'), { recursive: true }); + + emitFakeAngularCore(outDir); + + const configPath = path.join(outDir, 'tsconfig.json'); + fs.writeFileSync(configPath, JSON.stringify(TSCONFIG, null, 2)); + + const componentPaths: string[] = []; + for (let i = 0; i < options.components; i++) { + const componentPath = path.join(outDir, 'src', 'app', `comp${i}.component.ts`); + fs.writeFileSync(componentPath, angularComponentSource(i, options.props)); + componentPaths.push(componentPath); + } + + return { outDir, configPath, componentPaths }; +} + +function parseArgs(argv: string[]): AngularGenerateOptions { + const get = (flag: string, fallback: string) => { + const idx = argv.indexOf(flag); + return idx >= 0 && argv[idx + 1] ? argv[idx + 1] : fallback; + }; + return { + outDir: get('--out', '../storybook-sandboxes/docgen-perf-angular'), + components: Number(get('--components', '100')), + props: Number(get('--props', '8')), + }; +} + +if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) { + const options = parseArgs(process.argv.slice(2)); + const start = Date.now(); + const result = generateAngularProject(options); + console.log( + `Generated ${options.components} Angular components into ${result.outDir} in ${Date.now() - start}ms` + ); +} diff --git a/scripts/bench/docgen-perf/generators/vue.ts b/scripts/bench/docgen-perf/generators/vue.ts new file mode 100644 index 000000000000..5b0e1e8619ea --- /dev/null +++ b/scripts/bench/docgen-perf/generators/vue.ts @@ -0,0 +1,293 @@ +/** + * Generator for a synthetic Vue 3 project shaped like the monorepos where vue-component-meta + * struggles: a `packages/*` workspace with per-package tsconfigs (`extends` + `paths` aliases, root + * `references`), cross-package prop-type import chains, and an optional fake heavy `.d.ts` library + * inside the generated tree's own node_modules (hermetic - no npm install). + * + * Structure levers: + * - packages: workspace width; each `packages/pkg{p}` extends the previous package's props type, + * so the type chain across packages has depth = packages. `packages/types` holds the shared + * base type every package ultimately extends - the widely-imported type the base-type-touch + * scenario mutates. + * - chainDepth: intra-package type-alias hops between a package's props interface and its parent + * type, adding resolution steps without new files. + * - fanOut: distinct auxiliary types from `packages/types` referenced by every package's props. + * - heavyLib: emits `node_modules/(at)bench/heavy-ui` with a large generated `.d.ts` surface + * imitating a vuetify-scale component library; package props reference it when enabled. + * + * Generated trees are bench-time scratch output (under the sandbox directory); only this generator + * and the scenario configs are committed. + * + * Run directly: + * node --import jiti/register scripts/bench/docgen-perf/generators/vue.ts --out ../storybook-sandboxes/docgen-perf-vue --packages 4 --components-per-package 10 + */ +import { createRequire } from 'node:module'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { pathToFileURL } from 'node:url'; + +const require = createRequire(import.meta.url); + +export interface VueGenerateOptions { + outDir: string; + /** Workspace packages (excluding the shared `types` package). 1 = flat single-package layout. */ + packages: number; + componentsPerPackage: number; + /** Type-alias hops between each package's props interface and the type it extends. */ + chainDepth: number; + /** Auxiliary types from the shared types package referenced by every package's props. */ + fanOut: number; + /** Emit and reference the fake heavy `.d.ts` library. */ + heavyLib: boolean; +} + +export interface GeneratedVueProject { + outDir: string; + /** Root tsconfig (references only). Absent in flat layout. */ + rootConfigPath?: string; + /** Per-package tsconfig paths, in package order. Absent in flat layout. */ + packageConfigPaths: string[]; + /** Path of the shared base-type module (the base-type-touch scenario's save target). */ + baseTypesPath: string; + /** Absolute SFC paths, in (package, component) order. */ + componentPaths: string[]; +} + +/** Components in the fake heavy library; each carries a fat literal-union prop surface. */ +const HEAVY_LIB_COMPONENTS = 40; +const HEAVY_LIB_PROPS = 20; +const HEAVY_LIB_UNION_MEMBERS = 30; + +/** Copy the minimal node_modules packages the checker needs for Vue type resolution. */ +function copyVueNodeModules(projectDir: string): void { + const dest = path.join(projectDir, 'node_modules'); + const rootNodeModules = path.resolve(require.resolve('vue/package.json'), '../..'); + fs.mkdirSync(path.join(dest, '@vue'), { recursive: true }); + + for (const pkg of ['vue', 'csstype']) { + fs.cpSync(path.join(rootNodeModules, pkg), path.join(dest, pkg), { + recursive: true, + dereference: true, + }); + } + for (const pkg of ['runtime-core', 'runtime-dom', 'shared', 'reactivity', 'compiler-dom']) { + fs.cpSync(path.join(rootNodeModules, '@vue', pkg), path.join(dest, '@vue', pkg), { + recursive: true, + dereference: true, + }); + } +} + +function heavyLibSource(): string { + const components: string[] = []; + for (let c = 0; c < HEAVY_LIB_COMPONENTS; c++) { + const props: string[] = []; + for (let p = 0; p < HEAVY_LIB_PROPS; p++) { + const members = Array.from( + { length: HEAVY_LIB_UNION_MEMBERS }, + (_, m) => `'hc${c}_p${p}_v${m}'` + ).join(' | '); + props.push(` prop${p}?: ${members};`); + } + components.push( + `export interface HeavyComponent${c}Props {\n${props.join('\n')}\n}\n` + + `export declare const HeavyComponent${c}: (props: HeavyComponent${c}Props) => unknown;` + ); + } + return `${components.join('\n\n')}\n`; +} + +function emitHeavyLib(projectDir: string): void { + const libDir = path.join(projectDir, 'node_modules', '@bench', 'heavy-ui'); + fs.mkdirSync(libDir, { recursive: true }); + fs.writeFileSync( + path.join(libDir, 'package.json'), + JSON.stringify({ name: '@bench/heavy-ui', version: '1.0.0', types: 'index.d.ts' }, null, 2) + ); + fs.writeFileSync(path.join(libDir, 'index.d.ts'), heavyLibSource()); +} + +/** + * The shared types package's module source. `extraBaseProps` grows `BaseProps` by one optional + * prop per base-type-touch save, so every dependent package's props type genuinely changes. + */ +export function baseTypesSource(fanOut: number, extraBaseProps: number): string { + const extras = Array.from( + { length: extraBaseProps }, + (_, k) => ` /** Extra base prop ${k}. */\n extraBase${k}?: string;` + ).join('\n'); + const auxTypes = Array.from( + { length: fanOut }, + (_, k) => `export interface Aux${k} {\n aux${k}Value: string;\n aux${k}Count?: number;\n}` + ).join('\n\n'); + return `export interface BaseProps { + /** Stable identifier. */ + id: string; + /** Semantic kind token. */ + kind?: 'alpha' | 'beta' | 'gamma'; + /** Arbitrary metadata bag. */ + meta?: Record; +${extras ? `${extras}\n` : ''}} + +${auxTypes} +`; +} + +function packageTypesSource(p: number, options: VueGenerateOptions): string { + const auxNames = Array.from({ length: options.fanOut }, (_, k) => `Aux${k}`); + const parentImport = + p === 0 + ? `import type { BaseProps, ${auxNames.join(', ')} } from '@bench/types';` + : `import type { Pkg${p - 1}Props } from '@bench/pkg${p - 1}';\nimport type { ${auxNames.join(', ')} } from '@bench/types';`; + const heavyImport = options.heavyLib + ? `import type { HeavyComponent${p % HEAVY_LIB_COMPONENTS}Props } from '@bench/heavy-ui';\n` + : ''; + const parent = p === 0 ? 'BaseProps' : `Pkg${p - 1}Props`; + + const hops: string[] = []; + let current = parent; + for (let h = 0; h < options.chainDepth - 1; h++) { + hops.push(`type Hop${h} = ${current};`); + current = `Hop${h}`; + } + + const fanProps = auxNames + .map((aux, k) => ` /** Fan-out reference ${k}. */\n pkg${p}Fan${k}?: ${aux};`) + .join('\n'); + const heavyProp = options.heavyLib + ? `\n /** Heavy library surface reference. */\n pkg${p}Heavy?: HeavyComponent${p % HEAVY_LIB_COMPONENTS}Props;` + : ''; + + return `${parentImport} +${heavyImport} +${hops.length ? `${hops.join('\n')}\n` : ''} +export interface Pkg${p}Props extends ${current} { +${fanProps}${heavyProp} +} +`; +} + +/** + * An SFC's source. `extraProps` grows the intersection type by one optional prop per save, so a + * simulated save genuinely changes the component's type. + */ +export function vueComponentSource(p: number, i: number, extraProps: number): string { + const extras = Array.from({ length: extraProps }, (_, k) => ` extra${k}?: string;`).join(''); + const propsType = extras ? `Pkg${p}Props & {${extras} }` : `Pkg${p}Props`; + return ` + + +`; +} + +const BASE_COMPILER_OPTIONS = { + target: 'ESNext', + module: 'ESNext', + moduleResolution: 'Bundler', + strict: true, + skipLibCheck: true, + jsx: 'preserve', + lib: ['ESNext', 'DOM'], +}; + +export function generateVueProject(options: VueGenerateOptions): GeneratedVueProject { + const outDir = path.resolve(options.outDir); + fs.rmSync(outDir, { recursive: true, force: true }); + fs.mkdirSync(outDir, { recursive: true }); + + copyVueNodeModules(outDir); + if (options.heavyLib) { + emitHeavyLib(outDir); + } + + const typesDir = path.join(outDir, 'packages', 'types'); + fs.mkdirSync(typesDir, { recursive: true }); + const baseTypesPath = path.join(typesDir, 'index.ts'); + fs.writeFileSync(baseTypesPath, baseTypesSource(options.fanOut, 0)); + + const paths: Record = { '@bench/types': ['packages/types/index.ts'] }; + for (let p = 0; p < options.packages; p++) { + paths[`@bench/pkg${p}`] = [`packages/pkg${p}/types.ts`]; + } + fs.writeFileSync( + path.join(outDir, 'tsconfig.base.json'), + JSON.stringify({ compilerOptions: { ...BASE_COMPILER_OPTIONS, baseUrl: '.', paths } }, null, 2) + ); + + const componentPaths: string[] = []; + const packageConfigPaths: string[] = []; + + for (let p = 0; p < options.packages; p++) { + const pkgDir = path.join(outDir, 'packages', `pkg${p}`); + fs.mkdirSync(path.join(pkgDir, 'src'), { recursive: true }); + fs.writeFileSync(path.join(pkgDir, 'types.ts'), packageTypesSource(p, options)); + + const pkgConfigPath = path.join(pkgDir, 'tsconfig.json'); + fs.writeFileSync( + pkgConfigPath, + JSON.stringify( + { + extends: '../../tsconfig.base.json', + include: ['types.ts', 'src/**/*.ts', 'src/**/*.vue'], + }, + null, + 2 + ) + ); + packageConfigPaths.push(pkgConfigPath); + + for (let i = 0; i < options.componentsPerPackage; i++) { + const componentPath = path.join(pkgDir, 'src', `Comp${p}x${i}.vue`); + fs.writeFileSync(componentPath, vueComponentSource(p, i, 0)); + componentPaths.push(componentPath); + } + } + + const rootConfigPath = path.join(outDir, 'tsconfig.json'); + fs.writeFileSync( + rootConfigPath, + JSON.stringify( + { + files: [], + references: Array.from({ length: options.packages }, (_, p) => ({ + path: `./packages/pkg${p}`, + })), + }, + null, + 2 + ) + ); + + return { outDir, rootConfigPath, packageConfigPaths, baseTypesPath, componentPaths }; +} + +function parseArgs(argv: string[]): VueGenerateOptions { + const get = (flag: string, fallback: string) => { + const idx = argv.indexOf(flag); + return idx >= 0 && argv[idx + 1] ? argv[idx + 1] : fallback; + }; + return { + outDir: get('--out', '../storybook-sandboxes/docgen-perf-vue'), + packages: Number(get('--packages', '4')), + componentsPerPackage: Number(get('--components-per-package', '10')), + chainDepth: Number(get('--chain-depth', '3')), + fanOut: Number(get('--fan-out', '4')), + heavyLib: argv.includes('--heavy-lib'), + }; +} + +if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) { + const options = parseArgs(process.argv.slice(2)); + const start = Date.now(); + const result = generateVueProject(options); + console.log( + `Generated ${options.packages}×${options.componentsPerPackage} Vue components ` + + `into ${result.outDir} in ${Date.now() - start}ms` + ); +} diff --git a/scripts/bench/docgen-perf/paths.ts b/scripts/bench/docgen-perf/paths.ts new file mode 100644 index 000000000000..717094dd89ca --- /dev/null +++ b/scripts/bench/docgen-perf/paths.ts @@ -0,0 +1,13 @@ +/** + * Mirrors SANDBOX_DIRECTORY from scripts/utils/constants.ts. That module deliberately stays on the + * CJS-only `__dirname` global for Playwright compatibility, so the native-Node harness entry points + * here cannot import it. + */ +import * as path from 'node:path'; + +const ROOT_DIRECTORY = path.join(import.meta.dirname, '..', '..', '..'); + +export const SANDBOX_DIRECTORY = + process.env.STORYBOOK_SANDBOX_ROOT && path.isAbsolute(process.env.STORYBOOK_SANDBOX_ROOT) + ? process.env.STORYBOOK_SANDBOX_ROOT + : path.join(ROOT_DIRECTORY, process.env.STORYBOOK_SANDBOX_ROOT || '../storybook-sandboxes'); diff --git a/scripts/bench/docgen-perf/run.ts b/scripts/bench/docgen-perf/run.ts new file mode 100644 index 000000000000..7cd36a5538ae --- /dev/null +++ b/scripts/bench/docgen-perf/run.ts @@ -0,0 +1,457 @@ +/** + * Orchestrator for the per-engine docgen performance suite. Implements the measurement contract in + * scripts/bench/PERF-METHODOLOGY.md: five floor metrics per engine, plain Node, fresh child process + * per measurement, median-of-N with one pinned N recorded in the results. + * + * Topology per engine: + * - Cold medians come from N fresh spawns; warm is the median of the per-save durations inside + * the first repetition's save series; memory metrics also read the first repetition. + * - react-legacy and react-osa are the calibration control pair: both run inside this one + * invocation, their spawn order alternates across repetitions, and the results carry the + * legacy-vs-osa ratio of medians for cold and warm. + * - compodoc is a one-shot CLI, so its adapter runs in-process here and spawns the compodoc CLI + * itself as the measured child; it skips with an explicit message when no binary is resolvable. + * + * A failed engine does not abort the suite: it is recorded as failed, the remaining engines keep + * measuring, and the process exits 1 after the summary. + * + * Run: + * yarn bench:docgen-perf # from scripts/, full profile + * yarn bench:docgen-perf --quick # smoke profile; results marked non-comparable + * yarn bench:docgen-perf --engine react-legacy --engine react-osa + */ +import { spawnSync } from 'node:child_process'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +import { SANDBOX_DIRECTORY } from './paths.ts'; +import { + DEFAULT_PROFILE, + QUICK_PROFILE, + RSS_POLL_INTERVAL_MS, + type ReactScenarioConfig, + type SuiteProfile, + type VueScenarioConfig, +} from './config.ts'; +import { + type CompodocRepetition, + resolveCompodocBinary, + runCompodocRepetition, +} from './engines/compodoc.ts'; +import { mean, median } from './stats.ts'; +import { + type EngineId, + type EngineMetrics, + type EngineResult, + NOT_APPLICABLE, + type SeriesHarnessResult, + type SuiteResults, +} from './types.ts'; + +const ALL_ENGINES: EngineId[] = [ + 'react-legacy', + 'react-legacy-rdt', + 'react-osa', + 'vue-component-meta', + 'compodoc', +]; + +/** react-legacy-rdt carries no budget row and only runs when named via --engine. */ +const DEFAULT_ENGINES: EngineId[] = ['react-legacy', 'react-osa', 'vue-component-meta', 'compodoc']; + +const WORK_ROOT = path.join(SANDBOX_DIRECTORY, 'docgen-perf'); +const OSA_HARNESS = path.join(import.meta.dirname, '../docgen-memory/memory-harness.ts'); +const LEGACY_HARNESS = path.join(import.meta.dirname, 'engines', 'react-legacy.ts'); +const VUE_HARNESS = path.join(import.meta.dirname, 'engines', 'vue-component-meta.ts'); + +interface CliOptions { + quick: boolean; + engines: EngineId[]; + jsonOut: string; +} + +function parseArgs(argv: string[]): CliOptions { + const engines: EngineId[] = []; + for (let i = 0; i < argv.length; i++) { + if (argv[i] === '--engine' && argv[i + 1]) { + const engine = argv[i + 1] as EngineId; + if (!ALL_ENGINES.includes(engine)) { + throw new Error(`--engine must be one of ${ALL_ENGINES.join(', ')}, got "${argv[i + 1]}"`); + } + engines.push(engine); + } + } + const jsonIdx = argv.indexOf('--json'); + return { + quick: argv.includes('--quick'), + engines: engines.length ? engines : DEFAULT_ENGINES, + jsonOut: + jsonIdx >= 0 && argv[jsonIdx + 1] + ? argv[jsonIdx + 1] + : path.join(WORK_ROOT, 'results.json'), + }; +} + +/** + * The control pair alternates spawn order across repetitions so cache warming and thermal drift do + * not consistently favor one side. + */ +function engineOrderForRep(engines: EngineId[], rep: number): EngineId[] { + const order = [...engines]; + const legacyIdx = order.indexOf('react-legacy'); + const osaIdx = order.indexOf('react-osa'); + if (legacyIdx >= 0 && osaIdx >= 0 && rep % 2 === 0) { + order[legacyIdx] = 'react-osa'; + order[osaIdx] = 'react-legacy'; + } + return order; +} + +function reactChildInvocation(engine: EngineId, cfg: ReactScenarioConfig) { + const sizeArgs = [ + '--components', String(cfg.components), + '--variants', String(cfg.variants), + '--props', String(cfg.props), + '--saves', String(cfg.saves), + ]; + if (engine === 'react-osa') { + return { childPath: OSA_HARNESS, args: [...sizeArgs, '--mode', 'refresh', '--scope', 'changed'] }; + } + const parser = engine === 'react-legacy-rdt' ? 'react-docgen-typescript' : 'react-docgen'; + return { childPath: LEGACY_HARNESS, args: [...sizeArgs, '--parser', parser] }; +} + +function vueChildInvocation(scenario: VueScenarioConfig) { + return { + childPath: VUE_HARNESS, + args: [ + '--scenario', scenario.name, + '--packages', String(scenario.packages), + '--components-per-package', String(scenario.componentsPerPackage), + '--chain-depth', String(scenario.chainDepth), + '--fan-out', String(scenario.fanOut), + ...(scenario.heavyLib ? ['--heavy-lib'] : []), + '--saves', String(scenario.saves), + ], + }; +} + +/** + * Spawn one fresh series-harness child (the gate.ts pattern) and read back its result JSON. A + * non-zero exit or a missing/unparseable JSON is a failure. + * + * The new engine children run on native Node type stripping; only the reused docgen-memory harness + * still needs the jiti loader. Under jiti, react-docgen's browserslist dependency fails on its JSON + * data require ("jsReleases.map is not a function"), so the legacy child must NOT run under jiti. + */ +function runSeriesChild( + childPath: string, + childArgs: string[], + outDir: string, + jsonPath: string +): SeriesHarnessResult { + fs.mkdirSync(path.dirname(jsonPath), { recursive: true }); + // Remove any stale result so a crashed run cannot be mistaken for a prior success. + fs.rmSync(jsonPath, { force: true }); + + const nodeArgs = [ + '--expose-gc', + ...(childPath === OSA_HARNESS ? ['--import', 'jiti/register'] : []), + childPath, + ...childArgs, + '--out', + outDir, + '--json', + jsonPath, + ]; + const proc = spawnSync(process.execPath, nodeArgs, { encoding: 'utf8' }); + const output = `${proc.stdout ?? ''}${proc.stderr ?? ''}`; + const tail = output.trim().split('\n').slice(-4).map((line) => ` ${line}`).join('\n'); + + if (proc.status !== 0) { + throw new Error(`child exited with status ${proc.status}:\n${tail}`); + } + if (!fs.existsSync(jsonPath)) { + throw new Error(`child wrote no result JSON at ${jsonPath}:\n${tail}`); + } + return JSON.parse(fs.readFileSync(jsonPath, 'utf8')) as SeriesHarnessResult; +} + +interface SeriesCollector { + coldSamples: number[]; + /** The first repetition's full result; warm and memory metrics read this run. */ + designated?: SeriesHarnessResult; +} + +function seriesMetrics(collector: SeriesCollector | undefined): EngineMetrics { + if (!collector?.designated) { + throw new Error('no completed repetition recorded'); + } + const { designated } = collector; + const warmSamples = designated.samples.map((s) => s.durMs); + if (warmSamples.some((v) => v === undefined)) { + throw new Error('save series carries no durations (child predates latency persistence)'); + } + const transients = designated.samples + .map((s) => (s.retainedHeapMb !== undefined ? s.heapUsedMb - s.retainedHeapMb : undefined)) + .filter((v): v is number => v !== undefined); + if ( + transients.length === 0 || + designated.retainedGrowth === undefined || + designated.retainedSlope === undefined + ) { + throw new Error('retained metrics missing (child must run under --expose-gc)'); + } + return { + coldExtractionMs: { + status: 'measured', + samples: collector.coldSamples, + median: median(collector.coldSamples), + }, + warmExtractionMs: { status: 'measured', samples: warmSamples, median: median(warmSamples) }, + wholeProjectScanMs: NOT_APPLICABLE, + peakTransientMb: { status: 'measured', samples: transients, mean: mean(transients) }, + retainedGrowthMb: { status: 'measured', value: designated.retainedGrowth }, + retainedSlopeMbPerSave: { status: 'measured', value: designated.retainedSlope }, + }; +} + +function compodocMetrics(reps: CompodocRepetition[]): EngineMetrics { + if (reps.length === 0) { + throw new Error('no completed repetition recorded'); + } + const coldSamples = reps.map((r) => r.coldMs); + const warmSamples = reps.map((r) => r.warmMs); + const peaks = reps.map((r) => r.peakRssMb); + return { + coldExtractionMs: { status: 'measured', samples: coldSamples, median: median(coldSamples) }, + warmExtractionMs: { status: 'measured', samples: warmSamples, median: median(warmSamples) }, + // Cold extraction and the whole-project scan are the same full-project run for a one-shot CLI. + wholeProjectScanMs: { status: 'measured', samples: coldSamples, median: median(coldSamples) }, + peakTransientMb: { status: 'measured', samples: peaks, mean: mean(peaks) }, + retainedGrowthMb: NOT_APPLICABLE, + retainedSlopeMbPerSave: NOT_APPLICABLE, + }; +} + +function formatCell(metric: EngineMetrics[keyof EngineMetrics], unit: 'ms' | 'MB'): string { + if (metric.status === 'n/a') { + return 'n/a'; + } + if ('median' in metric) { + return `${metric.median.toFixed(0)}${unit}`; + } + if ('mean' in metric) { + return `${metric.mean.toFixed(0)}${unit}`; + } + return `${metric.value.toFixed(unit === 'MB' ? 1 : 0)}${unit}`; +} + +async function main() { + const options = parseArgs(process.argv.slice(2)); + const profile: SuiteProfile = options.quick ? QUICK_PROFILE : DEFAULT_PROFILE; + + console.log('docgen-perf suite'); + console.log( + ` engines=${options.engines.join(',')} n=${profile.n} comparable=${profile.comparable}` + ); + if (!profile.comparable) { + console.log(' QUICK PROFILE: results are non-comparable smoke numbers'); + } + + const state = new Map(); + const failed = new Map(); + const skipped = new Map(); + const compodocReps: CompodocRepetition[] = []; + + const compodocBinary = options.engines.includes('compodoc') ? resolveCompodocBinary() : undefined; + if (options.engines.includes('compodoc') && !compodocBinary) { + skipped.set( + 'compodoc', + 'no compodoc binary found (workspace node_modules/.bin or PATH); @compodoc/compodoc is a user-project dependency' + ); + console.log(` compodoc: SKIPPED - ${skipped.get('compodoc')}`); + } + + const runSeries = (engine: EngineId, scenarioName: string, rep: number) => { + const invocation = + engine === 'vue-component-meta' + ? vueChildInvocation(profile.vue.find((s) => s.name === scenarioName)!) + : reactChildInvocation(engine, profile.react); + const key = `${engine}/${scenarioName}`; + const scenarioDir = path.join(WORK_ROOT, engine, scenarioName); + console.log(` ${key} (rep ${rep}/${profile.n})…`); + const result = runSeriesChild( + invocation.childPath, + invocation.args, + path.join(scenarioDir, 'project'), + path.join(scenarioDir, `rep${rep}.json`) + ); + const collector = state.get(key) ?? { coldSamples: [] }; + collector.coldSamples.push(result.coldMs); + if (collector.designated === undefined) { + collector.designated = result; + } + state.set(key, collector); + console.log(` cold=${result.coldMs}ms`); + }; + + for (let rep = 1; rep <= profile.n; rep++) { + console.log(`\n=== repetition ${rep}/${profile.n} ===`); + for (const engine of engineOrderForRep(options.engines, rep)) { + if (failed.has(engine) || skipped.has(engine)) { + continue; + } + try { + if (engine === 'compodoc') { + const repetition = await runCompodocRepetition( + compodocBinary as string, + profile.angular, + path.join(WORK_ROOT, 'compodoc'), + RSS_POLL_INTERVAL_MS + ); + compodocReps.push(repetition); + console.log( + ` compodoc: cold=${repetition.coldMs}ms warm=${repetition.warmMs}ms ` + + `peakRss=${repetition.peakRssMb.toFixed(0)}MB` + ); + } else if (engine === 'vue-component-meta') { + for (const scenario of profile.vue) { + runSeries(engine, scenario.name, rep); + } + } else { + runSeries(engine, 'default', rep); + } + } catch (err) { + const reason = err instanceof Error ? err.message : String(err); + failed.set(engine, reason); + console.error(` ${engine} FAILED: ${reason}`); + } + } + } + + const engineResults: SuiteResults['engines'] = {}; + for (const engine of options.engines) { + const skipReason = skipped.get(engine); + if (skipReason) { + engineResults[engine] = { status: 'skipped', reason: skipReason }; + continue; + } + try { + if (engine === 'compodoc') { + engineResults[engine] = { + status: 'measured', + scenarios: { default: { params: { ...profile.angular }, metrics: compodocMetrics(compodocReps) } }, + }; + } else if (engine === 'vue-component-meta') { + const scenarios: Extract['scenarios'] = {}; + for (const scenario of profile.vue) { + scenarios[scenario.name] = { + params: { ...scenario }, + metrics: seriesMetrics(state.get(`${engine}/${scenario.name}`)), + }; + } + engineResults[engine] = { status: 'measured', scenarios }; + } else { + engineResults[engine] = { + status: 'measured', + scenarios: { + default: { params: { ...profile.react }, metrics: seriesMetrics(state.get(`${engine}/default`)) }, + }, + }; + } + } catch (err) { + const reason = failed.get(engine) ?? (err instanceof Error ? err.message : String(err)); + failed.set(engine, reason); + engineResults[engine] = { status: 'failed', reason }; + } + } + + // The calibration reference: react-docgen vs react-osa, both measured in this same invocation. + const ratios: SuiteResults['ratios'] = {}; + const legacy = engineResults['react-legacy']; + const osa = engineResults['react-osa']; + if (legacy?.status === 'measured' && osa?.status === 'measured') { + const legacyMetrics = legacy.scenarios.default.metrics; + const osaMetrics = osa.scenarios.default.metrics; + if (legacyMetrics.coldExtractionMs.status === 'measured' && osaMetrics.coldExtractionMs.status === 'measured') { + ratios.coldLegacyVsOsa = legacyMetrics.coldExtractionMs.median / osaMetrics.coldExtractionMs.median; + } + if (legacyMetrics.warmExtractionMs.status === 'measured' && osaMetrics.warmExtractionMs.status === 'measured') { + ratios.warmLegacyVsOsa = legacyMetrics.warmExtractionMs.median / osaMetrics.warmExtractionMs.median; + } + } + + const results: SuiteResults = { + generatedAt: new Date().toISOString(), + nodeVersion: process.version, + pinnedN: profile.n, + comparable: profile.comparable, + rssPollIntervalMs: RSS_POLL_INTERVAL_MS, + engines: engineResults, + ratios, + }; + + console.log('\nresults'); + const header = ['engine/scenario', 'cold', 'warm', 'scan', 'peak', 'ret-growth', 'ret-slope']; + const rows: string[][] = [header]; + const statusLines: string[] = []; + for (const engine of options.engines) { + const result = engineResults[engine]; + if (!result) { + continue; + } + if (result.status !== 'measured') { + statusLines.push(` ${engine}: ${result.status.toUpperCase()} - ${result.reason.split('\n')[0]}`); + continue; + } + for (const [scenarioName, scenario] of Object.entries(result.scenarios)) { + const m = scenario.metrics; + rows.push([ + `${engine}/${scenarioName}`, + formatCell(m.coldExtractionMs, 'ms'), + formatCell(m.warmExtractionMs, 'ms'), + formatCell(m.wholeProjectScanMs, 'ms'), + formatCell(m.peakTransientMb, 'MB'), + formatCell(m.retainedGrowthMb, 'MB'), + m.retainedSlopeMbPerSave.status === 'measured' + ? `${m.retainedSlopeMbPerSave.value.toFixed(2)}MB/save` + : 'n/a', + ]); + } + } + const widths = header.map((_, col) => Math.max(...rows.map((row) => (row[col] ?? '').length))); + for (const row of rows) { + console.log(` ${row.map((cell, col) => (cell ?? '').padEnd(widths[col])).join(' ')}`); + } + for (const line of statusLines) { + console.log(line); + } + if (ratios.coldLegacyVsOsa !== undefined) { + console.log(` ratio cold legacy/osa: ${ratios.coldLegacyVsOsa.toFixed(2)}`); + } + if (ratios.warmLegacyVsOsa !== undefined) { + console.log(` ratio warm legacy/osa: ${ratios.warmLegacyVsOsa.toFixed(2)}`); + } + if (!profile.comparable) { + console.log(' QUICK PROFILE: results are non-comparable smoke numbers'); + } + + fs.mkdirSync(path.dirname(options.jsonOut), { recursive: true }); + fs.writeFileSync(options.jsonOut, JSON.stringify(results, null, 2)); + console.log(` wrote ${options.jsonOut}`); + + if (failed.size > 0) { + console.error('\ndocgen-perf suite FAILED:'); + for (const [engine, reason] of failed) { + console.error(` - ${engine}: ${reason}`); + } + process.exit(1); + } + console.log('\ndocgen-perf suite completed.'); +} + +main().catch((err) => { + console.error(err); + process.exitCode = 1; +}); diff --git a/scripts/bench/docgen-perf/sampling.ts b/scripts/bench/docgen-perf/sampling.ts new file mode 100644 index 000000000000..eda1048da16c --- /dev/null +++ b/scripts/bench/docgen-perf/sampling.ts @@ -0,0 +1,29 @@ +/** + * Memory sampling shared by the series-harness engine children. Mirrors the docgen-memory harness: + * pre-GC rss/heapUsed is the transient-pressure signal, post-GC heapUsed (only under + * `node --expose-gc`) is the retained signal. + */ +import type { MemorySample } from './types.ts'; + +export const MB = 1024 * 1024; + +export function gc(): void { + if (typeof global.gc === 'function') { + global.gc(); + global.gc(); + } +} + +export function gcAvailable(): boolean { + return typeof global.gc === 'function'; +} + +export function sampleMemory(forceGc: boolean): MemorySample { + const pre = process.memoryUsage(); + let retainedHeapMb: number | undefined; + if (forceGc && gcAvailable()) { + gc(); + retainedHeapMb = process.memoryUsage().heapUsed / MB; + } + return { rssMb: pre.rss / MB, heapUsedMb: pre.heapUsed / MB, retainedHeapMb }; +} diff --git a/scripts/bench/docgen-perf/stats.test.ts b/scripts/bench/docgen-perf/stats.test.ts new file mode 100644 index 000000000000..f06ccd2cd5ed --- /dev/null +++ b/scripts/bench/docgen-perf/stats.test.ts @@ -0,0 +1,56 @@ +import { describe, expect, it } from 'vitest'; + +import { leastSquaresSlope, mean, median } from './stats.ts'; + +describe('median', () => { + it('returns the middle value for odd-length input', () => { + expect(median([3, 1, 2])).toBe(2); + }); + + it('averages the two middle values for even-length input', () => { + expect(median([4, 1, 3, 2])).toBe(2.5); + }); + + it('returns the value itself for a single sample', () => { + expect(median([7])).toBe(7); + }); + + it('does not mutate its input', () => { + const values = [3, 1, 2]; + median(values); + expect(values).toEqual([3, 1, 2]); + }); + + it('throws on empty input', () => { + expect(() => median([])).toThrow('median() requires at least one value'); + }); +}); + +describe('mean', () => { + it('averages the values', () => { + expect(mean([1, 2, 3, 4])).toBe(2.5); + }); + + it('throws on empty input', () => { + expect(() => mean([])).toThrow('mean() requires at least one value'); + }); +}); + +describe('leastSquaresSlope', () => { + it('returns 0 for fewer than two points', () => { + expect(leastSquaresSlope([])).toBe(0); + expect(leastSquaresSlope([5])).toBe(0); + }); + + it('recovers the slope of a perfect line', () => { + expect(leastSquaresSlope([1, 3, 5, 7])).toBe(2); + }); + + it('returns 0 for a flat series', () => { + expect(leastSquaresSlope([4, 4, 4])).toBe(0); + }); + + it('fits noisy data to the least-squares line', () => { + expect(leastSquaresSlope([0, 2, 1, 3])).toBeCloseTo(0.8, 5); + }); +}); diff --git a/scripts/bench/docgen-perf/stats.ts b/scripts/bench/docgen-perf/stats.ts new file mode 100644 index 000000000000..3987009e0d03 --- /dev/null +++ b/scripts/bench/docgen-perf/stats.ts @@ -0,0 +1,36 @@ +/** Statistics helpers for the per-engine docgen performance suite. */ + +/** Median of `values`. Throws on an empty input so a missing series fails loudly. */ +export function median(values: number[]): number { + if (values.length === 0) { + throw new Error('median() requires at least one value'); + } + const sorted = [...values].sort((a, b) => a - b); + const mid = Math.floor(sorted.length / 2); + return sorted.length % 2 === 1 ? sorted[mid] : (sorted[mid - 1] + sorted[mid]) / 2; +} + +/** Arithmetic mean of `values`. Throws on an empty input so a missing series fails loudly. */ +export function mean(values: number[]): number { + if (values.length === 0) { + throw new Error('mean() requires at least one value'); + } + return values.reduce((a, b) => a + b, 0) / values.length; +} + +/** Least-squares slope of `values` vs index, in units-per-step. 0 for fewer than two points. */ +export function leastSquaresSlope(values: number[]): number { + const n = values.length; + if (n < 2) { + return 0; + } + const meanX = (n - 1) / 2; + const meanY = values.reduce((a, b) => a + b, 0) / n; + let num = 0; + let den = 0; + for (let i = 0; i < n; i++) { + num += (i - meanX) * (values[i] - meanY); + den += (i - meanX) ** 2; + } + return den === 0 ? 0 : num / den; +} diff --git a/scripts/bench/docgen-perf/types.ts b/scripts/bench/docgen-perf/types.ts new file mode 100644 index 000000000000..1d288cb501af --- /dev/null +++ b/scripts/bench/docgen-perf/types.ts @@ -0,0 +1,99 @@ +/** + * Shared shapes for the per-engine docgen performance suite. The measurement contract behind these + * shapes is scripts/bench/PERF-METHODOLOGY.md. + */ + +export type EngineId = + | 'react-legacy' + | 'react-legacy-rdt' + | 'react-osa' + | 'vue-component-meta' + | 'compodoc'; + +export interface MemorySample { + rssMb: number; + heapUsedMb: number; + retainedHeapMb?: number; +} + +export interface SaveSample extends MemorySample { + save: number; + durMs: number; +} + +/** + * Subset of a series-harness child's `--json` output that the orchestrator consumes. The + * docgen-memory harness writes a superset of this; the new engine children write exactly this plus + * their own options echo. + */ +export interface SeriesHarnessResult { + coldMs: number; + baseline: MemorySample; + samples: SaveSample[]; + retainedSlope?: number; + retainedGrowth?: number; + avgTransient?: number; +} + +/** A latency metric: median of repeated samples (fresh process each, for cold/scan). */ +export interface LatencyMetric { + status: 'measured'; + samples: number[]; + median: number; +} + +/** A memory metric aggregated as the mean of a per-save (or per-run) series. */ +export interface SeriesMeanMetric { + status: 'measured'; + samples: number[]; + mean: number; +} + +/** A single-valued metric read from one run's series (retained growth, retained slope). */ +export interface ValueMetric { + status: 'measured'; + value: number; +} + +/** The explicit marker for a metric that does not apply to an engine; never a faked equivalent. */ +export interface NotApplicable { + status: 'n/a'; +} + +export const NOT_APPLICABLE: NotApplicable = { status: 'n/a' }; + +export interface EngineMetrics { + coldExtractionMs: LatencyMetric | NotApplicable; + warmExtractionMs: LatencyMetric | NotApplicable; + wholeProjectScanMs: LatencyMetric | NotApplicable; + peakTransientMb: SeriesMeanMetric | NotApplicable; + retainedGrowthMb: ValueMetric | NotApplicable; + retainedSlopeMbPerSave: ValueMetric | NotApplicable; +} + +export interface ScenarioResult { + params: Record; + metrics: EngineMetrics; +} + +export type EngineResult = + | { status: 'measured'; scenarios: Record } + | { status: 'skipped'; reason: string } + | { status: 'failed'; reason: string }; + +export interface SuiteResults { + generatedAt: string; + nodeVersion: string; + /** The one pinned N; numbers taken at different N are not comparable. */ + pinnedN: number; + /** False for --quick smoke runs, whose numbers must never be compared against real runs. */ + comparable: boolean; + /** Sampling interval for the compodoc child's externally-polled peak RSS. */ + rssPollIntervalMs: number; + engines: Partial>; + /** + * React legacy (react-docgen) median divided by react-osa median, both measured in this same + * invocation. The calibration reference for every other engine's budgets. + */ + ratios: { coldLegacyVsOsa?: number; warmLegacyVsOsa?: number }; +} diff --git a/scripts/package.json b/scripts/package.json index 685753fdef20..f448da8dc779 100644 --- a/scripts/package.json +++ b/scripts/package.json @@ -6,6 +6,7 @@ "scripts": { "bench-packages": "jiti ./bench/bench-packages.ts", "bench:docgen-memory": "node --import jiti/register ./bench/docgen-memory/gate.ts", + "bench:docgen-perf": "node ./bench/docgen-perf/run.ts", "build-package": "jiti ./build-package.ts", "check": "node ./check/check-package.ts", "check-package": "jiti ./check-package.ts", @@ -153,6 +154,7 @@ "typescript-native": "npm:typescript@^7.0.2", "uuid": "^9.0.1", "vitest": "^4.1.5", + "vue-component-meta": "^3.2.7", "wait-on": "^8.0.3", "window-size": "^1.1.1", "yaml": "^2.8.1", diff --git a/yarn.lock b/yarn.lock index 4f92e9aaeda6..859eb2484ddb 100644 --- a/yarn.lock +++ b/yarn.lock @@ -10512,6 +10512,7 @@ __metadata: verdaccio: "npm:^5.33.0" verdaccio-auth-memory: "npm:^10.3.1" vitest: "npm:^4.1.5" + vue-component-meta: "npm:^3.2.7" wait-on: "npm:^8.0.3" window-size: "npm:^1.1.1" yaml: "npm:^2.8.1" From 98e69691425af6e27b19f61ad3338a1dc1992229 Mon Sep 17 00:00:00 2001 From: Valentin Palkovic Date: Fri, 24 Jul 2026 22:17:43 +0200 Subject: [PATCH 04/14] Bench: Fix what the docgen perf suite measures Review of the harness turned up four ways it reported numbers that did not mean what they claimed. Warm latency and both memory metrics read repetition 1's save series. That repetition pays for a cold module graph and a cold page cache and runs several times slower than the rest: react-osa cold measured 2199ms on rep 1 and 552ms on rep 2. Only cold latency was protected, by its median. The series now comes from the repetition whose cold sample is the median. An engine that failed part-way through its repetitions still landed in the results as `measured`, carrying fewer samples than the pinned N the same file recorded. It now stays failed. The compodoc RSS poller called `.trim()` on `spawnSync().stdout`, which is undefined when `ps` is missing. That throws from a timer callback, where no per-engine catch can reach it, so one missing binary killed the whole run. `--json` took the next argv entry without checking it, so `--json --quick` wrote a file named `--quick`. Flag values are now validated. The legacy React warm sample re-extracts one component, but the legacy parsers are reachable only through generator.ts, which invalidates every cache and re-extracts every component; buildDocgen.ts hardcodes react-component-meta, so the single-file worker never runs them. The warm ratio therefore understates legacy's real per-save cost by roughly the component count. react-legacy.ts gains `--scope all` for the production-shaped number, and the note says what the equal-work ratio does and does not describe. Also: the note claimed as missing several things this PR ships, and left Vue's entry point unpinned after Vue's harness landed. Retained/transient derivation was written out three times and now lives in stats.ts. `vue` was resolved by hoisting alone and is now declared. Dropped three fields nothing reads. One thing left alone: oxfmt's ignore list contains a bare `bench`, which exempts all of scripts/bench from format checks. Narrowing it would reformat files outside this PR. --- scripts/bench/PERF-METHODOLOGY.md | 25 +++- scripts/bench/docgen-memory/gate.ts | 4 +- scripts/bench/docgen-memory/memory-harness.ts | 38 +----- scripts/bench/docgen-perf/budgets.ts | 18 ++- scripts/bench/docgen-perf/engines/compodoc.ts | 5 + .../bench/docgen-perf/engines/react-legacy.ts | 37 +++--- .../docgen-perf/engines/vue-component-meta.ts | 15 +-- .../bench/docgen-perf/generators/angular.ts | 9 +- scripts/bench/docgen-perf/generators/vue.ts | 13 +- scripts/bench/docgen-perf/run.ts | 115 ++++++++++++------ scripts/bench/docgen-perf/stats.ts | 38 ++++++ scripts/bench/docgen-perf/types.ts | 1 - scripts/package.json | 1 + yarn.lock | 1 + 14 files changed, 195 insertions(+), 125 deletions(-) diff --git a/scripts/bench/PERF-METHODOLOGY.md b/scripts/bench/PERF-METHODOLOGY.md index 03aaefd77175..85b67dcb452b 100644 --- a/scripts/bench/PERF-METHODOLOGY.md +++ b/scripts/bench/PERF-METHODOLOGY.md @@ -16,7 +16,11 @@ Each engine is measured through a directly callable entry point: Both wrappers cache per file for the life of the process and expose only global invalidation; a simulated save must invalidate before re-parsing or the warm sample is a cache hit. `react-docgen` is the budgeted legacy control; `react-docgen-typescript` is measurable through the same wrappers but carries no budget row. - Angular: a standalone Compodoc CLI run. -- Vue, Svelte, and web components: their extraction engines are plain functions callable from Node; exact entry points get pinned when each engine's harness lands. + Storybook shells out to the same CLI (`runCompodoc`, `code/frameworks/angular/src/builders/utils/run-compodoc.ts`), so the harness measures the tool the framework actually runs. +- Vue: `createChecker` and `createCheckerByJson` from `vue-component-meta`, with the checker options the Vite plugin passes (`code/frameworks/vue3-vite/src/plugins/vue-component-meta.ts`). + One caveat the budgets must respect: that plugin falls back to `createCheckerByJson` whenever the project's root tsconfig declares `references`, because `vue-component-meta` does not resolve them. + The harness's `workspace` and `base-type-touch` scenarios drive `createChecker` at a package tsconfig, which measures the engine rather than the plugin's path for that project shape. +- Svelte and web components: their extraction engines are plain functions callable from Node; exact entry points get pinned when each engine's harness lands. ## Metrics @@ -33,8 +37,9 @@ Five metrics per engine, all candidates for gating: Compodoc maps onto these differently because it is a fresh CLI process per run: cold extraction and whole-project scan are the same full-project measurement, warm extraction is a second full run after touching one file, and peak memory is the child's peak RSS sampled from outside the process. Whether the retained-series metrics mean anything for a fresh-process engine is settled when its baselines are recorded; until then its leak cells stay placeholders. -The memory and leak half exists today in `scripts/bench/docgen-memory/` (`yarn bench:docgen-memory` from `scripts/`). -The latency half is new: the current harness measures per-save duration but throws it away, so recording cold, warm, and scan timings is work for the per-engine harness. +Two harnesses implement this. +`scripts/bench/docgen-memory/` (`yarn bench:docgen-memory`) gates React memory and is the only tier that fails a build today. +`scripts/bench/docgen-perf/` (`yarn bench:docgen-perf`) records all five metrics for every engine and gates nothing yet, because no budget values exist. ## Determinism method @@ -49,8 +54,10 @@ The latency half is new: the current harness measures per-save duration but thro Each latency metric records a median, never a single-run number. Cold extraction and whole-project scan yield one sample per fresh process, so their N samples come from N spawns. Warm extraction records the median of the per-save durations inside a single run's save series. + That run is the repetition whose cold sample is the median, never the first: repetition 1 pays for a cold module graph and a cold page cache, and measures several times slower than the rest. The harness pins one N for all engines and records it with the results; numbers taken at different N are not comparable. - None of this exists today - the current gate runs each configuration exactly once. + An engine that fails part-way through its repetitions is reported as failed, never as measured at an N it did not reach. + `scripts/bench/docgen-perf/run.ts` implements this; the memory gate still runs each configuration exactly once. - **Series statistics for leak metrics.** Retained slope is a least-squares fit over the save series; retained growth is the delta between the final retained sample and the pre-run baseline. Both read one run's series instead of repeated runs. @@ -73,6 +80,13 @@ The latency half is new: the current harness measures per-save duration but thro Timing budgets are ratios or slopes, never absolute milliseconds; absolute wall-clock on shared CI executors is too noisy to gate. A timing ratio divides the median of one side by the median of the other, both measured in the same job. The calibration reference for React is the legacy-vs-new-engine ratio measured in the same run on the same machine. + +The warm half of that ratio carries a caveat the baseline work must settle. +Both sides re-extract one changed component per save, which compares the engines on equal work. +Only the new engine has a production path shaped that way: `buildDocgenPayload` hardcodes `react-component-meta`, so the single-component docgen worker never runs a legacy parser. +The legacy parsers are reachable only through `generator.ts`, which invalidates every cache and re-extracts every component on each manifest build. +A real legacy save therefore costs the whole project, not one file, and the recorded warm ratio understates legacy's true per-save cost by roughly the component count. +Run `react-legacy.ts --scope all` for the production-shaped number; the equal-work ratio is the engine comparison, not a saving a user would feel. Engines without a second implementation in the same job get their timing reference picked when their baselines are recorded; until then their timing budgets stay placeholders. Memory budgets stay absolute megabytes with generous headroom: budgets sit well above observed values so the gate is not flaky, while still failing hard on a real regression. Every engine must also carry its own negative control - a configuration that must fail, proving the gate can catch the regression class it exists for. @@ -112,11 +126,12 @@ All three build on existing tooling and add no new dependencies: ## Budgets table skeleton The baseline work fills in the values; until then every cell is a placeholder. +The exception is react-osa's memory row, which the docgen memory gate already enforces from `scripts/bench/docgen-perf/budgets.ts`. | Engine | Cold extraction | Warm extraction | Whole-project scan | Peak memory (transient) | Retained growth | Retained slope | Negative control | Tier | | ----------------------------------------- | --------------- | --------------- | ------------------ | ----------------------- | --------------- | -------------- | ---------------- | ---------- | | react-legacy (react-docgen, control) | TBD (1.12) | TBD (1.12) | n/a | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | -| react-osa (react-component-meta, control) | TBD (1.12) | TBD (1.12) | n/a | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | +| react-osa (ComponentMetaManager, control) | TBD (1.12) | TBD (1.12) | n/a | 90MB | 60MB | 3MB/save | OOM (gate.ts) | daily | | vue-component-meta | TBD (1.12) | TBD (1.12) | n/a | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | | compodoc | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | | svelte (stretch) | TBD (1.12) | TBD (1.12) | n/a | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | diff --git a/scripts/bench/docgen-memory/gate.ts b/scripts/bench/docgen-memory/gate.ts index 97ca2d41bc69..01af6b4d775a 100644 --- a/scripts/bench/docgen-memory/gate.ts +++ b/scripts/bench/docgen-memory/gate.ts @@ -23,7 +23,7 @@ import { spawnSync } from 'node:child_process'; import * as fs from 'node:fs'; import * as path from 'node:path'; -import { MEMORY_BUDGETS, type MemoryBudgets } from '../docgen-perf/budgets.ts'; +import { type MemoryBudgets, memoryBudgetsFor } from '../docgen-perf/budgets.ts'; import { SANDBOX_DIRECTORY } from '../../utils/constants.ts'; interface HarnessResult { @@ -70,7 +70,7 @@ const CONFIGS: GateConfig[] = [ '--scope', 'changed', ], outSubdir: 'changed-scope', - budgets: MEMORY_BUDGETS['react-osa'], + budgets: memoryBudgetsFor('react-osa'), }, { // Positive control: the live per-edit workload survives under a tight cap BECAUSE the shared diff --git a/scripts/bench/docgen-memory/memory-harness.ts b/scripts/bench/docgen-memory/memory-harness.ts index 928ab7782312..5c5c56eb20e0 100644 --- a/scripts/bench/docgen-memory/memory-harness.ts +++ b/scripts/bench/docgen-memory/memory-harness.ts @@ -37,6 +37,7 @@ import * as path from 'node:path'; import ts from 'typescript'; import { SANDBOX_DIRECTORY } from '../../utils/constants.ts'; +import { leastSquaresSlope, summarizeSeries } from '../docgen-perf/stats.ts'; import { componentSource, generateProject } from './generate-project.ts'; /** @@ -173,23 +174,6 @@ function sampleMemory(forceGc: boolean): { rssMb: number; heapUsedMb: number; re return { rssMb: pre.rss / MB, heapUsedMb: pre.heapUsed / MB, retainedHeapMb }; } -/** Least-squares slope of `values` vs index, in units-per-save. */ -function slopePerSave(values: number[]): number { - const n = values.length; - if (n < 2) { - return 0; - } - const meanX = (n - 1) / 2; - const meanY = values.reduce((a, b) => a + b, 0) / n; - let num = 0; - let den = 0; - for (let i = 0; i < n; i++) { - num += (i - meanX) * (values[i] - meanY); - den += (i - meanX) ** 2; - } - return den === 0 ? 0 : num / den; -} - function buildEntries(componentPaths: string[], storyPaths: string[]): StoryRef[] { return componentPaths.map((componentPath, i) => ({ storyPath: storyPaths[i], @@ -367,23 +351,9 @@ async function main() { const rssValues = samples.map((s) => s.rssMb); const peakRss = Math.max(baseline.rssMb, ...rssValues); const finalRss = rssValues.at(-1) ?? baseline.rssMb; - const rssSlope = slopePerSave(rssValues); - - const retainedValues = samples - .map((s) => s.retainedHeapMb) - .filter((v): v is number => v !== undefined); - const retainedSlope = retainedValues.length ? slopePerSave(retainedValues) : undefined; - const retainedGrowth = - retainedValues.length && baseline.retainedHeapMb !== undefined - ? (retainedValues.at(-1) as number) - baseline.retainedHeapMb - : undefined; - - const transients = samples - .map((s) => (s.retainedHeapMb !== undefined ? s.heapUsedMb - s.retainedHeapMb : undefined)) - .filter((v): v is number => v !== undefined); - const avgTransient = transients.length - ? transients.reduce((a, b) => a + b, 0) / transients.length - : undefined; + const rssSlope = leastSquaresSlope(rssValues); + + const { retainedSlope, retainedGrowth, avgTransient } = summarizeSeries(samples, baseline); console.log('\nsummary'); console.log(` peak rss: ${peakRss.toFixed(0)}MB`); diff --git a/scripts/bench/docgen-perf/budgets.ts b/scripts/bench/docgen-perf/budgets.ts index cdcd5a4adf9e..a5d6789bfe4e 100644 --- a/scripts/bench/docgen-perf/budgets.ts +++ b/scripts/bench/docgen-perf/budgets.ts @@ -1,7 +1,7 @@ /** - * Budget values shared between the docgen-memory gate and the per-engine perf suite. Budgets are - * derived per engine from that engine's own baseline runs and are never ported between engines; - * missing engines get their rows when their baselines are recorded. + * Memory budget values for the engine gates. Budgets are derived per engine from that engine's own + * baseline runs and are never ported between engines; missing engines get their rows when their + * baselines are recorded. */ import type { EngineId } from './types.ts'; @@ -21,3 +21,15 @@ export interface MemoryBudgets { export const MEMORY_BUDGETS: Partial> = { 'react-osa': { maxRetainedGrowthMb: 60, maxTransientMb: 90, maxRetainedSlopeMb: 3 }, }; + +/** + * Budgets for `engine`, or a hard failure. A gate that reads a missing row would assert nothing and + * still pass, so an engine without recorded budgets must not reach a gate config at all. + */ +export function memoryBudgetsFor(engine: EngineId): MemoryBudgets { + const budgets = MEMORY_BUDGETS[engine]; + if (!budgets) { + throw new Error(`no memory budgets recorded for "${engine}"`); + } + return budgets; +} diff --git a/scripts/bench/docgen-perf/engines/compodoc.ts b/scripts/bench/docgen-perf/engines/compodoc.ts index 5c11b5478ac8..f4f616195593 100644 --- a/scripts/bench/docgen-perf/engines/compodoc.ts +++ b/scripts/bench/docgen-perf/engines/compodoc.ts @@ -59,7 +59,12 @@ function runCompodocOnce( if (child.pid === undefined) { return; } + // `ps` is absent on Windows, and it exits non-zero once the child is gone. Either way this + // throws from a timer callback if it is not guarded, which no per-engine catch can reach. const ps = spawnSync('ps', ['-o', 'rss=', '-p', String(child.pid)], { encoding: 'utf8' }); + if (ps.error || typeof ps.stdout !== 'string') { + return; + } const rssKb = Number(ps.stdout.trim()); if (Number.isFinite(rssKb) && rssKb > peakRssKb) { peakRssKb = rssKb; diff --git a/scripts/bench/docgen-perf/engines/react-legacy.ts b/scripts/bench/docgen-perf/engines/react-legacy.ts index a228747f8517..0e85ea135ffa 100644 --- a/scripts/bench/docgen-perf/engines/react-legacy.ts +++ b/scripts/bench/docgen-perf/engines/react-legacy.ts @@ -22,7 +22,7 @@ import * as path from 'node:path'; import { SANDBOX_DIRECTORY } from '../paths.ts'; import { componentSource, generateProject } from '../../docgen-memory/generate-project.ts'; import { gcAvailable, sampleMemory } from '../sampling.ts'; -import { leastSquaresSlope, mean } from '../stats.ts'; +import { summarizeSeries } from '../stats.ts'; import type { SaveSample } from '../types.ts'; /** @@ -75,6 +75,12 @@ interface HarnessOptions { variants: number; props: number; saves: number; + /** + * Which components to re-extract per save. Mirrors the docgen-memory harness's flag. + * all – re-extract every component, the shape `generator.ts` actually runs. + * changed – re-extract only the component whose file changed. + */ + scope: 'all' | 'changed'; outDir: string; jsonOut?: string; } @@ -88,12 +94,17 @@ function parseArgs(argv: string[]): HarnessOptions { if (parser !== 'react-docgen' && parser !== 'react-docgen-typescript') { throw new Error(`--parser must be "react-docgen" or "react-docgen-typescript", got "${parser}"`); } + const scope = get('--scope', 'changed'); + if (scope !== 'all' && scope !== 'changed') { + throw new Error(`--scope must be "all" or "changed", got "${scope}"`); + } return { parser, components: Number(get('--components', '300')), variants: Number(get('--variants', '4')), props: Number(get('--props', '10')), saves: Number(get('--saves', '20')), + scope, outDir: get('--out', path.join(SANDBOX_DIRECTORY, 'docgen-perf', 'react-legacy', 'project')), jsonOut: argv.indexOf('--json') >= 0 ? get('--json', '') : undefined, }; @@ -115,7 +126,8 @@ async function main() { console.log(`react-legacy harness (${options.parser})`); console.log( - ` components=${options.components} variants=${options.variants} props=${options.props} saves=${options.saves}` + ` components=${options.components} variants=${options.variants} props=${options.props} ` + + `saves=${options.saves} scope=${options.scope}` ); if (!gcAvailable()) { console.log(' (run with `node --expose-gc` to measure retained heap; continuing without it)'); @@ -187,7 +199,13 @@ async function main() { invalidate(); const saveStart = Date.now(); - await extractOne(i); + if (options.scope === 'all') { + for (let c = 0; c < options.components; c++) { + await extractOne(c); + } + } else { + await extractOne(i); + } const durMs = Date.now() - saveStart; const mem = sampleMemory(true); @@ -201,18 +219,7 @@ async function main() { ); } - const retainedValues = samples - .map((s) => s.retainedHeapMb) - .filter((v): v is number => v !== undefined); - const retainedSlope = retainedValues.length ? leastSquaresSlope(retainedValues) : undefined; - const retainedGrowth = - retainedValues.length && baseline.retainedHeapMb !== undefined - ? (retainedValues.at(-1) as number) - baseline.retainedHeapMb - : undefined; - const transients = samples - .map((s) => (s.retainedHeapMb !== undefined ? s.heapUsedMb - s.retainedHeapMb : undefined)) - .filter((v): v is number => v !== undefined); - const avgTransient = transients.length ? mean(transients) : undefined; + const { retainedSlope, retainedGrowth, avgTransient } = summarizeSeries(samples, baseline); console.log('\nsummary'); console.log(` cold pass: ${coldMs}ms`); diff --git a/scripts/bench/docgen-perf/engines/vue-component-meta.ts b/scripts/bench/docgen-perf/engines/vue-component-meta.ts index 79d031fe48b0..4fcf73d131ea 100644 --- a/scripts/bench/docgen-perf/engines/vue-component-meta.ts +++ b/scripts/bench/docgen-perf/engines/vue-component-meta.ts @@ -33,7 +33,7 @@ import { import { SANDBOX_DIRECTORY } from '../paths.ts'; import { baseTypesSource, generateVueProject, vueComponentSource } from '../generators/vue.ts'; import { gcAvailable, sampleMemory } from '../sampling.ts'; -import { leastSquaresSlope, mean } from '../stats.ts'; +import { summarizeSeries } from '../stats.ts'; import type { SaveSample } from '../types.ts'; type Scenario = 'flat' | 'workspace' | 'base-type-touch'; @@ -178,18 +178,7 @@ async function main() { ); } - const retainedValues = samples - .map((s) => s.retainedHeapMb) - .filter((v): v is number => v !== undefined); - const retainedSlope = retainedValues.length ? leastSquaresSlope(retainedValues) : undefined; - const retainedGrowth = - retainedValues.length && baseline.retainedHeapMb !== undefined - ? (retainedValues.at(-1) as number) - baseline.retainedHeapMb - : undefined; - const transients = samples - .map((s) => (s.retainedHeapMb !== undefined ? s.heapUsedMb - s.retainedHeapMb : undefined)) - .filter((v): v is number => v !== undefined); - const avgTransient = transients.length ? mean(transients) : undefined; + const { retainedSlope, retainedGrowth, avgTransient } = summarizeSeries(samples, baseline); console.log('\nsummary'); console.log(` cold pass: ${coldMs}ms`); diff --git a/scripts/bench/docgen-perf/generators/angular.ts b/scripts/bench/docgen-perf/generators/angular.ts index eb8b5fdab106..aa4328e20341 100644 --- a/scripts/bench/docgen-perf/generators/angular.ts +++ b/scripts/bench/docgen-perf/generators/angular.ts @@ -6,7 +6,7 @@ * real framework) just so type resolution stays clean. * * Run directly: - * node --import jiti/register scripts/bench/docgen-perf/generators/angular.ts --out ../storybook-sandboxes/docgen-perf-angular --components 100 + * node scripts/bench/docgen-perf/generators/angular.ts --out ../storybook-sandboxes/docgen-perf-angular --components 100 */ import * as fs from 'node:fs'; import * as path from 'node:path'; @@ -21,7 +21,6 @@ export interface AngularGenerateOptions { export interface GeneratedAngularProject { outDir: string; - configPath: string; /** Absolute component file paths, in component order. */ componentPaths: string[]; } @@ -104,8 +103,8 @@ export function generateAngularProject(options: AngularGenerateOptions): Generat emitFakeAngularCore(outDir); - const configPath = path.join(outDir, 'tsconfig.json'); - fs.writeFileSync(configPath, JSON.stringify(TSCONFIG, null, 2)); + // The compodoc adapter passes `-p tsconfig.json` with the project dir as cwd. + fs.writeFileSync(path.join(outDir, 'tsconfig.json'), JSON.stringify(TSCONFIG, null, 2)); const componentPaths: string[] = []; for (let i = 0; i < options.components; i++) { @@ -114,7 +113,7 @@ export function generateAngularProject(options: AngularGenerateOptions): Generat componentPaths.push(componentPath); } - return { outDir, configPath, componentPaths }; + return { outDir, componentPaths }; } function parseArgs(argv: string[]): AngularGenerateOptions { diff --git a/scripts/bench/docgen-perf/generators/vue.ts b/scripts/bench/docgen-perf/generators/vue.ts index 5b0e1e8619ea..e8054bab9d7d 100644 --- a/scripts/bench/docgen-perf/generators/vue.ts +++ b/scripts/bench/docgen-perf/generators/vue.ts @@ -19,7 +19,7 @@ * and the scenario configs are committed. * * Run directly: - * node --import jiti/register scripts/bench/docgen-perf/generators/vue.ts --out ../storybook-sandboxes/docgen-perf-vue --packages 4 --components-per-package 10 + * node scripts/bench/docgen-perf/generators/vue.ts --out ../storybook-sandboxes/docgen-perf-vue --packages 4 --components-per-package 10 */ import { createRequire } from 'node:module'; import * as fs from 'node:fs'; @@ -43,9 +43,7 @@ export interface VueGenerateOptions { export interface GeneratedVueProject { outDir: string; - /** Root tsconfig (references only). Absent in flat layout. */ - rootConfigPath?: string; - /** Per-package tsconfig paths, in package order. Absent in flat layout. */ + /** Per-package tsconfig paths, in package order. */ packageConfigPaths: string[]; /** Path of the shared base-type module (the base-type-touch scenario's save target). */ baseTypesPath: string; @@ -249,9 +247,10 @@ export function generateVueProject(options: VueGenerateOptions): GeneratedVuePro } } - const rootConfigPath = path.join(outDir, 'tsconfig.json'); + // The root config declares `references` but no package sets `composite`, so `tsc -b` would reject + // this tree. It is here because the production Vite plugin branches on `references` being present. fs.writeFileSync( - rootConfigPath, + path.join(outDir, 'tsconfig.json'), JSON.stringify( { files: [], @@ -264,7 +263,7 @@ export function generateVueProject(options: VueGenerateOptions): GeneratedVuePro ) ); - return { outDir, rootConfigPath, packageConfigPaths, baseTypesPath, componentPaths }; + return { outDir, packageConfigPaths, baseTypesPath, componentPaths }; } function parseArgs(argv: string[]): VueGenerateOptions { diff --git a/scripts/bench/docgen-perf/run.ts b/scripts/bench/docgen-perf/run.ts index 7cd36a5538ae..ac40953e4ffc 100644 --- a/scripts/bench/docgen-perf/run.ts +++ b/scripts/bench/docgen-perf/run.ts @@ -5,7 +5,9 @@ * * Topology per engine: * - Cold medians come from N fresh spawns; warm is the median of the per-save durations inside - * the first repetition's save series; memory metrics also read the first repetition. + * one repetition's save series, and the memory metrics read that same repetition. The one that + * supplies them is the repetition whose cold sample is the median, because repetition 1 pays + * for a cold module graph and a cold page cache and is not representative. * - react-legacy and react-osa are the calibration control pair: both run inside this one * invocation, their spawn order alternates across repetitions, and the results carry the * legacy-vs-osa ratio of medians for cold and warm. @@ -70,25 +72,37 @@ interface CliOptions { jsonOut: string; } +/** + * The value after `flag`, or undefined when the flag is absent. A missing value - end of argv, or + * the next flag - throws instead of silently consuming the next flag as a value. + */ +function valueAfter(argv: string[], flag: string, at = argv.indexOf(flag)): string | undefined { + if (at < 0) { + return undefined; + } + const value = argv[at + 1]; + if (value === undefined || value.startsWith('--')) { + throw new Error(`${flag} requires a value`); + } + return value; +} + function parseArgs(argv: string[]): CliOptions { const engines: EngineId[] = []; for (let i = 0; i < argv.length; i++) { - if (argv[i] === '--engine' && argv[i + 1]) { - const engine = argv[i + 1] as EngineId; - if (!ALL_ENGINES.includes(engine)) { - throw new Error(`--engine must be one of ${ALL_ENGINES.join(', ')}, got "${argv[i + 1]}"`); - } - engines.push(engine); + if (argv[i] !== '--engine') { + continue; + } + const engine = valueAfter(argv, '--engine', i) as EngineId; + if (!ALL_ENGINES.includes(engine)) { + throw new Error(`--engine must be one of ${ALL_ENGINES.join(', ')}, got "${engine}"`); } + engines.push(engine); } - const jsonIdx = argv.indexOf('--json'); return { quick: argv.includes('--quick'), engines: engines.length ? engines : DEFAULT_ENGINES, - jsonOut: - jsonIdx >= 0 && argv[jsonIdx + 1] - ? argv[jsonIdx + 1] - : path.join(WORK_ROOT, 'results.json'), + jsonOut: valueAfter(argv, '--json') ?? path.join(WORK_ROOT, 'results.json'), }; } @@ -118,7 +132,9 @@ function reactChildInvocation(engine: EngineId, cfg: ReactScenarioConfig) { return { childPath: OSA_HARNESS, args: [...sizeArgs, '--mode', 'refresh', '--scope', 'changed'] }; } const parser = engine === 'react-legacy-rdt' ? 'react-docgen-typescript' : 'react-docgen'; - return { childPath: LEGACY_HARNESS, args: [...sizeArgs, '--parser', parser] }; + // Both sides re-extract one component per save, which measures the engines against each other. + // It is not what a real legacy save costs - see the calibration caveat in PERF-METHODOLOGY.md. + return { childPath: LEGACY_HARNESS, args: [...sizeArgs, '--parser', parser, '--scope', 'changed'] }; } function vueChildInvocation(scenario: VueScenarioConfig) { @@ -177,21 +193,27 @@ function runSeriesChild( return JSON.parse(fs.readFileSync(jsonPath, 'utf8')) as SeriesHarnessResult; } -interface SeriesCollector { - coldSamples: number[]; - /** The first repetition's full result; warm and memory metrics read this run. */ - designated?: SeriesHarnessResult; +/** + * Warm latency and both memory metrics read one repetition's save series. Repetition 1 is + * systematically the slowest - it pays for a cold module graph and a cold OS page cache - so the + * representative run is the one whose cold sample is the median, never simply the first. Cold + * latency is already protected by taking the median across repetitions. + */ +function designatedRep(reps: SeriesHarnessResult[]): SeriesHarnessResult { + const byCold = [...reps].sort((a, b) => a.coldMs - b.coldMs); + return byCold[Math.floor((byCold.length - 1) / 2)]; } -function seriesMetrics(collector: SeriesCollector | undefined): EngineMetrics { - if (!collector?.designated) { +function seriesMetrics(reps: SeriesHarnessResult[] | undefined, expectedN: number): EngineMetrics { + if (!reps || reps.length === 0) { throw new Error('no completed repetition recorded'); } - const { designated } = collector; - const warmSamples = designated.samples.map((s) => s.durMs); - if (warmSamples.some((v) => v === undefined)) { - throw new Error('save series carries no durations (child predates latency persistence)'); + if (reps.length !== expectedN) { + throw new Error(`recorded ${reps.length} repetitions, expected the pinned ${expectedN}`); } + const designated = designatedRep(reps); + const coldSamples = reps.map((r) => r.coldMs); + const warmSamples = designated.samples.map((s) => s.durMs); const transients = designated.samples .map((s) => (s.retainedHeapMb !== undefined ? s.heapUsedMb - s.retainedHeapMb : undefined)) .filter((v): v is number => v !== undefined); @@ -203,11 +225,7 @@ function seriesMetrics(collector: SeriesCollector | undefined): EngineMetrics { throw new Error('retained metrics missing (child must run under --expose-gc)'); } return { - coldExtractionMs: { - status: 'measured', - samples: collector.coldSamples, - median: median(collector.coldSamples), - }, + coldExtractionMs: { status: 'measured', samples: coldSamples, median: median(coldSamples) }, warmExtractionMs: { status: 'measured', samples: warmSamples, median: median(warmSamples) }, wholeProjectScanMs: NOT_APPLICABLE, peakTransientMb: { status: 'measured', samples: transients, mean: mean(transients) }, @@ -216,10 +234,13 @@ function seriesMetrics(collector: SeriesCollector | undefined): EngineMetrics { }; } -function compodocMetrics(reps: CompodocRepetition[]): EngineMetrics { +function compodocMetrics(reps: CompodocRepetition[], expectedN: number): EngineMetrics { if (reps.length === 0) { throw new Error('no completed repetition recorded'); } + if (reps.length !== expectedN) { + throw new Error(`recorded ${reps.length} repetitions, expected the pinned ${expectedN}`); + } const coldSamples = reps.map((r) => r.coldMs); const warmSamples = reps.map((r) => r.warmMs); const peaks = reps.map((r) => r.peakRssMb); @@ -259,7 +280,7 @@ async function main() { console.log(' QUICK PROFILE: results are non-comparable smoke numbers'); } - const state = new Map(); + const state = new Map(); const failed = new Map(); const skipped = new Map(); const compodocReps: CompodocRepetition[] = []; @@ -287,12 +308,9 @@ async function main() { path.join(scenarioDir, 'project'), path.join(scenarioDir, `rep${rep}.json`) ); - const collector = state.get(key) ?? { coldSamples: [] }; - collector.coldSamples.push(result.coldMs); - if (collector.designated === undefined) { - collector.designated = result; - } - state.set(key, collector); + const reps = state.get(key) ?? []; + reps.push(result); + state.set(key, reps); console.log(` cold=${result.coldMs}ms`); }; @@ -304,8 +322,11 @@ async function main() { } try { if (engine === 'compodoc') { + if (!compodocBinary) { + throw new Error('compodoc binary unresolved'); + } const repetition = await runCompodocRepetition( - compodocBinary as string, + compodocBinary, profile.angular, path.join(WORK_ROOT, 'compodoc'), RSS_POLL_INTERVAL_MS @@ -337,18 +358,26 @@ async function main() { engineResults[engine] = { status: 'skipped', reason: skipReason }; continue; } + // An engine that failed part-way through the repetitions holds fewer samples than the pinned N. + // Reporting it as 'measured' would put numbers taken at an unrecorded N into the results, so a + // mid-suite failure stays failed no matter how many repetitions it managed. + const failReason = failed.get(engine); + if (failReason) { + engineResults[engine] = { status: 'failed', reason: failReason }; + continue; + } try { if (engine === 'compodoc') { engineResults[engine] = { status: 'measured', - scenarios: { default: { params: { ...profile.angular }, metrics: compodocMetrics(compodocReps) } }, + scenarios: { default: { params: { ...profile.angular }, metrics: compodocMetrics(compodocReps, profile.n) } }, }; } else if (engine === 'vue-component-meta') { const scenarios: Extract['scenarios'] = {}; for (const scenario of profile.vue) { scenarios[scenario.name] = { params: { ...scenario }, - metrics: seriesMetrics(state.get(`${engine}/${scenario.name}`)), + metrics: seriesMetrics(state.get(`${engine}/${scenario.name}`), profile.n), }; } engineResults[engine] = { status: 'measured', scenarios }; @@ -356,7 +385,10 @@ async function main() { engineResults[engine] = { status: 'measured', scenarios: { - default: { params: { ...profile.react }, metrics: seriesMetrics(state.get(`${engine}/default`)) }, + default: { + params: { ...profile.react }, + metrics: seriesMetrics(state.get(`${engine}/default`), profile.n), + }, }, }; } @@ -433,6 +465,9 @@ async function main() { if (ratios.warmLegacyVsOsa !== undefined) { console.log(` ratio warm legacy/osa: ${ratios.warmLegacyVsOsa.toFixed(2)}`); } + if (ratios.coldLegacyVsOsa === undefined && ratios.warmLegacyVsOsa === undefined) { + console.log(' no calibration ratio: it needs react-legacy and react-osa measured in one run'); + } if (!profile.comparable) { console.log(' QUICK PROFILE: results are non-comparable smoke numbers'); } diff --git a/scripts/bench/docgen-perf/stats.ts b/scripts/bench/docgen-perf/stats.ts index 3987009e0d03..7b8677366a64 100644 --- a/scripts/bench/docgen-perf/stats.ts +++ b/scripts/bench/docgen-perf/stats.ts @@ -1,4 +1,5 @@ /** Statistics helpers for the per-engine docgen performance suite. */ +import type { MemorySample, SaveSample } from './types.ts'; /** Median of `values`. Throws on an empty input so a missing series fails loudly. */ export function median(values: number[]): number { @@ -34,3 +35,40 @@ export function leastSquaresSlope(values: number[]): number { } return den === 0 ? 0 : num / den; } + +/** The retained- and transient-memory figures every series harness derives from its save series. */ +export interface SeriesSummary { + /** Post-GC retained heap per save. Empty unless the child ran under `--expose-gc`. */ + retainedValues: number[]; + /** Least-squares slope of retained heap over the save series, MB per save. */ + retainedSlope?: number; + /** Final retained sample minus the pre-series baseline, MB. */ + retainedGrowth?: number; + /** Per-save allocation spike above the retained baseline, MB. */ + transients: number[]; + /** Mean of {@link transients}. */ + avgTransient?: number; +} + +/** + * Derive the retained/transient series figures shared by every series harness. Kept here so the + * memory harness and the per-engine harnesses cannot drift apart in how they compute them. + */ +export function summarizeSeries(samples: SaveSample[], baseline: MemorySample): SeriesSummary { + const retainedValues = samples + .map((s) => s.retainedHeapMb) + .filter((v): v is number => v !== undefined); + const transients = samples + .map((s) => (s.retainedHeapMb !== undefined ? s.heapUsedMb - s.retainedHeapMb : undefined)) + .filter((v): v is number => v !== undefined); + return { + retainedValues, + retainedSlope: retainedValues.length ? leastSquaresSlope(retainedValues) : undefined, + retainedGrowth: + retainedValues.length && baseline.retainedHeapMb !== undefined + ? (retainedValues.at(-1) as number) - baseline.retainedHeapMb + : undefined, + transients, + avgTransient: transients.length ? mean(transients) : undefined, + }; +} diff --git a/scripts/bench/docgen-perf/types.ts b/scripts/bench/docgen-perf/types.ts index 1d288cb501af..a93e1b779d7e 100644 --- a/scripts/bench/docgen-perf/types.ts +++ b/scripts/bench/docgen-perf/types.ts @@ -32,7 +32,6 @@ export interface SeriesHarnessResult { samples: SaveSample[]; retainedSlope?: number; retainedGrowth?: number; - avgTransient?: number; } /** A latency metric: median of repeated samples (fresh process each, for cold/scan). */ diff --git a/scripts/package.json b/scripts/package.json index f448da8dc779..cd7471580c93 100644 --- a/scripts/package.json +++ b/scripts/package.json @@ -154,6 +154,7 @@ "typescript-native": "npm:typescript@^7.0.2", "uuid": "^9.0.1", "vitest": "^4.1.5", + "vue": "^3.2.47", "vue-component-meta": "^3.2.7", "wait-on": "^8.0.3", "window-size": "^1.1.1", diff --git a/yarn.lock b/yarn.lock index 859eb2484ddb..b5178e8c5fc8 100644 --- a/yarn.lock +++ b/yarn.lock @@ -10512,6 +10512,7 @@ __metadata: verdaccio: "npm:^5.33.0" verdaccio-auth-memory: "npm:^10.3.1" vitest: "npm:^4.1.5" + vue: "npm:^3.2.47" vue-component-meta: "npm:^3.2.7" wait-on: "npm:^8.0.3" window-size: "npm:^1.1.1" From ecdd2b3b963158871f6ad3ca4a8ad1ac7f2ac31a Mon Sep 17 00:00:00 2001 From: Valentin Palkovic Date: Fri, 24 Jul 2026 22:28:03 +0200 Subject: [PATCH 05/14] Bench: Add vue-docgen-api as Vue's legacy control Vue had no legacy baseline. The budget shape asks every engine for a legacy-vs-new ratio measured in the same run, React had one, and Vue did not even though the second implementation was sitting there: vue-docgen-api is still the default plugin, and vue-component-meta is the opt-in successor. The new engine calls parse() per .vue file the way the Vite plugin does. The parser re-reads from disk and keeps no cache, so a save costs one parse of one file and the per-save sample needs no invalidation step. Both Vue engines run over the same generated project and alternate spawn order across repetitions, so their medians divide into a ratio. The ratio needs reading with care, which is why both engines now report how many members their cold pass documented. vue-docgen-api does not resolve tsconfig paths aliases, and the plugin calls parse(id) without an alias map, so on these projects it documents a fraction of what vue-component-meta does. At the pinned profile it finished a cold pass in 49-71ms against 437-456ms while documenting 50-80 members against 320-350. The suite prints both counts beside the ratio and marks them not like-for-like when they disagree, so nobody reads a 0.11 as a speed win over identical work. --- scripts/bench/PERF-METHODOLOGY.md | 12 +- .../docgen-perf/engines/vue-component-meta.ts | 7 +- .../docgen-perf/engines/vue-docgen-api.ts | 195 ++++++++++++++++++ scripts/bench/docgen-perf/run.ts | 96 +++++++-- scripts/bench/docgen-perf/types.ts | 19 +- 5 files changed, 305 insertions(+), 24 deletions(-) create mode 100644 scripts/bench/docgen-perf/engines/vue-docgen-api.ts diff --git a/scripts/bench/PERF-METHODOLOGY.md b/scripts/bench/PERF-METHODOLOGY.md index 85b67dcb452b..b8195570b56b 100644 --- a/scripts/bench/PERF-METHODOLOGY.md +++ b/scripts/bench/PERF-METHODOLOGY.md @@ -17,7 +17,10 @@ Each engine is measured through a directly callable entry point: `react-docgen` is the budgeted legacy control; `react-docgen-typescript` is measurable through the same wrappers but carries no budget row. - Angular: a standalone Compodoc CLI run. Storybook shells out to the same CLI (`runCompodoc`, `code/frameworks/angular/src/builders/utils/run-compodoc.ts`), so the harness measures the tool the framework actually runs. -- Vue: `createChecker` and `createCheckerByJson` from `vue-component-meta`, with the checker options the Vite plugin passes (`code/frameworks/vue3-vite/src/plugins/vue-component-meta.ts`). +- Vue, legacy engine: `parse` from `vue-docgen-api`, called per `.vue` file the way the Vite plugin calls it (`code/frameworks/vue3-vite/src/plugins/vue-docgen.ts`). + This is still the default plugin (`resolveDocgenOptions`, `code/frameworks/vue3-vite/src/preset.ts`), so it is Vue's legacy control the way `react-docgen` is React's. + The parser reads the file from disk on every call and keeps no cache, so one save costs one `parse` of one file - the per-save sample needs no invalidation step and matches production. +- Vue, new engine: `createChecker` and `createCheckerByJson` from `vue-component-meta`, with the checker options the Vite plugin passes (`code/frameworks/vue3-vite/src/plugins/vue-component-meta.ts`). One caveat the budgets must respect: that plugin falls back to `createCheckerByJson` whenever the project's root tsconfig declares `references`, because `vue-component-meta` does not resolve them. The harness's `workspace` and `base-type-touch` scenarios drive `createChecker` at a package tsconfig, which measures the engine rather than the plugin's path for that project shape. - Svelte and web components: their extraction engines are plain functions callable from Node; exact entry points get pinned when each engine's harness lands. @@ -87,6 +90,12 @@ Only the new engine has a production path shaped that way: `buildDocgenPayload` The legacy parsers are reachable only through `generator.ts`, which invalidates every cache and re-extracts every component on each manifest build. A real legacy save therefore costs the whole project, not one file, and the recorded warm ratio understates legacy's true per-save cost by roughly the component count. Run `react-legacy.ts --scope all` for the production-shaped number; the equal-work ratio is the engine comparison, not a saving a user would feel. + +Vue's pair carries a different caveat, and a sharper one. +`vue-docgen-api` does not resolve tsconfig `paths` aliases, and the Vite plugin calls `parse(id)` without an alias map, so on the generated projects it documents a fraction of what `vue-component-meta` documents. +Measured on the flat scenario, the legacy engine finished a cold pass in 23ms against 425ms, but recorded 6 documented members where the new engine recorded the full surface, and 0 on the save it was timed on. +The engines are not doing the same work, so the Vue ratio measures resolution depth as much as speed. +The suite prints both engines' documented-member counts next to the ratio for that reason, and the ratio must not become a budget while the counts disagree. Engines without a second implementation in the same job get their timing reference picked when their baselines are recorded; until then their timing budgets stay placeholders. Memory budgets stay absolute megabytes with generous headroom: budgets sit well above observed values so the gate is not flaky, while still failing hard on a real regression. Every engine must also carry its own negative control - a configuration that must fail, proving the gate can catch the regression class it exists for. @@ -132,6 +141,7 @@ The exception is react-osa's memory row, which the docgen memory gate already en | ----------------------------------------- | --------------- | --------------- | ------------------ | ----------------------- | --------------- | -------------- | ---------------- | ---------- | | react-legacy (react-docgen, control) | TBD (1.12) | TBD (1.12) | n/a | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | | react-osa (ComponentMetaManager, control) | TBD (1.12) | TBD (1.12) | n/a | 90MB | 60MB | 3MB/save | OOM (gate.ts) | daily | +| vue-docgen-api (legacy, current default) | TBD (1.12) | TBD (1.12) | n/a | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | | vue-component-meta | TBD (1.12) | TBD (1.12) | n/a | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | | compodoc | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | | svelte (stretch) | TBD (1.12) | TBD (1.12) | n/a | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | diff --git a/scripts/bench/docgen-perf/engines/vue-component-meta.ts b/scripts/bench/docgen-perf/engines/vue-component-meta.ts index 4fcf73d131ea..52c5274a9f8c 100644 --- a/scripts/bench/docgen-perf/engines/vue-component-meta.ts +++ b/scripts/bench/docgen-perf/engines/vue-component-meta.ts @@ -131,11 +131,12 @@ async function main() { const checker = flat ? createCheckerByJson(project.outDir, { include: ['**/*'] }, CHECKER_OPTIONS) : createChecker(project.packageConfigPaths[targetPackage], CHECKER_OPTIONS); + let coldMembers = 0; for (const sfcPath of targetPaths) { - extractOne(checker, sfcPath); + coldMembers += extractOne(checker, sfcPath); } const coldMs = Date.now() - coldStart; - console.log(` cold pass: ${coldMs}ms`); + console.log(` cold pass: ${coldMs}ms (${coldMembers} documented members)`); const baseline = sampleMemory(true); @@ -194,7 +195,7 @@ async function main() { fs.writeFileSync( options.jsonOut, JSON.stringify( - { options, coldMs, baseline, samples, retainedSlope, retainedGrowth, avgTransient }, + { options, coldMs, coldMembers, baseline, samples, retainedSlope, retainedGrowth, avgTransient }, null, 2 ) diff --git a/scripts/bench/docgen-perf/engines/vue-docgen-api.ts b/scripts/bench/docgen-perf/engines/vue-docgen-api.ts new file mode 100644 index 000000000000..4340c8b7c1ca --- /dev/null +++ b/scripts/bench/docgen-perf/engines/vue-docgen-api.ts @@ -0,0 +1,195 @@ +/** + * Series harness for the vue-docgen-api engine - Vue's legacy docgen and still the default one + * (`resolveDocgenOptions` in `code/frameworks/vue3-vite/src/preset.ts`). It is the legacy half of the + * Vue control pair, the way react-docgen is for React. + * + * Production calls `parse(id)` once per `.vue` file from a Vite `transform` hook + * (`code/frameworks/vue3-vite/src/plugins/vue-docgen.ts`), and the parser reads the file from disk + * on every call with no cache of its own. A save therefore costs exactly one `parse` of one file, + * so the per-save sample here is production-shaped without any invalidation step. + * + * The generated project is the same one the vue-component-meta harness measures, with the same + * scenario levers, so the two engines' medians divide into a ratio that means something. + * + * Run: + * node --expose-gc scripts/bench/docgen-perf/engines/vue-docgen-api.ts \ + * --scenario workspace --packages 4 --components-per-package 10 --heavy-lib --json /tmp/result.json + */ +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +import { parse } from 'vue-docgen-api'; + +import { SANDBOX_DIRECTORY } from '../paths.ts'; +import { baseTypesSource, generateVueProject, vueComponentSource } from '../generators/vue.ts'; +import { gcAvailable, sampleMemory } from '../sampling.ts'; +import { summarizeSeries } from '../stats.ts'; +import type { SaveSample } from '../types.ts'; + +type Scenario = 'flat' | 'workspace' | 'base-type-touch'; + +interface HarnessOptions { + scenario: Scenario; + packages: number; + componentsPerPackage: number; + chainDepth: number; + fanOut: number; + heavyLib: boolean; + saves: number; + outDir: string; + jsonOut?: string; +} + +function parseArgs(argv: string[]): HarnessOptions { + const get = (flag: string, fallback: string) => { + const idx = argv.indexOf(flag); + return idx >= 0 && argv[idx + 1] ? argv[idx + 1] : fallback; + }; + const scenario = get('--scenario', 'workspace'); + if (scenario !== 'flat' && scenario !== 'workspace' && scenario !== 'base-type-touch') { + throw new Error(`--scenario must be "flat", "workspace" or "base-type-touch", got "${scenario}"`); + } + return { + scenario, + packages: Number(get('--packages', '4')), + componentsPerPackage: Number(get('--components-per-package', '10')), + chainDepth: Number(get('--chain-depth', '3')), + fanOut: Number(get('--fan-out', '4')), + heavyLib: argv.includes('--heavy-lib'), + saves: Number(get('--saves', '15')), + outDir: get('--out', path.join(SANDBOX_DIRECTORY, 'docgen-perf', 'vue-docgen-api', 'project')), + jsonOut: argv.indexOf('--json') >= 0 ? get('--json', '') : undefined, + }; +} + +async function main() { + const options = parseArgs(process.argv.slice(2)); + const flat = options.scenario === 'flat'; + const packages = flat ? 1 : options.packages; + + console.log(`vue-docgen-api harness (${options.scenario})`); + console.log( + ` packages=${packages} componentsPerPackage=${options.componentsPerPackage} ` + + `chainDepth=${options.chainDepth} fanOut=${options.fanOut} heavyLib=${options.heavyLib} saves=${options.saves}` + ); + if (!gcAvailable()) { + console.log(' (run with `node --expose-gc` to measure retained heap; continuing without it)'); + } + + const genStart = Date.now(); + const project = generateVueProject({ + outDir: options.outDir, + packages, + componentsPerPackage: options.componentsPerPackage, + chainDepth: options.chainDepth, + fanOut: options.fanOut, + heavyLib: options.heavyLib, + }); + console.log(` generated project in ${Date.now() - genStart}ms at ${project.outDir}`); + + // Same measured set as the vue-component-meta harness: every component in flat layout, the + // deepest package's components otherwise. + const targetPackage = packages - 1; + const targetPaths = flat + ? project.componentPaths + : project.componentPaths.slice(-options.componentsPerPackage); + + /** + * Returns how many documented members the parse produced. The two Vue engines resolve imported + * prop types to different depths, so a timing ratio is only meaningful next to this count - + * a parse that resolved nothing is fast for the wrong reason. + */ + const extractOne = async (sfcPath: string): Promise => { + const doc = await parse(sfcPath); + return ( + Object.keys(doc.props ?? {}).length + + (doc.events ? Object.keys(doc.events).length : 0) + + (doc.slots ? Object.keys(doc.slots).length : 0) + + (doc.expose ? Object.keys(doc.expose).length : 0) + ); + }; + + console.log(` full extraction over ${targetPaths.length} components (cold pass)…`); + const coldStart = Date.now(); + let coldMembers = 0; + for (const sfcPath of targetPaths) { + coldMembers += await extractOne(sfcPath); + } + const coldMs = Date.now() - coldStart; + console.log(` cold pass: ${coldMs}ms (${coldMembers} documented members)`); + + const baseline = sampleMemory(true); + + const samples: SaveSample[] = []; + const extraByTarget = new Array(targetPaths.length).fill(0); + let extraBaseProps = 0; + let warmMembers = 0; + + for (let save = 1; save <= options.saves; save++) { + let measuredPath: string; + if (options.scenario === 'base-type-touch') { + extraBaseProps += 1; + fs.writeFileSync(project.baseTypesPath, baseTypesSource(options.fanOut, extraBaseProps)); + measuredPath = targetPaths[0]; + } else { + const t = (save - 1) % targetPaths.length; + extraByTarget[t] += 1; + fs.writeFileSync(targetPaths[t], vueComponentSource(targetPackage, t, extraByTarget[t])); + measuredPath = targetPaths[t]; + } + + const saveStart = Date.now(); + warmMembers = await extractOne(measuredPath); + const durMs = Date.now() - saveStart; + + const mem = sampleMemory(true); + samples.push({ save, durMs, ...mem }); + console.log( + ` save ${String(save).padStart(3)}: ${String(durMs).padStart(5)}ms ` + + `rss=${mem.rssMb.toFixed(0).padStart(5)}MB heapUsed=${mem.heapUsedMb.toFixed(0).padStart(5)}MB` + + (mem.retainedHeapMb !== undefined + ? ` retained=${mem.retainedHeapMb.toFixed(0).padStart(5)}MB` + : '') + ); + } + + const { retainedSlope, retainedGrowth, avgTransient } = summarizeSeries(samples, baseline); + + console.log('\nsummary'); + console.log(` cold pass: ${coldMs}ms`); + console.log(` documented members: ${coldMembers} cold, ${warmMembers} on the last save`); + if (avgTransient !== undefined) { + console.log(` avg transient/save: ${avgTransient.toFixed(0)}MB`); + } + if (retainedSlope !== undefined && retainedGrowth !== undefined) { + console.log(` retained slope: ${retainedSlope.toFixed(2)}MB/save`); + console.log(` retained growth: ${retainedGrowth.toFixed(0)}MB over ${options.saves} saves`); + } + + if (options.jsonOut) { + fs.writeFileSync( + options.jsonOut, + JSON.stringify( + { + options, + coldMs, + coldMembers, + warmMembers, + baseline, + samples, + retainedSlope, + retainedGrowth, + avgTransient, + }, + null, + 2 + ) + ); + console.log(` wrote ${options.jsonOut}`); + } +} + +main().catch((err) => { + console.error(err); + process.exitCode = 1; +}); diff --git a/scripts/bench/docgen-perf/run.ts b/scripts/bench/docgen-perf/run.ts index ac40953e4ffc..d281e7ae4c2f 100644 --- a/scripts/bench/docgen-perf/run.ts +++ b/scripts/bench/docgen-perf/run.ts @@ -54,17 +54,28 @@ const ALL_ENGINES: EngineId[] = [ 'react-legacy', 'react-legacy-rdt', 'react-osa', + 'vue-docgen-api', 'vue-component-meta', 'compodoc', ]; /** react-legacy-rdt carries no budget row and only runs when named via --engine. */ -const DEFAULT_ENGINES: EngineId[] = ['react-legacy', 'react-osa', 'vue-component-meta', 'compodoc']; +const DEFAULT_ENGINES: EngineId[] = [ + 'react-legacy', + 'react-osa', + 'vue-docgen-api', + 'vue-component-meta', + 'compodoc', +]; + +/** Engines driven by a per-scenario Vue child over the same generated project. */ +const VUE_ENGINES: EngineId[] = ['vue-docgen-api', 'vue-component-meta']; const WORK_ROOT = path.join(SANDBOX_DIRECTORY, 'docgen-perf'); const OSA_HARNESS = path.join(import.meta.dirname, '../docgen-memory/memory-harness.ts'); const LEGACY_HARNESS = path.join(import.meta.dirname, 'engines', 'react-legacy.ts'); -const VUE_HARNESS = path.join(import.meta.dirname, 'engines', 'vue-component-meta.ts'); +const VUE_META_HARNESS = path.join(import.meta.dirname, 'engines', 'vue-component-meta.ts'); +const VUE_DOCGEN_HARNESS = path.join(import.meta.dirname, 'engines', 'vue-docgen-api.ts'); interface CliOptions { quick: boolean; @@ -107,16 +118,26 @@ function parseArgs(argv: string[]): CliOptions { } /** - * The control pair alternates spawn order across repetitions so cache warming and thermal drift do - * not consistently favor one side. + * Every control pair alternates spawn order across repetitions so cache warming and thermal drift + * do not consistently favor one side. */ +const CONTROL_PAIRS: Array<[EngineId, EngineId]> = [ + ['react-legacy', 'react-osa'], + ['vue-docgen-api', 'vue-component-meta'], +]; + function engineOrderForRep(engines: EngineId[], rep: number): EngineId[] { const order = [...engines]; - const legacyIdx = order.indexOf('react-legacy'); - const osaIdx = order.indexOf('react-osa'); - if (legacyIdx >= 0 && osaIdx >= 0 && rep % 2 === 0) { - order[legacyIdx] = 'react-osa'; - order[osaIdx] = 'react-legacy'; + if (rep % 2 !== 0) { + return order; + } + for (const [legacy, next] of CONTROL_PAIRS) { + const legacyIdx = order.indexOf(legacy); + const nextIdx = order.indexOf(next); + if (legacyIdx >= 0 && nextIdx >= 0) { + order[legacyIdx] = next; + order[nextIdx] = legacy; + } } return order; } @@ -137,9 +158,9 @@ function reactChildInvocation(engine: EngineId, cfg: ReactScenarioConfig) { return { childPath: LEGACY_HARNESS, args: [...sizeArgs, '--parser', parser, '--scope', 'changed'] }; } -function vueChildInvocation(scenario: VueScenarioConfig) { +function vueChildInvocation(engine: EngineId, scenario: VueScenarioConfig) { return { - childPath: VUE_HARNESS, + childPath: engine === 'vue-docgen-api' ? VUE_DOCGEN_HARNESS : VUE_META_HARNESS, args: [ '--scenario', scenario.name, '--packages', String(scenario.packages), @@ -295,10 +316,9 @@ async function main() { } const runSeries = (engine: EngineId, scenarioName: string, rep: number) => { - const invocation = - engine === 'vue-component-meta' - ? vueChildInvocation(profile.vue.find((s) => s.name === scenarioName)!) - : reactChildInvocation(engine, profile.react); + const invocation = VUE_ENGINES.includes(engine) + ? vueChildInvocation(engine, profile.vue.find((s) => s.name === scenarioName)!) + : reactChildInvocation(engine, profile.react); const key = `${engine}/${scenarioName}`; const scenarioDir = path.join(WORK_ROOT, engine, scenarioName); console.log(` ${key} (rep ${rep}/${profile.n})…`); @@ -336,7 +356,7 @@ async function main() { ` compodoc: cold=${repetition.coldMs}ms warm=${repetition.warmMs}ms ` + `peakRss=${repetition.peakRssMb.toFixed(0)}MB` ); - } else if (engine === 'vue-component-meta') { + } else if (VUE_ENGINES.includes(engine)) { for (const scenario of profile.vue) { runSeries(engine, scenario.name, rep); } @@ -372,7 +392,7 @@ async function main() { status: 'measured', scenarios: { default: { params: { ...profile.angular }, metrics: compodocMetrics(compodocReps, profile.n) } }, }; - } else if (engine === 'vue-component-meta') { + } else if (VUE_ENGINES.includes(engine)) { const scenarios: Extract['scenarios'] = {}; for (const scenario of profile.vue) { scenarios[scenario.name] = { @@ -414,6 +434,31 @@ async function main() { } } + // The Vue equivalent, per scenario: vue-docgen-api is still the default plugin, vue-component-meta + // is the opt-in successor, and both ran over the same generated project in this invocation. + const vueLegacy = engineResults['vue-docgen-api']; + const vueMeta = engineResults['vue-component-meta']; + if (vueLegacy?.status === 'measured' && vueMeta?.status === 'measured') { + for (const [scenarioName, legacyScenario] of Object.entries(vueLegacy.scenarios)) { + const metaScenario = vueMeta.scenarios[scenarioName]; + if (!metaScenario) { + continue; + } + const l = legacyScenario.metrics; + const m = metaScenario.metrics; + if (l.coldExtractionMs.status === 'measured' && m.coldExtractionMs.status === 'measured') { + ratios.coldVueLegacyVsMeta ??= {}; + ratios.coldVueLegacyVsMeta[scenarioName] = + l.coldExtractionMs.median / m.coldExtractionMs.median; + } + if (l.warmExtractionMs.status === 'measured' && m.warmExtractionMs.status === 'measured') { + ratios.warmVueLegacyVsMeta ??= {}; + ratios.warmVueLegacyVsMeta[scenarioName] = + l.warmExtractionMs.median / m.warmExtractionMs.median; + } + } + } + const results: SuiteResults = { generatedAt: new Date().toISOString(), nodeVersion: process.version, @@ -465,6 +510,23 @@ async function main() { if (ratios.warmLegacyVsOsa !== undefined) { console.log(` ratio warm legacy/osa: ${ratios.warmLegacyVsOsa.toFixed(2)}`); } + for (const [scenarioName, value] of Object.entries(ratios.coldVueLegacyVsMeta ?? {})) { + // vue-docgen-api does not resolve tsconfig `paths` aliases, so on these generated projects it + // documents far less than vue-component-meta does. Printing the ratio without the member counts + // beside it would read as a 20x speed win over identical work, which it is not. + const legacyMembers = state.get(`vue-docgen-api/${scenarioName}`)?.[0]?.coldMembers; + const metaMembers = state.get(`vue-component-meta/${scenarioName}`)?.[0]?.coldMembers; + const members = + legacyMembers !== undefined && metaMembers !== undefined + ? ` [documented members ${legacyMembers} vs ${metaMembers}${legacyMembers === metaMembers ? '' : ' - NOT like-for-like'}]` + : ''; + console.log( + ` ratio cold vue-docgen-api/component-meta (${scenarioName}): ${value.toFixed(2)}${members}` + ); + } + for (const [scenarioName, value] of Object.entries(ratios.warmVueLegacyVsMeta ?? {})) { + console.log(` ratio warm vue-docgen-api/component-meta (${scenarioName}): ${value.toFixed(2)}`); + } if (ratios.coldLegacyVsOsa === undefined && ratios.warmLegacyVsOsa === undefined) { console.log(' no calibration ratio: it needs react-legacy and react-osa measured in one run'); } diff --git a/scripts/bench/docgen-perf/types.ts b/scripts/bench/docgen-perf/types.ts index a93e1b779d7e..3cbf7f031eff 100644 --- a/scripts/bench/docgen-perf/types.ts +++ b/scripts/bench/docgen-perf/types.ts @@ -7,6 +7,7 @@ export type EngineId = | 'react-legacy' | 'react-legacy-rdt' | 'react-osa' + | 'vue-docgen-api' | 'vue-component-meta' | 'compodoc'; @@ -32,6 +33,11 @@ export interface SeriesHarnessResult { samples: SaveSample[]; retainedSlope?: number; retainedGrowth?: number; + /** + * Documented members the cold pass produced. Two engines over the same project can differ by an + * order of magnitude here, and a timing ratio between them means nothing without it. + */ + coldMembers?: number; } /** A latency metric: median of repeated samples (fresh process each, for cold/scan). */ @@ -91,8 +97,15 @@ export interface SuiteResults { rssPollIntervalMs: number; engines: Partial>; /** - * React legacy (react-docgen) median divided by react-osa median, both measured in this same - * invocation. The calibration reference for every other engine's budgets. + * Each framework's legacy-engine median divided by its new-engine median, both measured in this + * same invocation. React pairs react-docgen against react-osa; Vue pairs vue-docgen-api (still + * the default plugin) against vue-component-meta. These are the calibration references budgets + * are derived from. */ - ratios: { coldLegacyVsOsa?: number; warmLegacyVsOsa?: number }; + ratios: { + coldLegacyVsOsa?: number; + warmLegacyVsOsa?: number; + coldVueLegacyVsMeta?: Record; + warmVueLegacyVsMeta?: Record; + }; } From 8a0e723e01c97e27e6a8a4a08a804ec4ba1537a9 Mon Sep 17 00:00:00 2001 From: Valentin Palkovic Date: Mon, 27 Jul 2026 11:28:40 +0200 Subject: [PATCH 06/14] Bench: Pin compodoc so the Angular engine runs by default The compodoc engine was in the default engine list but always skipped, because nothing in the repo declared the binary. Measuring Angular took a manual install, so in practice nobody measured it. scripts/package.json now pins @compodoc/compodoc at exactly 2.0.0 - the version the Angular docgen baselines already capture against, per code/lib/docgen-harness/README.md. Sharing one version keeps the perf suite and the baseline harness describing the same tool. The pin is exact rather than a range because compodoc's cost moves across versions, and a caret would drift the numbers without anyone choosing to. The resolved version is recorded with the results, read by walking up from the binary that actually ran rather than from a fixed path, since the package hoists to different node_modules depending on the install. The skip path stays. It is still what a partial install looks like, so its message now points at yarn install instead of calling compodoc a user-project dependency. Measured at the smoke profile: cold 574ms, warm 710ms, scan 574ms, peak 222MB. Warm sits near cold, which is what the note predicts for a one-shot CLI. --- scripts/bench/PERF-METHODOLOGY.md | 2 + scripts/bench/docgen-perf/engines/compodoc.ts | 31 +- scripts/bench/docgen-perf/run.ts | 4 +- scripts/bench/docgen-perf/types.ts | 2 + scripts/package.json | 1 + yarn.lock | 547 +++++++++++++++++- 6 files changed, 565 insertions(+), 22 deletions(-) diff --git a/scripts/bench/PERF-METHODOLOGY.md b/scripts/bench/PERF-METHODOLOGY.md index b8195570b56b..c5e0a186cf31 100644 --- a/scripts/bench/PERF-METHODOLOGY.md +++ b/scripts/bench/PERF-METHODOLOGY.md @@ -17,6 +17,8 @@ Each engine is measured through a directly callable entry point: `react-docgen` is the budgeted legacy control; `react-docgen-typescript` is measurable through the same wrappers but carries no budget row. - Angular: a standalone Compodoc CLI run. Storybook shells out to the same CLI (`runCompodoc`, `code/frameworks/angular/src/builders/utils/run-compodoc.ts`), so the harness measures the tool the framework actually runs. + `@compodoc/compodoc` is pinned exactly at 2.0.0 in `scripts/package.json`, the version the Angular docgen baselines capture against (`code/lib/docgen-harness/README.md`). + The pin is exact rather than a range because compodoc's cost moves across versions, and the resolved version is recorded with the results. - Vue, legacy engine: `parse` from `vue-docgen-api`, called per `.vue` file the way the Vite plugin calls it (`code/frameworks/vue3-vite/src/plugins/vue-docgen.ts`). This is still the default plugin (`resolveDocgenOptions`, `code/frameworks/vue3-vite/src/preset.ts`), so it is Vue's legacy control the way `react-docgen` is React's. The parser reads the file from disk on every call and keeps no cache, so one save costs one `parse` of one file - the per-save sample needs no invalidation step and matches production. diff --git a/scripts/bench/docgen-perf/engines/compodoc.ts b/scripts/bench/docgen-perf/engines/compodoc.ts index f4f616195593..b1f570401838 100644 --- a/scripts/bench/docgen-perf/engines/compodoc.ts +++ b/scripts/bench/docgen-perf/engines/compodoc.ts @@ -5,9 +5,11 @@ * full run after touching one component file, and peak memory is the child's peak RSS sampled from * outside the process. * - * The compodoc binary is resolved at runtime: it is a user-project dependency in this repo, not a - * workspace package, so the engine skips with an explicit message when no binary is found. Version - * pinning is baseline work. + * `@compodoc/compodoc` is pinned exactly in `scripts/package.json`, at the same 2.0.0 the Angular + * docgen baselines capture against (`code/lib/docgen-harness/README.md`). The pin is exact because + * compodoc's output and cost both move across versions, so a caret range would drift the numbers + * without anyone deciding to. The binary is still resolved at runtime and the engine still skips + * with an explicit message if it is missing, which is what a partial install looks like. */ import { spawn, spawnSync } from 'node:child_process'; import * as fs from 'node:fs'; @@ -16,6 +18,29 @@ import * as path from 'node:path'; import { angularComponentSource, generateAngularProject } from '../generators/angular.ts'; import type { AngularScenarioConfig } from '../config.ts'; +/** + * The version of the binary that actually ran, walked up from the resolved executable rather than + * read from a fixed path - the package hoists to different node_modules depending on the install. + */ +export function compodocVersion(binary: string): string | undefined { + try { + let dir = path.dirname(fs.realpathSync(binary)); + for (let up = 0; up < 5; up++) { + const candidate = path.join(dir, 'package.json'); + if (fs.existsSync(candidate)) { + const pkg = JSON.parse(fs.readFileSync(candidate, 'utf8')); + if (pkg.name === '@compodoc/compodoc') { + return pkg.version as string; + } + } + dir = path.dirname(dir); + } + } catch { + // A binary on PATH may not sit inside a package at all; an unknown version is not a failure. + } + return undefined; +} + /** Locations probed for a compodoc binary, in order: workspace .bin dirs, then PATH. */ export function resolveCompodocBinary(): string | undefined { const candidates = [ diff --git a/scripts/bench/docgen-perf/run.ts b/scripts/bench/docgen-perf/run.ts index d281e7ae4c2f..925e1ba1b913 100644 --- a/scripts/bench/docgen-perf/run.ts +++ b/scripts/bench/docgen-perf/run.ts @@ -37,6 +37,7 @@ import { } from './config.ts'; import { type CompodocRepetition, + compodocVersion, resolveCompodocBinary, runCompodocRepetition, } from './engines/compodoc.ts'; @@ -310,7 +311,7 @@ async function main() { if (options.engines.includes('compodoc') && !compodocBinary) { skipped.set( 'compodoc', - 'no compodoc binary found (workspace node_modules/.bin or PATH); @compodoc/compodoc is a user-project dependency' + 'no compodoc binary found (workspace node_modules/.bin or PATH); @compodoc/compodoc is pinned in scripts/package.json, so run yarn install' ); console.log(` compodoc: SKIPPED - ${skipped.get('compodoc')}`); } @@ -465,6 +466,7 @@ async function main() { pinnedN: profile.n, comparable: profile.comparable, rssPollIntervalMs: RSS_POLL_INTERVAL_MS, + compodocVersion: compodocBinary ? compodocVersion(compodocBinary) : undefined, engines: engineResults, ratios, }; diff --git a/scripts/bench/docgen-perf/types.ts b/scripts/bench/docgen-perf/types.ts index 3cbf7f031eff..3732826207cd 100644 --- a/scripts/bench/docgen-perf/types.ts +++ b/scripts/bench/docgen-perf/types.ts @@ -95,6 +95,8 @@ export interface SuiteResults { comparable: boolean; /** Sampling interval for the compodoc child's externally-polled peak RSS. */ rssPollIntervalMs: number; + /** Resolved compodoc version, when that engine ran. Its cost moves across versions. */ + compodocVersion?: string; engines: Partial>; /** * Each framework's legacy-engine median divided by its new-engine median, both measured in this diff --git a/scripts/package.json b/scripts/package.json index cd7471580c93..b6ae6ec2e8db 100644 --- a/scripts/package.json +++ b/scripts/package.json @@ -50,6 +50,7 @@ "dependencies": { "@actions/core": "^1.11.1", "@anthropic-ai/claude-agent-sdk": "^0.2.85", + "@compodoc/compodoc": "2.0.0", "@fal-works/esbuild-plugin-global-externals": "^2.1.2", "@google-cloud/bigquery": "^6.2.1", "@octokit/graphql": "^5.0.6", diff --git a/yarn.lock b/yarn.lock index 5a82ec4949a6..e62ad7e60082 100644 --- a/yarn.lock +++ b/yarn.lock @@ -48,6 +48,13 @@ __metadata: languageName: node linkType: hard +"@aduh95/viz.js@npm:3.4.0": + version: 3.4.0 + resolution: "@aduh95/viz.js@npm:3.4.0" + checksum: 10c0/6acb40e881dc0b7ac7a6800a1508e8c420e1f6b09bd42e3289aa2199345729af8aec6ffc52dc72cfe678476fcc341f4a83bc2c3d59cd223232e8495f3a414ab0 + languageName: node + linkType: hard + "@ampproject/remapping@npm:2.3.0, @ampproject/remapping@npm:^2.2.0, @ampproject/remapping@npm:^2.3.0": version: 2.3.0 resolution: "@ampproject/remapping@npm:2.3.0" @@ -434,6 +441,38 @@ __metadata: languageName: node linkType: hard +"@angular-devkit/core@npm:22.0.4": + version: 22.0.4 + resolution: "@angular-devkit/core@npm:22.0.4" + dependencies: + ajv: "npm:8.20.0" + ajv-formats: "npm:3.0.1" + jsonc-parser: "npm:3.3.1" + picomatch: "npm:4.0.4" + rxjs: "npm:7.8.2" + source-map: "npm:0.7.6" + peerDependencies: + chokidar: ^5.0.0 + peerDependenciesMeta: + chokidar: + optional: true + checksum: 10c0/0d84969ad31113a48202c644038386f7dd120af950490125d7b8d56c4ebc9ca362cff56be70f90eb3c3be1c15cbcec11aad648e46caebabba82efc91e75dff20 + languageName: node + linkType: hard + +"@angular-devkit/schematics@npm:22.0.4": + version: 22.0.4 + resolution: "@angular-devkit/schematics@npm:22.0.4" + dependencies: + "@angular-devkit/core": "npm:22.0.4" + jsonc-parser: "npm:3.3.1" + magic-string: "npm:0.30.21" + ora: "npm:9.4.0" + rxjs: "npm:7.8.2" + checksum: 10c0/02288bf8f636d3517d03e97942ac1fab4ec8e4448d0d918abd96b06b1a8d0e043482993b51fc69e6c2fe43a3459fb4bbf8c3cccc803f17649ebc2f61c656d94d + languageName: node + linkType: hard + "@angular/animations@npm:^19.1.1": version: 19.2.15 resolution: "@angular/animations@npm:19.2.15" @@ -855,6 +894,13 @@ __metadata: languageName: node linkType: hard +"@arr/every@npm:^1.0.0": + version: 1.0.1 + resolution: "@arr/every@npm:1.0.1" + checksum: 10c0/f9769bc97eca37d1c0dedfda4c17506c0e1269d95c15a7478a49c4d84d43512a3f350b71c95e045cf4113d3398ebd6d298f2eea09196a1ed61c311bc4e6964e7 + languageName: node + linkType: hard + "@aw-web-design/x-default-browser@npm:1.4.126": version: 1.4.126 resolution: "@aw-web-design/x-default-browser@npm:1.4.126" @@ -2644,6 +2690,69 @@ __metadata: languageName: node linkType: hard +"@compodoc/compodoc@npm:2.0.0": + version: 2.0.0 + resolution: "@compodoc/compodoc@npm:2.0.0" + dependencies: + "@angular-devkit/schematics": "npm:22.0.4" + "@compodoc/ngd-transformer": "npm:^2.1.3" + body-parser: "npm:^2.3.0" + bootstrap.native: "npm:^5.1.10" + cheerio: "npm:1.2.0" + chokidar: "npm:^5.0.0" + commander: "npm:^15.0.0" + cosmiconfig: "npm:^9.0.2" + es6-shim: "npm:^0.35.8" + fs-extra: "npm:^11.3.5" + handlebars: "npm:^4.7.9" + html-entities: "npm:^2.6.0" + i18next: "npm:26.3.3" + json5: "npm:^2.2.3" + loglevel: "npm:^1.9.2" + loglevel-plugin-prefix: "npm:^0.8.4" + lunr: "npm:^2.3.9" + marked: "npm:18.0.5" + neotraverse: "npm:^1.0.1" + picocolors: "npm:^1.1.1" + polka: "npm:^0.5.2" + prismjs: "npm:^1.30.0" + semver: "npm:^7.8.5" + sirv: "npm:^3.0.2" + svg-pan-zoom: "npm:^3.6.2" + tablesort: "npm:^5.7.1" + tinyglobby: "npm:^0.2.17" + ts-morph: "npm:^28.0.0" + uuid: "npm:14.0.1" + vis-network: "npm:^10.1.0" + bin: + compodoc: bin/index-cli.js + checksum: 10c0/327e6b7d173f75be1996b59b89d3832467599782530f1e2b1ff8bcc73e977833e69e35e5ab57bfd79ddfb02e31ea58e5ee9a58201829751b330601f74af0efc8 + languageName: node + linkType: hard + +"@compodoc/ngd-core@npm:~2.1.1": + version: 2.1.1 + resolution: "@compodoc/ngd-core@npm:2.1.1" + dependencies: + ansi-colors: "npm:^4.1.3" + fancy-log: "npm:^2.0.0" + typescript: "npm:^5.0.4" + checksum: 10c0/f78f4231e39d96e397bb39614cfbe8dfc6b985eb9b2795cd5188a40f65eb60eb096dc29de61baa6d914fbc86ec7f21b5a2d02e6000b26d82ae3cd682aea79275 + languageName: node + linkType: hard + +"@compodoc/ngd-transformer@npm:^2.1.3": + version: 2.1.3 + resolution: "@compodoc/ngd-transformer@npm:2.1.3" + dependencies: + "@aduh95/viz.js": "npm:3.4.0" + "@compodoc/ngd-core": "npm:~2.1.1" + dot: "npm:^2.0.0-beta.1" + fs-extra: "npm:^11.1.1" + checksum: 10c0/fb7b13860fc022cda284ca0bae55f8c6610621a060c943740838b5fa2f6be7957c99c2750024cf949b2c8f1c5f41bc64cd8e28ab333f985627f5991bcaf5cdcf + languageName: node + linkType: hard + "@cypress/request@npm:3.0.6": version: 3.0.6 resolution: "@cypress/request@npm:3.0.6" @@ -6487,6 +6596,13 @@ __metadata: languageName: node linkType: hard +"@polka/url@npm:^0.5.0": + version: 0.5.0 + resolution: "@polka/url@npm:0.5.0" + checksum: 10c0/76324126e1608f4cdc5c6b11b618ca67612c37d45401f3fbcce4967294093761b7568cc6ae74ade8afbe744547e72e9d50bc38f690ddea0904a1d66a6b8a0093 + languageName: node + linkType: hard + "@polka/url@npm:^1.0.0-next.21, @polka/url@npm:^1.0.0-next.24": version: 1.0.0-next.29 resolution: "@polka/url@npm:1.0.0-next.29" @@ -10404,6 +10520,7 @@ __metadata: dependencies: "@actions/core": "npm:^1.11.1" "@anthropic-ai/claude-agent-sdk": "npm:^0.2.85" + "@compodoc/compodoc": "npm:2.0.0" "@fal-works/esbuild-plugin-global-externals": "npm:^2.1.2" "@google-cloud/bigquery": "npm:^6.2.1" "@octokit/graphql": "npm:^5.0.6" @@ -11197,6 +11314,29 @@ __metadata: languageName: node linkType: hard +"@thednp/event-listener@npm:^2.0.15": + version: 2.0.15 + resolution: "@thednp/event-listener@npm:2.0.15" + checksum: 10c0/02ce4402985f72af44049b6557443dd96a47f37301ea2dbc82f009aa56f22d9d5f06b45e4cb07f6c3bc5ff30c0faadd118095212f908b40b9521348aee231c39 + languageName: node + linkType: hard + +"@thednp/position-observer@npm:^1.1.3": + version: 1.1.3 + resolution: "@thednp/position-observer@npm:1.1.3" + dependencies: + "@thednp/shorty": "npm:^2.0.14" + checksum: 10c0/1fc95a4d8a67e5accab98a6aac571053190bd491702d09759242357d35d3b7d035998470d86abe6f113304b1a5ea44722e5caa1832ef51c36d8654e3890d1951 + languageName: node + linkType: hard + +"@thednp/shorty@npm:^2.0.14": + version: 2.0.14 + resolution: "@thednp/shorty@npm:2.0.14" + checksum: 10c0/72a4ecdbdbeb05953649721f412c1301f950450906fe5914cf4225b83e49f18dc0573f3526aa6efaaff2e442753b5e5b897ed91f5be0792ad4766c3462a0277d + languageName: node + linkType: hard + "@tmcp/adapter-valibot@npm:^0.1.5": version: 0.1.6 resolution: "@tmcp/adapter-valibot@npm:0.1.6" @@ -11262,6 +11402,17 @@ __metadata: languageName: node linkType: hard +"@ts-morph/common@npm:~0.29.0": + version: 0.29.0 + resolution: "@ts-morph/common@npm:0.29.0" + dependencies: + minimatch: "npm:^10.0.1" + path-browserify: "npm:^1.0.1" + tinyglobby: "npm:^0.2.14" + checksum: 10c0/97ab8ca66558b817e5475a8893ee1476ab760a3ac71fa9868413d95ddbaaf909eda81716e3a8d14291cbf449e624af66de81bcddd3ec2e36525728b4edf5e8ee + languageName: node + linkType: hard + "@tybys/wasm-util@npm:^0.10.3": version: 0.10.3 resolution: "@tybys/wasm-util@npm:0.10.3" @@ -13677,6 +13828,18 @@ __metadata: languageName: node linkType: hard +"ajv@npm:8.20.0": + version: 8.20.0 + resolution: "ajv@npm:8.20.0" + dependencies: + fast-deep-equal: "npm:^3.1.3" + fast-uri: "npm:^3.0.1" + json-schema-traverse: "npm:^1.0.0" + require-from-string: "npm:^2.0.2" + checksum: 10c0/5df9a1c8f83863cde1bd3a9ddb426f599718f88e3dc9153616c79fb28e0be455335830d7f21d745576519f057b371352daa31047b6a33d7036fe08777d60cf2a + languageName: node + linkType: hard + "ajv@npm:^6.10.0, ajv@npm:^6.12.3, ajv@npm:^6.12.4, ajv@npm:^6.12.5, ajv@npm:^6.12.6": version: 6.12.6 resolution: "ajv@npm:6.12.6" @@ -13713,7 +13876,7 @@ __metadata: languageName: node linkType: hard -"ansi-colors@npm:4.1.3, ansi-colors@npm:^4.1.1": +"ansi-colors@npm:4.1.3, ansi-colors@npm:^4.1.1, ansi-colors@npm:^4.1.3": version: 4.1.3 resolution: "ansi-colors@npm:4.1.3" checksum: 10c0/ec87a2f59902f74e61eada7f6e6fe20094a628dab765cfdbd03c3477599368768cffccdb5d3bb19a1b6c99126783a143b1fee31aab729b31ffe5836c7e5e28b9 @@ -14821,6 +14984,23 @@ __metadata: languageName: node linkType: hard +"body-parser@npm:^2.3.0": + version: 2.3.0 + resolution: "body-parser@npm:2.3.0" + dependencies: + bytes: "npm:^3.1.2" + content-type: "npm:^2.0.0" + debug: "npm:^4.4.3" + http-errors: "npm:^2.0.1" + iconv-lite: "npm:^0.7.2" + on-finished: "npm:^2.4.1" + qs: "npm:^6.15.2" + raw-body: "npm:^3.0.2" + type-is: "npm:^2.1.0" + checksum: 10c0/2a8fbbdc471b588338555a3e1a597d1eb0ad0c21cf20fdc3bac5d3f8d9c3a4b19b4163575ab852a43a5dfc0df7770ced5284f979e561f1a24c2f851ce89e695a + languageName: node + linkType: hard + "body-parser@npm:~1.20.5": version: 1.20.5 resolution: "body-parser@npm:1.20.5" @@ -14858,6 +15038,17 @@ __metadata: languageName: node linkType: hard +"bootstrap.native@npm:^5.1.10": + version: 5.1.10 + resolution: "bootstrap.native@npm:5.1.10" + dependencies: + "@thednp/event-listener": "npm:^2.0.15" + "@thednp/position-observer": "npm:^1.1.3" + "@thednp/shorty": "npm:^2.0.14" + checksum: 10c0/6484411fc4ba5b7d1bdd9da039d07efc32c30b5b216b1442cea4ef235b6398b22ccf721b7216258113f8e9cca86558bf6155abb98ebe7ff55a51e4345130e997 + languageName: node + linkType: hard + "bplist-parser@npm:^0.2.0": version: 0.2.0 resolution: "bplist-parser@npm:0.2.0" @@ -15317,7 +15508,7 @@ __metadata: languageName: node linkType: hard -"bytes@npm:3.1.2, bytes@npm:~3.1.2": +"bytes@npm:3.1.2, bytes@npm:^3.1.2, bytes@npm:~3.1.2": version: 3.1.2 resolution: "bytes@npm:3.1.2" checksum: 10c0/76d1c43cbd602794ad8ad2ae94095cddeb1de78c5dddaa7005c51af10b0176c69971a6d88e805a90c2b6550d76636e43c40d8427a808b8645ede885de4a0358e @@ -15637,7 +15828,7 @@ __metadata: languageName: node linkType: hard -"cheerio@npm:^1.0.0": +"cheerio@npm:1.2.0, cheerio@npm:^1.0.0": version: 1.2.0 resolution: "cheerio@npm:1.2.0" dependencies: @@ -15993,6 +16184,13 @@ __metadata: languageName: node linkType: hard +"code-block-writer@npm:^13.0.3": + version: 13.0.3 + resolution: "code-block-writer@npm:13.0.3" + checksum: 10c0/87db97b37583f71cfd7eced8bf3f0a0a0ca53af912751a734372b36c08cd27f3e8a4878ec05591c0cd9ae11bea8add1423e132d660edd86aab952656dd41fd66 + languageName: node + linkType: hard + "code-point-at@npm:^1.0.0": version: 1.1.0 resolution: "code-point-at@npm:1.1.0" @@ -16114,6 +16312,13 @@ __metadata: languageName: node linkType: hard +"commander@npm:^15.0.0": + version: 15.0.0 + resolution: "commander@npm:15.0.0" + checksum: 10c0/539229c171914ea1ccd45ee5f10d924289a12a684ea3a7a44147abe54003c35ed6de9ae4ad198d88c29fdf403dca0428451a388234e6dc01b6faa978fb207206 + languageName: node + linkType: hard + "commander@npm:^2.18.0, commander@npm:^2.19.0, commander@npm:^2.20.0, commander@npm:^2.8.1": version: 2.20.3 resolution: "commander@npm:2.20.3" @@ -16282,6 +16487,13 @@ __metadata: languageName: node linkType: hard +"content-type@npm:^2.0.0": + version: 2.0.0 + resolution: "content-type@npm:2.0.0" + checksum: 10c0/491539fff707d7594b0ca4fabcc084bef2a31ffa754ff0a4f80c4377e3963cff0394317f9271c24087596c97fa675bc123d61fa34ffe65b4904e7d3d3098de72 + languageName: node + linkType: hard + "content-type@npm:~1.0.4, content-type@npm:~1.0.5": version: 1.0.5 resolution: "content-type@npm:1.0.5" @@ -16509,6 +16721,23 @@ __metadata: languageName: node linkType: hard +"cosmiconfig@npm:^9.0.2": + version: 9.0.2 + resolution: "cosmiconfig@npm:9.0.2" + dependencies: + env-paths: "npm:^2.2.1" + import-fresh: "npm:^3.3.0" + js-yaml: "npm:^4.1.0" + parse-json: "npm:^5.2.0" + peerDependencies: + typescript: ">=4.9.5" + peerDependenciesMeta: + typescript: + optional: true + checksum: 10c0/132d32863c0b3d9ace7010a10011e09089532b36e3b11e02004d671dcbd8b8f2f16293b82d510d9c15890f30358baf8722127c7b1c011449c90b6a178f9c6594 + languageName: node + linkType: hard + "create-ecdh@npm:^4.0.4": version: 4.0.4 resolution: "create-ecdh@npm:4.0.4" @@ -17513,6 +17742,13 @@ __metadata: languageName: node linkType: hard +"dot@npm:^2.0.0-beta.1": + version: 2.0.0-beta.1 + resolution: "dot@npm:2.0.0-beta.1" + checksum: 10c0/28876e214576210384fd49bcf2f3c4afc6478cec7de18e923c07a085dd1f3c565ff162c53911040267fc9ff96fa732dde8676c503ffd60d5a522101bc1ad1e21 + languageName: node + linkType: hard + "dotenv-expand@npm:^10.0.0": version: 10.0.0 resolution: "dotenv-expand@npm:10.0.0" @@ -18313,6 +18549,13 @@ __metadata: languageName: node linkType: hard +"es6-shim@npm:^0.35.8": + version: 0.35.8 + resolution: "es6-shim@npm:0.35.8" + checksum: 10c0/e54e147677b9cd57c96bbc2332c3096ab861040b22daa27728204cacf9aa656d6a3eeb5367a037b276bfbe442ec866938a95b4612accf0cd24d01c209e48eaf9 + languageName: node + linkType: hard + "esast-util-from-estree@npm:^2.0.0": version: 2.0.0 resolution: "esast-util-from-estree@npm:2.0.0" @@ -19311,6 +19554,15 @@ __metadata: languageName: node linkType: hard +"fancy-log@npm:^2.0.0": + version: 2.0.0 + resolution: "fancy-log@npm:2.0.0" + dependencies: + color-support: "npm:^1.1.3" + checksum: 10c0/a6e116f3346756a7363eea343b551c1375d2cd2afc2a92d224feb78589b6b3cff85db9dc5d5df89792a0f7c1e17f731f52cb3d2807302f0516422be6269b95a8 + languageName: node + linkType: hard + "fast-content-type-parse@npm:^2.0.0": version: 2.0.1 resolution: "fast-content-type-parse@npm:2.0.1" @@ -19968,6 +20220,17 @@ __metadata: languageName: node linkType: hard +"fs-extra@npm:^11.1.1, fs-extra@npm:^11.3.5": + version: 11.4.0 + resolution: "fs-extra@npm:11.4.0" + dependencies: + graceful-fs: "npm:^4.2.0" + jsonfile: "npm:^6.0.1" + universalify: "npm:^2.0.0" + checksum: 10c0/1e311eca820a12d3d795f2043dcd5628d017f4d21e634f3f9ef314ae68bee9979758262fb9cb35ffa6f26454c3fffe8b15558733e91e8fc729b07b10bb98d8b6 + languageName: node + linkType: hard + "fs-extra@npm:^5.0.0": version: 5.0.0 resolution: "fs-extra@npm:5.0.0" @@ -20712,6 +20975,24 @@ __metadata: languageName: node linkType: hard +"handlebars@npm:^4.7.9": + version: 4.7.9 + resolution: "handlebars@npm:4.7.9" + dependencies: + minimist: "npm:^1.2.5" + neo-async: "npm:^2.6.2" + source-map: "npm:^0.6.1" + uglify-js: "npm:^3.1.4" + wordwrap: "npm:^1.0.0" + dependenciesMeta: + uglify-js: + optional: true + bin: + handlebars: bin/handlebars + checksum: 10c0/22f8105a7e68e81aff2662bb434edf05f757d21d850731d71cec886d69c10cd33d3c43e34b2892968ec62de8241611851d3d0674c8ef324ea3e01dc66262faa9 + languageName: node + linkType: hard + "happy-dom@npm:^17.6.3": version: 17.6.3 resolution: "happy-dom@npm:17.6.3" @@ -21097,7 +21378,7 @@ __metadata: languageName: node linkType: hard -"html-entities@npm:^2.1.0": +"html-entities@npm:^2.1.0, html-entities@npm:^2.6.0": version: 2.6.0 resolution: "html-entities@npm:2.6.0" checksum: 10c0/7c8b15d9ea0cd00dc9279f61bab002ba6ca8a7a0f3c36ed2db3530a67a9621c017830d1d2c1c65beb9b8e3436ea663e9cf8b230472e0e413359399413b27c8b7 @@ -21219,19 +21500,7 @@ __metadata: languageName: node linkType: hard -"http-errors@npm:~1.6.2": - version: 1.6.3 - resolution: "http-errors@npm:1.6.3" - dependencies: - depd: "npm:~1.1.2" - inherits: "npm:2.0.3" - setprototypeof: "npm:1.1.0" - statuses: "npm:>= 1.4.0 < 2" - checksum: 10c0/17ec4046ee974477778bfdd525936c254b872054703ec2caa4d6f099566b8adade636ae6aeeacb39302c5cd6e28fb407ebd937f500f5010d0b6850750414ff78 - languageName: node - linkType: hard - -"http-errors@npm:~2.0.0, http-errors@npm:~2.0.1": +"http-errors@npm:^2.0.1, http-errors@npm:~2.0.0, http-errors@npm:~2.0.1": version: 2.0.1 resolution: "http-errors@npm:2.0.1" dependencies: @@ -21244,6 +21513,18 @@ __metadata: languageName: node linkType: hard +"http-errors@npm:~1.6.2": + version: 1.6.3 + resolution: "http-errors@npm:1.6.3" + dependencies: + depd: "npm:~1.1.2" + inherits: "npm:2.0.3" + setprototypeof: "npm:1.1.0" + statuses: "npm:>= 1.4.0 < 2" + checksum: 10c0/17ec4046ee974477778bfdd525936c254b872054703ec2caa4d6f099566b8adade636ae6aeeacb39302c5cd6e28fb407ebd937f500f5010d0b6850750414ff78 + languageName: node + linkType: hard + "http-parser-js@npm:>=0.5.1": version: 0.5.10 resolution: "http-parser-js@npm:0.5.10" @@ -21442,6 +21723,18 @@ __metadata: languageName: node linkType: hard +"i18next@npm:26.3.3": + version: 26.3.3 + resolution: "i18next@npm:26.3.3" + peerDependencies: + typescript: ^5 || ^6 + peerDependenciesMeta: + typescript: + optional: true + checksum: 10c0/7a6676a8bade1683d23c3187b92d571528eeadf9f3d6776b4ef1bd27789d2ee7594a9aa7ca05394d8e11c931c8c8f224d85b2b5059a5613cb79b04b8d46a291e + languageName: node + linkType: hard + "iconv-lite@npm:0.4.24, iconv-lite@npm:~0.4.24": version: 0.4.24 resolution: "iconv-lite@npm:0.4.24" @@ -21460,6 +21753,15 @@ __metadata: languageName: node linkType: hard +"iconv-lite@npm:^0.7.2, iconv-lite@npm:~0.7.0": + version: 0.7.3 + resolution: "iconv-lite@npm:0.7.3" + dependencies: + safer-buffer: "npm:>= 2.1.2 < 3.0.0" + checksum: 10c0/be2fd2414f7e94be3a63063fa0ad5919c16bc7b6e00c77eea0765ced6db405739f38dd51016583058fba25cc1d0106ba30b83fbb7a7e32f824d4db76f23d8d33 + languageName: node + linkType: hard + "icss-utils@npm:^5.0.0, icss-utils@npm:^5.1.0": version: 5.1.0 resolution: "icss-utils@npm:5.1.0" @@ -23854,6 +24156,20 @@ __metadata: languageName: node linkType: hard +"loglevel-plugin-prefix@npm:^0.8.4": + version: 0.8.4 + resolution: "loglevel-plugin-prefix@npm:0.8.4" + checksum: 10c0/357524eec4c165ff823b5bbf72e8373ff529e5cb95c1f4b20749847bd5b5b16ab328d6d33d1a9019f1a2dc52e28fca5d595e52f2ee20e24986182a6f9552a9ec + languageName: node + linkType: hard + +"loglevel@npm:^1.9.2": + version: 1.9.2 + resolution: "loglevel@npm:1.9.2" + checksum: 10c0/1e317fa4648fe0b4a4cffef6de037340592cee8547b07d4ce97a487abe9153e704b98451100c799b032c72bb89c9366d71c9fb8192ada8703269263ae77acdc7 + languageName: node + linkType: hard + "longest-streak@npm:^3.0.0": version: 3.1.0 resolution: "longest-streak@npm:3.1.0" @@ -23957,6 +24273,13 @@ __metadata: languageName: node linkType: hard +"lunr@npm:^2.3.9": + version: 2.3.9 + resolution: "lunr@npm:2.3.9" + checksum: 10c0/77d7dbb4fbd602aac161e2b50887d8eda28c0fa3b799159cee380fbb311f1e614219126ecbbd2c3a9c685f1720a8109b3c1ca85cc893c39b6c9cc6a62a1d8a8b + languageName: node + linkType: hard + "lz-string@npm:^1.5.0": version: 1.5.0 resolution: "lz-string@npm:1.5.0" @@ -24123,6 +24446,15 @@ __metadata: languageName: node linkType: hard +"marked@npm:18.0.5": + version: 18.0.5 + resolution: "marked@npm:18.0.5" + bin: + marked: bin/marked.js + checksum: 10c0/22763935a0a243c41851b010a086ea85a09951e379948b6aa629b6cecdcd66e3a5e8c4acac2f6b29c183c20609d3e5102495270207dfd5f4c46ddd773a4ddb9b + languageName: node + linkType: hard + "matcher-collection@npm:^1.0.0, matcher-collection@npm:^1.1.1": version: 1.1.2 resolution: "matcher-collection@npm:1.1.2" @@ -24142,6 +24474,15 @@ __metadata: languageName: node linkType: hard +"matchit@npm:^1.0.0": + version: 1.1.0 + resolution: "matchit@npm:1.1.0" + dependencies: + "@arr/every": "npm:^1.0.0" + checksum: 10c0/f2cd8702ea37dd9859cf1df762581c822180eb72372b9240397a80320cb4ffb4828b3d8ed0beff2429daf500d1ec746c2963dbb3c6386f1b5b9eb969e0f1caaa + languageName: node + linkType: hard + "math-intrinsics@npm:^1.1.0": version: 1.1.0 resolution: "math-intrinsics@npm:1.1.0" @@ -24301,6 +24642,13 @@ __metadata: languageName: node linkType: hard +"media-typer@npm:^1.1.0": + version: 1.1.1 + resolution: "media-typer@npm:1.1.1" + checksum: 10c0/924644de220c3107fc53ec0299b7c1488503470265d32f1492535bd798cc369ba7ceaf9c2a4533361590dcf7208d1cc7b7032dbc1a5918773f5f5d912cab5211 + languageName: node + linkType: hard + "memfs-or-file-map-to-github-branch@npm:^1.3.0": version: 1.3.0 resolution: "memfs-or-file-map-to-github-branch@npm:1.3.0" @@ -24778,6 +25126,15 @@ __metadata: languageName: node linkType: hard +"mime-types@npm:^3.0.0": + version: 3.0.2 + resolution: "mime-types@npm:3.0.2" + dependencies: + mime-db: "npm:^1.54.0" + checksum: 10c0/35a0dd1035d14d185664f346efcdb72e93ef7a9b6e9ae808bd1f6358227010267fab52657b37562c80fc888ff76becb2b2938deb5e730818b7983bf8bd359767 + languageName: node + linkType: hard + "mime-types@npm:^3.0.1": version: 3.0.1 resolution: "mime-types@npm:3.0.1" @@ -25336,6 +25693,13 @@ __metadata: languageName: node linkType: hard +"neotraverse@npm:^1.0.1": + version: 1.0.1 + resolution: "neotraverse@npm:1.0.1" + checksum: 10c0/98e5d2d1974f91d527851a9dcfceb62701a1d31970b324b9f35fbdd244d173edc7a379ea1773f64771a9dc282486e3d3b693b24e14b1e969291612b2ff6b5054 + languageName: node + linkType: hard + "next@npm:^15.2.3": version: 15.5.6 resolution: "next@npm:15.5.6" @@ -26165,6 +26529,22 @@ __metadata: languageName: node linkType: hard +"ora@npm:9.4.0": + version: 9.4.0 + resolution: "ora@npm:9.4.0" + dependencies: + chalk: "npm:^5.6.2" + cli-cursor: "npm:^5.0.0" + cli-spinners: "npm:^3.2.0" + is-interactive: "npm:^2.0.0" + is-unicode-supported: "npm:^2.1.0" + log-symbols: "npm:^7.0.1" + stdin-discarder: "npm:^0.3.2" + string-width: "npm:^8.1.0" + checksum: 10c0/8f2d7a8869cd68607797ac0ffe9f5cdceeb6009437672510d9920aea794cdd055164e3fe804248624c4940a71b22f94f1ffd94ce8fecf0746baef97a5c121a91 + languageName: node + linkType: hard + "ordered-binary@npm:^1.5.3": version: 1.6.0 resolution: "ordered-binary@npm:1.6.0" @@ -27491,6 +27871,16 @@ __metadata: languageName: node linkType: hard +"polka@npm:^0.5.2": + version: 0.5.2 + resolution: "polka@npm:0.5.2" + dependencies: + "@polka/url": "npm:^0.5.0" + trouter: "npm:^2.0.1" + checksum: 10c0/113005fe146e1ee67bc6e1a5438c71e1b59ac05a8bc435d593be210e21f712e6b9ba4d163072e3c941f8efbb7516df9c55bf18934948f91623d03dc9657a1410 + languageName: node + linkType: hard + "polka@npm:^1.0.0-next.28": version: 1.0.0-next.28 resolution: "polka@npm:1.0.0-next.28" @@ -28245,6 +28635,16 @@ __metadata: languageName: node linkType: hard +"qs@npm:^6.15.2": + version: 6.15.3 + resolution: "qs@npm:6.15.3" + dependencies: + es-define-property: "npm:^1.0.1" + side-channel: "npm:^1.1.1" + checksum: 10c0/8f3f6e45ece255347d57696628401cde29e9ec649fff698b53bd3150dea7cefdf33036e1bc1826b9f110bfa7cb0ec4ab9f5297eca628ce216c55af82c304e08e + languageName: node + linkType: hard + "qs@npm:~6.5.2": version: 6.5.3 resolution: "qs@npm:6.5.3" @@ -28329,6 +28729,18 @@ __metadata: languageName: node linkType: hard +"raw-body@npm:^3.0.2": + version: 3.0.2 + resolution: "raw-body@npm:3.0.2" + dependencies: + bytes: "npm:~3.1.2" + http-errors: "npm:~2.0.1" + iconv-lite: "npm:~0.7.0" + unpipe: "npm:~1.0.0" + checksum: 10c0/d266678d08e1e7abea62c0ce5864344e980fa81c64f6b481e9842c5beaed2cdcf975f658a3ccd67ad35fc919c1f6664ccc106067801850286a6cbe101de89f29 + languageName: node + linkType: hard + "raw-body@npm:~2.5.3": version: 2.5.3 resolution: "raw-body@npm:2.5.3" @@ -30368,6 +30780,15 @@ __metadata: languageName: node linkType: hard +"semver@npm:^7.8.5": + version: 7.8.5 + resolution: "semver@npm:7.8.5" + bin: + semver: bin/semver.js + checksum: 10c0/b1f3127a5be8125a94f37188b361c212466c292c6910adce3ec106cff5dc211ccaedc4739c11bb70fda59d6fc1f040a9bca289f4e093451521a2372e5231fe0c + languageName: node + linkType: hard + "send@npm:0.19.0": version: 0.19.0 resolution: "send@npm:0.19.0" @@ -30722,6 +31143,16 @@ __metadata: languageName: node linkType: hard +"side-channel-list@npm:^1.0.1": + version: 1.0.1 + resolution: "side-channel-list@npm:1.0.1" + dependencies: + es-errors: "npm:^1.3.0" + object-inspect: "npm:^1.13.4" + checksum: 10c0/d346c787fd2f9f1c2fdea14f00e8250118db0e7596d85a6cb9faa75f105d31a73a8f7a341c93d7df2a2429098c3d37a77bd3be9e88c37094b8c01807bc77c7a2 + languageName: node + linkType: hard + "side-channel-map@npm:^1.0.1": version: 1.0.1 resolution: "side-channel-map@npm:1.0.1" @@ -30760,6 +31191,19 @@ __metadata: languageName: node linkType: hard +"side-channel@npm:^1.1.1": + version: 1.1.1 + resolution: "side-channel@npm:1.1.1" + dependencies: + es-errors: "npm:^1.3.0" + object-inspect: "npm:^1.13.4" + side-channel-list: "npm:^1.0.1" + side-channel-map: "npm:^1.0.1" + side-channel-weakmap: "npm:^1.0.2" + checksum: 10c0/dc0ab81d67f61bda9247d053ce93f41c3fd8ad2bdcb9cf9d8d2f8540d488f26d87a5e99ebfc07eea49ec025867b2452b705442d974b1478f0395e69f6bfb3270 + languageName: node + linkType: hard + "siginfo@npm:^2.0.0": version: 2.0.0 resolution: "siginfo@npm:2.0.0" @@ -31280,7 +31724,7 @@ __metadata: languageName: node linkType: hard -"stdin-discarder@npm:^0.3.1": +"stdin-discarder@npm:^0.3.1, stdin-discarder@npm:^0.3.2": version: 0.3.2 resolution: "stdin-discarder@npm:0.3.2" checksum: 10c0/5dbaba9efbcb447a4450d5ae19794641ea9166abe96dc4b5547a109db1bb6e8bdb17bbe1029e02ca8d9d8ee996b7c7cbcce12b12c18c121871cd4f574292381a @@ -32009,6 +32453,13 @@ __metadata: languageName: node linkType: hard +"svg-pan-zoom@npm:^3.6.2": + version: 3.6.2 + resolution: "svg-pan-zoom@npm:3.6.2" + checksum: 10c0/dc05030bd6545d523de71e38d79d6830f780a74416a609269cc378fe8778796e794cb3e27f0f5b20bd7ee930597820e5b4463e4e4e0708127bed81e020339ec0 + languageName: node + linkType: hard + "symlink-or-copy@npm:^1.0.0, symlink-or-copy@npm:^1.0.1, symlink-or-copy@npm:^1.1.8, symlink-or-copy@npm:^1.2.0, symlink-or-copy@npm:^1.3.1": version: 1.3.1 resolution: "symlink-or-copy@npm:1.3.1" @@ -32038,6 +32489,13 @@ __metadata: languageName: node linkType: hard +"tablesort@npm:^5.7.1": + version: 5.7.1 + resolution: "tablesort@npm:5.7.1" + checksum: 10c0/be69ec398438695b87589de702861d4ca126dba0c4a8c1f4bbc28ffa4ea257a623ee764abd769ec88d7f8347dc36b5157b9d36d282c181a8c43b19413cd88854 + languageName: node + linkType: hard + "tagged-tag@npm:^1.0.0": version: 1.0.0 resolution: "tagged-tag@npm:1.0.0" @@ -32651,6 +33109,15 @@ __metadata: languageName: node linkType: hard +"trouter@npm:^2.0.1": + version: 2.0.1 + resolution: "trouter@npm:2.0.1" + dependencies: + matchit: "npm:^1.0.0" + checksum: 10c0/9d294dbe74d45649d929d3abf0d4cb8e508985a2ea0f8be41b3b097a985352b14f65dd76b68ae651731128995eceddba375b15719356c419bbfe754c5fb5673c + languageName: node + linkType: hard + "trouter@npm:^4.0.0": version: 4.0.0 resolution: "trouter@npm:4.0.0" @@ -32700,6 +33167,16 @@ __metadata: languageName: node linkType: hard +"ts-morph@npm:^28.0.0": + version: 28.0.0 + resolution: "ts-morph@npm:28.0.0" + dependencies: + "@ts-morph/common": "npm:~0.29.0" + code-block-writer: "npm:^13.0.3" + checksum: 10c0/969570a5e983b4193735fff43878ce7b78787b4860bf3d23f330c86d7de707286969e5a82d486dce333eea27bf9d373a46f9de58a80c0bf86d2a462620563a55 + languageName: node + linkType: hard + "ts-pnp@npm:^1.1.6": version: 1.2.0 resolution: "ts-pnp@npm:1.2.0" @@ -32869,6 +33346,17 @@ __metadata: languageName: node linkType: hard +"type-is@npm:^2.1.0": + version: 2.1.0 + resolution: "type-is@npm:2.1.0" + dependencies: + content-type: "npm:^2.0.0" + media-typer: "npm:^1.1.0" + mime-types: "npm:^3.0.0" + checksum: 10c0/a6018f8f509de48f2c7429305e3a920e73b374fa93127dd0877ae1c2df65a5d33907caac8afb0c37a9b9fc7c49f29e3f55d668963dc845d966930b667c07f50e + languageName: node + linkType: hard + "type-is@npm:~1.6.18": version: 1.6.18 resolution: "type-is@npm:1.6.18" @@ -33541,6 +34029,15 @@ __metadata: languageName: node linkType: hard +"uuid@npm:14.0.1": + version: 14.0.1 + resolution: "uuid@npm:14.0.1" + bin: + uuid: dist-node/bin/uuid + checksum: 10c0/2d961289097cb68c37d93d1da0b5c3aafc1eb9948a455cca8d0ae53a099b2fe583b8933254a84097f700e6bac2e23033364904df0467c22551d5d9611febcaf6 + languageName: node + linkType: hard + "uuid@npm:^11.1.0": version: 11.1.0 resolution: "uuid@npm:11.1.0" @@ -33758,6 +34255,20 @@ __metadata: languageName: node linkType: hard +"vis-network@npm:^10.1.0": + version: 10.1.0 + resolution: "vis-network@npm:10.1.0" + peerDependencies: + "@egjs/hammerjs": ^2.0.0 + component-emitter: ^1.3.0 || ^2.0.0 + keycharm: ^0.2.0 || ^0.3.0 || ^0.4.0 + uuid: ^3.4.0 || ^7.0.0 || ^8.0.0 || ^9.0.0 || ^10.0.0 || ^11.0.0 || ^13.0.0 || ^14.0.0 + vis-data: ">=8.0.0" + vis-util: ">=6.0.0" + checksum: 10c0/c3f553f03f80e096c7f8cea4fb98011906dde67c65e7d1e8ca0ed5811dad0dc9f648dde93a6a61ce3ac24f153a7a4eb96e931104c8ad1801701a50db687f5bb6 + languageName: node + linkType: hard + "vite-dev-rpc@npm:^1.1.0": version: 1.1.0 resolution: "vite-dev-rpc@npm:1.1.0" From 6047c34ab89905252c3b423a42ab5f7baa5fdafd Mon Sep 17 00:00:00 2001 From: Valentin Palkovic Date: Tue, 28 Jul 2026 18:26:21 +0200 Subject: [PATCH 07/14] Bench: Declare the vue-docgen-api dependency the harness imports The Vue legacy engine harness imports vue-docgen-api, but only code/frameworks/vue3-vite declared it. It resolved by hoisting, which would break the moment the framework dropped it or the install layout changed. Declare it in scripts at the same range the framework uses. --- scripts/package.json | 1 + yarn.lock | 1 + 2 files changed, 2 insertions(+) diff --git a/scripts/package.json b/scripts/package.json index b6ae6ec2e8db..51ca20d9b19e 100644 --- a/scripts/package.json +++ b/scripts/package.json @@ -157,6 +157,7 @@ "vitest": "^4.1.5", "vue": "^3.2.47", "vue-component-meta": "^3.2.7", + "vue-docgen-api": "^4.75.1", "wait-on": "^8.0.3", "window-size": "^1.1.1", "yaml": "^2.8.1", diff --git a/yarn.lock b/yarn.lock index e62ad7e60082..2022fd24cb71 100644 --- a/yarn.lock +++ b/yarn.lock @@ -10631,6 +10631,7 @@ __metadata: vitest: "npm:^4.1.5" vue: "npm:^3.2.47" vue-component-meta: "npm:^3.2.7" + vue-docgen-api: "npm:^4.75.1" wait-on: "npm:^8.0.3" window-size: "npm:^1.1.1" yaml: "npm:^2.8.1" From c2c1afd04a798a1066665a9f9a814a3d7d11f279 Mon Sep 17 00:00:00 2001 From: Valentin Palkovic Date: Tue, 28 Jul 2026 18:26:41 +0200 Subject: [PATCH 08/14] Bench: Share the docgen harness plumbing between the two suites The two docgen bench suites had grown six copies of the argument parser, four copies of the save loop, and two copies each of the memory sampler, the renderer-module loader and the Vue scenario setup. The memory gate, which is the only one that fails a build, imported from the new perf directory, which gates nothing. Both suites now depend on scripts/bench/docgen-shared and neither depends on the other. Engine children implement cold, applySave and reextract; the shared series harness owns the stopwatch, so which part of a save is measured is fixed in one place rather than hand-placed in each child. The orchestrator loses its per-engine branches. An engine is a BenchEngine: SeriesChildEngine spawns a harness child, CompodocEngine drives the CLI and holds the resolved binary. React and Vue are the same class with a different child and different flags. run.ts drops from 556 lines to 195, and the aggregation, ordering, ratio and rendering rules move into modules that tests can reach. Two reporting fixes come with that. Member counts now come from the same repetition the warm and memory figures do, instead of repetition 1, which the suite otherwise avoids on purpose. And a warm ratio now carries the same not-like-for-like warning the cold one did: on the generated projects the legacy Vue parser documents nothing on the save it is timed on, so a bare warm ratio read as a clean speed win. The shared argument parser also rejects what the old one accepted. "--saves --json out.json" used to parse as saves="--json", become NaN, and reach the generator as a silently wrong project size. Tests go from 12 to 72, covering the rules the note relies on: the median-repetition choice, the pinned-N check, the pair alternation, the ratio guards, and that both sides of a control pair stay registered and run by default. No measurement behavior changes. yarn bench:docgen-perf --quick reports the same numbers, and the memory harness still writes every field the gate reads. --- scripts/bench/PERF-METHODOLOGY.md | 10 +- scripts/bench/docgen-memory/gate.ts | 2 +- scripts/bench/docgen-memory/memory-harness.ts | 364 +++++------- scripts/bench/docgen-perf/aggregate.test.ts | 108 ++++ scripts/bench/docgen-perf/aggregate.ts | 93 +++ scripts/bench/docgen-perf/cli.ts | 27 + scripts/bench/docgen-perf/engine.ts | 126 ++++ scripts/bench/docgen-perf/engines/compodoc.ts | 58 +- .../bench/docgen-perf/engines/react-legacy.ts | 259 +++----- .../docgen-perf/engines/vue-component-meta.ts | 244 ++------ .../docgen-perf/engines/vue-docgen-api.ts | 220 ++----- .../bench/docgen-perf/engines/vue-scenario.ts | 137 +++++ .../bench/docgen-perf/generators/angular.ts | 13 +- scripts/bench/docgen-perf/generators/vue.ts | 19 +- scripts/bench/docgen-perf/ratios.test.ts | 132 +++++ scripts/bench/docgen-perf/ratios.ts | 103 ++++ scripts/bench/docgen-perf/registry.test.ts | 123 ++++ scripts/bench/docgen-perf/registry.ts | 91 +++ scripts/bench/docgen-perf/report.test.ts | 97 +++ scripts/bench/docgen-perf/report.ts | 117 ++++ scripts/bench/docgen-perf/run.ts | 557 +++--------------- scripts/bench/docgen-perf/sampling.ts | 29 - scripts/bench/docgen-perf/spawn.ts | 67 +++ scripts/bench/docgen-perf/types.ts | 86 ++- scripts/bench/docgen-shared/args.test.ts | 75 +++ scripts/bench/docgen-shared/args.ts | 81 +++ .../{docgen-perf => docgen-shared}/budgets.ts | 2 +- scripts/bench/docgen-shared/engine-ids.ts | 11 + .../{docgen-perf => docgen-shared}/paths.ts | 0 .../bench/docgen-shared/renderer-module.ts | 64 ++ scripts/bench/docgen-shared/samples.ts | 13 + scripts/bench/docgen-shared/sampling.ts | 39 ++ scripts/bench/docgen-shared/series.ts | 159 +++++ .../stats.test.ts | 0 .../{docgen-perf => docgen-shared}/stats.ts | 4 +- 35 files changed, 2222 insertions(+), 1308 deletions(-) create mode 100644 scripts/bench/docgen-perf/aggregate.test.ts create mode 100644 scripts/bench/docgen-perf/aggregate.ts create mode 100644 scripts/bench/docgen-perf/cli.ts create mode 100644 scripts/bench/docgen-perf/engine.ts create mode 100644 scripts/bench/docgen-perf/engines/vue-scenario.ts create mode 100644 scripts/bench/docgen-perf/ratios.test.ts create mode 100644 scripts/bench/docgen-perf/ratios.ts create mode 100644 scripts/bench/docgen-perf/registry.test.ts create mode 100644 scripts/bench/docgen-perf/registry.ts create mode 100644 scripts/bench/docgen-perf/report.test.ts create mode 100644 scripts/bench/docgen-perf/report.ts delete mode 100644 scripts/bench/docgen-perf/sampling.ts create mode 100644 scripts/bench/docgen-perf/spawn.ts create mode 100644 scripts/bench/docgen-shared/args.test.ts create mode 100644 scripts/bench/docgen-shared/args.ts rename scripts/bench/{docgen-perf => docgen-shared}/budgets.ts (96%) create mode 100644 scripts/bench/docgen-shared/engine-ids.ts rename scripts/bench/{docgen-perf => docgen-shared}/paths.ts (100%) create mode 100644 scripts/bench/docgen-shared/renderer-module.ts create mode 100644 scripts/bench/docgen-shared/samples.ts create mode 100644 scripts/bench/docgen-shared/sampling.ts create mode 100644 scripts/bench/docgen-shared/series.ts rename scripts/bench/{docgen-perf => docgen-shared}/stats.test.ts (100%) rename scripts/bench/{docgen-perf => docgen-shared}/stats.ts (95%) diff --git a/scripts/bench/PERF-METHODOLOGY.md b/scripts/bench/PERF-METHODOLOGY.md index c5e0a186cf31..ba00d6f7ebf8 100644 --- a/scripts/bench/PERF-METHODOLOGY.md +++ b/scripts/bench/PERF-METHODOLOGY.md @@ -62,7 +62,7 @@ Two harnesses implement this. That run is the repetition whose cold sample is the median, never the first: repetition 1 pays for a cold module graph and a cold page cache, and measures several times slower than the rest. The harness pins one N for all engines and records it with the results; numbers taken at different N are not comparable. An engine that fails part-way through its repetitions is reported as failed, never as measured at an N it did not reach. - `scripts/bench/docgen-perf/run.ts` implements this; the memory gate still runs each configuration exactly once. + `scripts/bench/docgen-perf/aggregate.ts` implements this; the memory gate still runs each configuration exactly once. - **Series statistics for leak metrics.** Retained slope is a least-squares fit over the save series; retained growth is the delta between the final retained sample and the pre-run baseline. Both read one run's series instead of repeated runs. @@ -95,9 +95,11 @@ Run `react-legacy.ts --scope all` for the production-shaped number; the equal-wo Vue's pair carries a different caveat, and a sharper one. `vue-docgen-api` does not resolve tsconfig `paths` aliases, and the Vite plugin calls `parse(id)` without an alias map, so on the generated projects it documents a fraction of what `vue-component-meta` documents. -Measured on the flat scenario, the legacy engine finished a cold pass in 23ms against 425ms, but recorded 6 documented members where the new engine recorded the full surface, and 0 on the save it was timed on. +Every run so far has the legacy engine finishing its cold pass an order of magnitude faster while documenting an order of magnitude fewer members, and documenting nothing at all on the save it was timed on. The engines are not doing the same work, so the Vue ratio measures resolution depth as much as speed. -The suite prints both engines' documented-member counts next to the ratio for that reason, and the ratio must not become a budget while the counts disagree. +Both engines therefore report their documented-member counts, and the suite prints those counts beside every ratio - the warm one as much as the cold one - and marks the pair not like-for-like whenever they disagree. +A ratio marked not like-for-like must not become a budget. +The baseline work records the figures; quoting any here would pin numbers taken at one profile on one machine. Engines without a second implementation in the same job get their timing reference picked when their baselines are recorded; until then their timing budgets stay placeholders. Memory budgets stay absolute megabytes with generous headroom: budgets sit well above observed values so the gate is not flaky, while still failing hard on a real regression. Every engine must also carry its own negative control - a configuration that must fail, proving the gate can catch the regression class it exists for. @@ -137,7 +139,7 @@ All three build on existing tooling and add no new dependencies: ## Budgets table skeleton The baseline work fills in the values; until then every cell is a placeholder. -The exception is react-osa's memory row, which the docgen memory gate already enforces from `scripts/bench/docgen-perf/budgets.ts`. +The exception is react-osa's memory row, which the docgen memory gate already enforces from `scripts/bench/docgen-shared/budgets.ts`. | Engine | Cold extraction | Warm extraction | Whole-project scan | Peak memory (transient) | Retained growth | Retained slope | Negative control | Tier | | ----------------------------------------- | --------------- | --------------- | ------------------ | ----------------------- | --------------- | -------------- | ---------------- | ---------- | diff --git a/scripts/bench/docgen-memory/gate.ts b/scripts/bench/docgen-memory/gate.ts index 01af6b4d775a..68ebe7a2a118 100644 --- a/scripts/bench/docgen-memory/gate.ts +++ b/scripts/bench/docgen-memory/gate.ts @@ -23,7 +23,7 @@ import { spawnSync } from 'node:child_process'; import * as fs from 'node:fs'; import * as path from 'node:path'; -import { type MemoryBudgets, memoryBudgetsFor } from '../docgen-perf/budgets.ts'; +import { type MemoryBudgets, memoryBudgetsFor } from '../docgen-shared/budgets.ts'; import { SANDBOX_DIRECTORY } from '../../utils/constants.ts'; interface HarnessResult { diff --git a/scripts/bench/docgen-memory/memory-harness.ts b/scripts/bench/docgen-memory/memory-harness.ts index 5c5c56eb20e0..3a95f582ca41 100644 --- a/scripts/bench/docgen-memory/memory-harness.ts +++ b/scripts/bench/docgen-memory/memory-harness.ts @@ -36,29 +36,25 @@ import * as path from 'node:path'; import ts from 'typescript'; -import { SANDBOX_DIRECTORY } from '../../utils/constants.ts'; -import { leastSquaresSlope, summarizeSeries } from '../docgen-perf/stats.ts'; +import { Args } from '../docgen-shared/args.ts'; +import { SANDBOX_DIRECTORY } from '../docgen-shared/paths.ts'; +import { + type StoryRefLike, + buildStoryRefs, + loadRendererModule, +} from '../docgen-shared/renderer-module.ts'; +import { MB, gcAvailable } from '../docgen-shared/sampling.ts'; +import { + type SeriesEngine, + harnessMain, + printSeriesSummary, + runSeries, +} from '../docgen-shared/series.ts'; +import { leastSquaresSlope } from '../docgen-shared/stats.ts'; import { componentSource, generateProject } from './generate-project.ts'; -/** - * Minimal structural view of the entry shape consumed by `ComponentMetaManager.batchExtract`. Kept - * local so the `scripts` typecheck does not descend into `code/renderers` internals; the real types - * live in `code/renderers/react/src/componentManifest`. - */ -interface StoryRef { - storyPath: string; - component: { - componentName: string; - importName: string; - localImportName: string; - importId: string; - isPackage: boolean; - path: string; - }; -} - interface ComponentMetaManagerLike { - batchExtract(entries: StoryRef[]): void; + batchExtract(entries: StoryRefLike[]): void; onFilesChanged(changes: Array<{ filePath: string; type: 'changed' | 'created' | 'deleted' }>): void; dispose(): void; } @@ -68,35 +64,26 @@ type ComponentMetaManagerCtor = new ( recycleHeapPressureRatio?: number ) => ComponentMetaManagerLike; +const MODES = ['refresh', 'live'] as const; /** - * Load the real `ComponentMetaManager` at runtime. The specifier is built with `new URL` so the - * static `scripts` typecheck does not pull `code/renderers` source into its program. + * Which entries to re-extract per save. + * all - re-extract every component each save (the docgen service refreshing all extracted + * components on an empty change hint). + * changed - re-extract only the component whose file changed. */ -async function loadComponentMetaManager(): Promise { - const url = new URL( - '../../../code/renderers/react/src/componentManifest/componentMeta/ComponentMetaManager.ts', - import.meta.url - ).href; - const mod = (await import(url)) as { ComponentMetaManager: ComponentMetaManagerCtor }; - return mod.ComponentMetaManager; -} +const SCOPES = ['all', 'changed'] as const; +const RECYCLE = ['on', 'off'] as const; interface HarnessOptions { components: number; variants: number; props: number; saves: number; - mode: 'refresh' | 'live'; + mode: (typeof MODES)[number]; heavyTypes: boolean; heavyFactor: number; base64Kb: number; - /** - * Which entries to re-extract per save. - * all – re-extract every component each save (the docgen service refreshing all extracted - * components on an empty change hint). - * changed – re-extract only the component whose file changed. - */ - scope: 'all' | 'changed'; + scope: (typeof SCOPES)[number]; forceGc: boolean; outDir: string; reuse: boolean; @@ -111,81 +98,64 @@ interface HarnessOptions { recycleHeapPressureRatio?: number; } -interface Sample { - save: number; - durMs: number; - rssMb: number; - heapUsedMb: number; - retainedHeapMb?: number; -} - -const MB = 1024 * 1024; - -function parseArgs(argv: string[]): HarnessOptions { - const get = (flag: string, fallback: string) => { - const idx = argv.indexOf(flag); - return idx >= 0 && argv[idx + 1] ? argv[idx + 1] : fallback; - }; - const mode = get('--mode', 'refresh'); - if (mode !== 'refresh' && mode !== 'live') { - throw new Error(`--mode must be "refresh" or "live", got "${mode}"`); - } - const scope = get('--scope', 'all'); - if (scope !== 'all' && scope !== 'changed') { - throw new Error(`--scope must be "all" or "changed", got "${scope}"`); - } - const recycle = get('--recycle', 'on'); - if (recycle !== 'on' && recycle !== 'off') { - throw new Error(`--recycle must be "on" or "off", got "${recycle}"`); - } +function parseOptions(argv: string[]): HarnessOptions { + const args = new Args(argv); return { - components: Number(get('--components', '600')), - variants: Number(get('--variants', '4')), - props: Number(get('--props', '8')), - saves: Number(get('--saves', '25')), - mode, - scope: scope as 'all' | 'changed', - recycleHeapPressureRatio: recycle === 'off' ? Number.POSITIVE_INFINITY : undefined, - heavyTypes: argv.includes('--heavy'), - heavyFactor: Number(get('--heavy-factor', '1')), - base64Kb: Number(get('--base64-kb', '0')), - forceGc: !argv.includes('--no-force-gc'), - outDir: get('--out', path.join(SANDBOX_DIRECTORY, 'docgen-memory-stress')), - reuse: argv.includes('--reuse'), - jsonOut: argv.indexOf('--json') >= 0 ? get('--json', '') : undefined, - maxRetainedGrowthMb: Number(get('--max-retained-growth', '400')), + components: args.count('components', 600), + variants: args.count('variants', 4), + props: args.count('props', 8), + saves: args.count('saves', 25), + mode: args.choice('mode', MODES, 'refresh'), + scope: args.choice('scope', SCOPES, 'all'), + recycleHeapPressureRatio: + args.choice('recycle', RECYCLE, 'on') === 'off' ? Number.POSITIVE_INFINITY : undefined, + heavyTypes: args.flag('heavy'), + heavyFactor: args.count('heavy-factor', 1), + base64Kb: args.count('base64-kb', 0), + forceGc: !args.flag('no-force-gc'), + outDir: args.string('out', path.join(SANDBOX_DIRECTORY, 'docgen-memory-stress')), + reuse: args.flag('reuse'), + jsonOut: args.optional('json'), + maxRetainedGrowthMb: args.count('max-retained-growth', 400), }; } -function gc(): void { - if (typeof global.gc === 'function') { - global.gc(); - global.gc(); - } +/** + * Load the real `ComponentMetaManager` at runtime. The specifier is built with `new URL` so the + * static `scripts` typecheck does not pull `code/renderers` source into its program. + */ +async function loadComponentMetaManager(): Promise { + const mod = await loadRendererModule<{ ComponentMetaManager: ComponentMetaManagerCtor }>( + 'componentMeta/ComponentMetaManager.ts' + ); + return mod.ComponentMetaManager; } -function sampleMemory(forceGc: boolean): { rssMb: number; heapUsedMb: number; retainedHeapMb?: number } { - const pre = process.memoryUsage(); - let retainedHeapMb: number | undefined; - if (forceGc) { - gc(); - retainedHeapMb = process.memoryUsage().heapUsed / MB; +function resolveProject(options: HarnessOptions): ReturnType { + const genStart = Date.now(); + if (options.reuse && fs.existsSync(path.join(path.resolve(options.outDir), 'tsconfig.json'))) { + const outDir = path.resolve(options.outDir); + const componentPaths: string[] = []; + const storyPaths: string[] = []; + for (let i = 0; i < options.components; i++) { + componentPaths.push(path.join(outDir, 'src', `Comp${i}`, `Comp${i}.tsx`)); + storyPaths.push(path.join(outDir, 'src', `Comp${i}`, `Comp${i}.stories.tsx`)); + } + console.log(` reusing generated project at ${outDir}`); + return { outDir, configPath: path.join(outDir, 'tsconfig.json'), componentPaths, storyPaths }; } - return { rssMb: pre.rss / MB, heapUsedMb: pre.heapUsed / MB, retainedHeapMb }; -} - -function buildEntries(componentPaths: string[], storyPaths: string[]): StoryRef[] { - return componentPaths.map((componentPath, i) => ({ - storyPath: storyPaths[i], - component: { - componentName: `Comp${i}`, - importName: `Comp${i}`, - localImportName: `Comp${i}`, - importId: `./Comp${i}`, - isPackage: false, - path: componentPath, - }, - })); + const project = generateProject({ + outDir: options.outDir, + components: options.components, + variants: options.variants, + props: options.props, + heavyTypes: options.heavyTypes, + heavyFactor: options.heavyFactor, + base64Kb: options.base64Kb, + withNodeModules: true, + }); + console.log(` generated project in ${Date.now() - genStart}ms at ${project.outDir}`); + return project; } /** @@ -202,7 +172,7 @@ function buildEntries(componentPaths: string[], storyPaths: string[]): StoryRef[ */ function runLiveMode( manager: ComponentMetaManagerLike, - entries: StoryRef[], + entries: StoryRefLike[], options: HarnessOptions ): void { const recycleEnabled = options.recycleHeapPressureRatio === undefined; @@ -242,144 +212,103 @@ function runLiveMode( } } -async function main() { - const options = parseArgs(process.argv.slice(2)); - const gcAvailable = typeof global.gc === 'function'; +/** + * Refresh-path mode as a save series. The cold pass is the *identical* operation a `scope=all` + * refresh save performs (extractPropsFromStories over every entry), so an OOM there is the same OOM + * every refresh-all save would hit; it simply lands on the first pass when the full-extraction + * working set already exceeds the heap cap. + */ +function refreshEngine( + manager: ComponentMetaManagerLike, + entries: StoryRefLike[], + project: ReturnType, + options: HarnessOptions +): SeriesEngine { + // Track how many extra props each component currently has, so each save grows the type. + const extraByComponent = new Array(options.components).fill(options.props); + const changedIndex = (save: number) => (save - 1) % options.components; + + return { + async cold() { + manager.batchExtract(entries); + return undefined; + }, + async applySave(save) { + const i = changedIndex(save); + const componentPath = project.componentPaths[i]; + // Mutate the component's props interface on disk so the type genuinely changes. + extraByComponent[i] += 1; + fs.writeFileSync( + componentPath, + componentSource(i, extraByComponent[i], { + heavyTypes: options.heavyTypes, + heavyFactor: options.heavyFactor, + base64Kb: options.base64Kb, + }) + ); + // Bump project versions so the next extraction re-reads the mutated file (matches the dev + // server's file-watcher → onFilesChanged flow); without this the program serves stale snapshots. + manager.onFilesChanged([{ filePath: componentPath, type: 'changed' }]); + }, + async reextract(save) { + manager.batchExtract(options.scope === 'changed' ? [entries[changedIndex(save)]] : entries); + return undefined; + }, + dispose: () => manager.dispose(), + }; +} + +harnessMain(async () => { + const options = parseOptions(process.argv.slice(2)); console.log('docgen-memory harness'); console.log( ` components=${options.components} variants=${options.variants} props=${options.props} ` + - `saves=${options.saves} mode=${options.mode} scope=${options.scope} forceGc=${options.forceGc && gcAvailable}` + `saves=${options.saves} mode=${options.mode} scope=${options.scope} ` + + `forceGc=${options.forceGc && gcAvailable()}` ); - if (options.forceGc && !gcAvailable) { + if (options.forceGc && !gcAvailable()) { console.log(' (run with `node --expose-gc` to measure retained heap; continuing without it)'); } - const genStart = Date.now(); - let project: ReturnType; - if (options.reuse && fs.existsSync(path.join(path.resolve(options.outDir), 'tsconfig.json'))) { - const outDir = path.resolve(options.outDir); - const componentPaths: string[] = []; - const storyPaths: string[] = []; - for (let i = 0; i < options.components; i++) { - componentPaths.push(path.join(outDir, 'src', `Comp${i}`, `Comp${i}.tsx`)); - storyPaths.push(path.join(outDir, 'src', `Comp${i}`, `Comp${i}.stories.tsx`)); - } - project = { outDir, configPath: path.join(outDir, 'tsconfig.json'), componentPaths, storyPaths }; - console.log(` reusing generated project at ${outDir}`); - } else { - project = generateProject({ - outDir: options.outDir, - components: options.components, - variants: options.variants, - props: options.props, - heavyTypes: options.heavyTypes, - heavyFactor: options.heavyFactor, - base64Kb: options.base64Kb, - withNodeModules: true, - }); - console.log(` generated project in ${Date.now() - genStart}ms at ${project.outDir}`); - } - + const project = resolveProject(options); const ComponentMetaManager = await loadComponentMetaManager(); const manager = new ComponentMetaManager(ts, options.recycleHeapPressureRatio); - const entries = buildEntries(project.componentPaths, project.storyPaths); + const entries = buildStoryRefs(project.componentPaths, project.storyPaths); if (options.mode === 'live') { runLiveMode(manager, entries, options); return; } - // Initial extraction — builds the full TS program once (the cold path). This is the *identical* - // operation a `scope=all` refresh save performs (extractPropsFromStories over every entry), so an - // OOM here is the same OOM every refresh-all save would hit; it simply lands on the first pass when - // the full-extraction working set already exceeds the heap cap. - const initialStart = Date.now(); - console.log(` full extraction over ${entries.length} components (cold pass == refresh-all save)…`); - manager.batchExtract(entries); - const coldMs = Date.now() - initialStart; - console.log(` initial batchExtract: ${coldMs}ms`); + const series = await runSeries(refreshEngine(manager, entries, project, options), { + saves: options.saves, + coldLabel: `${entries.length} components`, + forceGc: options.forceGc, + }); - const baseline = sampleMemory(options.forceGc && gcAvailable); - console.log( - ` baseline: rss=${baseline.rssMb.toFixed(0)}MB heapUsed=${baseline.heapUsedMb.toFixed(0)}MB` + - (baseline.retainedHeapMb !== undefined - ? ` retained=${baseline.retainedHeapMb.toFixed(0)}MB` - : '') - ); - - const samples: Sample[] = []; - // Track how many extra props each component currently has, so each save grows the type. - const extraByComponent = new Array(options.components).fill(options.props); - - for (let save = 1; save <= options.saves; save++) { - const i = (save - 1) % options.components; - const componentPath = project.componentPaths[i]; - - // Mutate the component's props interface on disk so the type genuinely changes. - extraByComponent[i] += 1; - fs.writeFileSync( - componentPath, - componentSource(i, extraByComponent[i], { - heavyTypes: options.heavyTypes, - heavyFactor: options.heavyFactor, - base64Kb: options.base64Kb, - }) - ); - // Bump project versions so the next extraction re-reads the mutated file (matches the dev - // server's file-watcher → onFilesChanged flow); without this the program serves stale snapshots. - manager.onFilesChanged([{ filePath: componentPath, type: 'changed' }]); - - const saveStart = Date.now(); - const toExtract = options.scope === 'changed' ? [entries[i]] : entries; - manager.batchExtract(toExtract); - const durMs = Date.now() - saveStart; - - const mem = sampleMemory(options.forceGc && gcAvailable); - samples.push({ save, durMs, ...mem }); - console.log( - ` save ${String(save).padStart(3)}: ${String(durMs).padStart(5)}ms ` + - `rss=${mem.rssMb.toFixed(0).padStart(5)}MB heapUsed=${mem.heapUsedMb.toFixed(0).padStart(5)}MB` + - (mem.retainedHeapMb !== undefined - ? ` retained=${mem.retainedHeapMb.toFixed(0).padStart(5)}MB` - : '') - ); - } - - manager.dispose(); - - const rssValues = samples.map((s) => s.rssMb); - const peakRss = Math.max(baseline.rssMb, ...rssValues); - const finalRss = rssValues.at(-1) ?? baseline.rssMb; + const rssValues = series.samples.map((s) => s.rssMb); + const peakRss = Math.max(series.baseline.rssMb, ...rssValues); + const finalRss = rssValues.at(-1) ?? series.baseline.rssMb; const rssSlope = leastSquaresSlope(rssValues); + const { retainedGrowth } = series; - const { retainedSlope, retainedGrowth, avgTransient } = summarizeSeries(samples, baseline); - - console.log('\nsummary'); + printSeriesSummary(series, options.saves); console.log(` peak rss: ${peakRss.toFixed(0)}MB`); - if (avgTransient !== undefined) { - console.log(` avg transient/save: ${avgTransient.toFixed(0)}MB (pre-GC heapUsed − post-GC heapUsed)`); - } console.log(` final rss: ${finalRss.toFixed(0)}MB`); console.log(` rss slope: ${rssSlope.toFixed(1)}MB/save`); - if (retainedSlope !== undefined) { - console.log(` retained slope: ${retainedSlope.toFixed(2)}MB/save (post-GC heapUsed)`); - console.log(` retained growth: ${retainedGrowth!.toFixed(0)}MB over ${options.saves} saves`); + if (retainedGrowth !== undefined) { console.log( - retainedGrowth! > 5 + retainedGrowth > 5 ? ' → classification: RETAINED leak (memory held between saves)' - : ' → classification: TRANSIENT pressure (post-GC heap flat; OOM is GC-can\'t-keep-up)' + : " → classification: TRANSIENT pressure (post-GC heap flat; OOM is GC-can't-keep-up)" ); } if (options.jsonOut) { fs.writeFileSync( options.jsonOut, - JSON.stringify( - { options, coldMs, baseline, samples, peakRss, finalRss, rssSlope, retainedSlope, retainedGrowth, avgTransient }, - null, - 2 - ) + JSON.stringify({ options, ...series, peakRss, finalRss, rssSlope }, null, 2) ); console.log(` wrote ${options.jsonOut}`); } @@ -390,9 +319,4 @@ async function main() { ); process.exitCode = 1; } -} - -main().catch((err) => { - console.error(err); - process.exitCode = 1; }); diff --git a/scripts/bench/docgen-perf/aggregate.test.ts b/scripts/bench/docgen-perf/aggregate.test.ts new file mode 100644 index 000000000000..dea15773852f --- /dev/null +++ b/scripts/bench/docgen-perf/aggregate.test.ts @@ -0,0 +1,108 @@ +import { describe, expect, it } from 'vitest'; + +import type { SaveSample } from '../docgen-shared/samples.ts'; +import type { SeriesResult } from '../docgen-shared/series.ts'; +import { designatedRep, oneShotMetrics, seriesMetrics } from './aggregate.ts'; + +function sample(save: number, durMs: number, heapUsedMb = 20, retainedHeapMb = 10): SaveSample { + return { save, durMs, rssMb: 100, heapUsedMb, retainedHeapMb }; +} + +function rep(coldMs: number, durations: number[]): SeriesResult { + return { + coldMs, + baseline: { rssMb: 90, heapUsedMb: 12, retainedHeapMb: 10 }, + samples: durations.map((d, i) => sample(i + 1, d)), + retainedSlope: 0.5, + retainedGrowth: 4, + }; +} + +describe('designatedRep', () => { + it('picks the repetition whose cold sample is the median', () => { + const reps = [rep(900, [1]), rep(100, [2]), rep(300, [3]), rep(200, [4]), rep(250, [5])]; + expect(designatedRep(reps).coldMs).toBe(250); + }); + + it('never picks the slow first repetition just because it came first', () => { + // Repetition 1 pays for a cold module graph; it must not supply the warm and memory metrics. + const reps = [rep(5000, [999]), rep(100, [10]), rep(110, [11]), rep(120, [12]), rep(130, [13])]; + expect(designatedRep(reps).coldMs).not.toBe(5000); + expect(designatedRep(reps).coldMs).toBe(120); + }); + + it('takes the lower middle for an even count', () => { + const reps = [rep(10, [1]), rep(20, [2]), rep(30, [3]), rep(40, [4])]; + expect(designatedRep(reps).coldMs).toBe(20); + }); + + it('does not mutate its input', () => { + const reps = [rep(30, [1]), rep(10, [2]), rep(20, [3])]; + designatedRep(reps); + expect(reps.map((r) => r.coldMs)).toEqual([30, 10, 20]); + }); +}); + +describe('seriesMetrics', () => { + const reps = [rep(300, [9, 9, 9]), rep(100, [1, 2, 3]), rep(200, [4, 5, 6])]; + + it('medians cold across every repetition', () => { + expect(seriesMetrics(reps, 3).coldExtractionMs).toMatchObject({ + samples: [300, 100, 200], + median: 200, + }); + }); + + it('reads warm from the designated repetition, not the first', () => { + expect(seriesMetrics(reps, 3).warmExtractionMs).toMatchObject({ samples: [4, 5, 6], median: 5 }); + }); + + it('marks whole-project scan as not applicable rather than faking one', () => { + expect(seriesMetrics(reps, 3).wholeProjectScanMs).toEqual({ status: 'n/a' }); + }); + + it('derives transient memory from the same repetition warm came from', () => { + expect(seriesMetrics(reps, 3).peakTransientMb).toMatchObject({ mean: 10 }); + }); + + it('rejects a repetition count below the pinned N', () => { + expect(() => seriesMetrics(reps, 5)).toThrow('recorded 3 repetitions, expected the pinned 5'); + }); + + it('rejects an empty set of repetitions', () => { + expect(() => seriesMetrics([], 5)).toThrow('no completed repetition recorded'); + expect(() => seriesMetrics(undefined, 5)).toThrow('no completed repetition recorded'); + }); + + it('rejects a run with no retained samples', () => { + const noGc: SeriesResult = { + coldMs: 100, + baseline: { rssMb: 90, heapUsedMb: 12 }, + samples: [{ save: 1, durMs: 5, rssMb: 100, heapUsedMb: 20 }], + }; + expect(() => seriesMetrics([noGc], 1)).toThrow('retained metrics missing'); + }); +}); + +describe('oneShotMetrics', () => { + const reps = [ + { coldMs: 300, warmMs: 280, peakRssMb: 500 }, + { coldMs: 100, warmMs: 120, peakRssMb: 400 }, + { coldMs: 200, warmMs: 200, peakRssMb: 450 }, + ]; + + it('reports cold and whole-project scan as the same measurement', () => { + const metrics = oneShotMetrics(reps, 3); + expect(metrics.wholeProjectScanMs).toEqual(metrics.coldExtractionMs); + }); + + it('has no retained series to report', () => { + const metrics = oneShotMetrics(reps, 3); + expect(metrics.retainedGrowthMb).toEqual({ status: 'n/a' }); + expect(metrics.retainedSlopeMbPerSave).toEqual({ status: 'n/a' }); + }); + + it('enforces the pinned repetition count', () => { + expect(() => oneShotMetrics(reps, 5)).toThrow('expected the pinned 5'); + }); +}); diff --git a/scripts/bench/docgen-perf/aggregate.ts b/scripts/bench/docgen-perf/aggregate.ts new file mode 100644 index 000000000000..db32980bdfe4 --- /dev/null +++ b/scripts/bench/docgen-perf/aggregate.ts @@ -0,0 +1,93 @@ +/** + * Turning N repetitions into the five floor metrics, per scripts/bench/PERF-METHODOLOGY.md. + * + * Cold latency is the median across repetitions. Warm latency and all three memory metrics read one + * repetition's save series, and they read the same one, so those four figures always describe the + * same run. + */ +import type { SeriesResult } from '../docgen-shared/series.ts'; +import { mean, median } from '../docgen-shared/stats.ts'; +import { type EngineMetrics, NOT_APPLICABLE } from './types.ts'; + +/** + * The repetition whose cold sample is the median. Repetition 1 is systematically the slowest - it + * pays for a cold module graph and a cold OS page cache - so taking it would bias warm latency and + * all three memory metrics at once. Cold latency needs no such protection: it is already a median + * across every repetition. + */ +export function designatedRep(reps: T[]): T { + const byCold = [...reps].sort((a, b) => a.coldMs - b.coldMs); + return byCold[Math.floor((byCold.length - 1) / 2)]; +} + +/** Per-save allocation spikes above the retained baseline, for the repetitions the child sampled. */ +function transientsOf(rep: SeriesResult): number[] { + return rep.samples + .map((s) => (s.retainedHeapMb !== undefined ? s.heapUsedMb - s.retainedHeapMb : undefined)) + .filter((v): v is number => v !== undefined); +} + +/** + * Reject a repetition count that is not the pinned N. An engine that failed part-way through holds + * fewer samples, and reporting those as a result would put numbers taken at an unrecorded N into the + * results file - which the whole comparison method rests on not happening. + */ +function assertRepetitionCount(reps: unknown[] | undefined, expectedN: number): void { + if (!reps || reps.length === 0) { + throw new Error('no completed repetition recorded'); + } + if (reps.length !== expectedN) { + throw new Error(`recorded ${reps.length} repetitions, expected the pinned ${expectedN}`); + } +} + +export function seriesMetrics(reps: SeriesResult[] | undefined, expectedN: number): EngineMetrics { + assertRepetitionCount(reps, expectedN); + const designated = designatedRep(reps!); + const coldSamples = reps!.map((r) => r.coldMs); + const warmSamples = designated.samples.map((s) => s.durMs); + const transients = transientsOf(designated); + + if ( + transients.length === 0 || + designated.retainedGrowth === undefined || + designated.retainedSlope === undefined + ) { + throw new Error('retained metrics missing (child must run under --expose-gc)'); + } + + return { + coldExtractionMs: { status: 'measured', samples: coldSamples, median: median(coldSamples) }, + warmExtractionMs: { status: 'measured', samples: warmSamples, median: median(warmSamples) }, + // Per-component engines have no batch pass; recording one would be a faked equivalent. + wholeProjectScanMs: NOT_APPLICABLE, + peakTransientMb: { status: 'measured', samples: transients, mean: mean(transients) }, + retainedGrowthMb: { status: 'measured', value: designated.retainedGrowth }, + retainedSlopeMbPerSave: { status: 'measured', value: designated.retainedSlope }, + }; +} + +export interface OneShotRepetition { + coldMs: number; + warmMs: number; + peakRssMb: number; +} + +/** + * Metrics for a one-shot CLI engine: a fresh process per run, so cold extraction and the + * whole-project scan are the same full-project measurement, and there is no retained series to read. + */ +export function oneShotMetrics(reps: OneShotRepetition[], expectedN: number): EngineMetrics { + assertRepetitionCount(reps, expectedN); + const coldSamples = reps.map((r) => r.coldMs); + const warmSamples = reps.map((r) => r.warmMs); + const peaks = reps.map((r) => r.peakRssMb); + return { + coldExtractionMs: { status: 'measured', samples: coldSamples, median: median(coldSamples) }, + warmExtractionMs: { status: 'measured', samples: warmSamples, median: median(warmSamples) }, + wholeProjectScanMs: { status: 'measured', samples: coldSamples, median: median(coldSamples) }, + peakTransientMb: { status: 'measured', samples: peaks, mean: mean(peaks) }, + retainedGrowthMb: NOT_APPLICABLE, + retainedSlopeMbPerSave: NOT_APPLICABLE, + }; +} diff --git a/scripts/bench/docgen-perf/cli.ts b/scripts/bench/docgen-perf/cli.ts new file mode 100644 index 000000000000..080ca2cadeb1 --- /dev/null +++ b/scripts/bench/docgen-perf/cli.ts @@ -0,0 +1,27 @@ +/** Command line for the docgen perf suite. */ +import * as path from 'node:path'; + +import { Args } from '../docgen-shared/args.ts'; +import { ALL_ENGINE_IDS, DEFAULT_ENGINE_IDS } from './registry.ts'; +import type { EngineId } from './types.ts'; + +export interface CliOptions { + quick: boolean; + engines: EngineId[]; + jsonOut: string; +} + +export function parseCliOptions(argv: string[], workRoot: string): CliOptions { + const args = new Args(argv); + const engines = args.all('engine'); + for (const engine of engines) { + if (!ALL_ENGINE_IDS.includes(engine as EngineId)) { + throw new Error(`--engine must be one of ${ALL_ENGINE_IDS.join(', ')}, got "${engine}"`); + } + } + return { + quick: args.flag('quick'), + engines: engines.length ? (engines as EngineId[]) : DEFAULT_ENGINE_IDS, + jsonOut: args.string('json', path.join(workRoot, 'results.json')), + }; +} diff --git a/scripts/bench/docgen-perf/engine.ts b/scripts/bench/docgen-perf/engine.ts new file mode 100644 index 000000000000..f48e206554aa --- /dev/null +++ b/scripts/bench/docgen-perf/engine.ts @@ -0,0 +1,126 @@ +/** + * What the orchestrator knows about an engine. + * + * Engines differ in exactly one way: how one repetition is produced. Five of them spawn a + * series-harness child and get a save series back; Compodoc runs an external CLI and gets one + * triple back. Everything downstream - which aggregation applies, whether there are member counts + * to report - follows from that choice, so it is the only thing a subclass decides. + * + * React and Vue are not separate kinds. They are the same {@link SeriesChildEngine} with a different + * child file and different flags, which is data, not behaviour. + */ +import * as path from 'node:path'; + +import type { EngineId } from '../docgen-shared/engine-ids.ts'; +import type { SeriesResult } from '../docgen-shared/series.ts'; +import { seriesMetrics } from './aggregate.ts'; +import type { SuiteProfile } from './config.ts'; +import type { SeriesChildSpec } from './spawn.ts'; +import type { EngineMetrics } from './types.ts'; + +export interface ScenarioSpec { + name: string; + /** Recorded verbatim in the results so a stored run is self-describing. */ + params: Record; +} + +export interface MeasureContext { + /** Scratch directory for this engine/scenario: generated project and per-repetition JSON. */ + scenarioDir: string; + rssPollIntervalMs: number; + runSeriesChild(spec: SeriesChildSpec, outDir: string, jsonPath: string): SeriesResult; +} + +/** `Sample` is whatever one repetition of this engine produces. */ +export abstract class BenchEngine { + abstract readonly id: EngineId; + + /** Engines outside the default run only measure when named with `--engine`. */ + inDefaultRun = true; + + abstract scenarios(profile: SuiteProfile): ScenarioSpec[]; + + abstract measure(ctx: MeasureContext, scenario: ScenarioSpec, rep: number): Promise; + + abstract aggregate(samples: Sample[], expectedN: number): EngineMetrics; + + /** + * Anything that must resolve before the run starts. A returned string is the skip reason, and a + * skip is reported instead of a failure: a missing external tool is what a partial install looks + * like, not a regression. + */ + preflight(): string | undefined { + return undefined; + } + + /** The resolved version of an externally installed engine, recorded with the results. */ + version(): string | undefined { + return undefined; + } + + /** Members the cold pass documented. Undefined when the engine cannot report a count. */ + coldMembers(_sample: Sample): number | undefined { + return undefined; + } + + /** Members the timed re-extraction documented. */ + warmMembers(_sample: Sample): number | undefined { + return undefined; + } +} + +export interface SeriesChildConfig { + id: EngineId; + /** Child entry point, relative to this directory. */ + child: string; + scenarios(profile: SuiteProfile): ScenarioSpec[]; + args(scenario: ScenarioSpec): string[]; + /** Only the reused docgen-memory harness runs under the jiti loader. */ + jiti?: boolean; + inDefaultRun?: boolean; +} + +/** + * An engine measured by spawning a series-harness child, one fresh process per repetition. The + * child produces the save series; aggregation and member counts are the same for every engine of + * this kind because they all share the series harness. + */ +export class SeriesChildEngine extends BenchEngine { + readonly id: EngineId; + readonly #config: SeriesChildConfig; + + constructor(config: SeriesChildConfig) { + super(); + this.id = config.id; + this.inDefaultRun = config.inDefaultRun ?? true; + this.#config = config; + } + + scenarios(profile: SuiteProfile): ScenarioSpec[] { + return this.#config.scenarios(profile); + } + + async measure(ctx: MeasureContext, scenario: ScenarioSpec, rep: number): Promise { + return ctx.runSeriesChild( + { + childPath: path.join(import.meta.dirname, this.#config.child), + args: this.#config.args(scenario), + jiti: this.#config.jiti, + }, + path.join(ctx.scenarioDir, 'project'), + path.join(ctx.scenarioDir, `rep${rep}.json`) + ); + } + + aggregate(samples: SeriesResult[], expectedN: number): EngineMetrics { + return seriesMetrics(samples, expectedN); + } + + coldMembers(sample: SeriesResult): number | undefined { + return sample.coldMembers; + } + + warmMembers(sample: SeriesResult): number | undefined { + return sample.warmMembers; + } +} diff --git a/scripts/bench/docgen-perf/engines/compodoc.ts b/scripts/bench/docgen-perf/engines/compodoc.ts index b1f570401838..5a67d57f0237 100644 --- a/scripts/bench/docgen-perf/engines/compodoc.ts +++ b/scripts/bench/docgen-perf/engines/compodoc.ts @@ -15,8 +15,12 @@ import { spawn, spawnSync } from 'node:child_process'; import * as fs from 'node:fs'; import * as path from 'node:path'; +import { type OneShotRepetition, oneShotMetrics } from '../aggregate.ts'; +import { BenchEngine, type MeasureContext, type ScenarioSpec } from '../engine.ts'; import { angularComponentSource, generateAngularProject } from '../generators/angular.ts'; -import type { AngularScenarioConfig } from '../config.ts'; +import type { AngularScenarioConfig, SuiteProfile } from '../config.ts'; +import type { EngineId } from '../../docgen-shared/engine-ids.ts'; +import type { EngineMetrics } from '../types.ts'; /** * The version of the binary that actually ran, walked up from the resolved executable rather than @@ -118,12 +122,6 @@ function runCompodocOnce( }); } -export interface CompodocRepetition { - coldMs: number; - warmMs: number; - peakRssMb: number; -} - /** * One repetition: fresh project, cold full run, touch one component, warm full run. Both runs are * fresh compodoc processes, matching the one-sample-per-fresh-process topology. @@ -133,7 +131,7 @@ export async function runCompodocRepetition( scenario: AngularScenarioConfig, workDir: string, pollIntervalMs: number -): Promise { +): Promise { const projectDir = path.join(workDir, 'project'); const docsOutDir = path.join(workDir, 'docs'); const project = generateAngularProject({ @@ -154,3 +152,47 @@ export async function runCompodocRepetition( peakRssMb: Math.max(cold.peakRssMb, warm.peakRssMb), }; } + +/** + * Compodoc as an engine. Unlike the series engines it has no child harness to spawn: the CLI is the + * measured process, and this class drives it directly. + * + * It is also the only engine with state - the resolved binary, found once in {@link preflight} and + * reused for every repetition and for the recorded version. + */ +export class CompodocEngine extends BenchEngine { + readonly id: EngineId = 'compodoc'; + + #binary: string | undefined; + + scenarios(profile: SuiteProfile): ScenarioSpec[] { + return [{ name: 'default', params: { ...profile.angular } }]; + } + + preflight(): string | undefined { + this.#binary = resolveCompodocBinary(); + return this.#binary + ? undefined + : 'no compodoc binary found (workspace node_modules/.bin or PATH); @compodoc/compodoc is pinned in scripts/package.json, so run yarn install'; + } + + version(): string | undefined { + return this.#binary ? compodocVersion(this.#binary) : undefined; + } + + async measure(ctx: MeasureContext, scenario: ScenarioSpec): Promise { + if (!this.#binary) { + throw new Error('compodoc binary unresolved; preflight must run first'); + } + return runCompodocRepetition( + this.#binary, + { components: Number(scenario.params.components), props: Number(scenario.params.props) }, + ctx.scenarioDir, + ctx.rssPollIntervalMs + ); + } + + aggregate(samples: OneShotRepetition[], expectedN: number): EngineMetrics { + return oneShotMetrics(samples, expectedN); + } +} diff --git a/scripts/bench/docgen-perf/engines/react-legacy.ts b/scripts/bench/docgen-perf/engines/react-legacy.ts index 0e85ea135ffa..5dd468f3a405 100644 --- a/scripts/bench/docgen-perf/engines/react-legacy.ts +++ b/scripts/bench/docgen-perf/engines/react-legacy.ts @@ -1,11 +1,9 @@ /** * Series harness for the legacy React docgen engines: `react-docgen` (the budgeted legacy control) - * and `react-docgen-typescript` (measurable, no budget row). Same shape as the docgen-memory - * harness: generate a project, one timed cold pass, then per-save touch + invalidate + re-extract - * with memory sampling around forced GC. + * and `react-docgen-typescript` (measurable, no budget row). * * Both engines cache per file for the life of the process and expose only global invalidation, so - * every simulated save calls the global invalidation first - skipping it would measure a cache hit. + * every simulated save invalidates before re-extracting - skipping it would measure a cache hit. * The react-docgen-typescript parser additionally keys its TS program off `process.cwd()`, so this * harness chdirs into the generated project before the first parse. * @@ -19,120 +17,59 @@ import * as fs from 'node:fs'; import * as path from 'node:path'; -import { SANDBOX_DIRECTORY } from '../paths.ts'; import { componentSource, generateProject } from '../../docgen-memory/generate-project.ts'; -import { gcAvailable, sampleMemory } from '../sampling.ts'; -import { summarizeSeries } from '../stats.ts'; -import type { SaveSample } from '../types.ts'; - +import { Args } from '../../docgen-shared/args.ts'; +import { SANDBOX_DIRECTORY } from '../../docgen-shared/paths.ts'; +import { + type ReactDocgenModule, + type ReactDocgenTypescriptModule, + type UtilsModule, + componentRef, + loadRendererModule, +} from '../../docgen-shared/renderer-module.ts'; +import { + type SeriesEngine, + harnessMain, + runSeriesHarness, +} from '../../docgen-shared/series.ts'; + +const PARSERS = ['react-docgen', 'react-docgen-typescript'] as const; /** - * Minimal structural views of the legacy engine modules, kept local so the `scripts` typecheck does - * not descend into `code/renderers` internals; the real types live in - * `code/renderers/react/src/componentManifest`. + * Which components to re-extract per save. + * all - re-extract every component, the shape `generator.ts` actually runs. + * changed - re-extract only the component whose file changed. */ -interface ComponentRefLike { - componentName: string; - importName: string; - localImportName: string; - importId: string; - isPackage: boolean; - path: string; -} - -interface ReactDocgenModule { - getReactDocgen( - path: string, - component: ComponentRefLike - ): { type: 'success' } | { type: 'error'; error: { name: string; message: string } }; -} - -interface UtilsModule { - invalidateCache(): void; -} - -interface ReactDocgenTypescriptModule { - parseWithReactDocgenTypescript(filePath: string): Promise>; - invalidateParser(): void; -} - -/** - * Load a `code/renderers/react` module at runtime. The specifier is built with `new URL` so the - * static `scripts` typecheck does not pull `code/renderers` source into its program. - */ -async function loadRendererModule(relativePath: string): Promise { - const url = new URL( - `../../../../code/renderers/react/src/componentManifest/${relativePath}`, - import.meta.url - ).href; - return (await import(url)) as T; -} - -type Parser = 'react-docgen' | 'react-docgen-typescript'; +const SCOPES = ['all', 'changed'] as const; interface HarnessOptions { - parser: Parser; + parser: (typeof PARSERS)[number]; components: number; variants: number; props: number; saves: number; - /** - * Which components to re-extract per save. Mirrors the docgen-memory harness's flag. - * all – re-extract every component, the shape `generator.ts` actually runs. - * changed – re-extract only the component whose file changed. - */ - scope: 'all' | 'changed'; + scope: (typeof SCOPES)[number]; outDir: string; jsonOut?: string; } -function parseArgs(argv: string[]): HarnessOptions { - const get = (flag: string, fallback: string) => { - const idx = argv.indexOf(flag); - return idx >= 0 && argv[idx + 1] ? argv[idx + 1] : fallback; - }; - const parser = get('--parser', 'react-docgen'); - if (parser !== 'react-docgen' && parser !== 'react-docgen-typescript') { - throw new Error(`--parser must be "react-docgen" or "react-docgen-typescript", got "${parser}"`); - } - const scope = get('--scope', 'changed'); - if (scope !== 'all' && scope !== 'changed') { - throw new Error(`--scope must be "all" or "changed", got "${scope}"`); - } - return { - parser, - components: Number(get('--components', '300')), - variants: Number(get('--variants', '4')), - props: Number(get('--props', '10')), - saves: Number(get('--saves', '20')), - scope, - outDir: get('--out', path.join(SANDBOX_DIRECTORY, 'docgen-perf', 'react-legacy', 'project')), - jsonOut: argv.indexOf('--json') >= 0 ? get('--json', '') : undefined, - }; -} - -function componentRef(i: number, componentPath: string): ComponentRefLike { +function parseOptions(argv: string[]): HarnessOptions { + const args = new Args(argv); return { - componentName: `Comp${i}`, - importName: `Comp${i}`, - localImportName: `Comp${i}`, - importId: `./Comp${i}`, - isPackage: false, - path: componentPath, + parser: args.choice('parser', PARSERS, 'react-docgen'), + components: args.count('components', 300), + variants: args.count('variants', 4), + props: args.count('props', 10), + saves: args.count('saves', 20), + scope: args.choice('scope', SCOPES, 'changed'), + outDir: args.string( + 'out', + path.join(SANDBOX_DIRECTORY, 'docgen-perf', 'react-legacy', 'project') + ), + jsonOut: args.optional('json'), }; } -async function main() { - const options = parseArgs(process.argv.slice(2)); - - console.log(`react-legacy harness (${options.parser})`); - console.log( - ` components=${options.components} variants=${options.variants} props=${options.props} ` + - `saves=${options.saves} scope=${options.scope}` - ); - if (!gcAvailable()) { - console.log(' (run with `node --expose-gc` to measure retained heap; continuing without it)'); - } - +async function createEngine(options: HarnessOptions): Promise { const genStart = Date.now(); const project = generateProject({ outDir: options.outDir, @@ -148,15 +85,18 @@ async function main() { const { invalidateCache } = await loadRendererModule('utils.ts'); - let extractOne: (i: number) => Promise | void; + let extractOne: (i: number) => Promise; let invalidate: () => void; if (options.parser === 'react-docgen') { const { getReactDocgen } = await loadRendererModule('reactDocgen.ts'); - extractOne = (i) => { - const result = getReactDocgen(project.componentPaths[i], componentRef(i, project.componentPaths[i])); + extractOne = async (i) => { + const componentPath = project.componentPaths[i]; + const result = getReactDocgen(componentPath, componentRef(i, componentPath)); if (result.type === 'error') { - throw new Error(`react-docgen failed on Comp${i}: ${result.error.name} ${result.error.message}`); + throw new Error( + `react-docgen failed on Comp${i}: ${result.error.name} ${result.error.message}` + ); } }; invalidate = invalidateCache; @@ -177,74 +117,55 @@ async function main() { }; } - console.log(` full extraction over ${options.components} components (cold pass)…`); - const coldStart = Date.now(); - for (let i = 0; i < options.components; i++) { - await extractOne(i); - } - const coldMs = Date.now() - coldStart; - console.log(` cold pass: ${coldMs}ms`); - - const baseline = sampleMemory(true); - - const samples: SaveSample[] = []; - const extraByComponent = new Array(options.components).fill(options.props); - - for (let save = 1; save <= options.saves; save++) { - const i = (save - 1) % options.components; - extraByComponent[i] += 1; - fs.writeFileSync(project.componentPaths[i], componentSource(i, extraByComponent[i])); - // Global invalidation is the only surface these engines expose; without it the re-extraction - // below is a cache hit. - invalidate(); - - const saveStart = Date.now(); - if (options.scope === 'all') { - for (let c = 0; c < options.components; c++) { - await extractOne(c); - } - } else { + const extractAll = async () => { + for (let i = 0; i < options.components; i++) { await extractOne(i); } - const durMs = Date.now() - saveStart; - - const mem = sampleMemory(true); - samples.push({ save, durMs, ...mem }); - console.log( - ` save ${String(save).padStart(3)}: ${String(durMs).padStart(5)}ms ` + - `rss=${mem.rssMb.toFixed(0).padStart(5)}MB heapUsed=${mem.heapUsedMb.toFixed(0).padStart(5)}MB` + - (mem.retainedHeapMb !== undefined - ? ` retained=${mem.retainedHeapMb.toFixed(0).padStart(5)}MB` - : '') - ); - } - - const { retainedSlope, retainedGrowth, avgTransient } = summarizeSeries(samples, baseline); + }; - console.log('\nsummary'); - console.log(` cold pass: ${coldMs}ms`); - if (avgTransient !== undefined) { - console.log(` avg transient/save: ${avgTransient.toFixed(0)}MB`); - } - if (retainedSlope !== undefined && retainedGrowth !== undefined) { - console.log(` retained slope: ${retainedSlope.toFixed(2)}MB/save`); - console.log(` retained growth: ${retainedGrowth.toFixed(0)}MB over ${options.saves} saves`); - } + // Track how many extra props each component currently has, so each save grows its type. + const extraByComponent = new Array(options.components).fill(options.props); + const changedIndex = (save: number) => (save - 1) % options.components; - if (options.jsonOut) { - fs.writeFileSync( - options.jsonOut, - JSON.stringify( - { options, coldMs, baseline, samples, retainedSlope, retainedGrowth, avgTransient }, - null, - 2 - ) - ); - console.log(` wrote ${options.jsonOut}`); - } + return { + async cold() { + await extractAll(); + return undefined; + }, + async applySave(save) { + const i = changedIndex(save); + extraByComponent[i] += 1; + fs.writeFileSync(project.componentPaths[i], componentSource(i, extraByComponent[i])); + // Global invalidation is the only surface these engines expose; without it the re-extraction + // below is a cache hit. + invalidate(); + }, + async reextract(save) { + if (options.scope === 'all') { + await extractAll(); + } else { + await extractOne(changedIndex(save)); + } + return undefined; + }, + }; } -main().catch((err) => { - console.error(err); - process.exitCode = 1; +harnessMain(async () => { + const options = parseOptions(process.argv.slice(2)); + await runSeriesHarness({ + title: `react-legacy harness (${options.parser})`, + options, + banner: { + components: options.components, + variants: options.variants, + props: options.props, + saves: options.saves, + scope: options.scope, + }, + saves: options.saves, + coldLabel: `${options.components} components`, + jsonOut: options.jsonOut, + setup: () => createEngine(options), + }); }); diff --git a/scripts/bench/docgen-perf/engines/vue-component-meta.ts b/scripts/bench/docgen-perf/engines/vue-component-meta.ts index 52c5274a9f8c..0b88c5994a76 100644 --- a/scripts/bench/docgen-perf/engines/vue-component-meta.ts +++ b/scripts/bench/docgen-perf/engines/vue-component-meta.ts @@ -1,54 +1,31 @@ /** - * Series harness for the vue-component-meta engine. Same shape as the docgen-memory harness: - * generate a project, one timed cold pass (checker creation included - it is part of the engine's - * first-extraction cost), then per-save touch + invalidate + re-extract with memory sampling. + * Series harness for the vue-component-meta engine - Vue's opt-in successor to vue-docgen-api. * * The checker never re-stats files on its own; a disk rewrite must be followed by * `checker.updateFile(path, content)` (the surface the production Vite plugin drives from HMR) or * the re-extraction measures a stale-cache no-op. * - * Scenarios: - * flat single package, checker over the project root (`createCheckerByJson`, the - * production fallback for tsconfig-less projects); saves touch components - * round-robin. - * workspace packages/* layout; the checker is driven at the deepest package's tsconfig - * (`createChecker`); saves touch that package's components round-robin. - * base-type-touch workspace layout; every save touches the widely-imported base type instead, - * then re-extracts one fixed dependent component, so each sample pays the wide - * checker invalidation. + * Checker creation is inside the cold pass because it is part of the engine's first-extraction cost. + * The `flat` scenario drives `createCheckerByJson` (the production fallback for tsconfig-less + * projects); the workspace scenarios drive `createChecker` at the deepest package's tsconfig. * * Run: * node --expose-gc scripts/bench/docgen-perf/engines/vue-component-meta.ts \ * --scenario workspace --packages 4 --components-per-package 10 --heavy-lib --json /tmp/result.json */ import * as fs from 'node:fs'; -import * as path from 'node:path'; -import { - type MetaCheckerOptions, - createChecker, - createCheckerByJson, -} from 'vue-component-meta'; - -import { SANDBOX_DIRECTORY } from '../paths.ts'; -import { baseTypesSource, generateVueProject, vueComponentSource } from '../generators/vue.ts'; -import { gcAvailable, sampleMemory } from '../sampling.ts'; -import { summarizeSeries } from '../stats.ts'; -import type { SaveSample } from '../types.ts'; +import { type MetaCheckerOptions, createChecker, createCheckerByJson } from 'vue-component-meta'; -type Scenario = 'flat' | 'workspace' | 'base-type-touch'; +import { type SeriesEngine, harnessMain, runSeriesHarness } from '../../docgen-shared/series.ts'; +import { + type VueHarnessOptions, + parseVueOptions, + setUpVueScenario, + vueBanner, +} from './vue-scenario.ts'; -interface HarnessOptions { - scenario: Scenario; - packages: number; - componentsPerPackage: number; - chainDepth: number; - fanOut: number; - heavyLib: boolean; - saves: number; - outDir: string; - jsonOut?: string; -} +type Checker = ReturnType; /** Mirrors the production Vite plugin's checker options. */ const CHECKER_OPTIONS: MetaCheckerOptions = { @@ -57,154 +34,63 @@ const CHECKER_OPTIONS: MetaCheckerOptions = { printer: { newLine: 1 }, }; -function parseArgs(argv: string[]): HarnessOptions { - const get = (flag: string, fallback: string) => { - const idx = argv.indexOf(flag); - return idx >= 0 && argv[idx + 1] ? argv[idx + 1] : fallback; - }; - const scenario = get('--scenario', 'workspace'); - if (scenario !== 'flat' && scenario !== 'workspace' && scenario !== 'base-type-touch') { - throw new Error(`--scenario must be "flat", "workspace" or "base-type-touch", got "${scenario}"`); +function extractOne(checker: Checker, sfcPath: string): number { + const exportNames = checker.getExportNames(sfcPath); + if (!exportNames.includes('default')) { + throw new Error( + `no default export found in ${sfcPath} (got: ${exportNames.join(', ') || 'none'})` + ); } - return { - scenario, - packages: Number(get('--packages', '4')), - componentsPerPackage: Number(get('--components-per-package', '10')), - chainDepth: Number(get('--chain-depth', '3')), - fanOut: Number(get('--fan-out', '4')), - heavyLib: argv.includes('--heavy-lib'), - saves: Number(get('--saves', '15')), - outDir: get('--out', path.join(SANDBOX_DIRECTORY, 'docgen-perf', 'vue-component-meta', 'project')), - jsonOut: argv.indexOf('--json') >= 0 ? get('--json', '') : undefined, - }; -} - -async function main() { - const options = parseArgs(process.argv.slice(2)); - const flat = options.scenario === 'flat'; - const packages = flat ? 1 : options.packages; - - console.log(`vue-component-meta harness (${options.scenario})`); - console.log( - ` packages=${packages} componentsPerPackage=${options.componentsPerPackage} ` + - `chainDepth=${options.chainDepth} fanOut=${options.fanOut} heavyLib=${options.heavyLib} saves=${options.saves}` - ); - if (!gcAvailable()) { - console.log(' (run with `node --expose-gc` to measure retained heap; continuing without it)'); + const meta = checker.getComponentMeta(sfcPath, 'default'); + if (meta.props.length === 0) { + throw new Error(`vue-component-meta returned zero props for ${sfcPath}`); } + // Consume the surfaces production consumes so their evaluation cost is inside the measurement. + return meta.props.length + meta.events.length + meta.slots.length + meta.exposed.length; +} - const genStart = Date.now(); - const project = generateVueProject({ - outDir: options.outDir, - packages, - componentsPerPackage: options.componentsPerPackage, - chainDepth: options.chainDepth, - fanOut: options.fanOut, - heavyLib: options.heavyLib, - }); - console.log(` generated project in ${Date.now() - genStart}ms at ${project.outDir}`); - - // The measured set: every component in flat layout, the deepest package's components otherwise - // (their prop types pull the whole cross-package chain). - const targetPackage = packages - 1; - const targetPaths = flat - ? project.componentPaths - : project.componentPaths.slice(-options.componentsPerPackage); +function createEngine(options: VueHarnessOptions): SeriesEngine { + const scenario = setUpVueScenario(options); + let checker: Checker | undefined; + let measuredPath = scenario.targetPaths[0]; - const extractOne = (checker: ReturnType, sfcPath: string) => { - const exportNames = checker.getExportNames(sfcPath); - if (!exportNames.includes('default')) { - throw new Error(`no default export found in ${sfcPath} (got: ${exportNames.join(', ') || 'none'})`); - } - const meta = checker.getComponentMeta(sfcPath, 'default'); - // Consume the surfaces production consumes so their evaluation cost is inside the measurement. - const propCount = meta.props.length + meta.events.length + meta.slots.length + meta.exposed.length; - if (meta.props.length === 0) { - throw new Error(`vue-component-meta returned zero props for ${sfcPath}`); - } - return propCount; + return { + async cold() { + // Checker creation is timed with the cold pass; it is part of first-extraction cost. + checker = + options.scenario === 'flat' + ? createCheckerByJson(scenario.project.outDir, { include: ['**/*'] }, CHECKER_OPTIONS) + : createChecker(scenario.project.packageConfigPaths[scenario.targetPackage], CHECKER_OPTIONS); + let members = 0; + for (const sfcPath of scenario.targetPaths) { + members += extractOne(checker, sfcPath); + } + return members; + }, + async applySave(save) { + const mutation = scenario.mutationFor(save); + fs.writeFileSync(mutation.filePath, mutation.content); + // The checker does not re-stat on its own; this is the HMR surface the Vite plugin drives. + checker!.updateFile(mutation.filePath, mutation.content); + measuredPath = mutation.measuredPath; + }, + async reextract() { + return extractOne(checker!, measuredPath); + }, }; - - // Cold pass: checker creation is part of the engine's first-extraction cost. - console.log(` full extraction over ${targetPaths.length} components (cold pass, checker included)…`); - const coldStart = Date.now(); - const checker = flat - ? createCheckerByJson(project.outDir, { include: ['**/*'] }, CHECKER_OPTIONS) - : createChecker(project.packageConfigPaths[targetPackage], CHECKER_OPTIONS); - let coldMembers = 0; - for (const sfcPath of targetPaths) { - coldMembers += extractOne(checker, sfcPath); - } - const coldMs = Date.now() - coldStart; - console.log(` cold pass: ${coldMs}ms (${coldMembers} documented members)`); - - const baseline = sampleMemory(true); - - const samples: SaveSample[] = []; - const extraByTarget = new Array(targetPaths.length).fill(0); - let extraBaseProps = 0; - - for (let save = 1; save <= options.saves; save++) { - let measuredPath: string; - if (options.scenario === 'base-type-touch') { - // Touch the widely-imported base type; every dependent package's props type changes. - extraBaseProps += 1; - const content = baseTypesSource(options.fanOut, extraBaseProps); - fs.writeFileSync(project.baseTypesPath, content); - checker.updateFile(project.baseTypesPath, content); - // Re-extract one fixed dependent - the first component of the deepest package - so each - // sample pays the wide invalidation. - measuredPath = targetPaths[0]; - } else { - const t = (save - 1) % targetPaths.length; - extraByTarget[t] += 1; - const content = vueComponentSource(targetPackage, t, extraByTarget[t]); - fs.writeFileSync(targetPaths[t], content); - checker.updateFile(targetPaths[t], content); - measuredPath = targetPaths[t]; - } - - const saveStart = Date.now(); - extractOne(checker, measuredPath); - const durMs = Date.now() - saveStart; - - const mem = sampleMemory(true); - samples.push({ save, durMs, ...mem }); - console.log( - ` save ${String(save).padStart(3)}: ${String(durMs).padStart(5)}ms ` + - `rss=${mem.rssMb.toFixed(0).padStart(5)}MB heapUsed=${mem.heapUsedMb.toFixed(0).padStart(5)}MB` + - (mem.retainedHeapMb !== undefined - ? ` retained=${mem.retainedHeapMb.toFixed(0).padStart(5)}MB` - : '') - ); - } - - const { retainedSlope, retainedGrowth, avgTransient } = summarizeSeries(samples, baseline); - - console.log('\nsummary'); - console.log(` cold pass: ${coldMs}ms`); - if (avgTransient !== undefined) { - console.log(` avg transient/save: ${avgTransient.toFixed(0)}MB`); - } - if (retainedSlope !== undefined && retainedGrowth !== undefined) { - console.log(` retained slope: ${retainedSlope.toFixed(2)}MB/save`); - console.log(` retained growth: ${retainedGrowth.toFixed(0)}MB over ${options.saves} saves`); - } - - if (options.jsonOut) { - fs.writeFileSync( - options.jsonOut, - JSON.stringify( - { options, coldMs, coldMembers, baseline, samples, retainedSlope, retainedGrowth, avgTransient }, - null, - 2 - ) - ); - console.log(` wrote ${options.jsonOut}`); - } } -main().catch((err) => { - console.error(err); - process.exitCode = 1; +harnessMain(async () => { + const options = parseVueOptions(process.argv.slice(2), 'vue-component-meta'); + await runSeriesHarness({ + title: `vue-component-meta harness (${options.scenario})`, + options, + banner: vueBanner(options), + saves: options.saves, + // Both layouts measure one package's worth of components: every component when flat, the + // deepest package's when not. + coldLabel: `${options.componentsPerPackage} components, checker creation included`, + jsonOut: options.jsonOut, + setup: async () => createEngine(options), + }); }); diff --git a/scripts/bench/docgen-perf/engines/vue-docgen-api.ts b/scripts/bench/docgen-perf/engines/vue-docgen-api.ts index 4340c8b7c1ca..ab527c7ed533 100644 --- a/scripts/bench/docgen-perf/engines/vue-docgen-api.ts +++ b/scripts/bench/docgen-perf/engines/vue-docgen-api.ts @@ -6,190 +6,74 @@ * Production calls `parse(id)` once per `.vue` file from a Vite `transform` hook * (`code/frameworks/vue3-vite/src/plugins/vue-docgen.ts`), and the parser reads the file from disk * on every call with no cache of its own. A save therefore costs exactly one `parse` of one file, - * so the per-save sample here is production-shaped without any invalidation step. + * so the per-save sample is production-shaped without any invalidation step. * - * The generated project is the same one the vue-component-meta harness measures, with the same - * scenario levers, so the two engines' medians divide into a ratio that means something. + * The generated project and the save sequence come from the shared scenario module, so they are + * identical to the ones vue-component-meta measures. * * Run: * node --expose-gc scripts/bench/docgen-perf/engines/vue-docgen-api.ts \ * --scenario workspace --packages 4 --components-per-package 10 --heavy-lib --json /tmp/result.json */ import * as fs from 'node:fs'; -import * as path from 'node:path'; import { parse } from 'vue-docgen-api'; -import { SANDBOX_DIRECTORY } from '../paths.ts'; -import { baseTypesSource, generateVueProject, vueComponentSource } from '../generators/vue.ts'; -import { gcAvailable, sampleMemory } from '../sampling.ts'; -import { summarizeSeries } from '../stats.ts'; -import type { SaveSample } from '../types.ts'; +import { type SeriesEngine, harnessMain, runSeriesHarness } from '../../docgen-shared/series.ts'; +import { + type VueHarnessOptions, + parseVueOptions, + setUpVueScenario, + vueBanner, +} from './vue-scenario.ts'; -type Scenario = 'flat' | 'workspace' | 'base-type-touch'; - -interface HarnessOptions { - scenario: Scenario; - packages: number; - componentsPerPackage: number; - chainDepth: number; - fanOut: number; - heavyLib: boolean; - saves: number; - outDir: string; - jsonOut?: string; +/** + * How many documented members the parse produced. The two Vue engines resolve imported prop types + * to different depths, so a timing ratio is only meaningful next to this count - a parse that + * resolved nothing is fast for the wrong reason. + */ +async function extractOne(sfcPath: string): Promise { + const doc = await parse(sfcPath); + return ( + Object.keys(doc.props ?? {}).length + + Object.keys(doc.events ?? {}).length + + Object.keys(doc.slots ?? {}).length + + Object.keys(doc.expose ?? {}).length + ); } -function parseArgs(argv: string[]): HarnessOptions { - const get = (flag: string, fallback: string) => { - const idx = argv.indexOf(flag); - return idx >= 0 && argv[idx + 1] ? argv[idx + 1] : fallback; - }; - const scenario = get('--scenario', 'workspace'); - if (scenario !== 'flat' && scenario !== 'workspace' && scenario !== 'base-type-touch') { - throw new Error(`--scenario must be "flat", "workspace" or "base-type-touch", got "${scenario}"`); - } +function createEngine(options: VueHarnessOptions): SeriesEngine { + const scenario = setUpVueScenario(options); + let measuredPath = scenario.targetPaths[0]; + return { - scenario, - packages: Number(get('--packages', '4')), - componentsPerPackage: Number(get('--components-per-package', '10')), - chainDepth: Number(get('--chain-depth', '3')), - fanOut: Number(get('--fan-out', '4')), - heavyLib: argv.includes('--heavy-lib'), - saves: Number(get('--saves', '15')), - outDir: get('--out', path.join(SANDBOX_DIRECTORY, 'docgen-perf', 'vue-docgen-api', 'project')), - jsonOut: argv.indexOf('--json') >= 0 ? get('--json', '') : undefined, + async cold() { + let members = 0; + for (const sfcPath of scenario.targetPaths) { + members += await extractOne(sfcPath); + } + return members; + }, + async applySave(save) { + const mutation = scenario.mutationFor(save); + fs.writeFileSync(mutation.filePath, mutation.content); + measuredPath = mutation.measuredPath; + }, + async reextract() { + return extractOne(measuredPath); + }, }; } -async function main() { - const options = parseArgs(process.argv.slice(2)); - const flat = options.scenario === 'flat'; - const packages = flat ? 1 : options.packages; - - console.log(`vue-docgen-api harness (${options.scenario})`); - console.log( - ` packages=${packages} componentsPerPackage=${options.componentsPerPackage} ` + - `chainDepth=${options.chainDepth} fanOut=${options.fanOut} heavyLib=${options.heavyLib} saves=${options.saves}` - ); - if (!gcAvailable()) { - console.log(' (run with `node --expose-gc` to measure retained heap; continuing without it)'); - } - - const genStart = Date.now(); - const project = generateVueProject({ - outDir: options.outDir, - packages, - componentsPerPackage: options.componentsPerPackage, - chainDepth: options.chainDepth, - fanOut: options.fanOut, - heavyLib: options.heavyLib, +harnessMain(async () => { + const options = parseVueOptions(process.argv.slice(2), 'vue-docgen-api'); + await runSeriesHarness({ + title: `vue-docgen-api harness (${options.scenario})`, + options, + banner: vueBanner(options), + saves: options.saves, + coldLabel: `${options.componentsPerPackage} components`, + jsonOut: options.jsonOut, + setup: async () => createEngine(options), }); - console.log(` generated project in ${Date.now() - genStart}ms at ${project.outDir}`); - - // Same measured set as the vue-component-meta harness: every component in flat layout, the - // deepest package's components otherwise. - const targetPackage = packages - 1; - const targetPaths = flat - ? project.componentPaths - : project.componentPaths.slice(-options.componentsPerPackage); - - /** - * Returns how many documented members the parse produced. The two Vue engines resolve imported - * prop types to different depths, so a timing ratio is only meaningful next to this count - - * a parse that resolved nothing is fast for the wrong reason. - */ - const extractOne = async (sfcPath: string): Promise => { - const doc = await parse(sfcPath); - return ( - Object.keys(doc.props ?? {}).length + - (doc.events ? Object.keys(doc.events).length : 0) + - (doc.slots ? Object.keys(doc.slots).length : 0) + - (doc.expose ? Object.keys(doc.expose).length : 0) - ); - }; - - console.log(` full extraction over ${targetPaths.length} components (cold pass)…`); - const coldStart = Date.now(); - let coldMembers = 0; - for (const sfcPath of targetPaths) { - coldMembers += await extractOne(sfcPath); - } - const coldMs = Date.now() - coldStart; - console.log(` cold pass: ${coldMs}ms (${coldMembers} documented members)`); - - const baseline = sampleMemory(true); - - const samples: SaveSample[] = []; - const extraByTarget = new Array(targetPaths.length).fill(0); - let extraBaseProps = 0; - let warmMembers = 0; - - for (let save = 1; save <= options.saves; save++) { - let measuredPath: string; - if (options.scenario === 'base-type-touch') { - extraBaseProps += 1; - fs.writeFileSync(project.baseTypesPath, baseTypesSource(options.fanOut, extraBaseProps)); - measuredPath = targetPaths[0]; - } else { - const t = (save - 1) % targetPaths.length; - extraByTarget[t] += 1; - fs.writeFileSync(targetPaths[t], vueComponentSource(targetPackage, t, extraByTarget[t])); - measuredPath = targetPaths[t]; - } - - const saveStart = Date.now(); - warmMembers = await extractOne(measuredPath); - const durMs = Date.now() - saveStart; - - const mem = sampleMemory(true); - samples.push({ save, durMs, ...mem }); - console.log( - ` save ${String(save).padStart(3)}: ${String(durMs).padStart(5)}ms ` + - `rss=${mem.rssMb.toFixed(0).padStart(5)}MB heapUsed=${mem.heapUsedMb.toFixed(0).padStart(5)}MB` + - (mem.retainedHeapMb !== undefined - ? ` retained=${mem.retainedHeapMb.toFixed(0).padStart(5)}MB` - : '') - ); - } - - const { retainedSlope, retainedGrowth, avgTransient } = summarizeSeries(samples, baseline); - - console.log('\nsummary'); - console.log(` cold pass: ${coldMs}ms`); - console.log(` documented members: ${coldMembers} cold, ${warmMembers} on the last save`); - if (avgTransient !== undefined) { - console.log(` avg transient/save: ${avgTransient.toFixed(0)}MB`); - } - if (retainedSlope !== undefined && retainedGrowth !== undefined) { - console.log(` retained slope: ${retainedSlope.toFixed(2)}MB/save`); - console.log(` retained growth: ${retainedGrowth.toFixed(0)}MB over ${options.saves} saves`); - } - - if (options.jsonOut) { - fs.writeFileSync( - options.jsonOut, - JSON.stringify( - { - options, - coldMs, - coldMembers, - warmMembers, - baseline, - samples, - retainedSlope, - retainedGrowth, - avgTransient, - }, - null, - 2 - ) - ); - console.log(` wrote ${options.jsonOut}`); - } -} - -main().catch((err) => { - console.error(err); - process.exitCode = 1; }); diff --git a/scripts/bench/docgen-perf/engines/vue-scenario.ts b/scripts/bench/docgen-perf/engines/vue-scenario.ts new file mode 100644 index 000000000000..d0c088609979 --- /dev/null +++ b/scripts/bench/docgen-perf/engines/vue-scenario.ts @@ -0,0 +1,137 @@ +/** + * The scenario plumbing both Vue engine harnesses share: option parsing, project generation, the + * measured component set, and the per-save mutation. + * + * The two engines must see byte-identical projects and touch the same files in the same order, or + * the ratio between their medians measures the generator instead of the engines. Keeping the shared + * half here is what makes that true by construction. + * + * Scenarios: + * flat single package; the measured set is every component. + * workspace packages/* layout; the measured set is the deepest package's components, whose + * prop types pull the whole cross-package chain. + * base-type-touch workspace layout, but every save touches the widely-imported base type instead + * of a component, so each sample pays a wide invalidation. + */ +import * as path from 'node:path'; + +import { Args } from '../../docgen-shared/args.ts'; +import { SANDBOX_DIRECTORY } from '../../docgen-shared/paths.ts'; +import { + type GeneratedVueProject, + baseTypesSource, + generateVueProject, + vueComponentSource, +} from '../generators/vue.ts'; + +export const VUE_SCENARIOS = ['flat', 'workspace', 'base-type-touch'] as const; +export type VueScenario = (typeof VUE_SCENARIOS)[number]; + +export interface VueHarnessOptions { + scenario: VueScenario; + packages: number; + componentsPerPackage: number; + chainDepth: number; + fanOut: number; + heavyLib: boolean; + saves: number; + outDir: string; + jsonOut?: string; +} + +export function parseVueOptions(argv: string[], engineDirName: string): VueHarnessOptions { + const args = new Args(argv); + return { + scenario: args.choice('scenario', VUE_SCENARIOS, 'workspace'), + packages: args.count('packages', 4), + componentsPerPackage: args.count('components-per-package', 10), + chainDepth: args.count('chain-depth', 3), + fanOut: args.count('fan-out', 4), + heavyLib: args.flag('heavy-lib'), + saves: args.count('saves', 15), + outDir: args.string('out', path.join(SANDBOX_DIRECTORY, 'docgen-perf', engineDirName, 'project')), + jsonOut: args.optional('json'), + }; +} + +export function vueBanner(options: VueHarnessOptions): Record { + return { + packages: effectivePackages(options), + componentsPerPackage: options.componentsPerPackage, + chainDepth: options.chainDepth, + fanOut: options.fanOut, + heavyLib: options.heavyLib, + saves: options.saves, + }; +} + +/** The `flat` scenario is a single-package layout whatever `--packages` says. */ +export function effectivePackages(options: VueHarnessOptions): number { + return options.scenario === 'flat' ? 1 : options.packages; +} + +/** One save's disk mutation: which file changes, and what it becomes. */ +export interface VueSaveMutation { + filePath: string; + content: string; + /** The component to re-extract afterwards. */ + measuredPath: string; +} + +export interface VueScenarioSetup { + project: GeneratedVueProject; + /** The components the cold pass extracts, and the saves rotate through. */ + targetPaths: string[]; + targetPackage: number; + /** The disk mutation for save `save`. Deterministic, so both engines apply the same one. */ + mutationFor(save: number): VueSaveMutation; +} + +export function setUpVueScenario(options: VueHarnessOptions): VueScenarioSetup { + const packages = effectivePackages(options); + + const genStart = Date.now(); + const project = generateVueProject({ + outDir: options.outDir, + packages, + componentsPerPackage: options.componentsPerPackage, + chainDepth: options.chainDepth, + fanOut: options.fanOut, + heavyLib: options.heavyLib, + }); + console.log(` generated project in ${Date.now() - genStart}ms at ${project.outDir}`); + + const targetPackage = packages - 1; + const targetPaths = + options.scenario === 'flat' + ? project.componentPaths + : project.componentPaths.slice(-options.componentsPerPackage); + + const extraByTarget = new Array(targetPaths.length).fill(0); + let extraBaseProps = 0; + + return { + project, + targetPaths, + targetPackage, + mutationFor(save) { + if (options.scenario === 'base-type-touch') { + // Touch the widely-imported base type; every dependent package's props type changes. The + // re-extraction target is fixed so each sample pays the same wide invalidation. + extraBaseProps += 1; + return { + filePath: project.baseTypesPath, + content: baseTypesSource(options.fanOut, extraBaseProps), + measuredPath: targetPaths[0], + }; + } + const t = (save - 1) % targetPaths.length; + extraByTarget[t] += 1; + return { + filePath: targetPaths[t], + content: vueComponentSource(targetPackage, t, extraByTarget[t]), + measuredPath: targetPaths[t], + }; + }, + }; +} diff --git a/scripts/bench/docgen-perf/generators/angular.ts b/scripts/bench/docgen-perf/generators/angular.ts index aa4328e20341..206f6a1faee1 100644 --- a/scripts/bench/docgen-perf/generators/angular.ts +++ b/scripts/bench/docgen-perf/generators/angular.ts @@ -12,6 +12,8 @@ import * as fs from 'node:fs'; import * as path from 'node:path'; import { pathToFileURL } from 'node:url'; +import { Args } from '../../docgen-shared/args.ts'; + export interface AngularGenerateOptions { outDir: string; components: number; @@ -117,14 +119,11 @@ export function generateAngularProject(options: AngularGenerateOptions): Generat } function parseArgs(argv: string[]): AngularGenerateOptions { - const get = (flag: string, fallback: string) => { - const idx = argv.indexOf(flag); - return idx >= 0 && argv[idx + 1] ? argv[idx + 1] : fallback; - }; + const args = new Args(argv); return { - outDir: get('--out', '../storybook-sandboxes/docgen-perf-angular'), - components: Number(get('--components', '100')), - props: Number(get('--props', '8')), + outDir: args.string('out', '../storybook-sandboxes/docgen-perf-angular'), + components: args.count('components', 100), + props: args.count('props', 8), }; } diff --git a/scripts/bench/docgen-perf/generators/vue.ts b/scripts/bench/docgen-perf/generators/vue.ts index e8054bab9d7d..c8fec3b76748 100644 --- a/scripts/bench/docgen-perf/generators/vue.ts +++ b/scripts/bench/docgen-perf/generators/vue.ts @@ -26,6 +26,8 @@ import * as fs from 'node:fs'; import * as path from 'node:path'; import { pathToFileURL } from 'node:url'; +import { Args } from '../../docgen-shared/args.ts'; + const require = createRequire(import.meta.url); export interface VueGenerateOptions { @@ -267,17 +269,14 @@ export function generateVueProject(options: VueGenerateOptions): GeneratedVuePro } function parseArgs(argv: string[]): VueGenerateOptions { - const get = (flag: string, fallback: string) => { - const idx = argv.indexOf(flag); - return idx >= 0 && argv[idx + 1] ? argv[idx + 1] : fallback; - }; + const args = new Args(argv); return { - outDir: get('--out', '../storybook-sandboxes/docgen-perf-vue'), - packages: Number(get('--packages', '4')), - componentsPerPackage: Number(get('--components-per-package', '10')), - chainDepth: Number(get('--chain-depth', '3')), - fanOut: Number(get('--fan-out', '4')), - heavyLib: argv.includes('--heavy-lib'), + outDir: args.string('out', '../storybook-sandboxes/docgen-perf-vue'), + packages: args.count('packages', 4), + componentsPerPackage: args.count('components-per-package', 10), + chainDepth: args.count('chain-depth', 3), + fanOut: args.count('fan-out', 4), + heavyLib: args.flag('heavy-lib'), }; } diff --git a/scripts/bench/docgen-perf/ratios.test.ts b/scripts/bench/docgen-perf/ratios.test.ts new file mode 100644 index 000000000000..0b1d3106bb8c --- /dev/null +++ b/scripts/bench/docgen-perf/ratios.test.ts @@ -0,0 +1,132 @@ +import { describe, expect, it } from 'vitest'; + +import { CONTROL_PAIRS, computeRatios, engineOrderForRep } from './ratios.ts'; +import { type EngineId, type EngineResult, NOT_APPLICABLE, type ScenarioResult } from './types.ts'; + +function scenario(cold: number, warm: number, members?: [number, number]): ScenarioResult { + return { + params: {}, + coldMembers: members?.[0], + warmMembers: members?.[1], + metrics: { + coldExtractionMs: { status: 'measured', samples: [cold], median: cold }, + warmExtractionMs: { status: 'measured', samples: [warm], median: warm }, + wholeProjectScanMs: NOT_APPLICABLE, + peakTransientMb: NOT_APPLICABLE, + retainedGrowthMb: NOT_APPLICABLE, + retainedSlopeMbPerSave: NOT_APPLICABLE, + }, + }; +} + +function measured(scenarios: Record): EngineResult { + return { status: 'measured', scenarios }; +} + +describe('engineOrderForRep', () => { + const engines: EngineId[] = ['react-legacy', 'react-osa', 'vue-docgen-api', 'vue-component-meta']; + + it('keeps the listed order on odd repetitions', () => { + expect(engineOrderForRep(engines, 1)).toEqual(engines); + expect(engineOrderForRep(engines, 3)).toEqual(engines); + }); + + it('swaps every control pair on even repetitions', () => { + expect(engineOrderForRep(engines, 2)).toEqual([ + 'react-osa', + 'react-legacy', + 'vue-component-meta', + 'vue-docgen-api', + ]); + }); + + it('leaves a pair alone when only one of its sides is running', () => { + expect(engineOrderForRep(['react-legacy', 'compodoc'], 2)).toEqual(['react-legacy', 'compodoc']); + }); + + it('does not mutate its input', () => { + const input: EngineId[] = ['react-legacy', 'react-osa']; + engineOrderForRep(input, 2); + expect(input).toEqual(['react-legacy', 'react-osa']); + }); + + it('gives each side of a pair the first slot at least once across the pinned five', () => { + const firsts = [1, 2, 3, 4, 5].map((rep) => engineOrderForRep(engines, rep)[0]); + expect(new Set(firsts)).toEqual(new Set(['react-legacy', 'react-osa'])); + }); +}); + +describe('computeRatios', () => { + it('divides the legacy median by the new engine median', () => { + const ratios = computeRatios({ + 'react-legacy': measured({ default: scenario(400, 40) }), + 'react-osa': measured({ default: scenario(100, 20) }), + }); + expect(ratios.react.default.cold).toBe(4); + expect(ratios.react.default.warm).toBe(2); + }); + + it('produces nothing when one side did not measure', () => { + expect( + computeRatios({ + 'react-legacy': measured({ default: scenario(400, 40) }), + 'react-osa': { status: 'failed', reason: 'boom' }, + }) + ).toEqual({}); + }); + + it('produces nothing when one side was skipped', () => { + expect( + computeRatios({ + 'vue-docgen-api': measured({ flat: scenario(20, 2) }), + 'vue-component-meta': { status: 'skipped', reason: 'not installed' }, + }) + ).toEqual({}); + }); + + it('pairs scenarios by name and ignores ones only one side ran', () => { + const ratios = computeRatios({ + 'vue-docgen-api': measured({ flat: scenario(20, 2), workspace: scenario(30, 3) }), + 'vue-component-meta': measured({ flat: scenario(400, 40) }), + }); + expect(Object.keys(ratios.vue)).toEqual(['flat']); + }); + + it('marks a pair like-for-like when both documented the same members', () => { + const ratios = computeRatios({ + 'vue-docgen-api': measured({ flat: scenario(20, 2, [50, 5]) }), + 'vue-component-meta': measured({ flat: scenario(400, 40, [50, 5]) }), + }); + expect(ratios.vue.flat.likeForLike).toBe(true); + }); + + it('marks a pair not like-for-like when the member counts disagree', () => { + // vue-docgen-api does not resolve tsconfig `paths` aliases, so it documents a fraction of what + // vue-component-meta does and is fast for the wrong reason. + const ratios = computeRatios({ + 'vue-docgen-api': measured({ flat: scenario(23, 1, [6, 0]) }), + 'vue-component-meta': measured({ flat: scenario(425, 40, [320, 32]) }), + }); + expect(ratios.vue.flat).toMatchObject({ + likeForLike: false, + legacyColdMembers: 6, + nextColdMembers: 320, + legacyWarmMembers: 0, + nextWarmMembers: 32, + }); + }); + + it('leaves like-for-like unknown when the engines report no member counts', () => { + // Unknown is not the same claim as unequal; only an explicit false earns a warning. + const ratios = computeRatios({ + 'react-legacy': measured({ default: scenario(400, 40) }), + 'react-osa': measured({ default: scenario(100, 20) }), + }); + expect(ratios.react.default.likeForLike).toBeUndefined(); + }); + + it('names every pair it knows uniquely', () => { + const names = CONTROL_PAIRS.map((p) => p.name); + expect(new Set(names).size).toBe(names.length); + }); +}); diff --git a/scripts/bench/docgen-perf/ratios.ts b/scripts/bench/docgen-perf/ratios.ts new file mode 100644 index 000000000000..8fd78ab9f2f9 --- /dev/null +++ b/scripts/bench/docgen-perf/ratios.ts @@ -0,0 +1,103 @@ +/** + * The calibration references: each framework's legacy-engine median divided by its new-engine + * median, both measured in the same invocation. + * + * Timing budgets are ratios rather than absolute milliseconds because absolute wall-clock on shared + * CI executors is too noisy to gate (PERF-METHODOLOGY.md, "Budget shape"). + */ +import type { EngineId } from '../docgen-shared/engine-ids.ts'; +import type { EngineResult, RatioEntry, Ratios, ScenarioResult } from './types.ts'; + +export interface ControlPair { + /** Key under which this pair's ratios appear in the results. */ + name: string; + legacy: EngineId; + next: EngineId; +} + +export const CONTROL_PAIRS: ControlPair[] = [ + { name: 'react', legacy: 'react-legacy', next: 'react-osa' }, + { name: 'vue', legacy: 'vue-docgen-api', next: 'vue-component-meta' }, +]; + +/** + * Spawn order for a repetition. Each control pair swaps sides on even repetitions so cache warming + * and thermal drift do not consistently favour whichever engine happens to be listed first. + */ +export function engineOrderForRep(engines: EngineId[], rep: number): EngineId[] { + const order = [...engines]; + if (rep % 2 !== 0) { + return order; + } + for (const { legacy, next } of CONTROL_PAIRS) { + const legacyIdx = order.indexOf(legacy); + const nextIdx = order.indexOf(next); + if (legacyIdx >= 0 && nextIdx >= 0) { + order[legacyIdx] = next; + order[nextIdx] = legacy; + } + } + return order; +} + +function ratioFor(legacy: ScenarioResult, next: ScenarioResult): RatioEntry { + const entry: RatioEntry = { + legacyColdMembers: legacy.coldMembers, + nextColdMembers: next.coldMembers, + legacyWarmMembers: legacy.warmMembers, + nextWarmMembers: next.warmMembers, + }; + + if ( + legacy.metrics.coldExtractionMs.status === 'measured' && + next.metrics.coldExtractionMs.status === 'measured' + ) { + entry.cold = legacy.metrics.coldExtractionMs.median / next.metrics.coldExtractionMs.median; + } + if ( + legacy.metrics.warmExtractionMs.status === 'measured' && + next.metrics.warmExtractionMs.status === 'measured' + ) { + entry.warm = legacy.metrics.warmExtractionMs.median / next.metrics.warmExtractionMs.median; + } + + // Two engines that documented different numbers of members did not do the same work, so the + // ratio measures resolution depth as much as speed. Engines that report no counts leave this + // undefined: unknown is not the same claim as unequal, and only an explicit false is worth + // warning about. + if (legacy.coldMembers !== undefined && next.coldMembers !== undefined) { + const warmAgrees = + legacy.warmMembers === undefined || + next.warmMembers === undefined || + legacy.warmMembers === next.warmMembers; + entry.likeForLike = legacy.coldMembers === next.coldMembers && warmAgrees; + } + + return entry; +} + +/** + * Ratios for every control pair whose two engines both measured in this invocation. A pair with one + * failed or skipped side yields nothing: dividing a fresh median by a stale one is not a comparison. + */ +export function computeRatios(results: Partial>): Ratios { + const ratios: Ratios = {}; + + for (const pair of CONTROL_PAIRS) { + const legacy = results[pair.legacy]; + const next = results[pair.next]; + if (legacy?.status !== 'measured' || next?.status !== 'measured') { + continue; + } + for (const [scenarioName, legacyScenario] of Object.entries(legacy.scenarios)) { + const nextScenario = next.scenarios[scenarioName]; + if (!nextScenario) { + continue; + } + ratios[pair.name] ??= {}; + ratios[pair.name][scenarioName] = ratioFor(legacyScenario, nextScenario); + } + } + + return ratios; +} diff --git a/scripts/bench/docgen-perf/registry.test.ts b/scripts/bench/docgen-perf/registry.test.ts new file mode 100644 index 000000000000..e46092d1e095 --- /dev/null +++ b/scripts/bench/docgen-perf/registry.test.ts @@ -0,0 +1,123 @@ +import { describe, expect, it } from 'vitest'; + +import type { SeriesResult } from '../docgen-shared/series.ts'; +import { QUICK_PROFILE } from './config.ts'; +import type { MeasureContext, ScenarioSpec } from './engine.ts'; +import { CONTROL_PAIRS } from './ratios.ts'; +import { ALL_ENGINE_IDS, DEFAULT_ENGINE_IDS, ENGINES, engineById } from './registry.ts'; +import type { EngineId } from './types.ts'; + +/** Captures what an engine would have spawned, without spawning it. */ +function captureSpawn() { + const calls: Array<{ spec: Parameters[0]; jsonPath: string }> = + []; + const ctx: MeasureContext = { + scenarioDir: '/tmp/scenario', + rssPollIntervalMs: 100, + runSeriesChild(spec, _outDir, jsonPath) { + calls.push({ spec, jsonPath }); + return {} as SeriesResult; + }, + }; + return { ctx, calls }; +} + +async function specFor(id: EngineId, scenario: ScenarioSpec, rep = 1) { + const { ctx, calls } = captureSpawn(); + await engineById(id).measure(ctx, scenario, rep); + return calls[0]; +} + +const reactScenario = engineById('react-legacy').scenarios(QUICK_PROFILE)[0]; + +describe('the engine table', () => { + it('registers each id exactly once', () => { + expect(new Set(ALL_ENGINE_IDS).size).toBe(ALL_ENGINE_IDS.length); + }); + + it('keeps the unbudgeted react-docgen-typescript engine out of the default run', () => { + expect(DEFAULT_ENGINE_IDS).not.toContain('react-legacy-rdt'); + expect(ALL_ENGINE_IDS).toContain('react-legacy-rdt'); + }); + + it('names an engine that is not registered', () => { + expect(() => engineById('svelte' as EngineId)).toThrow('no engine registered for "svelte"'); + }); + + it('gives every engine at least one scenario, with unique names', () => { + for (const engine of ENGINES) { + const names = engine.scenarios(QUICK_PROFILE).map((s) => s.name); + expect(names.length, engine.id).toBeGreaterThan(0); + expect(new Set(names).size, engine.id).toBe(names.length); + } + }); + + it('registers both sides of every control pair', () => { + // A pair naming an unregistered engine would silently never produce a ratio. + for (const pair of CONTROL_PAIRS) { + expect(ALL_ENGINE_IDS, pair.name).toContain(pair.legacy); + expect(ALL_ENGINE_IDS, pair.name).toContain(pair.next); + } + }); + + it('runs both sides of every control pair by default', () => { + // A ratio needs both sides measured in the same invocation. + for (const pair of CONTROL_PAIRS) { + expect(DEFAULT_ENGINE_IDS, pair.name).toContain(pair.legacy); + expect(DEFAULT_ENGINE_IDS, pair.name).toContain(pair.next); + } + }); +}); + +describe('what the series engines spawn', () => { + it('runs only the reused memory harness under jiti', async () => { + // Under jiti, react-docgen's browserslist dependency fails its JSON data require, so the + // legacy child must run on native type stripping. + expect((await specFor('react-osa', reactScenario)).spec.jiti).toBe(true); + expect((await specFor('react-legacy', reactScenario)).spec.jiti).toBeFalsy(); + expect((await specFor('react-legacy-rdt', reactScenario)).spec.jiti).toBeFalsy(); + }); + + it('sends the two React legacy engines to one child with different parsers', async () => { + const legacy = await specFor('react-legacy', reactScenario); + const rdt = await specFor('react-legacy-rdt', reactScenario); + expect(rdt.spec.childPath).toBe(legacy.spec.childPath); + expect(legacy.spec.args).toContain('react-docgen'); + expect(rdt.spec.args).toContain('react-docgen-typescript'); + }); + + it('measures both React engines on one changed component per save', async () => { + // Equal work on both sides is what makes the React ratio an engine comparison. + for (const id of ['react-legacy', 'react-osa'] as EngineId[]) { + const { spec } = await specFor(id, reactScenario); + expect(spec.args.slice(spec.args.indexOf('--scope')), id).toEqual(['--scope', 'changed']); + } + }); + + it('passes the scenario size through to the child', async () => { + const { spec } = await specFor('react-legacy', reactScenario); + const components = spec.args[spec.args.indexOf('--components') + 1]; + expect(components).toBe(String(QUICK_PROFILE.react.components)); + }); + + it('passes --heavy-lib only for scenarios that ask for it', async () => { + const scenarios = engineById('vue-component-meta').scenarios(QUICK_PROFILE); + const flat = scenarios.find((s) => s.name === 'flat')!; + const workspace = scenarios.find((s) => s.name === 'workspace')!; + expect((await specFor('vue-component-meta', flat)).spec.args).not.toContain('--heavy-lib'); + expect((await specFor('vue-component-meta', workspace)).spec.args).toContain('--heavy-lib'); + }); + + it('gives the two Vue engines identical flags, so the ratio compares engines not projects', async () => { + for (const scenario of engineById('vue-docgen-api').scenarios(QUICK_PROFILE)) { + const legacy = await specFor('vue-docgen-api', scenario); + const meta = await specFor('vue-component-meta', scenario); + expect(meta.spec.args, scenario.name).toEqual(legacy.spec.args); + expect(meta.spec.childPath).not.toBe(legacy.spec.childPath); + } + }); + + it('writes each repetition to its own result file', async () => { + expect((await specFor('react-legacy', reactScenario, 3)).jsonPath).toMatch(/rep3\.json$/); + }); +}); diff --git a/scripts/bench/docgen-perf/registry.ts b/scripts/bench/docgen-perf/registry.ts new file mode 100644 index 000000000000..6cf5a9ed4af4 --- /dev/null +++ b/scripts/bench/docgen-perf/registry.ts @@ -0,0 +1,91 @@ +/** + * The engine table. Adding an engine means adding one entry here; the orchestrator has no + * per-engine branches, so an engine cannot be half-added - listed in one place and forgotten in + * another. + * + * The machinery lives in engine.ts. Everything below is data: which child to spawn, with which + * flags, over which scenarios. + */ +import type { SuiteProfile } from './config.ts'; +import { type BenchEngine, type ScenarioSpec, SeriesChildEngine } from './engine.ts'; +import { CompodocEngine } from './engines/compodoc.ts'; +import type { EngineId } from './types.ts'; + +const reactScenarios = (profile: SuiteProfile): ScenarioSpec[] => [ + { name: 'default', params: { ...profile.react } }, +]; + +const vueScenarios = (profile: SuiteProfile): ScenarioSpec[] => + profile.vue.map((scenario) => ({ name: scenario.name, params: { ...scenario } })); + +const reactArgs = ({ params }: ScenarioSpec): string[] => [ + '--components', String(params.components), + '--variants', String(params.variants), + '--props', String(params.props), + '--saves', String(params.saves), +]; + +const vueArgs = ({ params }: ScenarioSpec): string[] => [ + '--scenario', String(params.name), + '--packages', String(params.packages), + '--components-per-package', String(params.componentsPerPackage), + '--chain-depth', String(params.chainDepth), + '--fan-out', String(params.fanOut), + ...(params.heavyLib ? ['--heavy-lib'] : []), + '--saves', String(params.saves), +]; + +export const ENGINES: BenchEngine[] = [ + // Both sides of the React pair re-extract one changed component per save, which compares the + // engines on equal work. It is not what a real legacy save costs - see the calibration caveat in + // PERF-METHODOLOGY.md. + new SeriesChildEngine({ + id: 'react-legacy', + child: 'engines/react-legacy.ts', + scenarios: reactScenarios, + args: (scenario) => [...reactArgs(scenario), '--parser', 'react-docgen', '--scope', 'changed'], + }), + // Measurable through the same child, but it carries no budget row, so it stays out by default. + new SeriesChildEngine({ + id: 'react-legacy-rdt', + child: 'engines/react-legacy.ts', + scenarios: reactScenarios, + inDefaultRun: false, + args: (scenario) => [ + ...reactArgs(scenario), + '--parser', 'react-docgen-typescript', + '--scope', 'changed', + ], + }), + new SeriesChildEngine({ + id: 'react-osa', + child: '../docgen-memory/memory-harness.ts', + scenarios: reactScenarios, + jiti: true, + args: (scenario) => [...reactArgs(scenario), '--mode', 'refresh', '--scope', 'changed'], + }), + new SeriesChildEngine({ + id: 'vue-docgen-api', + child: 'engines/vue-docgen-api.ts', + scenarios: vueScenarios, + args: vueArgs, + }), + new SeriesChildEngine({ + id: 'vue-component-meta', + child: 'engines/vue-component-meta.ts', + scenarios: vueScenarios, + args: vueArgs, + }), + new CompodocEngine(), +]; + +export const ALL_ENGINE_IDS: EngineId[] = ENGINES.map((engine) => engine.id); +export const DEFAULT_ENGINE_IDS: EngineId[] = ENGINES.filter((e) => e.inDefaultRun).map((e) => e.id); + +export function engineById(id: EngineId): BenchEngine { + const engine = ENGINES.find((candidate) => candidate.id === id); + if (!engine) { + throw new Error(`no engine registered for "${id}"`); + } + return engine; +} diff --git a/scripts/bench/docgen-perf/report.test.ts b/scripts/bench/docgen-perf/report.test.ts new file mode 100644 index 000000000000..29666ef568f6 --- /dev/null +++ b/scripts/bench/docgen-perf/report.test.ts @@ -0,0 +1,97 @@ +import { describe, expect, it } from 'vitest'; + +import { renderRatios, renderResults } from './report.ts'; +import { NOT_APPLICABLE, type Ratios } from './types.ts'; + +describe('renderRatios', () => { + const notLikeForLike: Ratios = { + vue: { + flat: { + cold: 18.48, + warm: 22.5, + legacyColdMembers: 6, + nextColdMembers: 320, + legacyWarmMembers: 0, + nextWarmMembers: 32, + likeForLike: false, + }, + }, + }; + + it('warns on the cold line when the engines documented different members', () => { + const [cold] = renderRatios(notLikeForLike); + expect(cold).toContain('documented members 6 vs 320'); + expect(cold).toContain('NOT like-for-like'); + }); + + it('warns on the warm line too', () => { + // The warm ratio is the more misleading of the two: the legacy Vue parser documented nothing + // on the save it was timed on, so a bare ratio reads as a 22x speed win over identical work. + const warm = renderRatios(notLikeForLike).find((line) => line.includes('warm')); + expect(warm).toContain('documented members 0 vs 32'); + expect(warm).toContain('NOT like-for-like'); + }); + + it('prints no warning when the engines did the same work', () => { + const lines = renderRatios({ + vue: { + flat: { cold: 1.2, warm: 1.1, legacyColdMembers: 50, nextColdMembers: 50, likeForLike: true }, + }, + }); + expect(lines.every((line) => !line.includes('NOT like-for-like'))).toBe(true); + expect(lines[0]).toContain('documented members 50 vs 50'); + }); + + it('prints a bare ratio when neither engine reports member counts', () => { + const lines = renderRatios({ react: { default: { cold: 4, warm: 2 } } }); + expect(lines).toEqual([ + ' ratio cold legacy/new (react/default): 4.00', + ' ratio warm legacy/new (react/default): 2.00', + ]); + }); + + it('says so when there is no ratio at all', () => { + expect(renderRatios({})).toEqual([ + ' no calibration ratio: it needs both sides of a control pair measured in one run', + ]); + }); + + it('omits a half of the pair that did not measure', () => { + const lines = renderRatios({ react: { default: { cold: 4 } } }); + expect(lines).toHaveLength(1); + expect(lines[0]).toContain('cold'); + }); +}); + +describe('renderResults', () => { + it('lists a failed engine as a status line rather than a table row', () => { + const { table, statusLines } = renderResults(['compodoc'], { + compodoc: { status: 'failed', reason: 'child exited with status 1:\nsecond line' }, + }); + expect(table).toHaveLength(1); // header only + expect(statusLines).toEqual([' compodoc: FAILED - child exited with status 1:']); + }); + + it('renders n/a metrics as n/a rather than zero', () => { + const { table } = renderResults(['react-osa'], { + 'react-osa': { + status: 'measured', + scenarios: { + default: { + params: {}, + metrics: { + coldExtractionMs: { status: 'measured', samples: [100], median: 100 }, + warmExtractionMs: { status: 'measured', samples: [10], median: 10 }, + wholeProjectScanMs: NOT_APPLICABLE, + peakTransientMb: { status: 'measured', samples: [5], mean: 5 }, + retainedGrowthMb: { status: 'measured', value: 3 }, + retainedSlopeMbPerSave: { status: 'measured', value: 0.25 }, + }, + }, + }, + }, + }); + expect(table[1]).toContain('n/a'); + expect(table[1]).toContain('0.25MB/save'); + }); +}); diff --git a/scripts/bench/docgen-perf/report.ts b/scripts/bench/docgen-perf/report.ts new file mode 100644 index 000000000000..c2c6ed6458f1 --- /dev/null +++ b/scripts/bench/docgen-perf/report.ts @@ -0,0 +1,117 @@ +/** + * Rendering the suite's terminal output. Pure string building, so the one rule that matters here - + * a ratio never appears without the caveat that applies to it - is testable. + */ +import type { EngineId, EngineMetrics, EngineResult, Ratios } from './types.ts'; + +const HEADER = ['engine/scenario', 'cold', 'warm', 'scan', 'peak', 'ret-growth', 'ret-slope']; + +export function formatCell(metric: EngineMetrics[keyof EngineMetrics], unit: 'ms' | 'MB'): string { + if (metric.status === 'n/a') { + return 'n/a'; + } + if ('median' in metric) { + return `${metric.median.toFixed(0)}${unit}`; + } + if ('mean' in metric) { + return `${metric.mean.toFixed(0)}${unit}`; + } + return `${metric.value.toFixed(unit === 'MB' ? 1 : 0)}${unit}`; +} + +export interface RenderedResults { + /** The metrics table, already padded into aligned lines. */ + table: string[]; + /** One line per engine that skipped or failed. */ + statusLines: string[]; +} + +export function renderResults( + engineIds: EngineId[], + results: Partial> +): RenderedResults { + const rows: string[][] = [HEADER]; + const statusLines: string[] = []; + + for (const engineId of engineIds) { + const result = results[engineId]; + if (!result) { + continue; + } + if (result.status !== 'measured') { + statusLines.push( + ` ${engineId}: ${result.status.toUpperCase()} - ${result.reason.split('\n')[0]}` + ); + continue; + } + for (const [scenarioName, scenario] of Object.entries(result.scenarios)) { + const m = scenario.metrics; + rows.push([ + `${engineId}/${scenarioName}`, + formatCell(m.coldExtractionMs, 'ms'), + formatCell(m.warmExtractionMs, 'ms'), + formatCell(m.wholeProjectScanMs, 'ms'), + formatCell(m.peakTransientMb, 'MB'), + formatCell(m.retainedGrowthMb, 'MB'), + m.retainedSlopeMbPerSave.status === 'measured' + ? `${m.retainedSlopeMbPerSave.value.toFixed(2)}MB/save` + : 'n/a', + ]); + } + } + + const widths = HEADER.map((_, col) => Math.max(...rows.map((row) => (row[col] ?? '').length))); + const table = rows.map( + (row) => ` ${row.map((cell, col) => (cell ?? '').padEnd(widths[col])).join(' ')}` + ); + return { table, statusLines }; +} + +/** + * Ratio lines, each carrying its own caveat. + * + * A ratio between engines that documented different numbers of members is not a speed comparison - + * the faster side was fast because it resolved less. Printing such a ratio bare reads as a clean + * win, so the member counts and the warning travel with it, on the warm line as much as the cold + * one. The warm line needs it more: on these projects the legacy Vue parser has documented zero + * members on the save it was timed on. + */ +export function renderRatios(ratios: Ratios): string[] { + const lines: string[] = []; + + for (const [pairName, scenarios] of Object.entries(ratios)) { + for (const [scenarioName, entry] of Object.entries(scenarios)) { + const notLikeForLike = entry.likeForLike === false; + const label = `${pairName}/${scenarioName}`; + + if (entry.cold !== undefined) { + lines.push( + ` ratio cold legacy/new (${label}): ${entry.cold.toFixed(2)}` + + memberNote(entry.legacyColdMembers, entry.nextColdMembers, notLikeForLike) + ); + } + if (entry.warm !== undefined) { + lines.push( + ` ratio warm legacy/new (${label}): ${entry.warm.toFixed(2)}` + + memberNote(entry.legacyWarmMembers, entry.nextWarmMembers, notLikeForLike) + ); + } + } + } + + if (lines.length === 0) { + lines.push(' no calibration ratio: it needs both sides of a control pair measured in one run'); + } + return lines; +} + +function memberNote( + legacy: number | undefined, + next: number | undefined, + notLikeForLike: boolean +): string { + if (legacy === undefined || next === undefined) { + return notLikeForLike ? ' [NOT like-for-like]' : ''; + } + return ` [documented members ${legacy} vs ${next}${notLikeForLike ? ' - NOT like-for-like' : ''}]`; +} diff --git a/scripts/bench/docgen-perf/run.ts b/scripts/bench/docgen-perf/run.ts index 925e1ba1b913..79ab7820e992 100644 --- a/scripts/bench/docgen-perf/run.ts +++ b/scripts/bench/docgen-perf/run.ts @@ -1,18 +1,11 @@ /** * Orchestrator for the per-engine docgen performance suite. Implements the measurement contract in - * scripts/bench/PERF-METHODOLOGY.md: five floor metrics per engine, plain Node, fresh child process - * per measurement, median-of-N with one pinned N recorded in the results. + * scripts/bench/PERF-METHODOLOGY.md: five floor metrics per engine, plain Node, a fresh child + * process per measurement, median-of-N with one pinned N recorded in the results. * - * Topology per engine: - * - Cold medians come from N fresh spawns; warm is the median of the per-save durations inside - * one repetition's save series, and the memory metrics read that same repetition. The one that - * supplies them is the repetition whose cold sample is the median, because repetition 1 pays - * for a cold module graph and a cold page cache and is not representative. - * - react-legacy and react-osa are the calibration control pair: both run inside this one - * invocation, their spawn order alternates across repetitions, and the results carry the - * legacy-vs-osa ratio of medians for cold and warm. - * - compodoc is a one-shot CLI, so its adapter runs in-process here and spawns the compodoc CLI - * itself as the measured child; it skips with an explicit message when no binary is resolvable. + * The loop below has no per-engine branches. What an engine measures and how its repetitions become + * metrics lives in its descriptor in registry.ts; spawn order alternation and the ratios live in + * ratios.ts; the aggregation rules live in aggregate.ts. * * A failed engine does not abort the suite: it is recorded as failed, the remaining engines keep * measuring, and the process exits 1 after the summary. @@ -22,441 +15,143 @@ * yarn bench:docgen-perf --quick # smoke profile; results marked non-comparable * yarn bench:docgen-perf --engine react-legacy --engine react-osa */ -import { spawnSync } from 'node:child_process'; import * as fs from 'node:fs'; import * as path from 'node:path'; -import { SANDBOX_DIRECTORY } from './paths.ts'; -import { - DEFAULT_PROFILE, - QUICK_PROFILE, - RSS_POLL_INTERVAL_MS, - type ReactScenarioConfig, - type SuiteProfile, - type VueScenarioConfig, -} from './config.ts'; -import { - type CompodocRepetition, - compodocVersion, - resolveCompodocBinary, - runCompodocRepetition, -} from './engines/compodoc.ts'; -import { mean, median } from './stats.ts'; -import { - type EngineId, - type EngineMetrics, - type EngineResult, - NOT_APPLICABLE, - type SeriesHarnessResult, - type SuiteResults, -} from './types.ts'; - -const ALL_ENGINES: EngineId[] = [ - 'react-legacy', - 'react-legacy-rdt', - 'react-osa', - 'vue-docgen-api', - 'vue-component-meta', - 'compodoc', -]; - -/** react-legacy-rdt carries no budget row and only runs when named via --engine. */ -const DEFAULT_ENGINES: EngineId[] = [ - 'react-legacy', - 'react-osa', - 'vue-docgen-api', - 'vue-component-meta', - 'compodoc', -]; - -/** Engines driven by a per-scenario Vue child over the same generated project. */ -const VUE_ENGINES: EngineId[] = ['vue-docgen-api', 'vue-component-meta']; +import { SANDBOX_DIRECTORY } from '../docgen-shared/paths.ts'; +import { designatedRep } from './aggregate.ts'; +import { parseCliOptions } from './cli.ts'; +import { DEFAULT_PROFILE, QUICK_PROFILE, RSS_POLL_INTERVAL_MS, type SuiteProfile } from './config.ts'; +import type { ScenarioSpec } from './engine.ts'; +import { computeRatios, engineOrderForRep } from './ratios.ts'; +import { engineById } from './registry.ts'; +import { renderRatios, renderResults } from './report.ts'; +import { runSeriesChild } from './spawn.ts'; +import type { EngineId, EngineResult, ScenarioResult, SuiteResults } from './types.ts'; const WORK_ROOT = path.join(SANDBOX_DIRECTORY, 'docgen-perf'); -const OSA_HARNESS = path.join(import.meta.dirname, '../docgen-memory/memory-harness.ts'); -const LEGACY_HARNESS = path.join(import.meta.dirname, 'engines', 'react-legacy.ts'); -const VUE_META_HARNESS = path.join(import.meta.dirname, 'engines', 'vue-component-meta.ts'); -const VUE_DOCGEN_HARNESS = path.join(import.meta.dirname, 'engines', 'vue-docgen-api.ts'); -interface CliOptions { - quick: boolean; - engines: EngineId[]; - jsonOut: string; -} +/** Raw repetition samples, keyed by `engine/scenario`. */ +type SampleStore = Map; -/** - * The value after `flag`, or undefined when the flag is absent. A missing value - end of argv, or - * the next flag - throws instead of silently consuming the next flag as a value. - */ -function valueAfter(argv: string[], flag: string, at = argv.indexOf(flag)): string | undefined { - if (at < 0) { - return undefined; - } - const value = argv[at + 1]; - if (value === undefined || value.startsWith('--')) { - throw new Error(`${flag} requires a value`); - } - return value; +function scenarioKey(engineId: EngineId, scenarioName: string): string { + return `${engineId}/${scenarioName}`; } -function parseArgs(argv: string[]): CliOptions { - const engines: EngineId[] = []; - for (let i = 0; i < argv.length; i++) { - if (argv[i] !== '--engine') { - continue; - } - const engine = valueAfter(argv, '--engine', i) as EngineId; - if (!ALL_ENGINES.includes(engine)) { - throw new Error(`--engine must be one of ${ALL_ENGINES.join(', ')}, got "${engine}"`); - } - engines.push(engine); - } - return { - quick: argv.includes('--quick'), - engines: engines.length ? engines : DEFAULT_ENGINES, - jsonOut: valueAfter(argv, '--json') ?? path.join(WORK_ROOT, 'results.json'), - }; -} - -/** - * Every control pair alternates spawn order across repetitions so cache warming and thermal drift - * do not consistently favor one side. - */ -const CONTROL_PAIRS: Array<[EngineId, EngineId]> = [ - ['react-legacy', 'react-osa'], - ['vue-docgen-api', 'vue-component-meta'], -]; - -function engineOrderForRep(engines: EngineId[], rep: number): EngineId[] { - const order = [...engines]; - if (rep % 2 !== 0) { - return order; - } - for (const [legacy, next] of CONTROL_PAIRS) { - const legacyIdx = order.indexOf(legacy); - const nextIdx = order.indexOf(next); - if (legacyIdx >= 0 && nextIdx >= 0) { - order[legacyIdx] = next; - order[nextIdx] = legacy; - } - } - return order; -} - -function reactChildInvocation(engine: EngineId, cfg: ReactScenarioConfig) { - const sizeArgs = [ - '--components', String(cfg.components), - '--variants', String(cfg.variants), - '--props', String(cfg.props), - '--saves', String(cfg.saves), - ]; - if (engine === 'react-osa') { - return { childPath: OSA_HARNESS, args: [...sizeArgs, '--mode', 'refresh', '--scope', 'changed'] }; - } - const parser = engine === 'react-legacy-rdt' ? 'react-docgen-typescript' : 'react-docgen'; - // Both sides re-extract one component per save, which measures the engines against each other. - // It is not what a real legacy save costs - see the calibration caveat in PERF-METHODOLOGY.md. - return { childPath: LEGACY_HARNESS, args: [...sizeArgs, '--parser', parser, '--scope', 'changed'] }; -} - -function vueChildInvocation(engine: EngineId, scenario: VueScenarioConfig) { - return { - childPath: engine === 'vue-docgen-api' ? VUE_DOCGEN_HARNESS : VUE_META_HARNESS, - args: [ - '--scenario', scenario.name, - '--packages', String(scenario.packages), - '--components-per-package', String(scenario.componentsPerPackage), - '--chain-depth', String(scenario.chainDepth), - '--fan-out', String(scenario.fanOut), - ...(scenario.heavyLib ? ['--heavy-lib'] : []), - '--saves', String(scenario.saves), - ], - }; -} - -/** - * Spawn one fresh series-harness child (the gate.ts pattern) and read back its result JSON. A - * non-zero exit or a missing/unparseable JSON is a failure. - * - * The new engine children run on native Node type stripping; only the reused docgen-memory harness - * still needs the jiti loader. Under jiti, react-docgen's browserslist dependency fails on its JSON - * data require ("jsReleases.map is not a function"), so the legacy child must NOT run under jiti. - */ -function runSeriesChild( - childPath: string, - childArgs: string[], - outDir: string, - jsonPath: string -): SeriesHarnessResult { - fs.mkdirSync(path.dirname(jsonPath), { recursive: true }); - // Remove any stale result so a crashed run cannot be mistaken for a prior success. - fs.rmSync(jsonPath, { force: true }); - - const nodeArgs = [ - '--expose-gc', - ...(childPath === OSA_HARNESS ? ['--import', 'jiti/register'] : []), - childPath, - ...childArgs, - '--out', - outDir, - '--json', - jsonPath, - ]; - const proc = spawnSync(process.execPath, nodeArgs, { encoding: 'utf8' }); - const output = `${proc.stdout ?? ''}${proc.stderr ?? ''}`; - const tail = output.trim().split('\n').slice(-4).map((line) => ` ${line}`).join('\n'); - - if (proc.status !== 0) { - throw new Error(`child exited with status ${proc.status}:\n${tail}`); - } - if (!fs.existsSync(jsonPath)) { - throw new Error(`child wrote no result JSON at ${jsonPath}:\n${tail}`); - } - return JSON.parse(fs.readFileSync(jsonPath, 'utf8')) as SeriesHarnessResult; -} - -/** - * Warm latency and both memory metrics read one repetition's save series. Repetition 1 is - * systematically the slowest - it pays for a cold module graph and a cold OS page cache - so the - * representative run is the one whose cold sample is the median, never simply the first. Cold - * latency is already protected by taking the median across repetitions. - */ -function designatedRep(reps: SeriesHarnessResult[]): SeriesHarnessResult { - const byCold = [...reps].sort((a, b) => a.coldMs - b.coldMs); - return byCold[Math.floor((byCold.length - 1) / 2)]; -} - -function seriesMetrics(reps: SeriesHarnessResult[] | undefined, expectedN: number): EngineMetrics { - if (!reps || reps.length === 0) { - throw new Error('no completed repetition recorded'); - } - if (reps.length !== expectedN) { - throw new Error(`recorded ${reps.length} repetitions, expected the pinned ${expectedN}`); - } - const designated = designatedRep(reps); - const coldSamples = reps.map((r) => r.coldMs); - const warmSamples = designated.samples.map((s) => s.durMs); - const transients = designated.samples - .map((s) => (s.retainedHeapMb !== undefined ? s.heapUsedMb - s.retainedHeapMb : undefined)) - .filter((v): v is number => v !== undefined); - if ( - transients.length === 0 || - designated.retainedGrowth === undefined || - designated.retainedSlope === undefined - ) { - throw new Error('retained metrics missing (child must run under --expose-gc)'); +/** One repetition of one engine, across every scenario that engine runs. */ +async function measureEngine( + engineId: EngineId, + profile: SuiteProfile, + rep: number, + store: SampleStore +): Promise { + const engine = engineById(engineId); + for (const scenario of engine.scenarios(profile)) { + const key = scenarioKey(engineId, scenario.name); + console.log(` ${key} (rep ${rep}/${profile.n})…`); + const sample = await engine.measure( + { + scenarioDir: path.join(WORK_ROOT, engineId, scenario.name), + rssPollIntervalMs: RSS_POLL_INTERVAL_MS, + runSeriesChild, + }, + scenario, + rep + ); + store.set(key, [...(store.get(key) ?? []), sample]); } - return { - coldExtractionMs: { status: 'measured', samples: coldSamples, median: median(coldSamples) }, - warmExtractionMs: { status: 'measured', samples: warmSamples, median: median(warmSamples) }, - wholeProjectScanMs: NOT_APPLICABLE, - peakTransientMb: { status: 'measured', samples: transients, mean: mean(transients) }, - retainedGrowthMb: { status: 'measured', value: designated.retainedGrowth }, - retainedSlopeMbPerSave: { status: 'measured', value: designated.retainedSlope }, - }; } -function compodocMetrics(reps: CompodocRepetition[], expectedN: number): EngineMetrics { - if (reps.length === 0) { - throw new Error('no completed repetition recorded'); - } - if (reps.length !== expectedN) { - throw new Error(`recorded ${reps.length} repetitions, expected the pinned ${expectedN}`); - } - const coldSamples = reps.map((r) => r.coldMs); - const warmSamples = reps.map((r) => r.warmMs); - const peaks = reps.map((r) => r.peakRssMb); +function assembleScenario( + engineId: EngineId, + scenario: ScenarioSpec, + profile: SuiteProfile, + store: SampleStore +): ScenarioResult { + const engine = engineById(engineId); + const samples = store.get(scenarioKey(engineId, scenario.name)) ?? []; + const metrics = engine.aggregate(samples, profile.n); + // Member counts are read from the same repetition the warm and memory metrics come from, so + // every figure reported for a scenario describes one run. + const designated = designatedRep(samples as Array<{ coldMs: number }>); return { - coldExtractionMs: { status: 'measured', samples: coldSamples, median: median(coldSamples) }, - warmExtractionMs: { status: 'measured', samples: warmSamples, median: median(warmSamples) }, - // Cold extraction and the whole-project scan are the same full-project run for a one-shot CLI. - wholeProjectScanMs: { status: 'measured', samples: coldSamples, median: median(coldSamples) }, - peakTransientMb: { status: 'measured', samples: peaks, mean: mean(peaks) }, - retainedGrowthMb: NOT_APPLICABLE, - retainedSlopeMbPerSave: NOT_APPLICABLE, + params: scenario.params, + metrics, + coldMembers: engine.coldMembers?.(designated), + warmMembers: engine.warmMembers?.(designated), }; } -function formatCell(metric: EngineMetrics[keyof EngineMetrics], unit: 'ms' | 'MB'): string { - if (metric.status === 'n/a') { - return 'n/a'; - } - if ('median' in metric) { - return `${metric.median.toFixed(0)}${unit}`; - } - if ('mean' in metric) { - return `${metric.mean.toFixed(0)}${unit}`; - } - return `${metric.value.toFixed(unit === 'MB' ? 1 : 0)}${unit}`; -} - async function main() { - const options = parseArgs(process.argv.slice(2)); + const options = parseCliOptions(process.argv.slice(2), WORK_ROOT); const profile: SuiteProfile = options.quick ? QUICK_PROFILE : DEFAULT_PROFILE; console.log('docgen-perf suite'); - console.log( - ` engines=${options.engines.join(',')} n=${profile.n} comparable=${profile.comparable}` - ); + console.log(` engines=${options.engines.join(',')} n=${profile.n} comparable=${profile.comparable}`); if (!profile.comparable) { console.log(' QUICK PROFILE: results are non-comparable smoke numbers'); } - const state = new Map(); + const store: SampleStore = new Map(); const failed = new Map(); const skipped = new Map(); - const compodocReps: CompodocRepetition[] = []; - const compodocBinary = options.engines.includes('compodoc') ? resolveCompodocBinary() : undefined; - if (options.engines.includes('compodoc') && !compodocBinary) { - skipped.set( - 'compodoc', - 'no compodoc binary found (workspace node_modules/.bin or PATH); @compodoc/compodoc is pinned in scripts/package.json, so run yarn install' - ); - console.log(` compodoc: SKIPPED - ${skipped.get('compodoc')}`); + for (const engineId of options.engines) { + const reason = engineById(engineId).preflight?.(); + if (reason) { + skipped.set(engineId, reason); + console.log(` ${engineId}: SKIPPED - ${reason}`); + } } - const runSeries = (engine: EngineId, scenarioName: string, rep: number) => { - const invocation = VUE_ENGINES.includes(engine) - ? vueChildInvocation(engine, profile.vue.find((s) => s.name === scenarioName)!) - : reactChildInvocation(engine, profile.react); - const key = `${engine}/${scenarioName}`; - const scenarioDir = path.join(WORK_ROOT, engine, scenarioName); - console.log(` ${key} (rep ${rep}/${profile.n})…`); - const result = runSeriesChild( - invocation.childPath, - invocation.args, - path.join(scenarioDir, 'project'), - path.join(scenarioDir, `rep${rep}.json`) - ); - const reps = state.get(key) ?? []; - reps.push(result); - state.set(key, reps); - console.log(` cold=${result.coldMs}ms`); - }; - for (let rep = 1; rep <= profile.n; rep++) { console.log(`\n=== repetition ${rep}/${profile.n} ===`); - for (const engine of engineOrderForRep(options.engines, rep)) { - if (failed.has(engine) || skipped.has(engine)) { + for (const engineId of engineOrderForRep(options.engines, rep)) { + if (failed.has(engineId) || skipped.has(engineId)) { continue; } try { - if (engine === 'compodoc') { - if (!compodocBinary) { - throw new Error('compodoc binary unresolved'); - } - const repetition = await runCompodocRepetition( - compodocBinary, - profile.angular, - path.join(WORK_ROOT, 'compodoc'), - RSS_POLL_INTERVAL_MS - ); - compodocReps.push(repetition); - console.log( - ` compodoc: cold=${repetition.coldMs}ms warm=${repetition.warmMs}ms ` + - `peakRss=${repetition.peakRssMb.toFixed(0)}MB` - ); - } else if (VUE_ENGINES.includes(engine)) { - for (const scenario of profile.vue) { - runSeries(engine, scenario.name, rep); - } - } else { - runSeries(engine, 'default', rep); - } + await measureEngine(engineId, profile, rep, store); } catch (err) { const reason = err instanceof Error ? err.message : String(err); - failed.set(engine, reason); - console.error(` ${engine} FAILED: ${reason}`); + failed.set(engineId, reason); + console.error(` ${engineId} FAILED: ${reason}`); } } } const engineResults: SuiteResults['engines'] = {}; - for (const engine of options.engines) { - const skipReason = skipped.get(engine); + const engineVersions: SuiteResults['engineVersions'] = {}; + + for (const engineId of options.engines) { + const version = engineById(engineId).version?.(); + if (version) { + engineVersions[engineId] = version; + } + + const skipReason = skipped.get(engineId); if (skipReason) { - engineResults[engine] = { status: 'skipped', reason: skipReason }; + engineResults[engineId] = { status: 'skipped', reason: skipReason }; continue; } - // An engine that failed part-way through the repetitions holds fewer samples than the pinned N. - // Reporting it as 'measured' would put numbers taken at an unrecorded N into the results, so a - // mid-suite failure stays failed no matter how many repetitions it managed. - const failReason = failed.get(engine); + // An engine that failed part-way holds fewer samples than the pinned N. Reporting it as + // measured would put numbers taken at an unrecorded N into the results, so a mid-suite failure + // stays failed no matter how many repetitions it managed. + const failReason = failed.get(engineId); if (failReason) { - engineResults[engine] = { status: 'failed', reason: failReason }; + engineResults[engineId] = { status: 'failed', reason: failReason }; continue; } try { - if (engine === 'compodoc') { - engineResults[engine] = { - status: 'measured', - scenarios: { default: { params: { ...profile.angular }, metrics: compodocMetrics(compodocReps, profile.n) } }, - }; - } else if (VUE_ENGINES.includes(engine)) { - const scenarios: Extract['scenarios'] = {}; - for (const scenario of profile.vue) { - scenarios[scenario.name] = { - params: { ...scenario }, - metrics: seriesMetrics(state.get(`${engine}/${scenario.name}`), profile.n), - }; - } - engineResults[engine] = { status: 'measured', scenarios }; - } else { - engineResults[engine] = { - status: 'measured', - scenarios: { - default: { - params: { ...profile.react }, - metrics: seriesMetrics(state.get(`${engine}/default`), profile.n), - }, - }, - }; + const scenarios: Record = {}; + for (const scenario of engineById(engineId).scenarios(profile)) { + scenarios[scenario.name] = assembleScenario(engineId, scenario, profile, store); } + engineResults[engineId] = { status: 'measured', scenarios } satisfies EngineResult; } catch (err) { - const reason = failed.get(engine) ?? (err instanceof Error ? err.message : String(err)); - failed.set(engine, reason); - engineResults[engine] = { status: 'failed', reason }; - } - } - - // The calibration reference: react-docgen vs react-osa, both measured in this same invocation. - const ratios: SuiteResults['ratios'] = {}; - const legacy = engineResults['react-legacy']; - const osa = engineResults['react-osa']; - if (legacy?.status === 'measured' && osa?.status === 'measured') { - const legacyMetrics = legacy.scenarios.default.metrics; - const osaMetrics = osa.scenarios.default.metrics; - if (legacyMetrics.coldExtractionMs.status === 'measured' && osaMetrics.coldExtractionMs.status === 'measured') { - ratios.coldLegacyVsOsa = legacyMetrics.coldExtractionMs.median / osaMetrics.coldExtractionMs.median; - } - if (legacyMetrics.warmExtractionMs.status === 'measured' && osaMetrics.warmExtractionMs.status === 'measured') { - ratios.warmLegacyVsOsa = legacyMetrics.warmExtractionMs.median / osaMetrics.warmExtractionMs.median; - } - } - - // The Vue equivalent, per scenario: vue-docgen-api is still the default plugin, vue-component-meta - // is the opt-in successor, and both ran over the same generated project in this invocation. - const vueLegacy = engineResults['vue-docgen-api']; - const vueMeta = engineResults['vue-component-meta']; - if (vueLegacy?.status === 'measured' && vueMeta?.status === 'measured') { - for (const [scenarioName, legacyScenario] of Object.entries(vueLegacy.scenarios)) { - const metaScenario = vueMeta.scenarios[scenarioName]; - if (!metaScenario) { - continue; - } - const l = legacyScenario.metrics; - const m = metaScenario.metrics; - if (l.coldExtractionMs.status === 'measured' && m.coldExtractionMs.status === 'measured') { - ratios.coldVueLegacyVsMeta ??= {}; - ratios.coldVueLegacyVsMeta[scenarioName] = - l.coldExtractionMs.median / m.coldExtractionMs.median; - } - if (l.warmExtractionMs.status === 'measured' && m.warmExtractionMs.status === 'measured') { - ratios.warmVueLegacyVsMeta ??= {}; - ratios.warmVueLegacyVsMeta[scenarioName] = - l.warmExtractionMs.median / m.warmExtractionMs.median; - } + const reason = err instanceof Error ? err.message : String(err); + failed.set(engineId, reason); + engineResults[engineId] = { status: 'failed', reason }; } } @@ -466,72 +161,16 @@ async function main() { pinnedN: profile.n, comparable: profile.comparable, rssPollIntervalMs: RSS_POLL_INTERVAL_MS, - compodocVersion: compodocBinary ? compodocVersion(compodocBinary) : undefined, + engineVersions, engines: engineResults, - ratios, + ratios: computeRatios(engineResults), }; console.log('\nresults'); - const header = ['engine/scenario', 'cold', 'warm', 'scan', 'peak', 'ret-growth', 'ret-slope']; - const rows: string[][] = [header]; - const statusLines: string[] = []; - for (const engine of options.engines) { - const result = engineResults[engine]; - if (!result) { - continue; - } - if (result.status !== 'measured') { - statusLines.push(` ${engine}: ${result.status.toUpperCase()} - ${result.reason.split('\n')[0]}`); - continue; - } - for (const [scenarioName, scenario] of Object.entries(result.scenarios)) { - const m = scenario.metrics; - rows.push([ - `${engine}/${scenarioName}`, - formatCell(m.coldExtractionMs, 'ms'), - formatCell(m.warmExtractionMs, 'ms'), - formatCell(m.wholeProjectScanMs, 'ms'), - formatCell(m.peakTransientMb, 'MB'), - formatCell(m.retainedGrowthMb, 'MB'), - m.retainedSlopeMbPerSave.status === 'measured' - ? `${m.retainedSlopeMbPerSave.value.toFixed(2)}MB/save` - : 'n/a', - ]); - } - } - const widths = header.map((_, col) => Math.max(...rows.map((row) => (row[col] ?? '').length))); - for (const row of rows) { - console.log(` ${row.map((cell, col) => (cell ?? '').padEnd(widths[col])).join(' ')}`); - } - for (const line of statusLines) { + const { table, statusLines } = renderResults(options.engines, engineResults); + for (const line of [...table, ...statusLines, ...renderRatios(results.ratios)]) { console.log(line); } - if (ratios.coldLegacyVsOsa !== undefined) { - console.log(` ratio cold legacy/osa: ${ratios.coldLegacyVsOsa.toFixed(2)}`); - } - if (ratios.warmLegacyVsOsa !== undefined) { - console.log(` ratio warm legacy/osa: ${ratios.warmLegacyVsOsa.toFixed(2)}`); - } - for (const [scenarioName, value] of Object.entries(ratios.coldVueLegacyVsMeta ?? {})) { - // vue-docgen-api does not resolve tsconfig `paths` aliases, so on these generated projects it - // documents far less than vue-component-meta does. Printing the ratio without the member counts - // beside it would read as a 20x speed win over identical work, which it is not. - const legacyMembers = state.get(`vue-docgen-api/${scenarioName}`)?.[0]?.coldMembers; - const metaMembers = state.get(`vue-component-meta/${scenarioName}`)?.[0]?.coldMembers; - const members = - legacyMembers !== undefined && metaMembers !== undefined - ? ` [documented members ${legacyMembers} vs ${metaMembers}${legacyMembers === metaMembers ? '' : ' - NOT like-for-like'}]` - : ''; - console.log( - ` ratio cold vue-docgen-api/component-meta (${scenarioName}): ${value.toFixed(2)}${members}` - ); - } - for (const [scenarioName, value] of Object.entries(ratios.warmVueLegacyVsMeta ?? {})) { - console.log(` ratio warm vue-docgen-api/component-meta (${scenarioName}): ${value.toFixed(2)}`); - } - if (ratios.coldLegacyVsOsa === undefined && ratios.warmLegacyVsOsa === undefined) { - console.log(' no calibration ratio: it needs react-legacy and react-osa measured in one run'); - } if (!profile.comparable) { console.log(' QUICK PROFILE: results are non-comparable smoke numbers'); } @@ -542,8 +181,8 @@ async function main() { if (failed.size > 0) { console.error('\ndocgen-perf suite FAILED:'); - for (const [engine, reason] of failed) { - console.error(` - ${engine}: ${reason}`); + for (const [engineId, reason] of failed) { + console.error(` - ${engineId}: ${reason}`); } process.exit(1); } diff --git a/scripts/bench/docgen-perf/sampling.ts b/scripts/bench/docgen-perf/sampling.ts deleted file mode 100644 index eda1048da16c..000000000000 --- a/scripts/bench/docgen-perf/sampling.ts +++ /dev/null @@ -1,29 +0,0 @@ -/** - * Memory sampling shared by the series-harness engine children. Mirrors the docgen-memory harness: - * pre-GC rss/heapUsed is the transient-pressure signal, post-GC heapUsed (only under - * `node --expose-gc`) is the retained signal. - */ -import type { MemorySample } from './types.ts'; - -export const MB = 1024 * 1024; - -export function gc(): void { - if (typeof global.gc === 'function') { - global.gc(); - global.gc(); - } -} - -export function gcAvailable(): boolean { - return typeof global.gc === 'function'; -} - -export function sampleMemory(forceGc: boolean): MemorySample { - const pre = process.memoryUsage(); - let retainedHeapMb: number | undefined; - if (forceGc && gcAvailable()) { - gc(); - retainedHeapMb = process.memoryUsage().heapUsed / MB; - } - return { rssMb: pre.rss / MB, heapUsedMb: pre.heapUsed / MB, retainedHeapMb }; -} diff --git a/scripts/bench/docgen-perf/spawn.ts b/scripts/bench/docgen-perf/spawn.ts new file mode 100644 index 000000000000..4b948ae2a6e4 --- /dev/null +++ b/scripts/bench/docgen-perf/spawn.ts @@ -0,0 +1,67 @@ +/** + * Spawning one measured child process and reading its result back. + * + * Every measurement runs in a fresh process so it starts from a clean heap; an engine measured after + * another engine's garbage would report that garbage as its own. + */ +import { spawnSync } from 'node:child_process'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +import type { SeriesResult } from '../docgen-shared/series.ts'; + +export interface SeriesChildSpec { + childPath: string; + args: string[]; + /** + * Run the child under the jiti loader. Only the reused docgen-memory harness needs it: under + * jiti, react-docgen's browserslist dependency fails on its JSON data require + * ("jsReleases.map is not a function"), so the legacy child must run on native type stripping. + */ + jiti?: boolean; +} + +function tailOf(output: string): string { + return output + .trim() + .split('\n') + .slice(-4) + .map((line) => ` ${line}`) + .join('\n'); +} + +/** + * Run one series-harness child and parse its result JSON. A non-zero exit, or a missing or + * unreadable result, is a failure - never a silently empty measurement. + */ +export function runSeriesChild( + spec: SeriesChildSpec, + outDir: string, + jsonPath: string +): SeriesResult { + fs.mkdirSync(path.dirname(jsonPath), { recursive: true }); + // Remove any stale result so a crashed run cannot be mistaken for a prior success. + fs.rmSync(jsonPath, { force: true }); + + const nodeArgs = [ + '--expose-gc', + ...(spec.jiti ? ['--import', 'jiti/register'] : []), + spec.childPath, + ...spec.args, + '--out', + outDir, + '--json', + jsonPath, + ]; + + const proc = spawnSync(process.execPath, nodeArgs, { encoding: 'utf8' }); + const output = `${proc.stdout ?? ''}${proc.stderr ?? ''}`; + + if (proc.status !== 0) { + throw new Error(`child exited with status ${proc.status}:\n${tailOf(output)}`); + } + if (!fs.existsSync(jsonPath)) { + throw new Error(`child wrote no result JSON at ${jsonPath}:\n${tailOf(output)}`); + } + return JSON.parse(fs.readFileSync(jsonPath, 'utf8')) as SeriesResult; +} diff --git a/scripts/bench/docgen-perf/types.ts b/scripts/bench/docgen-perf/types.ts index 3732826207cd..7ce13e7b5ed6 100644 --- a/scripts/bench/docgen-perf/types.ts +++ b/scripts/bench/docgen-perf/types.ts @@ -1,44 +1,10 @@ /** - * Shared shapes for the per-engine docgen performance suite. The measurement contract behind these + * Result shapes for the per-engine docgen performance suite. The measurement contract behind these * shapes is scripts/bench/PERF-METHODOLOGY.md. */ +import type { EngineId } from '../docgen-shared/engine-ids.ts'; -export type EngineId = - | 'react-legacy' - | 'react-legacy-rdt' - | 'react-osa' - | 'vue-docgen-api' - | 'vue-component-meta' - | 'compodoc'; - -export interface MemorySample { - rssMb: number; - heapUsedMb: number; - retainedHeapMb?: number; -} - -export interface SaveSample extends MemorySample { - save: number; - durMs: number; -} - -/** - * Subset of a series-harness child's `--json` output that the orchestrator consumes. The - * docgen-memory harness writes a superset of this; the new engine children write exactly this plus - * their own options echo. - */ -export interface SeriesHarnessResult { - coldMs: number; - baseline: MemorySample; - samples: SaveSample[]; - retainedSlope?: number; - retainedGrowth?: number; - /** - * Documented members the cold pass produced. Two engines over the same project can differ by an - * order of magnitude here, and a timing ratio between them means nothing without it. - */ - coldMembers?: number; -} +export type { EngineId }; /** A latency metric: median of repeated samples (fresh process each, for cold/scan). */ export interface LatencyMetric { @@ -79,6 +45,14 @@ export interface EngineMetrics { export interface ScenarioResult { params: Record; metrics: EngineMetrics; + /** + * Members the cold pass documented, when the engine reports it. Two engines over the same project + * can differ by an order of magnitude here, and a timing ratio between them means nothing without + * it. + */ + coldMembers?: number; + /** Members the timed re-extraction documented. Warm ratios need this for the same reason. */ + warmMembers?: number; } export type EngineResult = @@ -86,6 +60,27 @@ export type EngineResult = | { status: 'skipped'; reason: string } | { status: 'failed'; reason: string }; +/** + * One control pair's comparison for one scenario: the legacy engine's median over the new engine's, + * both measured in the same invocation. + * + * `likeForLike` is false when the two engines documented different numbers of members, which means + * they did not do the same work and the ratio measures resolution depth as much as speed. A ratio + * whose `likeForLike` is false must not become a budget. + */ +export interface RatioEntry { + cold?: number; + warm?: number; + legacyColdMembers?: number; + nextColdMembers?: number; + legacyWarmMembers?: number; + nextWarmMembers?: number; + likeForLike?: boolean; +} + +/** Keyed by control-pair name, then by scenario name. */ +export type Ratios = Record>; + export interface SuiteResults { generatedAt: string; nodeVersion: string; @@ -95,19 +90,8 @@ export interface SuiteResults { comparable: boolean; /** Sampling interval for the compodoc child's externally-polled peak RSS. */ rssPollIntervalMs: number; - /** Resolved compodoc version, when that engine ran. Its cost moves across versions. */ - compodocVersion?: string; + /** Resolved versions of externally-installed engines, when they ran. */ + engineVersions: Partial>; engines: Partial>; - /** - * Each framework's legacy-engine median divided by its new-engine median, both measured in this - * same invocation. React pairs react-docgen against react-osa; Vue pairs vue-docgen-api (still - * the default plugin) against vue-component-meta. These are the calibration references budgets - * are derived from. - */ - ratios: { - coldLegacyVsOsa?: number; - warmLegacyVsOsa?: number; - coldVueLegacyVsMeta?: Record; - warmVueLegacyVsMeta?: Record; - }; + ratios: Ratios; } diff --git a/scripts/bench/docgen-shared/args.test.ts b/scripts/bench/docgen-shared/args.test.ts new file mode 100644 index 000000000000..47b768babdc5 --- /dev/null +++ b/scripts/bench/docgen-shared/args.test.ts @@ -0,0 +1,75 @@ +import { describe, expect, it } from 'vitest'; + +import { Args } from './args.ts'; + +describe('Args', () => { + it('reads a string value', () => { + expect(new Args(['--parser', 'react-docgen']).string('parser', 'x')).toBe('react-docgen'); + }); + + it('falls back when the flag is absent', () => { + expect(new Args([]).string('parser', 'fallback')).toBe('fallback'); + }); + + it('detects boolean flags', () => { + expect(new Args(['--quick']).flag('quick')).toBe(true); + expect(new Args([]).flag('quick')).toBe(false); + }); + + it('throws when a flag is followed by another flag instead of a value', () => { + expect(() => new Args(['--saves', '--json', 'out.json']).count('saves', 1)).toThrow( + '--saves requires a value' + ); + }); + + it('throws when a flag is the last argument', () => { + expect(() => new Args(['--out']).string('out', 'x')).toThrow('--out requires a value'); + }); + + it('collects every value of a repeated flag', () => { + const args = new Args(['--engine', 'a', '--quick', '--engine', 'b']); + expect(args.all('engine')).toEqual(['a', 'b']); + }); + + it('returns an empty list when a repeated flag is absent', () => { + expect(new Args(['--quick']).all('engine')).toEqual([]); + }); + + describe('count', () => { + it('parses an integer', () => { + expect(new Args(['--saves', '20']).count('saves', 1)).toBe(20); + }); + + it('rejects a non-numeric value', () => { + expect(() => new Args(['--saves', 'lots']).count('saves', 1)).toThrow( + '--saves must be a non-negative integer, got "lots"' + ); + }); + + it('rejects a fraction', () => { + expect(() => new Args(['--saves', '2.5']).count('saves', 1)).toThrow( + 'must be a non-negative integer' + ); + }); + + it('rejects a negative', () => { + expect(() => new Args(['--saves', '-3']).count('saves', 1)).toThrow( + 'must be a non-negative integer' + ); + }); + }); + + describe('choice', () => { + it('accepts an allowed value', () => { + expect(new Args(['--scope', 'all']).choice('scope', ['all', 'changed'], 'changed')).toBe( + 'all' + ); + }); + + it('names the whole set when the value is not allowed', () => { + expect(() => + new Args(['--scope', 'some']).choice('scope', ['all', 'changed'], 'changed') + ).toThrow('--scope must be one of all, changed, got "some"'); + }); + }); +}); diff --git a/scripts/bench/docgen-shared/args.ts b/scripts/bench/docgen-shared/args.ts new file mode 100644 index 000000000000..e29d8ebb9d17 --- /dev/null +++ b/scripts/bench/docgen-shared/args.ts @@ -0,0 +1,81 @@ +/** + * Strict command-line parsing for the docgen bench harnesses. + * + * Every harness used to carry its own `get(flag, fallback)` helper that returned `argv[idx + 1]` + * without checking it. `--saves --json out.json` therefore parsed as `saves = "--json"`, and + * `Number()` turned that into `NaN`, which reached the generators as a silently wrong project size. + * Values are validated here instead, once. + */ + +export class Args { + // A plain field, not a constructor parameter property: these harnesses run on Node's strip-only + // TypeScript support, which rejects parameter properties outright. + private readonly argv: string[]; + + constructor(argv: string[]) { + this.argv = argv; + } + + /** True when `--name` is present. */ + flag(name: string): boolean { + return this.argv.includes(`--${name}`); + } + + /** + * The value after `--name`. Throws when the flag is present but its value is missing or is + * itself a flag, so a typo fails at the boundary instead of becoming a bad measurement. + */ + private valueAt(name: string, at: number): string { + const value = this.argv[at + 1]; + if (value === undefined || value.startsWith('--')) { + throw new Error(`--${name} requires a value`); + } + return value; + } + + /** The value after `--name`, or undefined when the flag is absent. */ + optional(name: string): string | undefined { + const at = this.argv.indexOf(`--${name}`); + return at < 0 ? undefined : this.valueAt(name, at); + } + + string(name: string, fallback: string): string { + return this.optional(name) ?? fallback; + } + + /** Every value of a repeatable flag, in the order given. */ + all(name: string): string[] { + const values: string[] = []; + for (let i = 0; i < this.argv.length; i++) { + if (this.argv[i] === `--${name}`) { + values.push(this.valueAt(name, i)); + } + } + return values; + } + + /** A non-negative integer. Rejects `NaN`, fractions and negatives rather than passing them on. */ + count(name: string, fallback: number): number { + const raw = this.optional(name); + if (raw === undefined) { + return fallback; + } + const value = Number(raw); + if (!Number.isInteger(value) || value < 0) { + throw new Error(`--${name} must be a non-negative integer, got "${raw}"`); + } + return value; + } + + /** One of `allowed`. The error names the whole set, so a typo is self-correcting. */ + choice(name: string, allowed: readonly T[], fallback: T): T { + const raw = this.optional(name); + if (raw === undefined) { + return fallback; + } + if (!(allowed as readonly string[]).includes(raw)) { + throw new Error(`--${name} must be one of ${allowed.join(', ')}, got "${raw}"`); + } + return raw as T; + } +} diff --git a/scripts/bench/docgen-perf/budgets.ts b/scripts/bench/docgen-shared/budgets.ts similarity index 96% rename from scripts/bench/docgen-perf/budgets.ts rename to scripts/bench/docgen-shared/budgets.ts index a5d6789bfe4e..a889f2de6a5d 100644 --- a/scripts/bench/docgen-perf/budgets.ts +++ b/scripts/bench/docgen-shared/budgets.ts @@ -3,7 +3,7 @@ * baseline runs and are never ported between engines; missing engines get their rows when their * baselines are recorded. */ -import type { EngineId } from './types.ts'; +import type { EngineId } from './engine-ids.ts'; export interface MemoryBudgets { /** Max allowed post-GC retained growth (MB) across the run. */ diff --git a/scripts/bench/docgen-shared/engine-ids.ts b/scripts/bench/docgen-shared/engine-ids.ts new file mode 100644 index 000000000000..bd6be5b43aa4 --- /dev/null +++ b/scripts/bench/docgen-shared/engine-ids.ts @@ -0,0 +1,11 @@ +/** + * The docgen engines under measurement. Lives here rather than with the perf suite's result types + * because the memory gate keys its budgets off the same ids. + */ +export type EngineId = + | 'react-legacy' + | 'react-legacy-rdt' + | 'react-osa' + | 'vue-docgen-api' + | 'vue-component-meta' + | 'compodoc'; diff --git a/scripts/bench/docgen-perf/paths.ts b/scripts/bench/docgen-shared/paths.ts similarity index 100% rename from scripts/bench/docgen-perf/paths.ts rename to scripts/bench/docgen-shared/paths.ts diff --git a/scripts/bench/docgen-shared/renderer-module.ts b/scripts/bench/docgen-shared/renderer-module.ts new file mode 100644 index 000000000000..bd9896c858fe --- /dev/null +++ b/scripts/bench/docgen-shared/renderer-module.ts @@ -0,0 +1,64 @@ +/** + * Runtime access to the React renderer's docgen internals. + * + * The specifiers are built with `new URL` rather than written as static imports so the `scripts` + * typecheck does not pull `code/renderers` source into its program. The structural interfaces below + * are the minimum each harness touches; the real types live in + * `code/renderers/react/src/componentManifest`. + */ + +const COMPONENT_MANIFEST = '../../../code/renderers/react/src/componentManifest/'; + +export async function loadRendererModule(relativePath: string): Promise { + const url = new URL(`${COMPONENT_MANIFEST}${relativePath}`, import.meta.url).href; + return (await import(url)) as T; +} + +export interface ComponentRefLike { + componentName: string; + importName: string; + localImportName: string; + importId: string; + isPackage: boolean; + path: string; +} + +export interface StoryRefLike { + storyPath: string; + component: ComponentRefLike; +} + +/** The component reference the generated project's `Comp{i}` corresponds to. */ +export function componentRef(i: number, componentPath: string): ComponentRefLike { + return { + componentName: `Comp${i}`, + importName: `Comp${i}`, + localImportName: `Comp${i}`, + importId: `./Comp${i}`, + isPackage: false, + path: componentPath, + }; +} + +export function buildStoryRefs(componentPaths: string[], storyPaths: string[]): StoryRefLike[] { + return componentPaths.map((componentPath, i) => ({ + storyPath: storyPaths[i], + component: componentRef(i, componentPath), + })); +} + +export interface ReactDocgenModule { + getReactDocgen( + path: string, + component: ComponentRefLike + ): { type: 'success' } | { type: 'error'; error: { name: string; message: string } }; +} + +export interface UtilsModule { + invalidateCache(): void; +} + +export interface ReactDocgenTypescriptModule { + parseWithReactDocgenTypescript(filePath: string): Promise>; + invalidateParser(): void; +} diff --git a/scripts/bench/docgen-shared/samples.ts b/scripts/bench/docgen-shared/samples.ts new file mode 100644 index 000000000000..4f630a01fab8 --- /dev/null +++ b/scripts/bench/docgen-shared/samples.ts @@ -0,0 +1,13 @@ +/** The per-sample shapes every docgen bench harness records. */ + +export interface MemorySample { + rssMb: number; + heapUsedMb: number; + /** Post-GC heap. Present only when the child runs under `node --expose-gc`. */ + retainedHeapMb?: number; +} + +export interface SaveSample extends MemorySample { + save: number; + durMs: number; +} diff --git a/scripts/bench/docgen-shared/sampling.ts b/scripts/bench/docgen-shared/sampling.ts new file mode 100644 index 000000000000..f49f851d4643 --- /dev/null +++ b/scripts/bench/docgen-shared/sampling.ts @@ -0,0 +1,39 @@ +/** + * Memory sampling shared by every docgen bench harness: pre-GC rss/heapUsed is the transient-pressure + * signal, post-GC heapUsed (only under `node --expose-gc`) is the retained signal. + */ +import type { MemorySample } from './samples.ts'; + +export const MB = 1024 * 1024; + +export function gc(): void { + if (typeof global.gc === 'function') { + global.gc(); + global.gc(); + } +} + +export function gcAvailable(): boolean { + return typeof global.gc === 'function'; +} + +export function sampleMemory(forceGc: boolean): MemorySample { + const pre = process.memoryUsage(); + let retainedHeapMb: number | undefined; + if (forceGc && gcAvailable()) { + gc(); + retainedHeapMb = process.memoryUsage().heapUsed / MB; + } + return { rssMb: pre.rss / MB, heapUsedMb: pre.heapUsed / MB, retainedHeapMb }; +} + +/** The one per-save log line every harness prints, so their output stays comparable by eye. */ +export function formatSampleLine(save: number, durMs: number, mem: MemorySample): string { + return ( + ` save ${String(save).padStart(3)}: ${String(durMs).padStart(5)}ms ` + + `rss=${mem.rssMb.toFixed(0).padStart(5)}MB heapUsed=${mem.heapUsedMb.toFixed(0).padStart(5)}MB` + + (mem.retainedHeapMb !== undefined + ? ` retained=${mem.retainedHeapMb.toFixed(0).padStart(5)}MB` + : '') + ); +} diff --git a/scripts/bench/docgen-shared/series.ts b/scripts/bench/docgen-shared/series.ts new file mode 100644 index 000000000000..2bb42f99bb07 --- /dev/null +++ b/scripts/bench/docgen-shared/series.ts @@ -0,0 +1,159 @@ +/** + * The save-series harness every docgen engine child runs. + * + * The shape is fixed by scripts/bench/PERF-METHODOLOGY.md: one timed cold pass, then K simulated + * saves, each of which mutates the project on disk, invalidates whatever the engine caches, and + * re-extracts. Memory is sampled around forced GC after every save. + * + * Engines differ only in what those three steps mean, so they implement {@link SeriesEngine} and this + * module owns the timing. That matters more than the deduplication: which part of a save is inside + * the stopwatch is the measurement contract, and it used to be hand-placed in each child. + * `applySave` is never timed, `reextract` always is. + */ +import * as fs from 'node:fs'; + +import type { MemorySample, SaveSample } from './samples.ts'; +import { formatSampleLine, gcAvailable, sampleMemory } from './sampling.ts'; +import { summarizeSeries } from './stats.ts'; + +export interface SeriesEngine { + /** + * One full extraction over the measured set, from a cold start. Returns how many members it + * documented, when the engine can report that; two engines over one project can differ by an order + * of magnitude, and a timing ratio between them means nothing without it. + */ + cold(): Promise; + /** Mutate the project on disk for save `save` and invalidate the engine's caches. Never timed. */ + applySave(save: number): Promise; + /** Re-extract after save `save`. This call, and only this call, is the warm sample. */ + reextract(save: number): Promise; + dispose?(): void; +} + +export interface SeriesResult { + coldMs: number; + coldMembers?: number; + /** Members documented by the last timed re-extraction. */ + warmMembers?: number; + baseline: MemorySample; + samples: SaveSample[]; + retainedSlope?: number; + retainedGrowth?: number; + avgTransient?: number; +} + +export interface SeriesOptions { + saves: number; + /** Describes the measured set for the cold-pass log line, e.g. "300 components". */ + coldLabel: string; + /** Off only for the memory harness's uncapped pressure configs, which measure pre-GC rss. */ + forceGc?: boolean; +} + +export async function runSeries( + engine: SeriesEngine, + options: SeriesOptions +): Promise { + const forceGc = options.forceGc ?? true; + + console.log(` full extraction over ${options.coldLabel} (cold pass)…`); + const coldStart = Date.now(); + const coldMembers = await engine.cold(); + const coldMs = Date.now() - coldStart; + console.log( + ` cold pass: ${coldMs}ms` + (coldMembers !== undefined ? ` (${coldMembers} documented members)` : '') + ); + + const baseline = sampleMemory(forceGc); + const samples: SaveSample[] = []; + let warmMembers: number | undefined; + + for (let save = 1; save <= options.saves; save++) { + await engine.applySave(save); + + const saveStart = Date.now(); + warmMembers = await engine.reextract(save); + const durMs = Date.now() - saveStart; + + const mem = sampleMemory(forceGc); + samples.push({ save, durMs, ...mem }); + console.log(formatSampleLine(save, durMs, mem)); + } + + engine.dispose?.(); + + const { retainedSlope, retainedGrowth, avgTransient } = summarizeSeries(samples, baseline); + return { + coldMs, + coldMembers, + warmMembers, + baseline, + samples, + retainedSlope, + retainedGrowth, + avgTransient, + }; +} + +export function printSeriesSummary(result: SeriesResult, saves: number): void { + console.log('\nsummary'); + console.log(` cold pass: ${result.coldMs}ms`); + if (result.coldMembers !== undefined) { + console.log( + ` documented members: ${result.coldMembers} cold, ${result.warmMembers ?? 0} on the last save` + ); + } + if (result.avgTransient !== undefined) { + console.log(` avg transient/save: ${result.avgTransient.toFixed(0)}MB`); + } + if (result.retainedSlope !== undefined && result.retainedGrowth !== undefined) { + console.log(` retained slope: ${result.retainedSlope.toFixed(2)}MB/save`); + console.log(` retained growth: ${result.retainedGrowth.toFixed(0)}MB over ${saves} saves`); + } +} + +export interface SeriesHarnessSpec extends SeriesOptions { + /** Banner line, e.g. `vue-component-meta harness (workspace)`. */ + title: string; + /** The resolved options, recorded in the result JSON so a stored run is self-describing. */ + options: object; + /** The subset echoed under the banner. Defaults to {@link options}. */ + banner?: Record; + jsonOut?: string; + /** Build the project and the engine. Runs before the cold pass, so its cost is not measured. */ + setup(): Promise; +} + +/** + * The whole body of an engine child: banner, setup, series, summary, result JSON. Children own only + * their {@link SeriesEngine}; anything the orchestrator reads back is produced here so the children + * cannot drift apart in what they report. + */ +export async function runSeriesHarness(spec: SeriesHarnessSpec): Promise { + console.log(spec.title); + console.log( + ` ${Object.entries(spec.banner ?? spec.options) + .map(([key, value]) => `${key}=${String(value)}`) + .join(' ')}` + ); + if ((spec.forceGc ?? true) && !gcAvailable()) { + console.log(' (run with `node --expose-gc` to measure retained heap; continuing without it)'); + } + + const engine = await spec.setup(); + const result = await runSeries(engine, spec); + printSeriesSummary(result, spec.saves); + + if (spec.jsonOut) { + fs.writeFileSync(spec.jsonOut, JSON.stringify({ options: spec.options, ...result }, null, 2)); + console.log(` wrote ${spec.jsonOut}`); + } +} + +/** Entry-point wrapper: any throw becomes a non-zero exit, which is how the orchestrator sees it. */ +export function harnessMain(run: () => Promise): void { + run().catch((err) => { + console.error(err); + process.exitCode = 1; + }); +} diff --git a/scripts/bench/docgen-perf/stats.test.ts b/scripts/bench/docgen-shared/stats.test.ts similarity index 100% rename from scripts/bench/docgen-perf/stats.test.ts rename to scripts/bench/docgen-shared/stats.test.ts diff --git a/scripts/bench/docgen-perf/stats.ts b/scripts/bench/docgen-shared/stats.ts similarity index 95% rename from scripts/bench/docgen-perf/stats.ts rename to scripts/bench/docgen-shared/stats.ts index 7b8677366a64..6ae8fed71165 100644 --- a/scripts/bench/docgen-perf/stats.ts +++ b/scripts/bench/docgen-shared/stats.ts @@ -1,5 +1,5 @@ -/** Statistics helpers for the per-engine docgen performance suite. */ -import type { MemorySample, SaveSample } from './types.ts'; +/** Statistics helpers shared by the docgen bench harnesses. */ +import type { MemorySample, SaveSample } from './samples.ts'; /** Median of `values`. Throws on an empty input so a missing series fails loudly. */ export function median(values: number[]): number { From 0722452503204e5d714c4780f95dd825b95e5431 Mon Sep 17 00:00:00 2001 From: Valentin Palkovic Date: Tue, 28 Jul 2026 19:46:59 +0200 Subject: [PATCH 09/14] Bench: Cut the docgen bench comments back to what is not obvious The comments explained too much of what the code already says, and several module docblocks argued for the design rather than describing how to use it. Facts were repeated up to four times: the reason a Vue ratio is not like-for-like appeared in two module blocks and two functions. Every comment that restates an identifier, narrates the code, or sells the architecture is gone. What stays is the part a reader cannot infer: that both legacy React parsers expose only global invalidation, so a save that skips it measures a cache hit; that vue-component-meta never re-stats a file, so a disk write without updateFile measures a stale cache; that this child cannot run under jiti because react-docgen's browserslist dependency fails its JSON data require; that ps is absent on Windows and would throw from a timer callback no catch can reach; that strip-only TypeScript rejects parameter properties. Each of those now appears once, at the site a reader reaches first. CLI invocation examples are kept everywhere they appeared - they are the API usage. 83 comment lines net, comments only. No code, type, signature or test assertion changed, verified by checking that every added and removed line in the diff is a comment. yarn bench:docgen-perf --quick still reports the same numbers. --- scripts/bench/docgen-memory/gate.ts | 23 ++++---- scripts/bench/docgen-memory/memory-harness.ts | 58 ++++++++----------- scripts/bench/docgen-perf/aggregate.test.ts | 1 - scripts/bench/docgen-perf/aggregate.ts | 21 +++---- scripts/bench/docgen-perf/engine.ts | 23 +++----- scripts/bench/docgen-perf/engines/compodoc.ts | 19 +++--- .../bench/docgen-perf/engines/react-legacy.ts | 10 ++-- .../docgen-perf/engines/vue-component-meta.ts | 14 +---- .../docgen-perf/engines/vue-docgen-api.ts | 20 +++---- .../bench/docgen-perf/engines/vue-scenario.ts | 10 +--- .../bench/docgen-perf/generators/angular.ts | 9 ++- scripts/bench/docgen-perf/generators/vue.ts | 17 ++---- scripts/bench/docgen-perf/ratios.ts | 16 ++--- scripts/bench/docgen-perf/registry.ts | 12 ++-- scripts/bench/docgen-perf/report.ts | 15 ++--- scripts/bench/docgen-perf/run.ts | 21 ++----- scripts/bench/docgen-shared/series.ts | 20 +++---- scripts/bench/docgen-shared/stats.ts | 4 +- 18 files changed, 115 insertions(+), 198 deletions(-) diff --git a/scripts/bench/docgen-memory/gate.ts b/scripts/bench/docgen-memory/gate.ts index 68ebe7a2a118..4a6644e33832 100644 --- a/scripts/bench/docgen-memory/gate.ts +++ b/scripts/bench/docgen-memory/gate.ts @@ -2,18 +2,15 @@ * CI regression gate for the docgen-server memory behavior * (https://github.com/storybookjs/storybook/issues/35260). * - * Runs {@link file://./memory-harness.ts} in fresh child processes (so each measurement starts from a - * clean heap) and asserts two independent things: + * Runs {@link file://./memory-harness.ts} in fresh child processes (one clean heap per measurement) + * and asserts two independent things: * - * 1. Steady-state churn/leak-freeness (`changed-scope` metric config): post-GC `heapUsed` must not - * climb across saves, and the per-save transient working set must stay under budget. A regression - * that re-extracts the whole index would blow past these. - * 2. The OOM fix itself, as a **crash → survive negative control** (`live` configs): the live path - * (many per-component `batchExtract` calls on the shared program) is run under a tight heap cap - * both WITH and WITHOUT program recycling. - * - recycle OFF (`--recycle off`, ratio Infinity) MUST OOM. - * - recycle ON (default) MUST survive. - * Asserting the flip (not just survival) is what guards the fix rather than the cap. + * 1. Steady-state leak-freeness (`changed-scope` metric config): post-GC `heapUsed` must not + * climb across saves, and the per-save transient working set must stay under budget. + * 2. The OOM fix itself, as a **crash → survive negative control** (`live` configs): the live + * path is run under a tight heap cap both WITH and WITHOUT program recycling - recycle OFF + * (`--recycle off`, ratio Infinity) MUST OOM, recycle ON (default) MUST survive. Asserting + * the flip, not just survival, is what guards the fix rather than the cap. * * Run: * yarn bench:docgen-memory # from scripts/ @@ -42,8 +39,8 @@ interface GateConfig { /** `--max-old-space-size` cap (MB) for the child. Omit to run uncapped. */ capMb?: number; /** - * Crash → survive negative control. `true` ⇒ the child MUST OOM (recycle disabled); `false` ⇒ it - * MUST survive (recycle on). Omit for metric-only configs. + * Crash → survive negative control: `true` ⇒ MUST OOM (recycle disabled), `false` ⇒ MUST + * survive (recycle on). Omit for metric-only configs. */ expectOom?: boolean; /** Post-GC metric budgets, asserted from `result.json` (metric configs only). */ diff --git a/scripts/bench/docgen-memory/memory-harness.ts b/scripts/bench/docgen-memory/memory-harness.ts index 3a95f582ca41..a45f83812b83 100644 --- a/scripts/bench/docgen-memory/memory-harness.ts +++ b/scripts/bench/docgen-memory/memory-harness.ts @@ -2,22 +2,21 @@ * Deterministic in-process memory harness for the docgen-server OOM * (https://github.com/storybookjs/storybook/issues/35260). * - * It reproduces the "re-extract on every save" behavior without a browser or dev server, so it runs - * fast and is fully deterministic. It drives the real {@link ComponentMetaManager} against a - * generated project of N components, then simulates K file saves and samples memory after each one. + * Drives the real {@link ComponentMetaManager} against a generated project of N components, then + * simulates K file saves and samples memory after each one. * - * Two failure signals are measured independently: - * - RETAINED growth: post-GC `heapUsed` trend across saves. A rising trend ⇒ a true leak (memory - * held between saves). A flat trend ⇒ the cost is transient allocation, not retained state. - * - PEAK pressure: pre-GC `rss` per save. With `--no-force-gc` and a `--max-old-space-size` cap at - * launch, this is what crashes the process when saves outpace GC (the reported OOM). + * Two independent failure signals: + * - RETAINED growth: post-GC `heapUsed` trend across saves. Rising ⇒ a true leak (memory held + * between saves), flat ⇒ transient allocation. + * - PEAK pressure: pre-GC `rss` per save. Under `--no-force-gc` and a `--max-old-space-size` cap, + * this is what crashes the process when saves outpace GC (the reported OOM). * * Modes: - * --mode refresh call manager.batchExtract(allEntries) synchronously each save. Mirrors the docgen + * --mode refresh synchronous manager.batchExtract(allEntries) each save. Mirrors the docgen * open-service "refresh all extracted components" path (server.ts). * --mode live many per-component batchExtract calls on the shared program, mirroring the - * docs-addon per-edit wave that drives the #35260 OOM. The program-recycle fix bounds - * this path. Use --recycle off to assert the OOM still happens without the fix. + * docs-addon per-edit wave that drives the #35260 OOM. The program-recycle fix + * bounds this path; use --recycle off to assert the OOM still happens without it. * * Run (diagnose retained vs transient): * node --expose-gc --import jiti/register scripts/bench/docgen-memory/memory-harness.ts \ @@ -66,10 +65,8 @@ type ComponentMetaManagerCtor = new ( const MODES = ['refresh', 'live'] as const; /** - * Which entries to re-extract per save. - * all - re-extract every component each save (the docgen service refreshing all extracted - * components on an empty change hint). - * changed - re-extract only the component whose file changed. + * Which entries to re-extract per save: `all` re-extracts every component (the docgen service + * refreshing on an empty change hint), `changed` re-extracts only the component whose file changed. */ const SCOPES = ['all', 'changed'] as const; const RECYCLE = ['on', 'off'] as const; @@ -92,8 +89,7 @@ interface HarnessOptions { maxRetainedGrowthMb: number; /** * Heap-pressure ratio forwarded to `ComponentMetaManager`. `Infinity` disables program recycling - * (the negative control: assert the OOM happens without the fix). `undefined` uses the product - * default. + * (negative control), `undefined` uses the product default. */ recycleHeapPressureRatio?: number; } @@ -121,8 +117,8 @@ function parseOptions(argv: string[]): HarnessOptions { } /** - * Load the real `ComponentMetaManager` at runtime. The specifier is built with `new URL` so the - * static `scripts` typecheck does not pull `code/renderers` source into its program. + * Loaded via `new URL` at runtime, not a static import, so the `scripts` typecheck does not pull + * `code/renderers` source into its program. */ async function loadComponentMetaManager(): Promise { const mod = await loadRendererModule<{ ComponentMetaManager: ComponentMetaManagerCtor }>( @@ -159,16 +155,10 @@ function resolveProject(options: HarnessOptions): ReturnType { }); it('never picks the slow first repetition just because it came first', () => { - // Repetition 1 pays for a cold module graph; it must not supply the warm and memory metrics. const reps = [rep(5000, [999]), rep(100, [10]), rep(110, [11]), rep(120, [12]), rep(130, [13])]; expect(designatedRep(reps).coldMs).not.toBe(5000); expect(designatedRep(reps).coldMs).toBe(120); diff --git a/scripts/bench/docgen-perf/aggregate.ts b/scripts/bench/docgen-perf/aggregate.ts index db32980bdfe4..2b7710a42091 100644 --- a/scripts/bench/docgen-perf/aggregate.ts +++ b/scripts/bench/docgen-perf/aggregate.ts @@ -1,19 +1,16 @@ /** - * Turning N repetitions into the five floor metrics, per scripts/bench/PERF-METHODOLOGY.md. - * - * Cold latency is the median across repetitions. Warm latency and all three memory metrics read one - * repetition's save series, and they read the same one, so those four figures always describe the - * same run. + * Turns N repetitions into the five floor metrics (see scripts/bench/PERF-METHODOLOGY.md). Warm + * latency and all three memory metrics read the same designated repetition's save series, so those + * four figures always describe one run; cold latency is a median across every repetition instead. */ import type { SeriesResult } from '../docgen-shared/series.ts'; import { mean, median } from '../docgen-shared/stats.ts'; import { type EngineMetrics, NOT_APPLICABLE } from './types.ts'; /** - * The repetition whose cold sample is the median. Repetition 1 is systematically the slowest - it - * pays for a cold module graph and a cold OS page cache - so taking it would bias warm latency and - * all three memory metrics at once. Cold latency needs no such protection: it is already a median - * across every repetition. + * Repetition 1 is systematically the slowest - it pays for a cold module graph and a cold OS page + * cache - so taking it would bias warm latency and all three memory metrics at once. Cold latency + * needs no such protection: it is already a median across every repetition. */ export function designatedRep(reps: T[]): T { const byCold = [...reps].sort((a, b) => a.coldMs - b.coldMs); @@ -28,9 +25,9 @@ function transientsOf(rep: SeriesResult): number[] { } /** - * Reject a repetition count that is not the pinned N. An engine that failed part-way through holds - * fewer samples, and reporting those as a result would put numbers taken at an unrecorded N into the - * results file - which the whole comparison method rests on not happening. + * An engine that failed part-way through holds fewer samples than expectedN. Reporting those would + * put numbers taken at an unrecorded N into the results file, which the comparison method rests on + * not happening. */ function assertRepetitionCount(reps: unknown[] | undefined, expectedN: number): void { if (!reps || reps.length === 0) { diff --git a/scripts/bench/docgen-perf/engine.ts b/scripts/bench/docgen-perf/engine.ts index f48e206554aa..f823e3cd9249 100644 --- a/scripts/bench/docgen-perf/engine.ts +++ b/scripts/bench/docgen-perf/engine.ts @@ -1,13 +1,6 @@ /** - * What the orchestrator knows about an engine. - * - * Engines differ in exactly one way: how one repetition is produced. Five of them spawn a - * series-harness child and get a save series back; Compodoc runs an external CLI and gets one - * triple back. Everything downstream - which aggregation applies, whether there are member counts - * to report - follows from that choice, so it is the only thing a subclass decides. - * - * React and Vue are not separate kinds. They are the same {@link SeriesChildEngine} with a different - * child file and different flags, which is data, not behaviour. + * What the orchestrator knows about an engine. Engines differ only in how one repetition is + * produced; everything downstream (aggregation, member counts) follows from that choice. */ import * as path from 'node:path'; @@ -45,9 +38,9 @@ export abstract class BenchEngine { abstract aggregate(samples: Sample[], expectedN: number): EngineMetrics; /** - * Anything that must resolve before the run starts. A returned string is the skip reason, and a - * skip is reported instead of a failure: a missing external tool is what a partial install looks - * like, not a regression. + * Anything that must resolve before the run starts. A returned string is the skip reason: a + * missing external tool is a partial install, not a regression, so it is reported as skipped + * rather than failed. */ preflight(): string | undefined { return undefined; @@ -81,9 +74,9 @@ export interface SeriesChildConfig { } /** - * An engine measured by spawning a series-harness child, one fresh process per repetition. The - * child produces the save series; aggregation and member counts are the same for every engine of - * this kind because they all share the series harness. + * An engine measured by spawning a series-harness child, one fresh process per repetition. All + * engines of this kind share the series harness, so aggregation and member counts are the same + * for each. */ export class SeriesChildEngine extends BenchEngine { readonly id: EngineId; diff --git a/scripts/bench/docgen-perf/engines/compodoc.ts b/scripts/bench/docgen-perf/engines/compodoc.ts index 5a67d57f0237..e7941f5f7310 100644 --- a/scripts/bench/docgen-perf/engines/compodoc.ts +++ b/scripts/bench/docgen-perf/engines/compodoc.ts @@ -1,15 +1,12 @@ /** * Adapter for the Compodoc engine. Compodoc is a one-shot CLI - a fresh process per run - so this - * adapter runs inside the orchestrator and spawns the compodoc CLI itself as the measured child: - * cold extraction and whole-project scan are the same full-project run, warm extraction is a second - * full run after touching one component file, and peak memory is the child's peak RSS sampled from - * outside the process. + * adapter spawns the compodoc CLI itself as the measured child: cold extraction and whole-project + * scan are the same full-project run, warm extraction is a second full run after touching one + * component file, and peak memory is the child's peak RSS sampled from outside the process. * - * `@compodoc/compodoc` is pinned exactly in `scripts/package.json`, at the same 2.0.0 the Angular - * docgen baselines capture against (`code/lib/docgen-harness/README.md`). The pin is exact because - * compodoc's output and cost both move across versions, so a caret range would drift the numbers - * without anyone deciding to. The binary is still resolved at runtime and the engine still skips - * with an explicit message if it is missing, which is what a partial install looks like. + * `@compodoc/compodoc` is pinned exactly in `scripts/package.json`, matching the version the Angular + * docgen baselines capture against (`code/lib/docgen-harness/README.md`) - a caret range would let + * the numbers drift across compodoc versions without anyone deciding to. */ import { spawn, spawnSync } from 'node:child_process'; import * as fs from 'node:fs'; @@ -23,8 +20,8 @@ import type { EngineId } from '../../docgen-shared/engine-ids.ts'; import type { EngineMetrics } from '../types.ts'; /** - * The version of the binary that actually ran, walked up from the resolved executable rather than - * read from a fixed path - the package hoists to different node_modules depending on the install. + * Walked up from the resolved binary rather than a fixed path, since the package hoists to + * different node_modules depending on the install. */ export function compodocVersion(binary: string): string | undefined { try { diff --git a/scripts/bench/docgen-perf/engines/react-legacy.ts b/scripts/bench/docgen-perf/engines/react-legacy.ts index 5dd468f3a405..1d74997d5859 100644 --- a/scripts/bench/docgen-perf/engines/react-legacy.ts +++ b/scripts/bench/docgen-perf/engines/react-legacy.ts @@ -2,13 +2,11 @@ * Series harness for the legacy React docgen engines: `react-docgen` (the budgeted legacy control) * and `react-docgen-typescript` (measurable, no budget row). * - * Both engines cache per file for the life of the process and expose only global invalidation, so - * every simulated save invalidates before re-extracting - skipping it would measure a cache hit. - * The react-docgen-typescript parser additionally keys its TS program off `process.cwd()`, so this - * harness chdirs into the generated project before the first parse. + * Both cache per file for the process lifetime and expose only global invalidation, so every save + * must invalidate before re-extracting or the sample is a cache hit. * - * Runs on native Node type stripping - NOT under jiti: react-docgen's browserslist dependency - * fails on its JSON data require under the jiti loader ("jsReleases.map is not a function"). + * Do not run this child under jiti - react-docgen's browserslist dependency fails its JSON data + * require under that loader ("jsReleases.map is not a function"). * * Run: * node --expose-gc scripts/bench/docgen-perf/engines/react-legacy.ts \ diff --git a/scripts/bench/docgen-perf/engines/vue-component-meta.ts b/scripts/bench/docgen-perf/engines/vue-component-meta.ts index 0b88c5994a76..678e3ffaf110 100644 --- a/scripts/bench/docgen-perf/engines/vue-component-meta.ts +++ b/scripts/bench/docgen-perf/engines/vue-component-meta.ts @@ -2,12 +2,9 @@ * Series harness for the vue-component-meta engine - Vue's opt-in successor to vue-docgen-api. * * The checker never re-stats files on its own; a disk rewrite must be followed by - * `checker.updateFile(path, content)` (the surface the production Vite plugin drives from HMR) or - * the re-extraction measures a stale-cache no-op. - * - * Checker creation is inside the cold pass because it is part of the engine's first-extraction cost. - * The `flat` scenario drives `createCheckerByJson` (the production fallback for tsconfig-less - * projects); the workspace scenarios drive `createChecker` at the deepest package's tsconfig. + * `checker.updateFile(path, content)` or the re-extraction measures a stale-cache no-op. Checker + * creation runs inside the cold pass, since it is part of first-extraction cost: `flat` uses + * `createCheckerByJson`, the workspace scenarios use `createChecker` on the deepest package. * * Run: * node --expose-gc scripts/bench/docgen-perf/engines/vue-component-meta.ts \ @@ -45,7 +42,6 @@ function extractOne(checker: Checker, sfcPath: string): number { if (meta.props.length === 0) { throw new Error(`vue-component-meta returned zero props for ${sfcPath}`); } - // Consume the surfaces production consumes so their evaluation cost is inside the measurement. return meta.props.length + meta.events.length + meta.slots.length + meta.exposed.length; } @@ -56,7 +52,6 @@ function createEngine(options: VueHarnessOptions): SeriesEngine { return { async cold() { - // Checker creation is timed with the cold pass; it is part of first-extraction cost. checker = options.scenario === 'flat' ? createCheckerByJson(scenario.project.outDir, { include: ['**/*'] }, CHECKER_OPTIONS) @@ -70,7 +65,6 @@ function createEngine(options: VueHarnessOptions): SeriesEngine { async applySave(save) { const mutation = scenario.mutationFor(save); fs.writeFileSync(mutation.filePath, mutation.content); - // The checker does not re-stat on its own; this is the HMR surface the Vite plugin drives. checker!.updateFile(mutation.filePath, mutation.content); measuredPath = mutation.measuredPath; }, @@ -87,8 +81,6 @@ harnessMain(async () => { options, banner: vueBanner(options), saves: options.saves, - // Both layouts measure one package's worth of components: every component when flat, the - // deepest package's when not. coldLabel: `${options.componentsPerPackage} components, checker creation included`, jsonOut: options.jsonOut, setup: async () => createEngine(options), diff --git a/scripts/bench/docgen-perf/engines/vue-docgen-api.ts b/scripts/bench/docgen-perf/engines/vue-docgen-api.ts index ab527c7ed533..255c098a3827 100644 --- a/scripts/bench/docgen-perf/engines/vue-docgen-api.ts +++ b/scripts/bench/docgen-perf/engines/vue-docgen-api.ts @@ -1,15 +1,10 @@ /** - * Series harness for the vue-docgen-api engine - Vue's legacy docgen and still the default one - * (`resolveDocgenOptions` in `code/frameworks/vue3-vite/src/preset.ts`). It is the legacy half of the - * Vue control pair, the way react-docgen is for React. + * Series harness for the vue-docgen-api engine - Vue's legacy docgen and still the default + * (`resolveDocgenOptions` in `code/frameworks/vue3-vite/src/preset.ts`), the Vue equivalent of the + * react-docgen control. * - * Production calls `parse(id)` once per `.vue` file from a Vite `transform` hook - * (`code/frameworks/vue3-vite/src/plugins/vue-docgen.ts`), and the parser reads the file from disk - * on every call with no cache of its own. A save therefore costs exactly one `parse` of one file, - * so the per-save sample is production-shaped without any invalidation step. - * - * The generated project and the save sequence come from the shared scenario module, so they are - * identical to the ones vue-component-meta measures. + * The parser reads from disk on every call and keeps no cache, so one save costs one `parse` and + * needs no invalidation step. * * Run: * node --expose-gc scripts/bench/docgen-perf/engines/vue-docgen-api.ts \ @@ -28,9 +23,8 @@ import { } from './vue-scenario.ts'; /** - * How many documented members the parse produced. The two Vue engines resolve imported prop types - * to different depths, so a timing ratio is only meaningful next to this count - a parse that - * resolved nothing is fast for the wrong reason. + * Member count is reported alongside timing: a parse that resolved nothing is fast for the wrong + * reason. */ async function extractOne(sfcPath: string): Promise { const doc = await parse(sfcPath); diff --git a/scripts/bench/docgen-perf/engines/vue-scenario.ts b/scripts/bench/docgen-perf/engines/vue-scenario.ts index d0c088609979..a6c813ce6f9b 100644 --- a/scripts/bench/docgen-perf/engines/vue-scenario.ts +++ b/scripts/bench/docgen-perf/engines/vue-scenario.ts @@ -1,10 +1,8 @@ /** * The scenario plumbing both Vue engine harnesses share: option parsing, project generation, the - * measured component set, and the per-save mutation. - * - * The two engines must see byte-identical projects and touch the same files in the same order, or - * the ratio between their medians measures the generator instead of the engines. Keeping the shared - * half here is what makes that true by construction. + * measured component set, and the per-save mutation. Both engines must see identical projects and + * touch the same files in the same order, or the ratio between their medians measures the generator + * instead of the engines. * * Scenarios: * flat single package; the measured set is every component. @@ -116,8 +114,6 @@ export function setUpVueScenario(options: VueHarnessOptions): VueScenarioSetup { targetPackage, mutationFor(save) { if (options.scenario === 'base-type-touch') { - // Touch the widely-imported base type; every dependent package's props type changes. The - // re-extraction target is fixed so each sample pays the same wide invalidation. extraBaseProps += 1; return { filePath: project.baseTypesPath, diff --git a/scripts/bench/docgen-perf/generators/angular.ts b/scripts/bench/docgen-perf/generators/angular.ts index 206f6a1faee1..ab1e47ef2f4e 100644 --- a/scripts/bench/docgen-perf/generators/angular.ts +++ b/scripts/bench/docgen-perf/generators/angular.ts @@ -1,9 +1,8 @@ /** * Generator for a synthetic Angular project consumable by a standalone Compodoc CLI run. * - * Compodoc documents inputs/outputs from the TypeScript AST, so the generated tree ships a minimal - * fake `(at)angular/core` type surface inside its own node_modules (hermetic - no npm install of the - * real framework) just so type resolution stays clean. + * Ships a minimal fake `(at)angular/core` type surface inside its own node_modules so the tree is + * hermetic - no npm install of the real framework is needed for type resolution. * * Run directly: * node scripts/bench/docgen-perf/generators/angular.ts --out ../storybook-sandboxes/docgen-perf-angular --components 100 @@ -51,8 +50,8 @@ function emitFakeAngularCore(projectDir: string): void { } /** - * A component's source. `extraProps` grows the input surface by one per warm-run touch, so the - * second Compodoc run sees a genuinely changed file. + * `extraProps` grows the input surface by one per warm-run touch, so the second Compodoc run sees + * a genuinely changed file. */ export function angularComponentSource(i: number, extraProps: number): string { const extras = Array.from( diff --git a/scripts/bench/docgen-perf/generators/vue.ts b/scripts/bench/docgen-perf/generators/vue.ts index c8fec3b76748..c0f927566b86 100644 --- a/scripts/bench/docgen-perf/generators/vue.ts +++ b/scripts/bench/docgen-perf/generators/vue.ts @@ -5,18 +5,13 @@ * inside the generated tree's own node_modules (hermetic - no npm install). * * Structure levers: - * - packages: workspace width; each `packages/pkg{p}` extends the previous package's props type, + * - packages: workspace width. Each `packages/pkg{p}` extends the previous package's props type, * so the type chain across packages has depth = packages. `packages/types` holds the shared - * base type every package ultimately extends - the widely-imported type the base-type-touch - * scenario mutates. - * - chainDepth: intra-package type-alias hops between a package's props interface and its parent - * type, adding resolution steps without new files. - * - fanOut: distinct auxiliary types from `packages/types` referenced by every package's props. - * - heavyLib: emits `node_modules/(at)bench/heavy-ui` with a large generated `.d.ts` surface - * imitating a vuetify-scale component library; package props reference it when enabled. - * - * Generated trees are bench-time scratch output (under the sandbox directory); only this generator - * and the scenario configs are committed. + * base type every package extends - the type the base-type-touch scenario mutates. + * - chainDepth: intra-package type-alias hops between a package's props interface and its parent. + * - fanOut: auxiliary types from `packages/types` referenced by every package's props. + * - heavyLib: emits a fake `(at)bench/heavy-ui` package with a large `.d.ts` surface; package + * props reference it when enabled. * * Run directly: * node scripts/bench/docgen-perf/generators/vue.ts --out ../storybook-sandboxes/docgen-perf-vue --packages 4 --components-per-package 10 diff --git a/scripts/bench/docgen-perf/ratios.ts b/scripts/bench/docgen-perf/ratios.ts index 8fd78ab9f2f9..5b28862c928b 100644 --- a/scripts/bench/docgen-perf/ratios.ts +++ b/scripts/bench/docgen-perf/ratios.ts @@ -1,9 +1,7 @@ /** * The calibration references: each framework's legacy-engine median divided by its new-engine - * median, both measured in the same invocation. - * - * Timing budgets are ratios rather than absolute milliseconds because absolute wall-clock on shared - * CI executors is too noisy to gate (PERF-METHODOLOGY.md, "Budget shape"). + * median, both measured in the same invocation. Ratios stand in for absolute milliseconds because + * wall-clock on shared CI executors is too noisy to gate (PERF-METHODOLOGY.md, "Budget shape"). */ import type { EngineId } from '../docgen-shared/engine-ids.ts'; import type { EngineResult, RatioEntry, Ratios, ScenarioResult } from './types.ts'; @@ -21,8 +19,8 @@ export const CONTROL_PAIRS: ControlPair[] = [ ]; /** - * Spawn order for a repetition. Each control pair swaps sides on even repetitions so cache warming - * and thermal drift do not consistently favour whichever engine happens to be listed first. + * Each control pair swaps sides on even repetitions so cache warming and thermal drift do not + * consistently favour whichever engine is listed first. */ export function engineOrderForRep(engines: EngineId[], rep: number): EngineId[] { const order = [...engines]; @@ -61,10 +59,8 @@ function ratioFor(legacy: ScenarioResult, next: ScenarioResult): RatioEntry { entry.warm = legacy.metrics.warmExtractionMs.median / next.metrics.warmExtractionMs.median; } - // Two engines that documented different numbers of members did not do the same work, so the - // ratio measures resolution depth as much as speed. Engines that report no counts leave this - // undefined: unknown is not the same claim as unequal, and only an explicit false is worth - // warning about. + // likeForLike is left undefined, not false, when an engine reports no member counts - see + // renderRatios for why that distinction matters. if (legacy.coldMembers !== undefined && next.coldMembers !== undefined) { const warmAgrees = legacy.warmMembers === undefined || diff --git a/scripts/bench/docgen-perf/registry.ts b/scripts/bench/docgen-perf/registry.ts index 6cf5a9ed4af4..b41c8e53ed49 100644 --- a/scripts/bench/docgen-perf/registry.ts +++ b/scripts/bench/docgen-perf/registry.ts @@ -1,10 +1,7 @@ /** * The engine table. Adding an engine means adding one entry here; the orchestrator has no - * per-engine branches, so an engine cannot be half-added - listed in one place and forgotten in - * another. - * - * The machinery lives in engine.ts. Everything below is data: which child to spawn, with which - * flags, over which scenarios. + * per-engine branches. Everything below is data: which child to spawn, with which flags, over + * which scenarios. */ import type { SuiteProfile } from './config.ts'; import { type BenchEngine, type ScenarioSpec, SeriesChildEngine } from './engine.ts'; @@ -36,9 +33,8 @@ const vueArgs = ({ params }: ScenarioSpec): string[] => [ ]; export const ENGINES: BenchEngine[] = [ - // Both sides of the React pair re-extract one changed component per save, which compares the - // engines on equal work. It is not what a real legacy save costs - see the calibration caveat in - // PERF-METHODOLOGY.md. + // --scope changed on both sides of the React pair compares the engines on equal work; that is + // not what a real legacy save costs - see the calibration caveat in PERF-METHODOLOGY.md. new SeriesChildEngine({ id: 'react-legacy', child: 'engines/react-legacy.ts', diff --git a/scripts/bench/docgen-perf/report.ts b/scripts/bench/docgen-perf/report.ts index c2c6ed6458f1..98712581e025 100644 --- a/scripts/bench/docgen-perf/report.ts +++ b/scripts/bench/docgen-perf/report.ts @@ -1,7 +1,4 @@ -/** - * Rendering the suite's terminal output. Pure string building, so the one rule that matters here - - * a ratio never appears without the caveat that applies to it - is testable. - */ +/** Renders the suite's terminal output. Pure string building, so it is testable without a runner. */ import type { EngineId, EngineMetrics, EngineResult, Ratios } from './types.ts'; const HEADER = ['engine/scenario', 'cold', 'warm', 'scan', 'peak', 'ret-growth', 'ret-slope']; @@ -20,9 +17,9 @@ export function formatCell(metric: EngineMetrics[keyof EngineMetrics], unit: 'ms } export interface RenderedResults { - /** The metrics table, already padded into aligned lines. */ + /** Already padded into aligned lines. */ table: string[]; - /** One line per engine that skipped or failed. */ + /** One line per engine that skipped or failed, in place of a table row. */ statusLines: string[]; } @@ -68,13 +65,11 @@ export function renderResults( } /** - * Ratio lines, each carrying its own caveat. - * * A ratio between engines that documented different numbers of members is not a speed comparison - * the faster side was fast because it resolved less. Printing such a ratio bare reads as a clean * win, so the member counts and the warning travel with it, on the warm line as much as the cold - * one. The warm line needs it more: on these projects the legacy Vue parser has documented zero - * members on the save it was timed on. + * one. The warm line needs it more: the legacy Vue parser documents zero members on the save it is + * timed on, so a bare warm ratio would read as a clean win over identical work. */ export function renderRatios(ratios: Ratios): string[] { const lines: string[] = []; diff --git a/scripts/bench/docgen-perf/run.ts b/scripts/bench/docgen-perf/run.ts index 79ab7820e992..dae3ae126552 100644 --- a/scripts/bench/docgen-perf/run.ts +++ b/scripts/bench/docgen-perf/run.ts @@ -1,14 +1,6 @@ /** - * Orchestrator for the per-engine docgen performance suite. Implements the measurement contract in - * scripts/bench/PERF-METHODOLOGY.md: five floor metrics per engine, plain Node, a fresh child - * process per measurement, median-of-N with one pinned N recorded in the results. - * - * The loop below has no per-engine branches. What an engine measures and how its repetitions become - * metrics lives in its descriptor in registry.ts; spawn order alternation and the ratios live in - * ratios.ts; the aggregation rules live in aggregate.ts. - * - * A failed engine does not abort the suite: it is recorded as failed, the remaining engines keep - * measuring, and the process exits 1 after the summary. + * Orchestrator for the per-engine docgen performance suite. See scripts/bench/PERF-METHODOLOGY.md + * for the measurement contract. * * Run: * yarn bench:docgen-perf # from scripts/, full profile @@ -71,8 +63,8 @@ function assembleScenario( const engine = engineById(engineId); const samples = store.get(scenarioKey(engineId, scenario.name)) ?? []; const metrics = engine.aggregate(samples, profile.n); - // Member counts are read from the same repetition the warm and memory metrics come from, so - // every figure reported for a scenario describes one run. + // Member counts come from the same repetition as the warm and memory metrics, so every figure + // reported for a scenario describes one run. const designated = designatedRep(samples as Array<{ coldMs: number }>); return { params: scenario.params, @@ -134,9 +126,8 @@ async function main() { engineResults[engineId] = { status: 'skipped', reason: skipReason }; continue; } - // An engine that failed part-way holds fewer samples than the pinned N. Reporting it as - // measured would put numbers taken at an unrecorded N into the results, so a mid-suite failure - // stays failed no matter how many repetitions it managed. + // An engine that failed part-way holds fewer samples than the pinned N, so it stays failed + // rather than being reported as measured at an unrecorded N. const failReason = failed.get(engineId); if (failReason) { engineResults[engineId] = { status: 'failed', reason: failReason }; diff --git a/scripts/bench/docgen-shared/series.ts b/scripts/bench/docgen-shared/series.ts index 2bb42f99bb07..8eb35d24f59b 100644 --- a/scripts/bench/docgen-shared/series.ts +++ b/scripts/bench/docgen-shared/series.ts @@ -1,14 +1,9 @@ /** - * The save-series harness every docgen engine child runs. - * - * The shape is fixed by scripts/bench/PERF-METHODOLOGY.md: one timed cold pass, then K simulated - * saves, each of which mutates the project on disk, invalidates whatever the engine caches, and - * re-extracts. Memory is sampled around forced GC after every save. - * - * Engines differ only in what those three steps mean, so they implement {@link SeriesEngine} and this - * module owns the timing. That matters more than the deduplication: which part of a save is inside - * the stopwatch is the measurement contract, and it used to be hand-placed in each child. - * `applySave` is never timed, `reextract` always is. + * The save-series harness every docgen engine child runs, per scripts/bench/PERF-METHODOLOGY.md: + * one timed cold pass, then K simulated saves that mutate the project on disk, invalidate the + * engine's caches, and re-extract, with memory sampled around forced GC after every save. Engines + * differ only in what those steps mean, so they implement {@link SeriesEngine} and this module owns + * the timing. */ import * as fs from 'node:fs'; @@ -125,9 +120,8 @@ export interface SeriesHarnessSpec extends SeriesOptions { } /** - * The whole body of an engine child: banner, setup, series, summary, result JSON. Children own only - * their {@link SeriesEngine}; anything the orchestrator reads back is produced here so the children - * cannot drift apart in what they report. + * Children own only their {@link SeriesEngine}; everything the orchestrator reads back is produced + * here so the children cannot drift apart in what they report. */ export async function runSeriesHarness(spec: SeriesHarnessSpec): Promise { console.log(spec.title); diff --git a/scripts/bench/docgen-shared/stats.ts b/scripts/bench/docgen-shared/stats.ts index 6ae8fed71165..26b0b2a7fb98 100644 --- a/scripts/bench/docgen-shared/stats.ts +++ b/scripts/bench/docgen-shared/stats.ts @@ -1,7 +1,7 @@ /** Statistics helpers shared by the docgen bench harnesses. */ import type { MemorySample, SaveSample } from './samples.ts'; -/** Median of `values`. Throws on an empty input so a missing series fails loudly. */ +/** Throws on an empty input so a missing series fails loudly. */ export function median(values: number[]): number { if (values.length === 0) { throw new Error('median() requires at least one value'); @@ -11,7 +11,7 @@ export function median(values: number[]): number { return sorted.length % 2 === 1 ? sorted[mid] : (sorted[mid - 1] + sorted[mid]) / 2; } -/** Arithmetic mean of `values`. Throws on an empty input so a missing series fails loudly. */ +/** Throws on an empty input so a missing series fails loudly. */ export function mean(values: number[]): number { if (values.length === 0) { throw new Error('mean() requires at least one value'); From c5a089dc9e60735165695ecb5a15dd4e7f2040a0 Mon Sep 17 00:00:00 2001 From: Valentin Palkovic Date: Tue, 28 Jul 2026 19:56:31 +0200 Subject: [PATCH 10/14] Bench: Parse harness options with node:util and Zod The bench harnesses had a hand-rolled argument parser. node:util's parseArgs already does the job, and scripts/eval/eval.ts already pairs it with Zod, so this follows a convention that was there before. It also closes a hole. The hand-rolled parser only validated flags it was asked about, so a typo passed straight through: `--component 5`, with the s missing, was ignored and the harness ran the default 300 components while the operator believed it ran 5. A benchmark that reports a confident number for the wrong project size is the failure this suite exists to prevent. parseArgs in strict mode rejects any undeclared flag, which means every harness now has to declare the flags the orchestrator sends it. Zod covers what parseArgs does not: coercing the string values to integers and rejecting fractions, negatives and NaN, and checking the enum flags. Options are typed by the harness interfaces rather than inferred from the schemas. Under this TypeScript setup zod 3 types a `.default()` key as optional even on parsed output, so inference would make every defaulted option optional at the call site; args.test.ts covers the gap by asserting a no-argument parse populates every field. Verified: yarn bench:docgen-perf --quick measures all five engines with no strict-mode mismatch, and the memory harness accepts both gate configs, including the live one with --recycle, --heavy and --no-force-gc. --- scripts/bench/docgen-memory/memory-harness.ts | 80 +++++++++---- scripts/bench/docgen-perf/cli.ts | 34 ++++-- .../bench/docgen-perf/engines/react-legacy.ts | 47 +++++--- .../bench/docgen-perf/engines/vue-scenario.ts | 55 ++++++--- .../bench/docgen-perf/generators/angular.ts | 24 ++-- scripts/bench/docgen-perf/generators/vue.ts | 44 +++++-- scripts/bench/docgen-shared/args.test.ts | 110 ++++++++++-------- scripts/bench/docgen-shared/args.ts | 108 ++++++----------- 8 files changed, 302 insertions(+), 200 deletions(-) diff --git a/scripts/bench/docgen-memory/memory-harness.ts b/scripts/bench/docgen-memory/memory-harness.ts index a45f83812b83..f25779ae62e1 100644 --- a/scripts/bench/docgen-memory/memory-harness.ts +++ b/scripts/bench/docgen-memory/memory-harness.ts @@ -35,7 +35,9 @@ import * as path from 'node:path'; import ts from 'typescript'; -import { Args } from '../docgen-shared/args.ts'; +import { z } from 'zod'; + +import { countOption, parseHarnessOptions } from '../docgen-shared/args.ts'; import { SANDBOX_DIRECTORY } from '../docgen-shared/paths.ts'; import { type StoryRefLike, @@ -94,26 +96,64 @@ interface HarnessOptions { recycleHeapPressureRatio?: number; } +const OPTIONS = { + components: { type: 'string' }, + variants: { type: 'string' }, + props: { type: 'string' }, + saves: { type: 'string' }, + mode: { type: 'string' }, + scope: { type: 'string' }, + recycle: { type: 'string' }, + heavy: { type: 'boolean' }, + 'heavy-factor': { type: 'string' }, + 'base64-kb': { type: 'string' }, + 'no-force-gc': { type: 'boolean' }, + out: { type: 'string' }, + reuse: { type: 'boolean' }, + json: { type: 'string' }, + 'max-retained-growth': { type: 'string' }, +} as const; + +const SCHEMA = z.object({ + components: countOption(600), + variants: countOption(4), + props: countOption(8), + saves: countOption(25), + mode: z.enum(MODES).default('refresh'), + scope: z.enum(SCOPES).default('all'), + // `Infinity` disables program recycling; `undefined` leaves the product default in place. + recycleHeapPressureRatio: z + .enum(RECYCLE) + .default('on') + .transform((recycle) => (recycle === 'off' ? Number.POSITIVE_INFINITY : undefined)), + heavyTypes: z.boolean().default(false), + heavyFactor: countOption(1), + base64Kb: countOption(0), + forceGc: z.boolean().default(false).transform((noForceGc) => !noForceGc), + outDir: z.string().default(path.join(SANDBOX_DIRECTORY, 'docgen-memory-stress')), + reuse: z.boolean().default(false), + jsonOut: z.string().optional(), + maxRetainedGrowthMb: countOption(400), +}); + function parseOptions(argv: string[]): HarnessOptions { - const args = new Args(argv); - return { - components: args.count('components', 600), - variants: args.count('variants', 4), - props: args.count('props', 8), - saves: args.count('saves', 25), - mode: args.choice('mode', MODES, 'refresh'), - scope: args.choice('scope', SCOPES, 'all'), - recycleHeapPressureRatio: - args.choice('recycle', RECYCLE, 'on') === 'off' ? Number.POSITIVE_INFINITY : undefined, - heavyTypes: args.flag('heavy'), - heavyFactor: args.count('heavy-factor', 1), - base64Kb: args.count('base64-kb', 0), - forceGc: !args.flag('no-force-gc'), - outDir: args.string('out', path.join(SANDBOX_DIRECTORY, 'docgen-memory-stress')), - reuse: args.flag('reuse'), - jsonOut: args.optional('json'), - maxRetainedGrowthMb: args.count('max-retained-growth', 400), - }; + return parseHarnessOptions(argv, OPTIONS, SCHEMA, (values) => ({ + components: values.components, + variants: values.variants, + props: values.props, + saves: values.saves, + mode: values.mode, + scope: values.scope, + recycleHeapPressureRatio: values.recycle, + heavyTypes: values.heavy, + heavyFactor: values['heavy-factor'], + base64Kb: values['base64-kb'], + forceGc: values['no-force-gc'], + outDir: values.out, + reuse: values.reuse, + jsonOut: values.json, + maxRetainedGrowthMb: values['max-retained-growth'], + })); } /** diff --git a/scripts/bench/docgen-perf/cli.ts b/scripts/bench/docgen-perf/cli.ts index 080ca2cadeb1..acdbe283bfd6 100644 --- a/scripts/bench/docgen-perf/cli.ts +++ b/scripts/bench/docgen-perf/cli.ts @@ -1,10 +1,18 @@ /** Command line for the docgen perf suite. */ import * as path from 'node:path'; -import { Args } from '../docgen-shared/args.ts'; +import { z } from 'zod'; + +import { parseHarnessOptions } from '../docgen-shared/args.ts'; import { ALL_ENGINE_IDS, DEFAULT_ENGINE_IDS } from './registry.ts'; import type { EngineId } from './types.ts'; +const OPTIONS = { + quick: { type: 'boolean' }, + engine: { type: 'string', multiple: true }, + json: { type: 'string' }, +} as const; + export interface CliOptions { quick: boolean; engines: EngineId[]; @@ -12,16 +20,20 @@ export interface CliOptions { } export function parseCliOptions(argv: string[], workRoot: string): CliOptions { - const args = new Args(argv); - const engines = args.all('engine'); - for (const engine of engines) { - if (!ALL_ENGINE_IDS.includes(engine as EngineId)) { - throw new Error(`--engine must be one of ${ALL_ENGINE_IDS.join(', ')}, got "${engine}"`); - } - } + const schema = z.object({ + quick: z.boolean().default(false), + // No --engine means the default set, which is why this defaults to empty rather than to + // DEFAULT_ENGINE_IDS: an explicit empty list and an absent flag are the same request. + engines: z.array(z.enum(ALL_ENGINE_IDS as [EngineId, ...EngineId[]])).default([]), + jsonOut: z.string().default(path.join(workRoot, 'results.json')), + }); + const options = parseHarnessOptions(argv, OPTIONS, schema, (values) => ({ + quick: values.quick, + engines: values.engine, + jsonOut: values.json, + })); return { - quick: args.flag('quick'), - engines: engines.length ? (engines as EngineId[]) : DEFAULT_ENGINE_IDS, - jsonOut: args.string('json', path.join(workRoot, 'results.json')), + ...options, + engines: options.engines.length ? options.engines : DEFAULT_ENGINE_IDS, }; } diff --git a/scripts/bench/docgen-perf/engines/react-legacy.ts b/scripts/bench/docgen-perf/engines/react-legacy.ts index 1d74997d5859..58c6f0676e16 100644 --- a/scripts/bench/docgen-perf/engines/react-legacy.ts +++ b/scripts/bench/docgen-perf/engines/react-legacy.ts @@ -15,8 +15,10 @@ import * as fs from 'node:fs'; import * as path from 'node:path'; +import { z } from 'zod'; + import { componentSource, generateProject } from '../../docgen-memory/generate-project.ts'; -import { Args } from '../../docgen-shared/args.ts'; +import { countOption, parseHarnessOptions } from '../../docgen-shared/args.ts'; import { SANDBOX_DIRECTORY } from '../../docgen-shared/paths.ts'; import { type ReactDocgenModule, @@ -39,6 +41,30 @@ const PARSERS = ['react-docgen', 'react-docgen-typescript'] as const; */ const SCOPES = ['all', 'changed'] as const; +const OPTIONS = { + parser: { type: 'string' }, + components: { type: 'string' }, + variants: { type: 'string' }, + props: { type: 'string' }, + saves: { type: 'string' }, + scope: { type: 'string' }, + out: { type: 'string' }, + json: { type: 'string' }, +} as const; + +const SCHEMA = z.object({ + parser: z.enum(PARSERS).default('react-docgen'), + components: countOption(300), + variants: countOption(4), + props: countOption(10), + saves: countOption(20), + scope: z.enum(SCOPES).default('changed'), + outDir: z + .string() + .default(path.join(SANDBOX_DIRECTORY, 'docgen-perf', 'react-legacy', 'project')), + jsonOut: z.string().optional(), +}); + interface HarnessOptions { parser: (typeof PARSERS)[number]; components: number; @@ -51,20 +77,11 @@ interface HarnessOptions { } function parseOptions(argv: string[]): HarnessOptions { - const args = new Args(argv); - return { - parser: args.choice('parser', PARSERS, 'react-docgen'), - components: args.count('components', 300), - variants: args.count('variants', 4), - props: args.count('props', 10), - saves: args.count('saves', 20), - scope: args.choice('scope', SCOPES, 'changed'), - outDir: args.string( - 'out', - path.join(SANDBOX_DIRECTORY, 'docgen-perf', 'react-legacy', 'project') - ), - jsonOut: args.optional('json'), - }; + return parseHarnessOptions(argv, OPTIONS, SCHEMA, (values) => ({ + ...values, + outDir: values.out, + jsonOut: values.json, + })); } async function createEngine(options: HarnessOptions): Promise { diff --git a/scripts/bench/docgen-perf/engines/vue-scenario.ts b/scripts/bench/docgen-perf/engines/vue-scenario.ts index a6c813ce6f9b..9a881b51a06d 100644 --- a/scripts/bench/docgen-perf/engines/vue-scenario.ts +++ b/scripts/bench/docgen-perf/engines/vue-scenario.ts @@ -13,7 +13,9 @@ */ import * as path from 'node:path'; -import { Args } from '../../docgen-shared/args.ts'; +import { z } from 'zod'; + +import { countOption, parseHarnessOptions } from '../../docgen-shared/args.ts'; import { SANDBOX_DIRECTORY } from '../../docgen-shared/paths.ts'; import { type GeneratedVueProject, @@ -25,6 +27,34 @@ import { export const VUE_SCENARIOS = ['flat', 'workspace', 'base-type-touch'] as const; export type VueScenario = (typeof VUE_SCENARIOS)[number]; +const OPTIONS = { + scenario: { type: 'string' }, + packages: { type: 'string' }, + 'components-per-package': { type: 'string' }, + 'chain-depth': { type: 'string' }, + 'fan-out': { type: 'string' }, + 'heavy-lib': { type: 'boolean' }, + saves: { type: 'string' }, + out: { type: 'string' }, + json: { type: 'string' }, +} as const; + +/** `engineDirName` only supplies the default scratch directory, so it is bound per harness. */ +const schemaFor = (engineDirName: string) => + z.object({ + scenario: z.enum(VUE_SCENARIOS).default('workspace'), + packages: countOption(4), + componentsPerPackage: countOption(10), + chainDepth: countOption(3), + fanOut: countOption(4), + heavyLib: z.boolean().default(false), + saves: countOption(15), + outDir: z + .string() + .default(path.join(SANDBOX_DIRECTORY, 'docgen-perf', engineDirName, 'project')), + jsonOut: z.string().optional(), + }); + export interface VueHarnessOptions { scenario: VueScenario; packages: number; @@ -38,18 +68,17 @@ export interface VueHarnessOptions { } export function parseVueOptions(argv: string[], engineDirName: string): VueHarnessOptions { - const args = new Args(argv); - return { - scenario: args.choice('scenario', VUE_SCENARIOS, 'workspace'), - packages: args.count('packages', 4), - componentsPerPackage: args.count('components-per-package', 10), - chainDepth: args.count('chain-depth', 3), - fanOut: args.count('fan-out', 4), - heavyLib: args.flag('heavy-lib'), - saves: args.count('saves', 15), - outDir: args.string('out', path.join(SANDBOX_DIRECTORY, 'docgen-perf', engineDirName, 'project')), - jsonOut: args.optional('json'), - }; + return parseHarnessOptions(argv, OPTIONS, schemaFor(engineDirName), (values) => ({ + scenario: values.scenario, + packages: values.packages, + componentsPerPackage: values['components-per-package'], + chainDepth: values['chain-depth'], + fanOut: values['fan-out'], + heavyLib: values['heavy-lib'], + saves: values.saves, + outDir: values.out, + jsonOut: values.json, + })); } export function vueBanner(options: VueHarnessOptions): Record { diff --git a/scripts/bench/docgen-perf/generators/angular.ts b/scripts/bench/docgen-perf/generators/angular.ts index ab1e47ef2f4e..d56351959acd 100644 --- a/scripts/bench/docgen-perf/generators/angular.ts +++ b/scripts/bench/docgen-perf/generators/angular.ts @@ -11,7 +11,9 @@ import * as fs from 'node:fs'; import * as path from 'node:path'; import { pathToFileURL } from 'node:url'; -import { Args } from '../../docgen-shared/args.ts'; +import { z } from 'zod'; + +import { countOption, parseHarnessOptions } from '../../docgen-shared/args.ts'; export interface AngularGenerateOptions { outDir: string; @@ -117,17 +119,21 @@ export function generateAngularProject(options: AngularGenerateOptions): Generat return { outDir, componentPaths }; } -function parseArgs(argv: string[]): AngularGenerateOptions { - const args = new Args(argv); - return { - outDir: args.string('out', '../storybook-sandboxes/docgen-perf-angular'), - components: args.count('components', 100), - props: args.count('props', 8), - }; +function parseOptions(argv: string[]): AngularGenerateOptions { + return parseHarnessOptions( + argv, + { out: { type: 'string' }, components: { type: 'string' }, props: { type: 'string' } } as const, + z.object({ + outDir: z.string().default('../storybook-sandboxes/docgen-perf-angular'), + components: countOption(100), + props: countOption(8), + }), + (values) => ({ ...values, outDir: values.out }) + ); } if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) { - const options = parseArgs(process.argv.slice(2)); + const options = parseOptions(process.argv.slice(2)); const start = Date.now(); const result = generateAngularProject(options); console.log( diff --git a/scripts/bench/docgen-perf/generators/vue.ts b/scripts/bench/docgen-perf/generators/vue.ts index c0f927566b86..002d038d0331 100644 --- a/scripts/bench/docgen-perf/generators/vue.ts +++ b/scripts/bench/docgen-perf/generators/vue.ts @@ -21,7 +21,9 @@ import * as fs from 'node:fs'; import * as path from 'node:path'; import { pathToFileURL } from 'node:url'; -import { Args } from '../../docgen-shared/args.ts'; +import { z } from 'zod'; + +import { countOption, parseHarnessOptions } from '../../docgen-shared/args.ts'; const require = createRequire(import.meta.url); @@ -263,20 +265,38 @@ export function generateVueProject(options: VueGenerateOptions): GeneratedVuePro return { outDir, packageConfigPaths, baseTypesPath, componentPaths }; } -function parseArgs(argv: string[]): VueGenerateOptions { - const args = new Args(argv); - return { - outDir: args.string('out', '../storybook-sandboxes/docgen-perf-vue'), - packages: args.count('packages', 4), - componentsPerPackage: args.count('components-per-package', 10), - chainDepth: args.count('chain-depth', 3), - fanOut: args.count('fan-out', 4), - heavyLib: args.flag('heavy-lib'), - }; +function parseOptions(argv: string[]): VueGenerateOptions { + return parseHarnessOptions( + argv, + { + out: { type: 'string' }, + packages: { type: 'string' }, + 'components-per-package': { type: 'string' }, + 'chain-depth': { type: 'string' }, + 'fan-out': { type: 'string' }, + 'heavy-lib': { type: 'boolean' }, + } as const, + z.object({ + outDir: z.string().default('../storybook-sandboxes/docgen-perf-vue'), + packages: countOption(4), + componentsPerPackage: countOption(10), + chainDepth: countOption(3), + fanOut: countOption(4), + heavyLib: z.boolean().default(false), + }), + (values) => ({ + outDir: values.out, + packages: values.packages, + componentsPerPackage: values['components-per-package'], + chainDepth: values['chain-depth'], + fanOut: values['fan-out'], + heavyLib: values['heavy-lib'], + }) + ); } if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) { - const options = parseArgs(process.argv.slice(2)); + const options = parseOptions(process.argv.slice(2)); const start = Date.now(); const result = generateVueProject(options); console.log( diff --git a/scripts/bench/docgen-shared/args.test.ts b/scripts/bench/docgen-shared/args.test.ts index 47b768babdc5..a8605dff440a 100644 --- a/scripts/bench/docgen-shared/args.test.ts +++ b/scripts/bench/docgen-shared/args.test.ts @@ -1,75 +1,91 @@ import { describe, expect, it } from 'vitest'; +import { z } from 'zod'; -import { Args } from './args.ts'; +import { countOption, parseHarnessOptions } from './args.ts'; -describe('Args', () => { - it('reads a string value', () => { - expect(new Args(['--parser', 'react-docgen']).string('parser', 'x')).toBe('react-docgen'); +const OPTIONS = { + components: { type: 'string' }, + scope: { type: 'string' }, + heavy: { type: 'boolean' }, + out: { type: 'string' }, + 'components-per-package': { type: 'string' }, +} as const; + +const SCHEMA = z.object({ + components: countOption(300), + scope: z.enum(['all', 'changed']).default('changed'), + heavy: z.boolean().default(false), + outDir: z.string().default('/default'), + componentsPerPackage: countOption(10), +}); + +interface Options { + components: number; + scope: 'all' | 'changed'; + heavy: boolean; + outDir: string; + componentsPerPackage: number; +} + +const parse = (argv: string[]) => + parseHarnessOptions(argv, OPTIONS, SCHEMA, (values) => ({ + ...values, + outDir: values.out, + componentsPerPackage: values['components-per-package'], + })); + +describe('parseHarnessOptions', () => { + it('applies defaults when nothing is passed', () => { + expect(parse([])).toEqual({ + components: 300, + scope: 'changed', + heavy: false, + outDir: '/default', + componentsPerPackage: 10, + }); }); - it('falls back when the flag is absent', () => { - expect(new Args([]).string('parser', 'fallback')).toBe('fallback'); + it('coerces numeric flags', () => { + expect(parse(['--components', '20']).components).toBe(20); }); - it('detects boolean flags', () => { - expect(new Args(['--quick']).flag('quick')).toBe(true); - expect(new Args([]).flag('quick')).toBe(false); + it('maps kebab-case flags onto the schema shape', () => { + expect(parse(['--components-per-package', '5']).componentsPerPackage).toBe(5); }); - it('throws when a flag is followed by another flag instead of a value', () => { - expect(() => new Args(['--saves', '--json', 'out.json']).count('saves', 1)).toThrow( - '--saves requires a value' - ); + it('reads boolean flags', () => { + expect(parse(['--heavy']).heavy).toBe(true); }); - it('throws when a flag is the last argument', () => { - expect(() => new Args(['--out']).string('out', 'x')).toThrow('--out requires a value'); + it('rejects an unknown flag', () => { + // The reason for strict mode: `--component 5` would otherwise be dropped silently and the + // harness would benchmark 300 components while reporting a run someone asked for at 5. + expect(() => parse(['--component', '5'])).toThrow(/Unknown option/); }); - it('collects every value of a repeated flag', () => { - const args = new Args(['--engine', 'a', '--quick', '--engine', 'b']); - expect(args.all('engine')).toEqual(['a', 'b']); + it('rejects a flag whose value is another flag', () => { + expect(() => parse(['--components', '--out', 'x'])).toThrow(/ambiguous/); }); - it('returns an empty list when a repeated flag is absent', () => { - expect(new Args(['--quick']).all('engine')).toEqual([]); + it('rejects a flag with no value at all', () => { + expect(() => parse(['--components'])).toThrow(/argument missing/); }); - describe('count', () => { - it('parses an integer', () => { - expect(new Args(['--saves', '20']).count('saves', 1)).toBe(20); - }); + it('rejects a value outside an enum, naming the flag', () => { + expect(() => parse(['--scope', 'some'])).toThrow(/--scope/); + }); + describe('countOption', () => { it('rejects a non-numeric value', () => { - expect(() => new Args(['--saves', 'lots']).count('saves', 1)).toThrow( - '--saves must be a non-negative integer, got "lots"' - ); + expect(() => parse(['--components', 'lots'])).toThrow(/--components/); }); it('rejects a fraction', () => { - expect(() => new Args(['--saves', '2.5']).count('saves', 1)).toThrow( - 'must be a non-negative integer' - ); + expect(() => parse(['--components', '2.5'])).toThrow(/--components/); }); it('rejects a negative', () => { - expect(() => new Args(['--saves', '-3']).count('saves', 1)).toThrow( - 'must be a non-negative integer' - ); - }); - }); - - describe('choice', () => { - it('accepts an allowed value', () => { - expect(new Args(['--scope', 'all']).choice('scope', ['all', 'changed'], 'changed')).toBe( - 'all' - ); - }); - - it('names the whole set when the value is not allowed', () => { - expect(() => - new Args(['--scope', 'some']).choice('scope', ['all', 'changed'], 'changed') - ).toThrow('--scope must be one of all, changed, got "some"'); + expect(() => parse(['--components', '-3'])).toThrow(/--components/); }); }); }); diff --git a/scripts/bench/docgen-shared/args.ts b/scripts/bench/docgen-shared/args.ts index e29d8ebb9d17..bcd548290d5e 100644 --- a/scripts/bench/docgen-shared/args.ts +++ b/scripts/bench/docgen-shared/args.ts @@ -1,81 +1,43 @@ /** - * Strict command-line parsing for the docgen bench harnesses. + * Option parsing for the docgen bench harnesses: `node:util` parseArgs in strict mode for the + * parse, then a Zod schema for coercion and validation. Same pairing as scripts/eval/eval.ts. * - * Every harness used to carry its own `get(flag, fallback)` helper that returned `argv[idx + 1]` - * without checking it. `--saves --json out.json` therefore parsed as `saves = "--json"`, and - * `Number()` turned that into `NaN`, which reached the generators as a silently wrong project size. - * Values are validated here instead, once. + * Strict mode is what makes this worth doing. It rejects a flag the harness does not declare, so + * `--component 5` with the `s` missing fails loudly instead of leaving the default in place and + * reporting a benchmark for a project size nobody asked for. */ +import { type ParseArgsConfig, parseArgs } from 'node:util'; -export class Args { - // A plain field, not a constructor parameter property: these harnesses run on Node's strip-only - // TypeScript support, which rejects parameter properties outright. - private readonly argv: string[]; +import { z } from 'zod'; - constructor(argv: string[]) { - this.argv = argv; - } - - /** True when `--name` is present. */ - flag(name: string): boolean { - return this.argv.includes(`--${name}`); - } - - /** - * The value after `--name`. Throws when the flag is present but its value is missing or is - * itself a flag, so a typo fails at the boundary instead of becoming a bad measurement. - */ - private valueAt(name: string, at: number): string { - const value = this.argv[at + 1]; - if (value === undefined || value.startsWith('--')) { - throw new Error(`--${name} requires a value`); - } - return value; - } - - /** The value after `--name`, or undefined when the flag is absent. */ - optional(name: string): string | undefined { - const at = this.argv.indexOf(`--${name}`); - return at < 0 ? undefined : this.valueAt(name, at); - } - - string(name: string, fallback: string): string { - return this.optional(name) ?? fallback; - } - - /** Every value of a repeatable flag, in the order given. */ - all(name: string): string[] { - const values: string[] = []; - for (let i = 0; i < this.argv.length; i++) { - if (this.argv[i] === `--${name}`) { - values.push(this.valueAt(name, i)); - } - } - return values; - } +type OptionsConfig = NonNullable; - /** A non-negative integer. Rejects `NaN`, fractions and negatives rather than passing them on. */ - count(name: string, fallback: number): number { - const raw = this.optional(name); - if (raw === undefined) { - return fallback; - } - const value = Number(raw); - if (!Number.isInteger(value) || value < 0) { - throw new Error(`--${name} must be a non-negative integer, got "${raw}"`); - } - return value; - } +/** parseArgs yields strings, so numeric flags are coerced here rather than at each call site. */ +export const countOption = (fallback: number) => + z.coerce.number().int().nonnegative().default(fallback); - /** One of `allowed`. The error names the whole set, so a typo is self-correcting. */ - choice(name: string, allowed: readonly T[], fallback: T): T { - const raw = this.optional(name); - if (raw === undefined) { - return fallback; - } - if (!(allowed as readonly string[]).includes(raw)) { - throw new Error(`--${name} must be one of ${allowed.join(', ')}, got "${raw}"`); - } - return raw as T; - } +/** + * Keys in `values` are the flags as written, so kebab-case flags arrive kebab-cased. `toInput` maps + * them onto the schema's shape, the way eval.ts spreads renamed keys into `safeParse`. + * + * `Out` is supplied by the caller rather than inferred: under this TypeScript setup zod 3 types a + * `.default()` key as optional even on the parsed output, so inference would make every defaulted + * option optional at the call site. The harness interfaces stay the source of truth, and + * args.test.ts asserts that a parse with no arguments really does populate every field. + */ +export function parseHarnessOptions( + argv: string[], + options: OptionsConfig, + schema: z.ZodTypeAny, + toInput?: (values: Record) => Record +): Out { + const { values } = parseArgs({ args: argv, options, strict: true }); + const result = schema.safeParse(toInput ? toInput(values) : values); + if (!result.success) { + const issues = result.error.issues + .map((issue) => ` --${issue.path.join('.')}: ${issue.message}`) + .join('\n'); + throw new Error(`invalid options:\n${issues}`); + } + return result.data as Out; } From cd5d1bbcd945064957c29dc6b23deddd9b2bc544 Mon Sep 17 00:00:00 2001 From: Valentin Palkovic Date: Tue, 28 Jul 2026 20:29:14 +0200 Subject: [PATCH 11/14] Bench: Simplify the docgen bench plumbing parseHarnessOptions now camel-cases kebab flags, so each harness spells out only the real renames such as --out onto outDir. The memory harness drops from fifteen mapped keys to six, and a typo in one can no longer leave a default silently in place. forceGc and withNodeModules lose their double negative: the inversion sits where the flag is renamed, not inside the Zod schema. SeriesResult extends SeriesSummary instead of restating its four memory fields, and runSeries spreads summarizeSeries. summarizeSeries walks the GC-sampled saves once rather than twice. ratioFor builds one object literal, with a likeForLike helper beside the existing medianRatio. One outputTail helper replaces three copies of the last-N-lines formatting. The react-legacy parser wiring moves into loadParser. run.ts drops optional calls on methods BenchEngine always defines. No behavior change. The bench unit tests pass, and the React, Vue and memory harnesses were run end to end. --- scripts/bench/docgen-memory/gate.ts | 11 +- .../bench/docgen-memory/generate-project.ts | 72 +++++++++--- scripts/bench/docgen-memory/memory-harness.ts | 34 ++---- scripts/bench/docgen-perf/aggregate.test.ts | 28 ++--- scripts/bench/docgen-perf/aggregate.ts | 23 ++-- scripts/bench/docgen-perf/cli.ts | 2 +- scripts/bench/docgen-perf/engine.ts | 3 +- scripts/bench/docgen-perf/engines/compodoc.ts | 9 +- .../bench/docgen-perf/engines/react-legacy.ts | 110 +++++++++++------- .../bench/docgen-perf/engines/vue-scenario.ts | 8 +- scripts/bench/docgen-perf/generators/vue.ts | 9 +- scripts/bench/docgen-perf/ratios.ts | 65 ++++++----- scripts/bench/docgen-perf/report.ts | 6 +- scripts/bench/docgen-perf/run.ts | 11 +- scripts/bench/docgen-perf/spawn.ts | 14 +-- scripts/bench/docgen-shared/args.test.ts | 1 - scripts/bench/docgen-shared/args.ts | 14 ++- scripts/bench/docgen-shared/child-output.ts | 12 ++ .../bench/docgen-shared/renderer-module.ts | 42 +------ scripts/bench/docgen-shared/series.ts | 19 +-- scripts/bench/docgen-shared/stats.ts | 23 ++-- 21 files changed, 259 insertions(+), 257 deletions(-) create mode 100644 scripts/bench/docgen-shared/child-output.ts diff --git a/scripts/bench/docgen-memory/gate.ts b/scripts/bench/docgen-memory/gate.ts index 4a6644e33832..c24d7636ef82 100644 --- a/scripts/bench/docgen-memory/gate.ts +++ b/scripts/bench/docgen-memory/gate.ts @@ -21,6 +21,7 @@ import * as fs from 'node:fs'; import * as path from 'node:path'; import { type MemoryBudgets, memoryBudgetsFor } from '../docgen-shared/budgets.ts'; +import { outputTail } from '../docgen-shared/child-output.ts'; import { SANDBOX_DIRECTORY } from '../../utils/constants.ts'; interface HarnessResult { @@ -142,15 +143,7 @@ function runHarness(config: GateConfig): RunOutcome { const proc = spawnSync(process.execPath, nodeArgs, { encoding: 'utf8' }); const output = `${proc.stdout ?? ''}${proc.stderr ?? ''}`; - // Echo a compact tail so CI logs show what happened without the full V8 crash dump. - console.log( - output - .trim() - .split('\n') - .slice(-8) - .map((line) => ` ${line}`) - .join('\n') - ); + console.log(outputTail(output, 8)); const result = fs.existsSync(jsonPath) ? (JSON.parse(fs.readFileSync(jsonPath, 'utf8')) as HarnessResult) diff --git a/scripts/bench/docgen-memory/generate-project.ts b/scripts/bench/docgen-memory/generate-project.ts index ab29f440ef79..a6e476996af7 100644 --- a/scripts/bench/docgen-memory/generate-project.ts +++ b/scripts/bench/docgen-memory/generate-project.ts @@ -19,6 +19,11 @@ import * as fs from 'node:fs'; import * as path from 'node:path'; import { pathToFileURL } from 'node:url'; +import { z } from 'zod'; + +import { countOption, parseHarnessOptions } from '../docgen-shared/args.ts'; +import type { ComponentRefLike, StoryRefLike } from '../docgen-shared/renderer-module.ts'; + const require = createRequire(import.meta.url); export interface GenerateOptions { @@ -239,6 +244,26 @@ ${variantExports} `; } +/** The renderer-side reference for the generated `Comp{i}`, matching the naming used above. */ +export function componentRef(i: number, componentPath: string): ComponentRefLike { + return { + componentName: `Comp${i}`, + importName: `Comp${i}`, + localImportName: `Comp${i}`, + importId: `./Comp${i}`, + isPackage: false, + path: componentPath, + }; +} + +/** Pairs {@link GeneratedProject.componentPaths} with `storyPaths`, which share an index. */ +export function buildStoryRefs(componentPaths: string[], storyPaths: string[]): StoryRefLike[] { + return componentPaths.map((componentPath, i) => ({ + storyPath: storyPaths[i], + component: componentRef(i, componentPath), + })); +} + export function generateProject(options: GenerateOptions): GeneratedProject { const outDir = path.resolve(options.outDir); fs.rmSync(outDir, { recursive: true, force: true }); @@ -275,25 +300,40 @@ export function generateProject(options: GenerateOptions): GeneratedProject { return { outDir, configPath, componentPaths, storyPaths }; } -function parseArgs(argv: string[]): GenerateOptions { - const get = (flag: string, fallback: string) => { - const idx = argv.indexOf(flag); - return idx >= 0 && argv[idx + 1] ? argv[idx + 1] : fallback; - }; - return { - outDir: get('--out', '../storybook-sandboxes/docgen-memory-stress'), - components: Number(get('--components', '500')), - variants: Number(get('--variants', '4')), - props: Number(get('--props', '8')), - heavyTypes: argv.includes('--heavy'), - heavyFactor: Number(get('--heavy-factor', '1')), - base64Kb: Number(get('--base64-kb', '0')), - withNodeModules: !argv.includes('--no-node-modules'), - }; +function parseOptions(argv: string[]): GenerateOptions { + return parseHarnessOptions( + argv, + { + out: { type: 'string' }, + components: { type: 'string' }, + variants: { type: 'string' }, + props: { type: 'string' }, + heavy: { type: 'boolean' }, + 'heavy-factor': { type: 'string' }, + 'base64-kb': { type: 'string' }, + 'no-node-modules': { type: 'boolean' }, + } as const, + z.object({ + outDir: z.string().default('../storybook-sandboxes/docgen-memory-stress'), + components: countOption(500), + variants: countOption(4), + props: countOption(8), + heavyTypes: z.boolean().default(false), + heavyFactor: countOption(1), + base64Kb: countOption(0), + withNodeModules: z.boolean().default(true), + }), + (values) => ({ + ...values, + outDir: values.out, + heavyTypes: values.heavy, + withNodeModules: !values.noNodeModules, + }) + ); } if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) { - const options = parseArgs(process.argv.slice(2)); + const options = parseOptions(process.argv.slice(2)); const start = Date.now(); const result = generateProject(options); console.log( diff --git a/scripts/bench/docgen-memory/memory-harness.ts b/scripts/bench/docgen-memory/memory-harness.ts index f25779ae62e1..0352c6331021 100644 --- a/scripts/bench/docgen-memory/memory-harness.ts +++ b/scripts/bench/docgen-memory/memory-harness.ts @@ -39,11 +39,7 @@ import { z } from 'zod'; import { countOption, parseHarnessOptions } from '../docgen-shared/args.ts'; import { SANDBOX_DIRECTORY } from '../docgen-shared/paths.ts'; -import { - type StoryRefLike, - buildStoryRefs, - loadRendererModule, -} from '../docgen-shared/renderer-module.ts'; +import { type StoryRefLike, loadRendererModule } from '../docgen-shared/renderer-module.ts'; import { MB, gcAvailable } from '../docgen-shared/sampling.ts'; import { type SeriesEngine, @@ -52,7 +48,7 @@ import { runSeries, } from '../docgen-shared/series.ts'; import { leastSquaresSlope } from '../docgen-shared/stats.ts'; -import { componentSource, generateProject } from './generate-project.ts'; +import { buildStoryRefs, componentSource, generateProject } from './generate-project.ts'; interface ComponentMetaManagerLike { batchExtract(entries: StoryRefLike[]): void; @@ -129,7 +125,7 @@ const SCHEMA = z.object({ heavyTypes: z.boolean().default(false), heavyFactor: countOption(1), base64Kb: countOption(0), - forceGc: z.boolean().default(false).transform((noForceGc) => !noForceGc), + forceGc: z.boolean().default(true), outDir: z.string().default(path.join(SANDBOX_DIRECTORY, 'docgen-memory-stress')), reuse: z.boolean().default(false), jsonOut: z.string().optional(), @@ -138,21 +134,12 @@ const SCHEMA = z.object({ function parseOptions(argv: string[]): HarnessOptions { return parseHarnessOptions(argv, OPTIONS, SCHEMA, (values) => ({ - components: values.components, - variants: values.variants, - props: values.props, - saves: values.saves, - mode: values.mode, - scope: values.scope, + ...values, recycleHeapPressureRatio: values.recycle, heavyTypes: values.heavy, - heavyFactor: values['heavy-factor'], - base64Kb: values['base64-kb'], - forceGc: values['no-force-gc'], + forceGc: !values.noForceGc, outDir: values.out, - reuse: values.reuse, jsonOut: values.json, - maxRetainedGrowthMb: values['max-retained-growth'], })); } @@ -168,9 +155,10 @@ async function loadComponentMetaManager(): Promise { } function resolveProject(options: HarnessOptions): ReturnType { - const genStart = Date.now(); - if (options.reuse && fs.existsSync(path.join(path.resolve(options.outDir), 'tsconfig.json'))) { - const outDir = path.resolve(options.outDir); + const outDir = path.resolve(options.outDir); + const configPath = path.join(outDir, 'tsconfig.json'); + + if (options.reuse && fs.existsSync(configPath)) { const componentPaths: string[] = []; const storyPaths: string[] = []; for (let i = 0; i < options.components; i++) { @@ -178,8 +166,10 @@ function resolveProject(options: HarnessOptions): ReturnType sample(i + 1, d)), - retainedSlope: 0.5, - retainedGrowth: 4, - }; + durations.map((d, i) => sample(i + 1, d)), + { rssMb: 90, heapUsedMb: 12, retainedHeapMb: 10 } + ); } describe('designatedRep', () => { @@ -70,15 +74,13 @@ describe('seriesMetrics', () => { it('rejects an empty set of repetitions', () => { expect(() => seriesMetrics([], 5)).toThrow('no completed repetition recorded'); - expect(() => seriesMetrics(undefined, 5)).toThrow('no completed repetition recorded'); }); it('rejects a run with no retained samples', () => { - const noGc: SeriesResult = { - coldMs: 100, - baseline: { rssMb: 90, heapUsedMb: 12 }, - samples: [{ save: 1, durMs: 5, rssMb: 100, heapUsedMb: 20 }], - }; + const noGc = repOf(100, [{ save: 1, durMs: 5, rssMb: 100, heapUsedMb: 20 }], { + rssMb: 90, + heapUsedMb: 12, + }); expect(() => seriesMetrics([noGc], 1)).toThrow('retained metrics missing'); }); }); diff --git a/scripts/bench/docgen-perf/aggregate.ts b/scripts/bench/docgen-perf/aggregate.ts index 2b7710a42091..0f69cabf6f85 100644 --- a/scripts/bench/docgen-perf/aggregate.ts +++ b/scripts/bench/docgen-perf/aggregate.ts @@ -17,20 +17,13 @@ export function designatedRep(reps: T[]): T { return byCold[Math.floor((byCold.length - 1) / 2)]; } -/** Per-save allocation spikes above the retained baseline, for the repetitions the child sampled. */ -function transientsOf(rep: SeriesResult): number[] { - return rep.samples - .map((s) => (s.retainedHeapMb !== undefined ? s.heapUsedMb - s.retainedHeapMb : undefined)) - .filter((v): v is number => v !== undefined); -} - /** * An engine that failed part-way through holds fewer samples than expectedN. Reporting those would * put numbers taken at an unrecorded N into the results file, which the comparison method rests on * not happening. */ -function assertRepetitionCount(reps: unknown[] | undefined, expectedN: number): void { - if (!reps || reps.length === 0) { +function assertRepetitionCount(reps: unknown[], expectedN: number): void { + if (reps.length === 0) { throw new Error('no completed repetition recorded'); } if (reps.length !== expectedN) { @@ -38,15 +31,15 @@ function assertRepetitionCount(reps: unknown[] | undefined, expectedN: number): } } -export function seriesMetrics(reps: SeriesResult[] | undefined, expectedN: number): EngineMetrics { +export function seriesMetrics(reps: SeriesResult[], expectedN: number): EngineMetrics { assertRepetitionCount(reps, expectedN); - const designated = designatedRep(reps!); - const coldSamples = reps!.map((r) => r.coldMs); + const designated = designatedRep(reps); + const coldSamples = reps.map((r) => r.coldMs); const warmSamples = designated.samples.map((s) => s.durMs); - const transients = transientsOf(designated); + const { transients, avgTransient } = designated; if ( - transients.length === 0 || + avgTransient === undefined || designated.retainedGrowth === undefined || designated.retainedSlope === undefined ) { @@ -58,7 +51,7 @@ export function seriesMetrics(reps: SeriesResult[] | undefined, expectedN: numbe warmExtractionMs: { status: 'measured', samples: warmSamples, median: median(warmSamples) }, // Per-component engines have no batch pass; recording one would be a faked equivalent. wholeProjectScanMs: NOT_APPLICABLE, - peakTransientMb: { status: 'measured', samples: transients, mean: mean(transients) }, + peakTransientMb: { status: 'measured', samples: transients, mean: avgTransient }, retainedGrowthMb: { status: 'measured', value: designated.retainedGrowth }, retainedSlopeMbPerSave: { status: 'measured', value: designated.retainedSlope }, }; diff --git a/scripts/bench/docgen-perf/cli.ts b/scripts/bench/docgen-perf/cli.ts index acdbe283bfd6..4fdfdedff700 100644 --- a/scripts/bench/docgen-perf/cli.ts +++ b/scripts/bench/docgen-perf/cli.ts @@ -28,7 +28,7 @@ export function parseCliOptions(argv: string[], workRoot: string): CliOptions { jsonOut: z.string().default(path.join(workRoot, 'results.json')), }); const options = parseHarnessOptions(argv, OPTIONS, schema, (values) => ({ - quick: values.quick, + ...values, engines: values.engine, jsonOut: values.json, })); diff --git a/scripts/bench/docgen-perf/engine.ts b/scripts/bench/docgen-perf/engine.ts index f823e3cd9249..398a7afb0ec6 100644 --- a/scripts/bench/docgen-perf/engine.ts +++ b/scripts/bench/docgen-perf/engine.ts @@ -4,12 +4,11 @@ */ import * as path from 'node:path'; -import type { EngineId } from '../docgen-shared/engine-ids.ts'; import type { SeriesResult } from '../docgen-shared/series.ts'; import { seriesMetrics } from './aggregate.ts'; import type { SuiteProfile } from './config.ts'; import type { SeriesChildSpec } from './spawn.ts'; -import type { EngineMetrics } from './types.ts'; +import type { EngineId, EngineMetrics } from './types.ts'; export interface ScenarioSpec { name: string; diff --git a/scripts/bench/docgen-perf/engines/compodoc.ts b/scripts/bench/docgen-perf/engines/compodoc.ts index e7941f5f7310..792f236d0e3c 100644 --- a/scripts/bench/docgen-perf/engines/compodoc.ts +++ b/scripts/bench/docgen-perf/engines/compodoc.ts @@ -12,12 +12,12 @@ import { spawn, spawnSync } from 'node:child_process'; import * as fs from 'node:fs'; import * as path from 'node:path'; +import { outputTail } from '../../docgen-shared/child-output.ts'; import { type OneShotRepetition, oneShotMetrics } from '../aggregate.ts'; +import type { AngularScenarioConfig, SuiteProfile } from '../config.ts'; import { BenchEngine, type MeasureContext, type ScenarioSpec } from '../engine.ts'; import { angularComponentSource, generateAngularProject } from '../generators/angular.ts'; -import type { AngularScenarioConfig, SuiteProfile } from '../config.ts'; -import type { EngineId } from '../../docgen-shared/engine-ids.ts'; -import type { EngineMetrics } from '../types.ts'; +import type { EngineId, EngineMetrics } from '../types.ts'; /** * Walked up from the resolved binary rather than a fixed path, since the package hoists to @@ -105,8 +105,7 @@ function runCompodocOnce( clearInterval(poll); const durMs = Date.now() - start; if (status !== 0) { - const tail = output.trim().split('\n').slice(-8).join('\n'); - reject(new Error(`compodoc exited with status ${status}:\n${tail}`)); + reject(new Error(`compodoc exited with status ${status}:\n${outputTail(output, 8)}`)); return; } if (!fs.existsSync(path.join(docsOutDir, 'documentation.json'))) { diff --git a/scripts/bench/docgen-perf/engines/react-legacy.ts b/scripts/bench/docgen-perf/engines/react-legacy.ts index 58c6f0676e16..7df062c05e8b 100644 --- a/scripts/bench/docgen-perf/engines/react-legacy.ts +++ b/scripts/bench/docgen-perf/engines/react-legacy.ts @@ -17,14 +17,16 @@ import * as path from 'node:path'; import { z } from 'zod'; -import { componentSource, generateProject } from '../../docgen-memory/generate-project.ts'; +import { + type GeneratedProject, + componentRef, + componentSource, + generateProject, +} from '../../docgen-memory/generate-project.ts'; import { countOption, parseHarnessOptions } from '../../docgen-shared/args.ts'; import { SANDBOX_DIRECTORY } from '../../docgen-shared/paths.ts'; import { - type ReactDocgenModule, - type ReactDocgenTypescriptModule, - type UtilsModule, - componentRef, + type ComponentRefLike, loadRendererModule, } from '../../docgen-shared/renderer-module.ts'; import { @@ -33,6 +35,23 @@ import { runSeriesHarness, } from '../../docgen-shared/series.ts'; +/** The renderer module surfaces this harness loads, narrowed to what it calls. */ +interface UtilsModule { + invalidateCache(): void; +} + +interface ReactDocgenModule { + getReactDocgen( + path: string, + component: ComponentRefLike + ): { type: 'success' } | { type: 'error'; error: { name: string; message: string } }; +} + +interface ReactDocgenTypescriptModule { + parseWithReactDocgenTypescript(filePath: string): Promise>; + invalidateParser(): void; +} + const PARSERS = ['react-docgen', 'react-docgen-typescript'] as const; /** * Which components to re-extract per save. @@ -84,6 +103,50 @@ function parseOptions(argv: string[]): HarnessOptions { })); } +interface Parser { + /** Extract `Comp{i}`. Throws on a failed or empty result, which is fast for the wrong reason. */ + extractOne(i: number): Promise; + /** Global invalidation is the only surface these engines expose. */ + invalidate(): void; +} + +async function loadParser(options: HarnessOptions, project: GeneratedProject): Promise { + const { componentPaths } = project; + const { invalidateCache } = await loadRendererModule('utils.ts'); + + if (options.parser === 'react-docgen') { + const { getReactDocgen } = await loadRendererModule('reactDocgen.ts'); + return { + async extractOne(i) { + const result = getReactDocgen(componentPaths[i], componentRef(i, componentPaths[i])); + if (result.type === 'error') { + throw new Error( + `react-docgen failed on Comp${i}: ${result.error.name} ${result.error.message}` + ); + } + }, + invalidate: invalidateCache, + }; + } + + const { parseWithReactDocgenTypescript, invalidateParser } = + await loadRendererModule('reactDocgenTypescript.ts'); + // The parser resolves its tsconfig from process.cwd(); point it at the generated project. + process.chdir(project.outDir); + return { + async extractOne(i) { + const docs = await parseWithReactDocgenTypescript(componentPaths[i]); + if (docs.length === 0) { + throw new Error(`react-docgen-typescript returned no docs for Comp${i}`); + } + }, + invalidate() { + invalidateParser(); + invalidateCache(); + }, + }; +} + async function createEngine(options: HarnessOptions): Promise { const genStart = Date.now(); const project = generateProject({ @@ -98,39 +161,7 @@ async function createEngine(options: HarnessOptions): Promise { }); console.log(` generated project in ${Date.now() - genStart}ms at ${project.outDir}`); - const { invalidateCache } = await loadRendererModule('utils.ts'); - - let extractOne: (i: number) => Promise; - let invalidate: () => void; - - if (options.parser === 'react-docgen') { - const { getReactDocgen } = await loadRendererModule('reactDocgen.ts'); - extractOne = async (i) => { - const componentPath = project.componentPaths[i]; - const result = getReactDocgen(componentPath, componentRef(i, componentPath)); - if (result.type === 'error') { - throw new Error( - `react-docgen failed on Comp${i}: ${result.error.name} ${result.error.message}` - ); - } - }; - invalidate = invalidateCache; - } else { - const { parseWithReactDocgenTypescript, invalidateParser } = - await loadRendererModule('reactDocgenTypescript.ts'); - // The parser resolves its tsconfig from process.cwd(); point it at the generated project. - process.chdir(project.outDir); - extractOne = async (i) => { - const docs = await parseWithReactDocgenTypescript(project.componentPaths[i]); - if (docs.length === 0) { - throw new Error(`react-docgen-typescript returned no docs for Comp${i}`); - } - }; - invalidate = () => { - invalidateParser(); - invalidateCache(); - }; - } + const { extractOne, invalidate } = await loadParser(options, project); const extractAll = async () => { for (let i = 0; i < options.components; i++) { @@ -151,8 +182,7 @@ async function createEngine(options: HarnessOptions): Promise { const i = changedIndex(save); extraByComponent[i] += 1; fs.writeFileSync(project.componentPaths[i], componentSource(i, extraByComponent[i])); - // Global invalidation is the only surface these engines expose; without it the re-extraction - // below is a cache hit. + // Without this the re-extraction below is a cache hit, not a measurement. invalidate(); }, async reextract(save) { diff --git a/scripts/bench/docgen-perf/engines/vue-scenario.ts b/scripts/bench/docgen-perf/engines/vue-scenario.ts index 9a881b51a06d..e1d554c3211c 100644 --- a/scripts/bench/docgen-perf/engines/vue-scenario.ts +++ b/scripts/bench/docgen-perf/engines/vue-scenario.ts @@ -69,13 +69,7 @@ export interface VueHarnessOptions { export function parseVueOptions(argv: string[], engineDirName: string): VueHarnessOptions { return parseHarnessOptions(argv, OPTIONS, schemaFor(engineDirName), (values) => ({ - scenario: values.scenario, - packages: values.packages, - componentsPerPackage: values['components-per-package'], - chainDepth: values['chain-depth'], - fanOut: values['fan-out'], - heavyLib: values['heavy-lib'], - saves: values.saves, + ...values, outDir: values.out, jsonOut: values.json, })); diff --git a/scripts/bench/docgen-perf/generators/vue.ts b/scripts/bench/docgen-perf/generators/vue.ts index 002d038d0331..730e6bae6017 100644 --- a/scripts/bench/docgen-perf/generators/vue.ts +++ b/scripts/bench/docgen-perf/generators/vue.ts @@ -284,14 +284,7 @@ function parseOptions(argv: string[]): VueGenerateOptions { fanOut: countOption(4), heavyLib: z.boolean().default(false), }), - (values) => ({ - outDir: values.out, - packages: values.packages, - componentsPerPackage: values['components-per-package'], - chainDepth: values['chain-depth'], - fanOut: values['fan-out'], - heavyLib: values['heavy-lib'], - }) + (values) => ({ ...values, outDir: values.out }) ); } diff --git a/scripts/bench/docgen-perf/ratios.ts b/scripts/bench/docgen-perf/ratios.ts index 5b28862c928b..5b461eefe5de 100644 --- a/scripts/bench/docgen-perf/ratios.ts +++ b/scripts/bench/docgen-perf/ratios.ts @@ -3,10 +3,17 @@ * median, both measured in the same invocation. Ratios stand in for absolute milliseconds because * wall-clock on shared CI executors is too noisy to gate (PERF-METHODOLOGY.md, "Budget shape"). */ -import type { EngineId } from '../docgen-shared/engine-ids.ts'; -import type { EngineResult, RatioEntry, Ratios, ScenarioResult } from './types.ts'; +import type { + EngineId, + EngineResult, + LatencyMetric, + NotApplicable, + RatioEntry, + Ratios, + ScenarioResult, +} from './types.ts'; -export interface ControlPair { +interface ControlPair { /** Key under which this pair's ratios appear in the results. */ name: string; legacy: EngineId; @@ -38,38 +45,38 @@ export function engineOrderForRep(engines: EngineId[], rep: number): EngineId[] return order; } +/** Undefined unless both sides measured: dividing by a skipped or failed side is not a comparison. */ +function medianRatio(legacy: LatencyMetric | NotApplicable, next: LatencyMetric | NotApplicable) { + return legacy.status === 'measured' && next.status === 'measured' + ? legacy.median / next.median + : undefined; +} + +/** + * Undefined, not false, when an engine reports no member counts - see renderRatios for why that + * distinction matters. + */ +function likeForLike(legacy: ScenarioResult, next: ScenarioResult): boolean | undefined { + if (legacy.coldMembers === undefined || next.coldMembers === undefined) { + return undefined; + } + const warmAgrees = + legacy.warmMembers === undefined || + next.warmMembers === undefined || + legacy.warmMembers === next.warmMembers; + return legacy.coldMembers === next.coldMembers && warmAgrees; +} + function ratioFor(legacy: ScenarioResult, next: ScenarioResult): RatioEntry { - const entry: RatioEntry = { + return { + cold: medianRatio(legacy.metrics.coldExtractionMs, next.metrics.coldExtractionMs), + warm: medianRatio(legacy.metrics.warmExtractionMs, next.metrics.warmExtractionMs), legacyColdMembers: legacy.coldMembers, nextColdMembers: next.coldMembers, legacyWarmMembers: legacy.warmMembers, nextWarmMembers: next.warmMembers, + likeForLike: likeForLike(legacy, next), }; - - if ( - legacy.metrics.coldExtractionMs.status === 'measured' && - next.metrics.coldExtractionMs.status === 'measured' - ) { - entry.cold = legacy.metrics.coldExtractionMs.median / next.metrics.coldExtractionMs.median; - } - if ( - legacy.metrics.warmExtractionMs.status === 'measured' && - next.metrics.warmExtractionMs.status === 'measured' - ) { - entry.warm = legacy.metrics.warmExtractionMs.median / next.metrics.warmExtractionMs.median; - } - - // likeForLike is left undefined, not false, when an engine reports no member counts - see - // renderRatios for why that distinction matters. - if (legacy.coldMembers !== undefined && next.coldMembers !== undefined) { - const warmAgrees = - legacy.warmMembers === undefined || - next.warmMembers === undefined || - legacy.warmMembers === next.warmMembers; - entry.likeForLike = legacy.coldMembers === next.coldMembers && warmAgrees; - } - - return entry; } /** diff --git a/scripts/bench/docgen-perf/report.ts b/scripts/bench/docgen-perf/report.ts index 98712581e025..3a6773e960d9 100644 --- a/scripts/bench/docgen-perf/report.ts +++ b/scripts/bench/docgen-perf/report.ts @@ -57,10 +57,8 @@ export function renderResults( } } - const widths = HEADER.map((_, col) => Math.max(...rows.map((row) => (row[col] ?? '').length))); - const table = rows.map( - (row) => ` ${row.map((cell, col) => (cell ?? '').padEnd(widths[col])).join(' ')}` - ); + const widths = HEADER.map((_, col) => Math.max(...rows.map((row) => row[col].length))); + const table = rows.map((row) => ` ${row.map((cell, col) => cell.padEnd(widths[col])).join(' ')}`); return { table, statusLines }; } diff --git a/scripts/bench/docgen-perf/run.ts b/scripts/bench/docgen-perf/run.ts index dae3ae126552..b3b6520f431b 100644 --- a/scripts/bench/docgen-perf/run.ts +++ b/scripts/bench/docgen-perf/run.ts @@ -69,8 +69,8 @@ function assembleScenario( return { params: scenario.params, metrics, - coldMembers: engine.coldMembers?.(designated), - warmMembers: engine.warmMembers?.(designated), + coldMembers: engine.coldMembers(designated), + warmMembers: engine.warmMembers(designated), }; } @@ -89,7 +89,7 @@ async function main() { const skipped = new Map(); for (const engineId of options.engines) { - const reason = engineById(engineId).preflight?.(); + const reason = engineById(engineId).preflight(); if (reason) { skipped.set(engineId, reason); console.log(` ${engineId}: SKIPPED - ${reason}`); @@ -116,7 +116,8 @@ async function main() { const engineVersions: SuiteResults['engineVersions'] = {}; for (const engineId of options.engines) { - const version = engineById(engineId).version?.(); + const engine = engineById(engineId); + const version = engine.version(); if (version) { engineVersions[engineId] = version; } @@ -135,7 +136,7 @@ async function main() { } try { const scenarios: Record = {}; - for (const scenario of engineById(engineId).scenarios(profile)) { + for (const scenario of engine.scenarios(profile)) { scenarios[scenario.name] = assembleScenario(engineId, scenario, profile, store); } engineResults[engineId] = { status: 'measured', scenarios } satisfies EngineResult; diff --git a/scripts/bench/docgen-perf/spawn.ts b/scripts/bench/docgen-perf/spawn.ts index 4b948ae2a6e4..b9e376957005 100644 --- a/scripts/bench/docgen-perf/spawn.ts +++ b/scripts/bench/docgen-perf/spawn.ts @@ -8,6 +8,7 @@ import { spawnSync } from 'node:child_process'; import * as fs from 'node:fs'; import * as path from 'node:path'; +import { outputTail } from '../docgen-shared/child-output.ts'; import type { SeriesResult } from '../docgen-shared/series.ts'; export interface SeriesChildSpec { @@ -21,15 +22,6 @@ export interface SeriesChildSpec { jiti?: boolean; } -function tailOf(output: string): string { - return output - .trim() - .split('\n') - .slice(-4) - .map((line) => ` ${line}`) - .join('\n'); -} - /** * Run one series-harness child and parse its result JSON. A non-zero exit, or a missing or * unreadable result, is a failure - never a silently empty measurement. @@ -58,10 +50,10 @@ export function runSeriesChild( const output = `${proc.stdout ?? ''}${proc.stderr ?? ''}`; if (proc.status !== 0) { - throw new Error(`child exited with status ${proc.status}:\n${tailOf(output)}`); + throw new Error(`child exited with status ${proc.status}:\n${outputTail(output, 4)}`); } if (!fs.existsSync(jsonPath)) { - throw new Error(`child wrote no result JSON at ${jsonPath}:\n${tailOf(output)}`); + throw new Error(`child wrote no result JSON at ${jsonPath}:\n${outputTail(output, 4)}`); } return JSON.parse(fs.readFileSync(jsonPath, 'utf8')) as SeriesResult; } diff --git a/scripts/bench/docgen-shared/args.test.ts b/scripts/bench/docgen-shared/args.test.ts index a8605dff440a..bf139f0fee1d 100644 --- a/scripts/bench/docgen-shared/args.test.ts +++ b/scripts/bench/docgen-shared/args.test.ts @@ -31,7 +31,6 @@ const parse = (argv: string[]) => parseHarnessOptions(argv, OPTIONS, SCHEMA, (values) => ({ ...values, outDir: values.out, - componentsPerPackage: values['components-per-package'], })); describe('parseHarnessOptions', () => { diff --git a/scripts/bench/docgen-shared/args.ts b/scripts/bench/docgen-shared/args.ts index bcd548290d5e..8a67dc343142 100644 --- a/scripts/bench/docgen-shared/args.ts +++ b/scripts/bench/docgen-shared/args.ts @@ -16,9 +16,14 @@ type OptionsConfig = NonNullable; export const countOption = (fallback: number) => z.coerce.number().int().nonnegative().default(fallback); +/** parseArgs keys flags as written, so `--fan-out` arrives as `fan-out`; schema keys are camelCase. */ +function toCamelCase(flag: string): string { + return flag.replace(/-([a-z])/g, (_, letter: string) => letter.toUpperCase()); +} + /** - * Keys in `values` are the flags as written, so kebab-case flags arrive kebab-cased. `toInput` maps - * them onto the schema's shape, the way eval.ts spreads renamed keys into `safeParse`. + * Flags reach the schema camel-cased, so only genuine renames (`--out` onto `outDir`) need + * `toInput`, which spreads renamed keys into `safeParse` the way eval.ts does. * * `Out` is supplied by the caller rather than inferred: under this TypeScript setup zod 3 types a * `.default()` key as optional even on the parsed output, so inference would make every defaulted @@ -32,7 +37,10 @@ export function parseHarnessOptions( toInput?: (values: Record) => Record ): Out { const { values } = parseArgs({ args: argv, options, strict: true }); - const result = schema.safeParse(toInput ? toInput(values) : values); + const flags = Object.fromEntries( + Object.entries(values).map(([flag, value]) => [toCamelCase(flag), value]) + ); + const result = schema.safeParse(toInput ? toInput(flags) : flags); if (!result.success) { const issues = result.error.issues .map((issue) => ` --${issue.path.join('.')}: ${issue.message}`) diff --git a/scripts/bench/docgen-shared/child-output.ts b/scripts/bench/docgen-shared/child-output.ts new file mode 100644 index 000000000000..b2a437fa9a07 --- /dev/null +++ b/scripts/bench/docgen-shared/child-output.ts @@ -0,0 +1,12 @@ +/** + * The last `count` lines of a measured child's combined stdout and stderr, indented as a log block. + * Enough to see why a run failed without spilling a full V8 crash dump into the output. + */ +export function outputTail(output: string, count: number): string { + return output + .trim() + .split('\n') + .slice(-count) + .map((line) => ` ${line}`) + .join('\n'); +} diff --git a/scripts/bench/docgen-shared/renderer-module.ts b/scripts/bench/docgen-shared/renderer-module.ts index bd9896c858fe..4eade1b4137f 100644 --- a/scripts/bench/docgen-shared/renderer-module.ts +++ b/scripts/bench/docgen-shared/renderer-module.ts @@ -2,9 +2,10 @@ * Runtime access to the React renderer's docgen internals. * * The specifiers are built with `new URL` rather than written as static imports so the `scripts` - * typecheck does not pull `code/renderers` source into its program. The structural interfaces below - * are the minimum each harness touches; the real types live in - * `code/renderers/react/src/componentManifest`. + * typecheck does not pull `code/renderers` source into its program. The ref interfaces below are the + * minimum the harnesses pass across that boundary; the real types live in + * `code/renderers/react/src/componentManifest`. Each harness declares the shape of the module it + * loads next to its own call. */ const COMPONENT_MANIFEST = '../../../code/renderers/react/src/componentManifest/'; @@ -27,38 +28,3 @@ export interface StoryRefLike { storyPath: string; component: ComponentRefLike; } - -/** The component reference the generated project's `Comp{i}` corresponds to. */ -export function componentRef(i: number, componentPath: string): ComponentRefLike { - return { - componentName: `Comp${i}`, - importName: `Comp${i}`, - localImportName: `Comp${i}`, - importId: `./Comp${i}`, - isPackage: false, - path: componentPath, - }; -} - -export function buildStoryRefs(componentPaths: string[], storyPaths: string[]): StoryRefLike[] { - return componentPaths.map((componentPath, i) => ({ - storyPath: storyPaths[i], - component: componentRef(i, componentPath), - })); -} - -export interface ReactDocgenModule { - getReactDocgen( - path: string, - component: ComponentRefLike - ): { type: 'success' } | { type: 'error'; error: { name: string; message: string } }; -} - -export interface UtilsModule { - invalidateCache(): void; -} - -export interface ReactDocgenTypescriptModule { - parseWithReactDocgenTypescript(filePath: string): Promise>; - invalidateParser(): void; -} diff --git a/scripts/bench/docgen-shared/series.ts b/scripts/bench/docgen-shared/series.ts index 8eb35d24f59b..fd1149abba3b 100644 --- a/scripts/bench/docgen-shared/series.ts +++ b/scripts/bench/docgen-shared/series.ts @@ -9,7 +9,7 @@ import * as fs from 'node:fs'; import type { MemorySample, SaveSample } from './samples.ts'; import { formatSampleLine, gcAvailable, sampleMemory } from './sampling.ts'; -import { summarizeSeries } from './stats.ts'; +import { type SeriesSummary, summarizeSeries } from './stats.ts'; export interface SeriesEngine { /** @@ -25,16 +25,13 @@ export interface SeriesEngine { dispose?(): void; } -export interface SeriesResult { +export interface SeriesResult extends SeriesSummary { coldMs: number; coldMembers?: number; /** Members documented by the last timed re-extraction. */ warmMembers?: number; baseline: MemorySample; samples: SaveSample[]; - retainedSlope?: number; - retainedGrowth?: number; - avgTransient?: number; } export interface SeriesOptions { @@ -77,17 +74,7 @@ export async function runSeries( engine.dispose?.(); - const { retainedSlope, retainedGrowth, avgTransient } = summarizeSeries(samples, baseline); - return { - coldMs, - coldMembers, - warmMembers, - baseline, - samples, - retainedSlope, - retainedGrowth, - avgTransient, - }; + return { coldMs, coldMembers, warmMembers, baseline, samples, ...summarizeSeries(samples, baseline) }; } export function printSeriesSummary(result: SeriesResult, saves: number): void { diff --git a/scripts/bench/docgen-shared/stats.ts b/scripts/bench/docgen-shared/stats.ts index 26b0b2a7fb98..1229e38de4fb 100644 --- a/scripts/bench/docgen-shared/stats.ts +++ b/scripts/bench/docgen-shared/stats.ts @@ -38,8 +38,6 @@ export function leastSquaresSlope(values: number[]): number { /** The retained- and transient-memory figures every series harness derives from its save series. */ export interface SeriesSummary { - /** Post-GC retained heap per save. Empty unless the child ran under `--expose-gc`. */ - retainedValues: number[]; /** Least-squares slope of retained heap over the save series, MB per save. */ retainedSlope?: number; /** Final retained sample minus the pre-series baseline, MB. */ @@ -55,18 +53,19 @@ export interface SeriesSummary { * memory harness and the per-engine harnesses cannot drift apart in how they compute them. */ export function summarizeSeries(samples: SaveSample[], baseline: MemorySample): SeriesSummary { - const retainedValues = samples - .map((s) => s.retainedHeapMb) - .filter((v): v is number => v !== undefined); - const transients = samples - .map((s) => (s.retainedHeapMb !== undefined ? s.heapUsedMb - s.retainedHeapMb : undefined)) - .filter((v): v is number => v !== undefined); + // Samples taken without --expose-gc carry no retained heap; dropping them keeps a missing reading + // from being averaged in as a zero. + const gcSampled = samples.filter( + (s): s is SaveSample & { retainedHeapMb: number } => s.retainedHeapMb !== undefined + ); + const retained = gcSampled.map((s) => s.retainedHeapMb); + const transients = gcSampled.map((s) => s.heapUsedMb - s.retainedHeapMb); + const last = retained.at(-1); return { - retainedValues, - retainedSlope: retainedValues.length ? leastSquaresSlope(retainedValues) : undefined, + retainedSlope: retained.length ? leastSquaresSlope(retained) : undefined, retainedGrowth: - retainedValues.length && baseline.retainedHeapMb !== undefined - ? (retainedValues.at(-1) as number) - baseline.retainedHeapMb + last !== undefined && baseline.retainedHeapMb !== undefined + ? last - baseline.retainedHeapMb : undefined, transients, avgTransient: transients.length ? mean(transients) : undefined, From 3d97e647ef181c404443d4fe7096eed9d75f3e93 Mon Sep 17 00:00:00 2001 From: Valentin Palkovic Date: Tue, 28 Jul 2026 20:46:06 +0200 Subject: [PATCH 12/14] Bench: Restore the memory harness --max-retained-growth flag Camel-casing the flag names gave `maxRetainedGrowth`, but the schema key is `maxRetainedGrowthMb`, so the value never landed and Zod fell back to the 400MB default. parseArgs accepted the flag, nothing warned, and the harness compared against a threshold nobody asked for. It needs the same explicit rename as --out and --json. args.test.ts now carries a flag with the same shape, so removing the rename fails the suite instead of going quiet. Also finish the react-renderer-module rename, which left generate-project.ts importing a deleted file, and correct the SaveSample comment: saves count from 1, not 0. --- .../bench/docgen-memory/generate-project.ts | 2 +- scripts/bench/docgen-memory/memory-harness.ts | 6 ++-- .../bench/docgen-perf/engines/react-legacy.ts | 10 +++---- scripts/bench/docgen-shared/args.test.ts | 12 ++++++++ scripts/bench/docgen-shared/args.ts | 25 ++++------------ scripts/bench/docgen-shared/budgets.ts | 15 +--------- scripts/bench/docgen-shared/child-output.ts | 4 --- scripts/bench/docgen-shared/engine-ids.ts | 3 +- .../docgen-shared/react-renderer-module.ts | 20 +++++++++++++ .../bench/docgen-shared/renderer-module.ts | 30 ------------------- scripts/bench/docgen-shared/samples.ts | 6 ++-- scripts/bench/docgen-shared/series.ts | 2 +- 12 files changed, 55 insertions(+), 80 deletions(-) create mode 100644 scripts/bench/docgen-shared/react-renderer-module.ts delete mode 100644 scripts/bench/docgen-shared/renderer-module.ts diff --git a/scripts/bench/docgen-memory/generate-project.ts b/scripts/bench/docgen-memory/generate-project.ts index a6e476996af7..f1c95b696b64 100644 --- a/scripts/bench/docgen-memory/generate-project.ts +++ b/scripts/bench/docgen-memory/generate-project.ts @@ -22,7 +22,7 @@ import { pathToFileURL } from 'node:url'; import { z } from 'zod'; import { countOption, parseHarnessOptions } from '../docgen-shared/args.ts'; -import type { ComponentRefLike, StoryRefLike } from '../docgen-shared/renderer-module.ts'; +import type { ComponentRefLike, StoryRefLike } from '../docgen-shared/react-renderer-module.ts'; const require = createRequire(import.meta.url); diff --git a/scripts/bench/docgen-memory/memory-harness.ts b/scripts/bench/docgen-memory/memory-harness.ts index 0352c6331021..3950313e2416 100644 --- a/scripts/bench/docgen-memory/memory-harness.ts +++ b/scripts/bench/docgen-memory/memory-harness.ts @@ -39,7 +39,7 @@ import { z } from 'zod'; import { countOption, parseHarnessOptions } from '../docgen-shared/args.ts'; import { SANDBOX_DIRECTORY } from '../docgen-shared/paths.ts'; -import { type StoryRefLike, loadRendererModule } from '../docgen-shared/renderer-module.ts'; +import { type StoryRefLike, loadReactRendererModule } from '../docgen-shared/react-renderer-module.ts'; import { MB, gcAvailable } from '../docgen-shared/sampling.ts'; import { type SeriesEngine, @@ -140,6 +140,8 @@ function parseOptions(argv: string[]): HarnessOptions { forceGc: !values.noForceGc, outDir: values.out, jsonOut: values.json, + // The schema key carries the unit the flag name leaves off, so camel-casing alone misses it. + maxRetainedGrowthMb: values.maxRetainedGrowth, })); } @@ -148,7 +150,7 @@ function parseOptions(argv: string[]): HarnessOptions { * `code/renderers` source into its program. */ async function loadComponentMetaManager(): Promise { - const mod = await loadRendererModule<{ ComponentMetaManager: ComponentMetaManagerCtor }>( + const mod = await loadReactRendererModule<{ ComponentMetaManager: ComponentMetaManagerCtor }>( 'componentMeta/ComponentMetaManager.ts' ); return mod.ComponentMetaManager; diff --git a/scripts/bench/docgen-perf/engines/react-legacy.ts b/scripts/bench/docgen-perf/engines/react-legacy.ts index 7df062c05e8b..9a0415585c82 100644 --- a/scripts/bench/docgen-perf/engines/react-legacy.ts +++ b/scripts/bench/docgen-perf/engines/react-legacy.ts @@ -27,8 +27,8 @@ import { countOption, parseHarnessOptions } from '../../docgen-shared/args.ts'; import { SANDBOX_DIRECTORY } from '../../docgen-shared/paths.ts'; import { type ComponentRefLike, - loadRendererModule, -} from '../../docgen-shared/renderer-module.ts'; + loadReactRendererModule, +} from '../../docgen-shared/react-renderer-module.ts'; import { type SeriesEngine, harnessMain, @@ -112,10 +112,10 @@ interface Parser { async function loadParser(options: HarnessOptions, project: GeneratedProject): Promise { const { componentPaths } = project; - const { invalidateCache } = await loadRendererModule('utils.ts'); + const { invalidateCache } = await loadReactRendererModule('utils.ts'); if (options.parser === 'react-docgen') { - const { getReactDocgen } = await loadRendererModule('reactDocgen.ts'); + const { getReactDocgen } = await loadReactRendererModule('reactDocgen.ts'); return { async extractOne(i) { const result = getReactDocgen(componentPaths[i], componentRef(i, componentPaths[i])); @@ -130,7 +130,7 @@ async function loadParser(options: HarnessOptions, project: GeneratedProject): P } const { parseWithReactDocgenTypescript, invalidateParser } = - await loadRendererModule('reactDocgenTypescript.ts'); + await loadReactRendererModule('reactDocgenTypescript.ts'); // The parser resolves its tsconfig from process.cwd(); point it at the generated project. process.chdir(project.outDir); return { diff --git a/scripts/bench/docgen-shared/args.test.ts b/scripts/bench/docgen-shared/args.test.ts index bf139f0fee1d..21694d3779c9 100644 --- a/scripts/bench/docgen-shared/args.test.ts +++ b/scripts/bench/docgen-shared/args.test.ts @@ -9,6 +9,7 @@ const OPTIONS = { heavy: { type: 'boolean' }, out: { type: 'string' }, 'components-per-package': { type: 'string' }, + 'max-retained-growth': { type: 'string' }, } as const; const SCHEMA = z.object({ @@ -17,6 +18,7 @@ const SCHEMA = z.object({ heavy: z.boolean().default(false), outDir: z.string().default('/default'), componentsPerPackage: countOption(10), + maxRetainedGrowthMb: countOption(400), }); interface Options { @@ -25,12 +27,14 @@ interface Options { heavy: boolean; outDir: string; componentsPerPackage: number; + maxRetainedGrowthMb: number; } const parse = (argv: string[]) => parseHarnessOptions(argv, OPTIONS, SCHEMA, (values) => ({ ...values, outDir: values.out, + maxRetainedGrowthMb: values.maxRetainedGrowth, })); describe('parseHarnessOptions', () => { @@ -41,6 +45,7 @@ describe('parseHarnessOptions', () => { heavy: false, outDir: '/default', componentsPerPackage: 10, + maxRetainedGrowthMb: 400, }); }); @@ -52,6 +57,13 @@ describe('parseHarnessOptions', () => { expect(parse(['--components-per-package', '5']).componentsPerPackage).toBe(5); }); + it('carries a flag whose schema key is not its camelCase', () => { + // `--max-retained-growth` camel-cases to `maxRetainedGrowth`, but the schema names the unit. + // Without the rename in `toInput` the flag is accepted and then silently ignored, which is how + // the memory harness lost its threshold flag once already. + expect(parse(['--max-retained-growth', '50']).maxRetainedGrowthMb).toBe(50); + }); + it('reads boolean flags', () => { expect(parse(['--heavy']).heavy).toBe(true); }); diff --git a/scripts/bench/docgen-shared/args.ts b/scripts/bench/docgen-shared/args.ts index 8a67dc343142..ad3bae5bd831 100644 --- a/scripts/bench/docgen-shared/args.ts +++ b/scripts/bench/docgen-shared/args.ts @@ -1,35 +1,22 @@ -/** - * Option parsing for the docgen bench harnesses: `node:util` parseArgs in strict mode for the - * parse, then a Zod schema for coercion and validation. Same pairing as scripts/eval/eval.ts. - * - * Strict mode is what makes this worth doing. It rejects a flag the harness does not declare, so - * `--component 5` with the `s` missing fails loudly instead of leaving the default in place and - * reporting a benchmark for a project size nobody asked for. - */ import { type ParseArgsConfig, parseArgs } from 'node:util'; import { z } from 'zod'; type OptionsConfig = NonNullable; -/** parseArgs yields strings, so numeric flags are coerced here rather than at each call site. */ +/** parseArgs yields strings, so numeric flags are coerced here */ export const countOption = (fallback: number) => z.coerce.number().int().nonnegative().default(fallback); -/** parseArgs keys flags as written, so `--fan-out` arrives as `fan-out`; schema keys are camelCase. */ +/** + * parseArgs keys flags as written, so `--fan-out` arrives as `fan-out`; schema keys are camelCase. + * A schema key that is not exactly its flag's camelCase — `maxRetainedGrowthMb` for + * `--max-retained-growth` — must still be renamed in `toInput`, or Zod silently uses its default. + */ function toCamelCase(flag: string): string { return flag.replace(/-([a-z])/g, (_, letter: string) => letter.toUpperCase()); } -/** - * Flags reach the schema camel-cased, so only genuine renames (`--out` onto `outDir`) need - * `toInput`, which spreads renamed keys into `safeParse` the way eval.ts does. - * - * `Out` is supplied by the caller rather than inferred: under this TypeScript setup zod 3 types a - * `.default()` key as optional even on the parsed output, so inference would make every defaulted - * option optional at the call site. The harness interfaces stay the source of truth, and - * args.test.ts asserts that a parse with no arguments really does populate every field. - */ export function parseHarnessOptions( argv: string[], options: OptionsConfig, diff --git a/scripts/bench/docgen-shared/budgets.ts b/scripts/bench/docgen-shared/budgets.ts index a889f2de6a5d..7526d68307a6 100644 --- a/scripts/bench/docgen-shared/budgets.ts +++ b/scripts/bench/docgen-shared/budgets.ts @@ -1,8 +1,3 @@ -/** - * Memory budget values for the engine gates. Budgets are derived per engine from that engine's own - * baseline runs and are never ported between engines; missing engines get their rows when their - * baselines are recorded. - */ import type { EngineId } from './engine-ids.ts'; export interface MemoryBudgets { @@ -14,18 +9,10 @@ export interface MemoryBudgets { maxRetainedSlopeMb: number; } -/** - * Budgets sit well above observed values so the gate is not flaky, while still failing hard on a - * real regression. - */ -export const MEMORY_BUDGETS: Partial> = { +const MEMORY_BUDGETS: Partial> = { 'react-osa': { maxRetainedGrowthMb: 60, maxTransientMb: 90, maxRetainedSlopeMb: 3 }, }; -/** - * Budgets for `engine`, or a hard failure. A gate that reads a missing row would assert nothing and - * still pass, so an engine without recorded budgets must not reach a gate config at all. - */ export function memoryBudgetsFor(engine: EngineId): MemoryBudgets { const budgets = MEMORY_BUDGETS[engine]; if (!budgets) { diff --git a/scripts/bench/docgen-shared/child-output.ts b/scripts/bench/docgen-shared/child-output.ts index b2a437fa9a07..c9778ba246ea 100644 --- a/scripts/bench/docgen-shared/child-output.ts +++ b/scripts/bench/docgen-shared/child-output.ts @@ -1,7 +1,3 @@ -/** - * The last `count` lines of a measured child's combined stdout and stderr, indented as a log block. - * Enough to see why a run failed without spilling a full V8 crash dump into the output. - */ export function outputTail(output: string, count: number): string { return output .trim() diff --git a/scripts/bench/docgen-shared/engine-ids.ts b/scripts/bench/docgen-shared/engine-ids.ts index bd6be5b43aa4..49934a8fa1e7 100644 --- a/scripts/bench/docgen-shared/engine-ids.ts +++ b/scripts/bench/docgen-shared/engine-ids.ts @@ -1,6 +1,5 @@ /** - * The docgen engines under measurement. Lives here rather than with the perf suite's result types - * because the memory gate keys its budgets off the same ids. + * The docgen engines under measurement. */ export type EngineId = | 'react-legacy' diff --git a/scripts/bench/docgen-shared/react-renderer-module.ts b/scripts/bench/docgen-shared/react-renderer-module.ts new file mode 100644 index 000000000000..84b902192e67 --- /dev/null +++ b/scripts/bench/docgen-shared/react-renderer-module.ts @@ -0,0 +1,20 @@ +const COMPONENT_MANIFEST = '../../../code/renderers/react/src/componentManifest/'; + +export async function loadReactRendererModule(relativePath: string): Promise { + const url = new URL(`${COMPONENT_MANIFEST}${relativePath}`, import.meta.url).href; + return (await import(url)) as T; +} + +export interface ComponentRefLike { + componentName: string; + importName: string; + localImportName: string; + importId: string; + isPackage: boolean; + path: string; +} + +export interface StoryRefLike { + storyPath: string; + component: ComponentRefLike; +} diff --git a/scripts/bench/docgen-shared/renderer-module.ts b/scripts/bench/docgen-shared/renderer-module.ts deleted file mode 100644 index 4eade1b4137f..000000000000 --- a/scripts/bench/docgen-shared/renderer-module.ts +++ /dev/null @@ -1,30 +0,0 @@ -/** - * Runtime access to the React renderer's docgen internals. - * - * The specifiers are built with `new URL` rather than written as static imports so the `scripts` - * typecheck does not pull `code/renderers` source into its program. The ref interfaces below are the - * minimum the harnesses pass across that boundary; the real types live in - * `code/renderers/react/src/componentManifest`. Each harness declares the shape of the module it - * loads next to its own call. - */ - -const COMPONENT_MANIFEST = '../../../code/renderers/react/src/componentManifest/'; - -export async function loadRendererModule(relativePath: string): Promise { - const url = new URL(`${COMPONENT_MANIFEST}${relativePath}`, import.meta.url).href; - return (await import(url)) as T; -} - -export interface ComponentRefLike { - componentName: string; - importName: string; - localImportName: string; - importId: string; - isPackage: boolean; - path: string; -} - -export interface StoryRefLike { - storyPath: string; - component: ComponentRefLike; -} diff --git a/scripts/bench/docgen-shared/samples.ts b/scripts/bench/docgen-shared/samples.ts index 4f630a01fab8..84cbe5a8254e 100644 --- a/scripts/bench/docgen-shared/samples.ts +++ b/scripts/bench/docgen-shared/samples.ts @@ -1,13 +1,15 @@ -/** The per-sample shapes every docgen bench harness records. */ - export interface MemorySample { + /** Resident Set Size, the total memory allocated for the process. */ rssMb: number; + /** The V8 heap used by the process. */ heapUsedMb: number; /** Post-GC heap. Present only when the child runs under `node --expose-gc`. */ retainedHeapMb?: number; } export interface SaveSample extends MemorySample { + /** The save number, counting from 1. */ save: number; + /** The duration of the save's re-extraction, in milliseconds. */ durMs: number; } diff --git a/scripts/bench/docgen-shared/series.ts b/scripts/bench/docgen-shared/series.ts index fd1149abba3b..4a5733f40c99 100644 --- a/scripts/bench/docgen-shared/series.ts +++ b/scripts/bench/docgen-shared/series.ts @@ -1,5 +1,5 @@ /** - * The save-series harness every docgen engine child runs, per scripts/bench/PERF-METHODOLOGY.md: + * The save-series harness every docgen engine child runs: * one timed cold pass, then K simulated saves that mutate the project on disk, invalidate the * engine's caches, and re-extract, with memory sampled around forced GC after every save. Engines * differ only in what those steps mean, so they implement {@link SeriesEngine} and this module owns From 90b45f041e5081a685949c7780ec1d7153f6ed49 Mon Sep 17 00:00:00 2001 From: Valentin Palkovic Date: Tue, 28 Jul 2026 20:53:11 +0200 Subject: [PATCH 13/14] Bench: Resolve compodoc through require.resolve The engine guessed at two node_modules/.bin paths by counting `../` from its own directory, then fell back to `which`. The version came from a separate walk up to five directories from the resolved binary. One require.resolve of the package.json replaces all of it and hands back the version for free, matching how the React and Vue generators already find their packages. Spawning `node ` rather than the .bin shim drops the dependency on the exec bit, and gives Windows a path that works at all: the shim is named compodoc.cmd there, so neither candidate matched and `which` does not exist. Dropping the PATH fallback is deliberate. @compodoc/compodoc is pinned in scripts/package.json so the numbers describe one known version, and a global compodoc of some other version would move them without anyone deciding to. Missing package now means skipped, with a reason that names the fix. --- scripts/bench/docgen-perf/engines/compodoc.ts | 97 +++++++------------ 1 file changed, 34 insertions(+), 63 deletions(-) diff --git a/scripts/bench/docgen-perf/engines/compodoc.ts b/scripts/bench/docgen-perf/engines/compodoc.ts index 792f236d0e3c..8a730912c372 100644 --- a/scripts/bench/docgen-perf/engines/compodoc.ts +++ b/scripts/bench/docgen-perf/engines/compodoc.ts @@ -1,15 +1,6 @@ -/** - * Adapter for the Compodoc engine. Compodoc is a one-shot CLI - a fresh process per run - so this - * adapter spawns the compodoc CLI itself as the measured child: cold extraction and whole-project - * scan are the same full-project run, warm extraction is a second full run after touching one - * component file, and peak memory is the child's peak RSS sampled from outside the process. - * - * `@compodoc/compodoc` is pinned exactly in `scripts/package.json`, matching the version the Angular - * docgen baselines capture against (`code/lib/docgen-harness/README.md`) - a caret range would let - * the numbers drift across compodoc versions without anyone deciding to. - */ import { spawn, spawnSync } from 'node:child_process'; import * as fs from 'node:fs'; +import { createRequire } from 'node:module'; import * as path from 'node:path'; import { outputTail } from '../../docgen-shared/child-output.ts'; @@ -19,43 +10,25 @@ import { BenchEngine, type MeasureContext, type ScenarioSpec } from '../engine.t import { angularComponentSource, generateAngularProject } from '../generators/angular.ts'; import type { EngineId, EngineMetrics } from '../types.ts'; -/** - * Walked up from the resolved binary rather than a fixed path, since the package hoists to - * different node_modules depending on the install. - */ -export function compodocVersion(binary: string): string | undefined { - try { - let dir = path.dirname(fs.realpathSync(binary)); - for (let up = 0; up < 5; up++) { - const candidate = path.join(dir, 'package.json'); - if (fs.existsSync(candidate)) { - const pkg = JSON.parse(fs.readFileSync(candidate, 'utf8')); - if (pkg.name === '@compodoc/compodoc') { - return pkg.version as string; - } - } - dir = path.dirname(dir); - } - } catch { - // A binary on PATH may not sit inside a package at all; an unknown version is not a failure. - } - return undefined; +const require = createRequire(import.meta.url); + +interface ResolvedCompodoc { + /** The CLI entry point, run as `node ` so no shell shim or exec bit is involved. */ + cli: string; + version: string; } -/** Locations probed for a compodoc binary, in order: workspace .bin dirs, then PATH. */ -export function resolveCompodocBinary(): string | undefined { - const candidates = [ - path.resolve(import.meta.dirname, '../../../node_modules/.bin/compodoc'), - path.resolve(import.meta.dirname, '../../../../node_modules/.bin/compodoc'), - ]; - for (const candidate of candidates) { - if (fs.existsSync(candidate)) { - return candidate; - } +function resolveCompodoc(): ResolvedCompodoc | undefined { + try { + const packagePath = require.resolve('@compodoc/compodoc/package.json'); + const pkg = JSON.parse(fs.readFileSync(packagePath, 'utf8')); + return { + cli: path.join(path.dirname(packagePath), pkg.bin.compodoc), + version: pkg.version, + }; + } catch { + return undefined; } - const which = spawnSync('which', ['compodoc'], { encoding: 'utf8' }); - const fromPath = which.status === 0 ? which.stdout.trim() : ''; - return fromPath || undefined; } interface CompodocRun { @@ -63,19 +36,18 @@ interface CompodocRun { peakRssMb: number; } -/** Run compodoc once over the project, polling the child's RSS from outside. */ function runCompodocOnce( - binary: string, + cli: string, projectDir: string, docsOutDir: string, pollIntervalMs: number ): Promise { fs.rmSync(docsOutDir, { recursive: true, force: true }); - const args = ['-p', 'tsconfig.json', '-e', 'json', '-d', docsOutDir, '--silent']; + const args = [cli, '-p', 'tsconfig.json', '-e', 'json', '-d', docsOutDir, '--silent']; return new Promise((resolve, reject) => { const start = Date.now(); - const child = spawn(binary, args, { cwd: projectDir }); + const child = spawn(process.execPath, args, { cwd: projectDir }); let peakRssKb = 0; let output = ''; child.stdout.on('data', (chunk: Buffer) => (output += chunk.toString())); @@ -85,8 +57,7 @@ function runCompodocOnce( if (child.pid === undefined) { return; } - // `ps` is absent on Windows, and it exits non-zero once the child is gone. Either way this - // throws from a timer callback if it is not guarded, which no per-engine catch can reach. + const ps = spawnSync('ps', ['-o', 'rss=', '-p', String(child.pid)], { encoding: 'utf8' }); if (ps.error || typeof ps.stdout !== 'string') { return; @@ -123,7 +94,7 @@ function runCompodocOnce( * fresh compodoc processes, matching the one-sample-per-fresh-process topology. */ export async function runCompodocRepetition( - binary: string, + cli: string, scenario: AngularScenarioConfig, workDir: string, pollIntervalMs: number @@ -136,11 +107,11 @@ export async function runCompodocRepetition( props: scenario.props, }); - const cold = await runCompodocOnce(binary, project.outDir, docsOutDir, pollIntervalMs); + const cold = await runCompodocOnce(cli, project.outDir, docsOutDir, pollIntervalMs); // Touch one component so the warm run sees a genuinely changed file. fs.writeFileSync(project.componentPaths[0], angularComponentSource(0, scenario.props + 1)); - const warm = await runCompodocOnce(binary, project.outDir, docsOutDir, pollIntervalMs); + const warm = await runCompodocOnce(cli, project.outDir, docsOutDir, pollIntervalMs); return { coldMs: cold.durMs, @@ -153,35 +124,35 @@ export async function runCompodocRepetition( * Compodoc as an engine. Unlike the series engines it has no child harness to spawn: the CLI is the * measured process, and this class drives it directly. * - * It is also the only engine with state - the resolved binary, found once in {@link preflight} and - * reused for every repetition and for the recorded version. + * It is also the only engine with state - the resolved CLI and version, looked up once in + * {@link preflight} and reused for every repetition. */ export class CompodocEngine extends BenchEngine { readonly id: EngineId = 'compodoc'; - #binary: string | undefined; + #resolved: ResolvedCompodoc | undefined; scenarios(profile: SuiteProfile): ScenarioSpec[] { return [{ name: 'default', params: { ...profile.angular } }]; } preflight(): string | undefined { - this.#binary = resolveCompodocBinary(); - return this.#binary + this.#resolved = resolveCompodoc(); + return this.#resolved ? undefined - : 'no compodoc binary found (workspace node_modules/.bin or PATH); @compodoc/compodoc is pinned in scripts/package.json, so run yarn install'; + : '@compodoc/compodoc did not resolve; it is pinned in scripts/package.json, so run yarn install'; } version(): string | undefined { - return this.#binary ? compodocVersion(this.#binary) : undefined; + return this.#resolved?.version; } async measure(ctx: MeasureContext, scenario: ScenarioSpec): Promise { - if (!this.#binary) { - throw new Error('compodoc binary unresolved; preflight must run first'); + if (!this.#resolved) { + throw new Error('compodoc unresolved; preflight must run first'); } return runCompodocRepetition( - this.#binary, + this.#resolved.cli, { components: Number(scenario.params.components), props: Number(scenario.params.props) }, ctx.scenarioDir, ctx.rssPollIntervalMs From 4391ae9b12a64e58431ceeebcf9700418130f0fc Mon Sep 17 00:00:00 2001 From: Valentin Palkovic Date: Tue, 28 Jul 2026 21:33:14 +0200 Subject: [PATCH 14/14] Bench: Record what compodoc documented, and how much of it it resolved The Angular row reported timings and nothing else, so there was no way to read them. Compodoc writes documentation.json on every run and the engine never opened it. It now reports documented members, counting a component's own four member arrays across components and directives. Classes are left out on purpose: compodoc copies an ancestor's members into every descendant, so counting the base class again would double every inherited member. A member count alone would still mislead here. Compodoc never resolves a named type, so `@Input() kind: ButtonKind` is stored as the name and an `@Output()` is stored as `EventEmitter` with the payload dropped, while the same union written inline is stored in full. Chain a type through twenty files and it documents every member at the same speed, having looked through none of them. So it also reports how many documented members carry a type it never resolved, which is what keeps a future Angular pair from reading equal counts as equal work. Measured on the quick profile: 90 members cold, 91 warm, 10 of them opaque - one EventEmitter per component. The methodology note gains that reasoning, plus the signals finding: compodoc matches input() and output() against the initializer's source text, so it documents them at the same speed and count as decorators. The cost is fidelity, not time, so it belongs in the harness fixtures rather than a bench scenario. --- scripts/bench/PERF-METHODOLOGY.md | 12 +++ scripts/bench/docgen-perf/aggregate.ts | 4 + scripts/bench/docgen-perf/engine.ts | 9 +++ .../docgen-perf/engines/compodoc-doc.test.ts | 53 +++++++++++++ .../bench/docgen-perf/engines/compodoc-doc.ts | 76 +++++++++++++++++++ scripts/bench/docgen-perf/engines/compodoc.ts | 34 ++++++--- scripts/bench/docgen-perf/run.ts | 1 + scripts/bench/docgen-perf/types.ts | 6 ++ 8 files changed, 185 insertions(+), 10 deletions(-) create mode 100644 scripts/bench/docgen-perf/engines/compodoc-doc.test.ts create mode 100644 scripts/bench/docgen-perf/engines/compodoc-doc.ts diff --git a/scripts/bench/PERF-METHODOLOGY.md b/scripts/bench/PERF-METHODOLOGY.md index ba00d6f7ebf8..f2a90375b4cd 100644 --- a/scripts/bench/PERF-METHODOLOGY.md +++ b/scripts/bench/PERF-METHODOLOGY.md @@ -99,6 +99,18 @@ Every run so far has the legacy engine finishing its cold pass an order of magni The engines are not doing the same work, so the Vue ratio measures resolution depth as much as speed. Both engines therefore report their documented-member counts, and the suite prints those counts beside every ratio - the warm one as much as the cold one - and marks the pair not like-for-like whenever they disagree. A ratio marked not like-for-like must not become a budget. + +Angular needs a second signal, because equal member counts there would not mean equal work. +Compodoc never resolves a named type: it records the name written at the property, so `@Input() kind: ButtonKind` is stored as `ButtonKind` and an `@Output()` is stored as `EventEmitter` with its payload dropped, while the same union written inline is stored in full. +Chain a type through twenty files and it still documents every member, at the same speed, having looked through none of them - so a member count alone would read as identical to an engine that expanded all of them. +Compodoc therefore also reports how many of its documented members carry a type it never resolved, and a future Angular pair must compare that alongside the counts before calling a ratio like-for-like. +Its warm member count is a second whole-project total rather than the one member re-extracted, so it is not comparable with a series engine's warm count either. + +Signals are documented, not benchmarked. +Compodoc reads `input()`, `output()` and `model()` by matching the initializer's source text, not by resolving the import or the type, and it documents them at the same speed and count as the decorator form - so there is no timing to measure. +What the textual match costs is fidelity: an aliased `import { input as ngInput }` stops being recognised as an input at all, and a non-literal default without an explicit generic loses its type. +Those belong in the docgen-harness fixtures that pin compodoc's behaviour, not in a bench scenario that would report two rows indistinguishable from the baseline. + The baseline work records the figures; quoting any here would pin numbers taken at one profile on one machine. Engines without a second implementation in the same job get their timing reference picked when their baselines are recorded; until then their timing budgets stay placeholders. Memory budgets stay absolute megabytes with generous headroom: budgets sit well above observed values so the gate is not flaky, while still failing hard on a real regression. diff --git a/scripts/bench/docgen-perf/aggregate.ts b/scripts/bench/docgen-perf/aggregate.ts index 0f69cabf6f85..10762817ce50 100644 --- a/scripts/bench/docgen-perf/aggregate.ts +++ b/scripts/bench/docgen-perf/aggregate.ts @@ -61,6 +61,10 @@ export interface OneShotRepetition { coldMs: number; warmMs: number; peakRssMb: number; + coldMembers?: number; + warmMembers?: number; + /** Of the cold pass's members, how many carry a type the engine never resolved. */ + coldOpaqueTypes?: number; } /** diff --git a/scripts/bench/docgen-perf/engine.ts b/scripts/bench/docgen-perf/engine.ts index 398a7afb0ec6..d41d8e46fe22 100644 --- a/scripts/bench/docgen-perf/engine.ts +++ b/scripts/bench/docgen-perf/engine.ts @@ -59,6 +59,15 @@ export abstract class BenchEngine { warmMembers(_sample: Sample): number | undefined { return undefined; } + + /** + * Of the cold pass's members, how many the engine documented under a type name it never resolved. + * Two engines can agree on {@link coldMembers} and still have done entirely different work, so a + * member count alone does not establish like-for-like; this is what separates them. + */ + coldOpaqueTypes(_sample: Sample): number | undefined { + return undefined; + } } export interface SeriesChildConfig { diff --git a/scripts/bench/docgen-perf/engines/compodoc-doc.test.ts b/scripts/bench/docgen-perf/engines/compodoc-doc.test.ts new file mode 100644 index 000000000000..4397481b7846 --- /dev/null +++ b/scripts/bench/docgen-perf/engines/compodoc-doc.test.ts @@ -0,0 +1,53 @@ +import * as path from 'node:path'; + +import { describe, expect, it } from 'vitest'; + +import { countDocumentation } from './compodoc-doc.ts'; + +/** Real compodoc output, committed as the Angular docgen baselines. */ +const fixture = (name: string) => + countDocumentation( + path.join( + import.meta.dirname, + '../../../../code/lib/docgen-harness/src/angular/__testfixtures__', + name, + 'compodoc-input.json' + ) + ); + +describe('countDocumentation', () => { + it('counts an inherited member once, not once per class that lists it', () => { + // The component declares `heading` and `dismissed` and inherits `dismissible`, which compodoc + // also lists separately under `classes`. Counting `classes` too would report it twice. + expect(fixture('cross-file-inheritance').members).toBe(3); + }); + + it('counts signal inputs and outputs like decorator ones', () => { + expect(fixture('signal-io').members).toBe(6); + }); + + describe('opaque types', () => { + it('separates a union written inline from the same union behind an alias', () => { + // This component's three inputs are `ButtonKind`, `"small" | "large"` and `ToneOption`. The + // inline one describes itself; the two aliases are names compodoc never looked through, and + // an engine that resolved them would document the same three members off far more work. + const counts = fixture('decorator-union-enum'); + expect(counts.members).toBe(3); + expect(counts.opaqueTypes).toBe(2); + }); + + it('counts an unresolved generic parameter', () => { + expect(fixture('decorator-generic')).toEqual({ members: 2, opaqueTypes: 2 }); + }); + + it('counts an output whose payload type was dropped', () => { + // compodoc records `EventEmitter`, losing the emitted shape. + expect(fixture('decorator-io-basics')).toEqual({ members: 5, opaqueTypes: 1 }); + }); + + it('reports none when every member describes itself', () => { + expect(fixture('properties-methods-noise').opaqueTypes).toBe(0); + expect(fixture('jsdoc-tags').opaqueTypes).toBe(0); + }); + }); +}); diff --git a/scripts/bench/docgen-perf/engines/compodoc-doc.ts b/scripts/bench/docgen-perf/engines/compodoc-doc.ts new file mode 100644 index 000000000000..ae0c7e890e42 --- /dev/null +++ b/scripts/bench/docgen-perf/engines/compodoc-doc.ts @@ -0,0 +1,76 @@ +/** + * Reading compodoc's `documentation.json`. + * + * Compodoc records how much it documented and how much of it it actually resolved, which is what + * makes its timings readable: it never expands a named type, so a chain of aliases costs it nothing + * and it still reports every member. Counting members alone would let it look equal to an engine + * that did far more work. + */ +import * as fs from 'node:fs'; + +interface MemberEntry { + type?: string; + subProperties?: unknown[]; +} + +interface MemberHolder { + inputsClass?: MemberEntry[]; + outputsClass?: MemberEntry[]; + propertiesClass?: MemberEntry[]; + methodsClass?: MemberEntry[]; +} + +interface Documentation { + components?: MemberHolder[]; + directives?: MemberHolder[]; +} + +const MEMBER_ARRAYS = ['inputsClass', 'outputsClass', 'propertiesClass', 'methodsClass'] as const; + +/** + * `classes` is deliberately left out. Compodoc copies an ancestor's members into every descendant's + * own arrays, so a base class documented under `classes` would be counted a second time. This also + * matches what Storybook reads, which is a component's own four arrays. + */ +function documentedMembers(doc: Documentation): MemberEntry[] { + return [...(doc.components ?? []), ...(doc.directives ?? [])].flatMap((holder) => + MEMBER_ARRAYS.flatMap((key) => holder[key] ?? []) + ); +} + +/** + * Names that describe themselves, so recording one is not a resolution an engine skipped. Keyword + * spellings are lowercase as compodoc emits them (`function`, not `Function`); both are listed + * because the two spellings reach the same member field. + */ +const RESOLVED_TYPES = new Set([ + 'string', 'number', 'boolean', 'any', 'unknown', 'void', 'never', 'null', 'undefined', + 'object', 'symbol', 'bigint', 'function', 'Date', 'Function', 'Array', 'Object', +]); + +/** + * A member whose type is a bare name the engine never looked through — `Hop19Shape`, not + * `{ x: string }`. Inline object literals, primitives and literal unions are all self-describing, + * so only a lone identifier counts. + * + * This is a floor: wrapped forms such as `Partial` are equally unresolved but do not match, + * so the true share of unresolved types is at least what this reports. + */ +function isOpaque(entry: MemberEntry): boolean { + if (entry.subProperties?.length || !entry.type) { + return false; + } + const named = entry.type.replace(/\[\]$/, '').trim(); + return /^[A-Za-z_$][\w$]*$/.test(named) && !RESOLVED_TYPES.has(named); +} + +export interface DocumentationCounts { + members: number; + opaqueTypes: number; +} + +export function countDocumentation(documentationJsonPath: string): DocumentationCounts { + const doc = JSON.parse(fs.readFileSync(documentationJsonPath, 'utf8')) as Documentation; + const members = documentedMembers(doc); + return { members: members.length, opaqueTypes: members.filter(isOpaque).length }; +} diff --git a/scripts/bench/docgen-perf/engines/compodoc.ts b/scripts/bench/docgen-perf/engines/compodoc.ts index 8a730912c372..4f9d681289f4 100644 --- a/scripts/bench/docgen-perf/engines/compodoc.ts +++ b/scripts/bench/docgen-perf/engines/compodoc.ts @@ -5,6 +5,7 @@ import * as path from 'node:path'; import { outputTail } from '../../docgen-shared/child-output.ts'; import { type OneShotRepetition, oneShotMetrics } from '../aggregate.ts'; +import { type DocumentationCounts, countDocumentation } from './compodoc-doc.ts'; import type { AngularScenarioConfig, SuiteProfile } from '../config.ts'; import { BenchEngine, type MeasureContext, type ScenarioSpec } from '../engine.ts'; import { angularComponentSource, generateAngularProject } from '../generators/angular.ts'; @@ -31,7 +32,7 @@ function resolveCompodoc(): ResolvedCompodoc | undefined { } } -interface CompodocRun { +interface CompodocRun extends DocumentationCounts { durMs: number; peakRssMb: number; } @@ -79,12 +80,15 @@ function runCompodocOnce( reject(new Error(`compodoc exited with status ${status}:\n${outputTail(output, 8)}`)); return; } - if (!fs.existsSync(path.join(docsOutDir, 'documentation.json'))) { + const documentationJson = path.join(docsOutDir, 'documentation.json'); + if (!fs.existsSync(documentationJson)) { reject(new Error('compodoc run produced no documentation.json')); return; } + // Counted before the next run overwrites the file, which both runs share. + const counts = countDocumentation(documentationJson); // The polled peak misses spikes shorter than the interval; the recorded value is a floor. - resolve({ durMs, peakRssMb: peakRssKb / 1024 }); + resolve({ durMs, peakRssMb: peakRssKb / 1024, ...counts }); }); }); } @@ -117,16 +121,14 @@ export async function runCompodocRepetition( coldMs: cold.durMs, warmMs: warm.durMs, peakRssMb: Math.max(cold.peakRssMb, warm.peakRssMb), + coldMembers: cold.members, + // A second whole-project pass, so this counts the project, not the one file that changed. It is + // not comparable with a series engine's warm count, which covers only the re-extracted member. + warmMembers: warm.members, + coldOpaqueTypes: cold.opaqueTypes, }; } -/** - * Compodoc as an engine. Unlike the series engines it has no child harness to spawn: the CLI is the - * measured process, and this class drives it directly. - * - * It is also the only engine with state - the resolved CLI and version, looked up once in - * {@link preflight} and reused for every repetition. - */ export class CompodocEngine extends BenchEngine { readonly id: EngineId = 'compodoc'; @@ -162,4 +164,16 @@ export class CompodocEngine extends BenchEngine { aggregate(samples: OneShotRepetition[], expectedN: number): EngineMetrics { return oneShotMetrics(samples, expectedN); } + + coldMembers(sample: OneShotRepetition): number | undefined { + return sample.coldMembers; + } + + warmMembers(sample: OneShotRepetition): number | undefined { + return sample.warmMembers; + } + + coldOpaqueTypes(sample: OneShotRepetition): number | undefined { + return sample.coldOpaqueTypes; + } } diff --git a/scripts/bench/docgen-perf/run.ts b/scripts/bench/docgen-perf/run.ts index b3b6520f431b..6366f4e97653 100644 --- a/scripts/bench/docgen-perf/run.ts +++ b/scripts/bench/docgen-perf/run.ts @@ -71,6 +71,7 @@ function assembleScenario( metrics, coldMembers: engine.coldMembers(designated), warmMembers: engine.warmMembers(designated), + coldOpaqueTypes: engine.coldOpaqueTypes(designated), }; } diff --git a/scripts/bench/docgen-perf/types.ts b/scripts/bench/docgen-perf/types.ts index 7ce13e7b5ed6..a8a7bf6366dc 100644 --- a/scripts/bench/docgen-perf/types.ts +++ b/scripts/bench/docgen-perf/types.ts @@ -53,6 +53,12 @@ export interface ScenarioResult { coldMembers?: number; /** Members the timed re-extraction documented. Warm ratios need this for the same reason. */ warmMembers?: number; + /** + * Of {@link coldMembers}, how many the engine documented under a type name it never resolved. + * An engine that prints `Hop19Shape` and one that expands it into its fields report the same + * member count off very different work, so equal counts alone do not make a ratio like-for-like. + */ + coldOpaqueTypes?: number; } export type EngineResult =