diff --git a/scripts/bench/PERF-METHODOLOGY.md b/scripts/bench/PERF-METHODOLOGY.md new file mode 100644 index 000000000000..f2a90375b4cd --- /dev/null +++ b/scripts/bench/PERF-METHODOLOGY.md @@ -0,0 +1,164 @@ +# Docgen performance methodology + +This note is the measurement contract for the per-engine docgen performance suite. +It fixes the metric set, the determinism method, the budget shape, and the CI tiers that the harness and its budgets implement. +The suite measures; it never optimizes. + +## Scope + +The gated suite runs in plain Node: no browser, no dev server, no Vite. +Each engine is measured through a directly callable entry point: + +- React, new engine: `ComponentMetaManager.batchExtract` (`code/renderers/react/src/componentManifest/componentMeta/ComponentMetaManager.ts`). + `buildDocgenPayload` (`code/renderers/react/src/docgen/buildDocgen.ts`) wraps it with story-file resolution and argTypes conversion, so it measures more work than the bare legacy parse calls; `batchExtract` is the metric of record. +- React, legacy engines: `getReactDocgen` (`code/renderers/react/src/componentManifest/reactDocgen.ts`) and `parseWithReactDocgenTypescript` (`code/renderers/react/src/componentManifest/reactDocgenTypescript.ts`). + The live legacy paths are builder plugins - a Vite plugin or a webpack loader, some third-party - and cannot run standalone; these wrappers share their extraction logic without being byte-identical to any of them, so legacy numbers describe the engines, not the builder pipelines. + Both wrappers cache per file for the life of the process and expose only global invalidation; a simulated save must invalidate before re-parsing or the warm sample is a cache hit. + `react-docgen` is the budgeted legacy control; `react-docgen-typescript` is measurable through the same wrappers but carries no budget row. +- Angular: a standalone Compodoc CLI run. + Storybook shells out to the same CLI (`runCompodoc`, `code/frameworks/angular/src/builders/utils/run-compodoc.ts`), so the harness measures the tool the framework actually runs. + `@compodoc/compodoc` is pinned exactly at 2.0.0 in `scripts/package.json`, the version the Angular docgen baselines capture against (`code/lib/docgen-harness/README.md`). + The pin is exact rather than a range because compodoc's cost moves across versions, and the resolved version is recorded with the results. +- Vue, legacy engine: `parse` from `vue-docgen-api`, called per `.vue` file the way the Vite plugin calls it (`code/frameworks/vue3-vite/src/plugins/vue-docgen.ts`). + This is still the default plugin (`resolveDocgenOptions`, `code/frameworks/vue3-vite/src/preset.ts`), so it is Vue's legacy control the way `react-docgen` is React's. + The parser reads the file from disk on every call and keeps no cache, so one save costs one `parse` of one file - the per-save sample needs no invalidation step and matches production. +- Vue, new engine: `createChecker` and `createCheckerByJson` from `vue-component-meta`, with the checker options the Vite plugin passes (`code/frameworks/vue3-vite/src/plugins/vue-component-meta.ts`). + One caveat the budgets must respect: that plugin falls back to `createCheckerByJson` whenever the project's root tsconfig declares `references`, because `vue-component-meta` does not resolve them. + The harness's `workspace` and `base-type-touch` scenarios drive `createChecker` at a package tsconfig, which measures the engine rather than the plugin's path for that project shape. +- Svelte and web components: their extraction engines are plain functions callable from Node; exact entry points get pinned when each engine's harness lands. + +## Metrics + +Five metrics per engine, all candidates for gating: + +- **Cold extraction**: time for the first full extraction in a fresh process. + For engines that build a TypeScript program, this includes building it. +- **Warm extraction**: time to re-extract one changed component after a simulated save, once the process is warm. +- **Whole-project scan**: time for one batch pass over the entire project. + Only Compodoc works this way, so this metric applies to Compodoc alone; per-component engines record it as n/a rather than a faked equivalent. +- **Peak memory (transient)**: the allocation spike a save causes above the retained baseline, sampled around forced GC. +- **Leak detection**: retained-heap growth after the save series, plus the least-squares slope of retained heap over that series. + +Compodoc maps onto these differently because it is a fresh CLI process per run: cold extraction and whole-project scan are the same full-project measurement, warm extraction is a second full run after touching one file, and peak memory is the child's peak RSS sampled from outside the process. +Whether the retained-series metrics mean anything for a fresh-process engine is settled when its baselines are recorded; until then its leak cells stay placeholders. + +Two harnesses implement this. +`scripts/bench/docgen-memory/` (`yarn bench:docgen-memory`) gates React memory and is the only tier that fails a build today. +`scripts/bench/docgen-perf/` (`yarn bench:docgen-perf`) records all five metrics for every engine and gates nothing yet, because no budget values exist. + +## Determinism method + +- **Fixed synthetic projects.** + Inputs come from a generator with fixed parameters (component count, props per component, type-heaviness levers), never from real-world checkouts. + The existing generator is `scripts/bench/docgen-memory/generate-project.ts`. +- **Warmup.** + Every run starts with one full cold pass. + That pass is the cold-extraction sample and is excluded from all warm samples. + This exists today in the refresh mode of `scripts/bench/docgen-memory/memory-harness.ts`; live mode skips it by design. +- **Median-of-N for latency.** + Each latency metric records a median, never a single-run number. + Cold extraction and whole-project scan yield one sample per fresh process, so their N samples come from N spawns. + Warm extraction records the median of the per-save durations inside a single run's save series. + That run is the repetition whose cold sample is the median, never the first: repetition 1 pays for a cold module graph and a cold page cache, and measures several times slower than the rest. + The harness pins one N for all engines and records it with the results; numbers taken at different N are not comparable. + An engine that fails part-way through its repetitions is reported as failed, never as measured at an N it did not reach. + `scripts/bench/docgen-perf/aggregate.ts` implements this; the memory gate still runs each configuration exactly once. +- **Series statistics for leak metrics.** + Retained slope is a least-squares fit over the save series; retained growth is the delta between the final retained sample and the pre-run baseline. + Both read one run's series instead of repeated runs. + This exists today. +- **Series mean for transient memory.** + Peak memory (transient) is the mean of the per-save spikes across the save series - not repeated and medianed like cold latency, not slope-fitted like retained slope. + Warm latency, transient memory, and the leak metrics all read the same fixed-length series from the same run. + This exists today in `scripts/bench/docgen-memory/memory-harness.ts`. +- **Fresh process per measurement.** + Every measured process is spawned fresh so it starts from a clean heap. + This exists today in `scripts/bench/docgen-memory/gate.ts`. +- **Relative comparison on the same machine.** + Perf questions are answered by ratios between runs executed sequentially inside a single CI job on one executor, or on one local machine. + "Same machine" means exactly that; cross-job, cross-run, and cross-PR comparisons are non-comparisons because CI executors are ephemeral and not guaranteed identical. + The two standing comparisons are docgen-server flag on vs off, and new engine vs legacy engine, both measured in the same run. + Paired runs alternate their order across repetitions so cache warming and thermal drift do not consistently favor one side. + +## Budget shape + +Timing budgets are ratios or slopes, never absolute milliseconds; absolute wall-clock on shared CI executors is too noisy to gate. +A timing ratio divides the median of one side by the median of the other, both measured in the same job. +The calibration reference for React is the legacy-vs-new-engine ratio measured in the same run on the same machine. + +The warm half of that ratio carries a caveat the baseline work must settle. +Both sides re-extract one changed component per save, which compares the engines on equal work. +Only the new engine has a production path shaped that way: `buildDocgenPayload` hardcodes `react-component-meta`, so the single-component docgen worker never runs a legacy parser. +The legacy parsers are reachable only through `generator.ts`, which invalidates every cache and re-extracts every component on each manifest build. +A real legacy save therefore costs the whole project, not one file, and the recorded warm ratio understates legacy's true per-save cost by roughly the component count. +Run `react-legacy.ts --scope all` for the production-shaped number; the equal-work ratio is the engine comparison, not a saving a user would feel. + +Vue's pair carries a different caveat, and a sharper one. +`vue-docgen-api` does not resolve tsconfig `paths` aliases, and the Vite plugin calls `parse(id)` without an alias map, so on the generated projects it documents a fraction of what `vue-component-meta` documents. +Every run so far has the legacy engine finishing its cold pass an order of magnitude faster while documenting an order of magnitude fewer members, and documenting nothing at all on the save it was timed on. +The engines are not doing the same work, so the Vue ratio measures resolution depth as much as speed. +Both engines therefore report their documented-member counts, and the suite prints those counts beside every ratio - the warm one as much as the cold one - and marks the pair not like-for-like whenever they disagree. +A ratio marked not like-for-like must not become a budget. + +Angular needs a second signal, because equal member counts there would not mean equal work. +Compodoc never resolves a named type: it records the name written at the property, so `@Input() kind: ButtonKind` is stored as `ButtonKind` and an `@Output()` is stored as `EventEmitter` with its payload dropped, while the same union written inline is stored in full. +Chain a type through twenty files and it still documents every member, at the same speed, having looked through none of them - so a member count alone would read as identical to an engine that expanded all of them. +Compodoc therefore also reports how many of its documented members carry a type it never resolved, and a future Angular pair must compare that alongside the counts before calling a ratio like-for-like. +Its warm member count is a second whole-project total rather than the one member re-extracted, so it is not comparable with a series engine's warm count either. + +Signals are documented, not benchmarked. +Compodoc reads `input()`, `output()` and `model()` by matching the initializer's source text, not by resolving the import or the type, and it documents them at the same speed and count as the decorator form - so there is no timing to measure. +What the textual match costs is fidelity: an aliased `import { input as ngInput }` stops being recognised as an input at all, and a non-literal default without an explicit generic loses its type. +Those belong in the docgen-harness fixtures that pin compodoc's behaviour, not in a bench scenario that would report two rows indistinguishable from the baseline. + +The baseline work records the figures; quoting any here would pin numbers taken at one profile on one machine. +Engines without a second implementation in the same job get their timing reference picked when their baselines are recorded; until then their timing budgets stay placeholders. +Memory budgets stay absolute megabytes with generous headroom: budgets sit well above observed values so the gate is not flaky, while still failing hard on a real regression. +Every engine must also carry its own negative control - a configuration that must fail, proving the gate can catch the regression class it exists for. +A control names its lever and the one metric it must trip; it does not need to trip every metric, and where no credible lever exists the gap is recorded instead of inventing a hollow control. +The only worked example today is the React memory gate's out-of-memory control. +Budgets and controls are derived per engine from that engine's own baseline runs; nothing is ported between engines. + +## CI tiers + +- **Per PR: report-only.** + Per-PR perf jobs attach to the unconditional job block of the generated CircleCI config (`scripts/ci/main.ts`), where the package benchmark job already runs on every build without gating. + They report numbers and never fail the build. +- **Daily: the only gating tier.** + "Daily" names the CircleCI `workflow=daily` pipeline parameter (tier order `normal < merged < daily`, `scripts/ci/utils/types.ts`); daily-only jobs attach in `scripts/ci/main.ts`, where the docgen memory gate already runs. +- **The daily cadence is unproven.** + The only trigger for the daily tier in version control is the `ci:daily` PR label (`.github/workflows/trigger-circle-ci-workflow.yml`); no scheduled trigger exists in the repo, and no CircleCI-side schedule is confirmed to exist. + Until that is resolved, gating on the daily tier means gating when someone applies the label. + The baseline-and-budget work must settle the cadence before this gate can be described as nightly protection. + +## User-perceived metrics + +Three user-perceived metrics join the suite as report-only extensions of the sandbox bench task; none joins the gated floor. +They need a real dev server and Chromium, which makes them variance-heavy; they exist to show user-visible impact, not to gate. +All three build on existing tooling and add no new dependencies: + +- **Dev-server startup delta.** + Run the sandbox bench task (`scripts/tasks/bench.ts` driving `scripts/bench/browse.ts` over Playwright/Chromium, with dev-server readiness timings from `scripts/tasks/dev.ts`) twice, docgen-server feature on and off, and report the delta. + Today only the internal Storybook config reads the on/off switch (`code/.storybook/main.ts`); generated sandboxes do not, so the harness work must inject the feature toggle into the sandbox config before the delta can be measured there. + No first-class on/off comparison mode exists yet; today this is two runs diffed by hand. +- **Time-to-Controls-populated.** + Adapt the existing Playwright flow that selects a story, opens the Controls panel, and waits for an args-table cell (`code/e2e-internal/docgen-hot-update.spec.ts`) into a timing probe, using the elapsed-time pattern from `browse.ts`. + Today that flow is a pass/fail assertion and records no elapsed time. +- **Docs-page props-table render.** + `benchAutodocs` in `scripts/bench/browse.ts` times the docs page today, but its wait on the component description text is a locator construction with no real wait condition, so the recorded number mostly reflects page load. + The probe waits on a props-table element with a real wait condition; it does not exist yet. + +## Budgets table skeleton + +The baseline work fills in the values; until then every cell is a placeholder. +The exception is react-osa's memory row, which the docgen memory gate already enforces from `scripts/bench/docgen-shared/budgets.ts`. + +| Engine | Cold extraction | Warm extraction | Whole-project scan | Peak memory (transient) | Retained growth | Retained slope | Negative control | Tier | +| ----------------------------------------- | --------------- | --------------- | ------------------ | ----------------------- | --------------- | -------------- | ---------------- | ---------- | +| react-legacy (react-docgen, control) | TBD (1.12) | TBD (1.12) | n/a | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | +| react-osa (ComponentMetaManager, control) | TBD (1.12) | TBD (1.12) | n/a | 90MB | 60MB | 3MB/save | OOM (gate.ts) | daily | +| vue-docgen-api (legacy, current default) | TBD (1.12) | TBD (1.12) | n/a | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | +| vue-component-meta | TBD (1.12) | TBD (1.12) | n/a | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | +| compodoc | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | +| svelte (stretch) | TBD (1.12) | TBD (1.12) | n/a | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | +| cem (stretch) | TBD (1.12) | TBD (1.12) | n/a | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | TBD (1.12) | diff --git a/scripts/bench/docgen-memory/gate.ts b/scripts/bench/docgen-memory/gate.ts index e70c0f71ff96..c24d7636ef82 100644 --- a/scripts/bench/docgen-memory/gate.ts +++ b/scripts/bench/docgen-memory/gate.ts @@ -2,18 +2,15 @@ * CI regression gate for the docgen-server memory behavior * (https://github.com/storybookjs/storybook/issues/35260). * - * Runs {@link file://./memory-harness.ts} in fresh child processes (so each measurement starts from a - * clean heap) and asserts two independent things: + * Runs {@link file://./memory-harness.ts} in fresh child processes (one clean heap per measurement) + * and asserts two independent things: * - * 1. Steady-state churn/leak-freeness (`changed-scope` metric config): post-GC `heapUsed` must not - * climb across saves, and the per-save transient working set must stay under budget. A regression - * that re-extracts the whole index would blow past these. - * 2. The OOM fix itself, as a **crash → survive negative control** (`live` configs): the live path - * (many per-component `batchExtract` calls on the shared program) is run under a tight heap cap - * both WITH and WITHOUT program recycling. - * - recycle OFF (`--recycle off`, ratio Infinity) MUST OOM. - * - recycle ON (default) MUST survive. - * Asserting the flip (not just survival) is what guards the fix rather than the cap. + * 1. Steady-state leak-freeness (`changed-scope` metric config): post-GC `heapUsed` must not + * climb across saves, and the per-save transient working set must stay under budget. + * 2. The OOM fix itself, as a **crash → survive negative control** (`live` configs): the live + * path is run under a tight heap cap both WITH and WITHOUT program recycling - recycle OFF + * (`--recycle off`, ratio Infinity) MUST OOM, recycle ON (default) MUST survive. Asserting + * the flip, not just survival, is what guards the fix rather than the cap. * * Run: * yarn bench:docgen-memory # from scripts/ @@ -23,6 +20,8 @@ import { spawnSync } from 'node:child_process'; import * as fs from 'node:fs'; import * as path from 'node:path'; +import { type MemoryBudgets, memoryBudgetsFor } from '../docgen-shared/budgets.ts'; +import { outputTail } from '../docgen-shared/child-output.ts'; import { SANDBOX_DIRECTORY } from '../../utils/constants.ts'; interface HarnessResult { @@ -33,15 +32,6 @@ interface HarnessResult { avgTransient?: number; } -interface MetricBudgets { - /** Max allowed post-GC retained growth (MB) across the run. */ - maxRetainedGrowthMb: number; - /** Max allowed average transient working set added per save (MB). */ - maxTransientMb: number; - /** Max allowed post-GC retained-heap slope (MB/save). */ - maxRetainedSlopeMb: number; -} - interface GateConfig { name: string; args: string[]; @@ -50,12 +40,12 @@ interface GateConfig { /** `--max-old-space-size` cap (MB) for the child. Omit to run uncapped. */ capMb?: number; /** - * Crash → survive negative control. `true` ⇒ the child MUST OOM (recycle disabled); `false` ⇒ it - * MUST survive (recycle on). Omit for metric-only configs. + * Crash → survive negative control: `true` ⇒ MUST OOM (recycle disabled), `false` ⇒ MUST + * survive (recycle on). Omit for metric-only configs. */ expectOom?: boolean; /** Post-GC metric budgets, asserted from `result.json` (metric configs only). */ - budgets?: MetricBudgets; + budgets?: MemoryBudgets; } const HARNESS = path.join(import.meta.dirname, 'memory-harness.ts'); @@ -78,7 +68,7 @@ const CONFIGS: GateConfig[] = [ '--scope', 'changed', ], outSubdir: 'changed-scope', - budgets: { maxRetainedGrowthMb: 60, maxTransientMb: 90, maxRetainedSlopeMb: 3 }, + budgets: memoryBudgetsFor('react-osa'), }, { // Positive control: the live per-edit workload survives under a tight cap BECAUSE the shared @@ -153,15 +143,7 @@ function runHarness(config: GateConfig): RunOutcome { const proc = spawnSync(process.execPath, nodeArgs, { encoding: 'utf8' }); const output = `${proc.stdout ?? ''}${proc.stderr ?? ''}`; - // Echo a compact tail so CI logs show what happened without the full V8 crash dump. - console.log( - output - .trim() - .split('\n') - .slice(-8) - .map((line) => ` ${line}`) - .join('\n') - ); + console.log(outputTail(output, 8)); const result = fs.existsSync(jsonPath) ? (JSON.parse(fs.readFileSync(jsonPath, 'utf8')) as HarnessResult) diff --git a/scripts/bench/docgen-memory/generate-project.ts b/scripts/bench/docgen-memory/generate-project.ts index ab29f440ef79..f1c95b696b64 100644 --- a/scripts/bench/docgen-memory/generate-project.ts +++ b/scripts/bench/docgen-memory/generate-project.ts @@ -19,6 +19,11 @@ import * as fs from 'node:fs'; import * as path from 'node:path'; import { pathToFileURL } from 'node:url'; +import { z } from 'zod'; + +import { countOption, parseHarnessOptions } from '../docgen-shared/args.ts'; +import type { ComponentRefLike, StoryRefLike } from '../docgen-shared/react-renderer-module.ts'; + const require = createRequire(import.meta.url); export interface GenerateOptions { @@ -239,6 +244,26 @@ ${variantExports} `; } +/** The renderer-side reference for the generated `Comp{i}`, matching the naming used above. */ +export function componentRef(i: number, componentPath: string): ComponentRefLike { + return { + componentName: `Comp${i}`, + importName: `Comp${i}`, + localImportName: `Comp${i}`, + importId: `./Comp${i}`, + isPackage: false, + path: componentPath, + }; +} + +/** Pairs {@link GeneratedProject.componentPaths} with `storyPaths`, which share an index. */ +export function buildStoryRefs(componentPaths: string[], storyPaths: string[]): StoryRefLike[] { + return componentPaths.map((componentPath, i) => ({ + storyPath: storyPaths[i], + component: componentRef(i, componentPath), + })); +} + export function generateProject(options: GenerateOptions): GeneratedProject { const outDir = path.resolve(options.outDir); fs.rmSync(outDir, { recursive: true, force: true }); @@ -275,25 +300,40 @@ export function generateProject(options: GenerateOptions): GeneratedProject { return { outDir, configPath, componentPaths, storyPaths }; } -function parseArgs(argv: string[]): GenerateOptions { - const get = (flag: string, fallback: string) => { - const idx = argv.indexOf(flag); - return idx >= 0 && argv[idx + 1] ? argv[idx + 1] : fallback; - }; - return { - outDir: get('--out', '../storybook-sandboxes/docgen-memory-stress'), - components: Number(get('--components', '500')), - variants: Number(get('--variants', '4')), - props: Number(get('--props', '8')), - heavyTypes: argv.includes('--heavy'), - heavyFactor: Number(get('--heavy-factor', '1')), - base64Kb: Number(get('--base64-kb', '0')), - withNodeModules: !argv.includes('--no-node-modules'), - }; +function parseOptions(argv: string[]): GenerateOptions { + return parseHarnessOptions( + argv, + { + out: { type: 'string' }, + components: { type: 'string' }, + variants: { type: 'string' }, + props: { type: 'string' }, + heavy: { type: 'boolean' }, + 'heavy-factor': { type: 'string' }, + 'base64-kb': { type: 'string' }, + 'no-node-modules': { type: 'boolean' }, + } as const, + z.object({ + outDir: z.string().default('../storybook-sandboxes/docgen-memory-stress'), + components: countOption(500), + variants: countOption(4), + props: countOption(8), + heavyTypes: z.boolean().default(false), + heavyFactor: countOption(1), + base64Kb: countOption(0), + withNodeModules: z.boolean().default(true), + }), + (values) => ({ + ...values, + outDir: values.out, + heavyTypes: values.heavy, + withNodeModules: !values.noNodeModules, + }) + ); } if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) { - const options = parseArgs(process.argv.slice(2)); + const options = parseOptions(process.argv.slice(2)); const start = Date.now(); const result = generateProject(options); console.log( diff --git a/scripts/bench/docgen-memory/memory-harness.ts b/scripts/bench/docgen-memory/memory-harness.ts index 7797d12abaf4..3950313e2416 100644 --- a/scripts/bench/docgen-memory/memory-harness.ts +++ b/scripts/bench/docgen-memory/memory-harness.ts @@ -2,22 +2,21 @@ * Deterministic in-process memory harness for the docgen-server OOM * (https://github.com/storybookjs/storybook/issues/35260). * - * It reproduces the "re-extract on every save" behavior without a browser or dev server, so it runs - * fast and is fully deterministic. It drives the real {@link ComponentMetaManager} against a - * generated project of N components, then simulates K file saves and samples memory after each one. + * Drives the real {@link ComponentMetaManager} against a generated project of N components, then + * simulates K file saves and samples memory after each one. * - * Two failure signals are measured independently: - * - RETAINED growth: post-GC `heapUsed` trend across saves. A rising trend ⇒ a true leak (memory - * held between saves). A flat trend ⇒ the cost is transient allocation, not retained state. - * - PEAK pressure: pre-GC `rss` per save. With `--no-force-gc` and a `--max-old-space-size` cap at - * launch, this is what crashes the process when saves outpace GC (the reported OOM). + * Two independent failure signals: + * - RETAINED growth: post-GC `heapUsed` trend across saves. Rising ⇒ a true leak (memory held + * between saves), flat ⇒ transient allocation. + * - PEAK pressure: pre-GC `rss` per save. Under `--no-force-gc` and a `--max-old-space-size` cap, + * this is what crashes the process when saves outpace GC (the reported OOM). * * Modes: - * --mode refresh call manager.batchExtract(allEntries) synchronously each save. Mirrors the docgen + * --mode refresh synchronous manager.batchExtract(allEntries) each save. Mirrors the docgen * open-service "refresh all extracted components" path (server.ts). * --mode live many per-component batchExtract calls on the shared program, mirroring the - * docs-addon per-edit wave that drives the #35260 OOM. The program-recycle fix bounds - * this path. Use --recycle off to assert the OOM still happens without the fix. + * docs-addon per-edit wave that drives the #35260 OOM. The program-recycle fix + * bounds this path; use --recycle off to assert the OOM still happens without it. * * Run (diagnose retained vs transient): * node --expose-gc --import jiti/register scripts/bench/docgen-memory/memory-harness.ts \ @@ -36,28 +35,23 @@ import * as path from 'node:path'; import ts from 'typescript'; -import { SANDBOX_DIRECTORY } from '../../utils/constants.ts'; -import { componentSource, generateProject } from './generate-project.ts'; - -/** - * Minimal structural view of the entry shape consumed by `ComponentMetaManager.batchExtract`. Kept - * local so the `scripts` typecheck does not descend into `code/renderers` internals; the real types - * live in `code/renderers/react/src/componentManifest`. - */ -interface StoryRef { - storyPath: string; - component: { - componentName: string; - importName: string; - localImportName: string; - importId: string; - isPackage: boolean; - path: string; - }; -} +import { z } from 'zod'; + +import { countOption, parseHarnessOptions } from '../docgen-shared/args.ts'; +import { SANDBOX_DIRECTORY } from '../docgen-shared/paths.ts'; +import { type StoryRefLike, loadReactRendererModule } from '../docgen-shared/react-renderer-module.ts'; +import { MB, gcAvailable } from '../docgen-shared/sampling.ts'; +import { + type SeriesEngine, + harnessMain, + printSeriesSummary, + runSeries, +} from '../docgen-shared/series.ts'; +import { leastSquaresSlope } from '../docgen-shared/stats.ts'; +import { buildStoryRefs, componentSource, generateProject } from './generate-project.ts'; interface ComponentMetaManagerLike { - batchExtract(entries: StoryRef[]): void; + batchExtract(entries: StoryRefLike[]): void; onFilesChanged(changes: Array<{ filePath: string; type: 'changed' | 'created' | 'deleted' }>): void; dispose(): void; } @@ -67,35 +61,24 @@ type ComponentMetaManagerCtor = new ( recycleHeapPressureRatio?: number ) => ComponentMetaManagerLike; +const MODES = ['refresh', 'live'] as const; /** - * Load the real `ComponentMetaManager` at runtime. The specifier is built with `new URL` so the - * static `scripts` typecheck does not pull `code/renderers` source into its program. + * Which entries to re-extract per save: `all` re-extracts every component (the docgen service + * refreshing on an empty change hint), `changed` re-extracts only the component whose file changed. */ -async function loadComponentMetaManager(): Promise { - const url = new URL( - '../../../code/renderers/react/src/componentManifest/componentMeta/ComponentMetaManager.ts', - import.meta.url - ).href; - const mod = (await import(url)) as { ComponentMetaManager: ComponentMetaManagerCtor }; - return mod.ComponentMetaManager; -} +const SCOPES = ['all', 'changed'] as const; +const RECYCLE = ['on', 'off'] as const; interface HarnessOptions { components: number; variants: number; props: number; saves: number; - mode: 'refresh' | 'live'; + mode: (typeof MODES)[number]; heavyTypes: boolean; heavyFactor: number; base64Kb: number; - /** - * Which entries to re-extract per save. - * all – re-extract every component each save (the docgen service refreshing all extracted - * components on an empty change hint). - * changed – re-extract only the component whose file changed. - */ - scope: 'all' | 'changed'; + scope: (typeof SCOPES)[number]; forceGc: boolean; outDir: string; reuse: boolean; @@ -104,120 +87,114 @@ interface HarnessOptions { maxRetainedGrowthMb: number; /** * Heap-pressure ratio forwarded to `ComponentMetaManager`. `Infinity` disables program recycling - * (the negative control: assert the OOM happens without the fix). `undefined` uses the product - * default. + * (negative control), `undefined` uses the product default. */ recycleHeapPressureRatio?: number; } -interface Sample { - save: number; - rssMb: number; - heapUsedMb: number; - retainedHeapMb?: number; -} - -const MB = 1024 * 1024; +const OPTIONS = { + components: { type: 'string' }, + variants: { type: 'string' }, + props: { type: 'string' }, + saves: { type: 'string' }, + mode: { type: 'string' }, + scope: { type: 'string' }, + recycle: { type: 'string' }, + heavy: { type: 'boolean' }, + 'heavy-factor': { type: 'string' }, + 'base64-kb': { type: 'string' }, + 'no-force-gc': { type: 'boolean' }, + out: { type: 'string' }, + reuse: { type: 'boolean' }, + json: { type: 'string' }, + 'max-retained-growth': { type: 'string' }, +} as const; + +const SCHEMA = z.object({ + components: countOption(600), + variants: countOption(4), + props: countOption(8), + saves: countOption(25), + mode: z.enum(MODES).default('refresh'), + scope: z.enum(SCOPES).default('all'), + // `Infinity` disables program recycling; `undefined` leaves the product default in place. + recycleHeapPressureRatio: z + .enum(RECYCLE) + .default('on') + .transform((recycle) => (recycle === 'off' ? Number.POSITIVE_INFINITY : undefined)), + heavyTypes: z.boolean().default(false), + heavyFactor: countOption(1), + base64Kb: countOption(0), + forceGc: z.boolean().default(true), + outDir: z.string().default(path.join(SANDBOX_DIRECTORY, 'docgen-memory-stress')), + reuse: z.boolean().default(false), + jsonOut: z.string().optional(), + maxRetainedGrowthMb: countOption(400), +}); -function parseArgs(argv: string[]): HarnessOptions { - const get = (flag: string, fallback: string) => { - const idx = argv.indexOf(flag); - return idx >= 0 && argv[idx + 1] ? argv[idx + 1] : fallback; - }; - const mode = get('--mode', 'refresh'); - if (mode !== 'refresh' && mode !== 'live') { - throw new Error(`--mode must be "refresh" or "live", got "${mode}"`); - } - const scope = get('--scope', 'all'); - if (scope !== 'all' && scope !== 'changed') { - throw new Error(`--scope must be "all" or "changed", got "${scope}"`); - } - const recycle = get('--recycle', 'on'); - if (recycle !== 'on' && recycle !== 'off') { - throw new Error(`--recycle must be "on" or "off", got "${recycle}"`); - } - return { - components: Number(get('--components', '600')), - variants: Number(get('--variants', '4')), - props: Number(get('--props', '8')), - saves: Number(get('--saves', '25')), - mode, - scope: scope as 'all' | 'changed', - recycleHeapPressureRatio: recycle === 'off' ? Number.POSITIVE_INFINITY : undefined, - heavyTypes: argv.includes('--heavy'), - heavyFactor: Number(get('--heavy-factor', '1')), - base64Kb: Number(get('--base64-kb', '0')), - forceGc: !argv.includes('--no-force-gc'), - outDir: get('--out', path.join(SANDBOX_DIRECTORY, 'docgen-memory-stress')), - reuse: argv.includes('--reuse'), - jsonOut: argv.indexOf('--json') >= 0 ? get('--json', '') : undefined, - maxRetainedGrowthMb: Number(get('--max-retained-growth', '400')), - }; +function parseOptions(argv: string[]): HarnessOptions { + return parseHarnessOptions(argv, OPTIONS, SCHEMA, (values) => ({ + ...values, + recycleHeapPressureRatio: values.recycle, + heavyTypes: values.heavy, + forceGc: !values.noForceGc, + outDir: values.out, + jsonOut: values.json, + // The schema key carries the unit the flag name leaves off, so camel-casing alone misses it. + maxRetainedGrowthMb: values.maxRetainedGrowth, + })); } -function gc(): void { - if (typeof global.gc === 'function') { - global.gc(); - global.gc(); - } +/** + * Loaded via `new URL` at runtime, not a static import, so the `scripts` typecheck does not pull + * `code/renderers` source into its program. + */ +async function loadComponentMetaManager(): Promise { + const mod = await loadReactRendererModule<{ ComponentMetaManager: ComponentMetaManagerCtor }>( + 'componentMeta/ComponentMetaManager.ts' + ); + return mod.ComponentMetaManager; } -function sampleMemory(forceGc: boolean): { rssMb: number; heapUsedMb: number; retainedHeapMb?: number } { - const pre = process.memoryUsage(); - let retainedHeapMb: number | undefined; - if (forceGc) { - gc(); - retainedHeapMb = process.memoryUsage().heapUsed / MB; - } - return { rssMb: pre.rss / MB, heapUsedMb: pre.heapUsed / MB, retainedHeapMb }; -} +function resolveProject(options: HarnessOptions): ReturnType { + const outDir = path.resolve(options.outDir); + const configPath = path.join(outDir, 'tsconfig.json'); -/** Least-squares slope of `values` vs index, in units-per-save. */ -function slopePerSave(values: number[]): number { - const n = values.length; - if (n < 2) { - return 0; - } - const meanX = (n - 1) / 2; - const meanY = values.reduce((a, b) => a + b, 0) / n; - let num = 0; - let den = 0; - for (let i = 0; i < n; i++) { - num += (i - meanX) * (values[i] - meanY); - den += (i - meanX) ** 2; + if (options.reuse && fs.existsSync(configPath)) { + const componentPaths: string[] = []; + const storyPaths: string[] = []; + for (let i = 0; i < options.components; i++) { + componentPaths.push(path.join(outDir, 'src', `Comp${i}`, `Comp${i}.tsx`)); + storyPaths.push(path.join(outDir, 'src', `Comp${i}`, `Comp${i}.stories.tsx`)); + } + console.log(` reusing generated project at ${outDir}`); + return { outDir, configPath, componentPaths, storyPaths }; } - return den === 0 ? 0 : num / den; -} -function buildEntries(componentPaths: string[], storyPaths: string[]): StoryRef[] { - return componentPaths.map((componentPath, i) => ({ - storyPath: storyPaths[i], - component: { - componentName: `Comp${i}`, - importName: `Comp${i}`, - localImportName: `Comp${i}`, - importId: `./Comp${i}`, - isPackage: false, - path: componentPath, - }, - })); + const genStart = Date.now(); + const project = generateProject({ + outDir: options.outDir, + components: options.components, + variants: options.variants, + props: options.props, + heavyTypes: options.heavyTypes, + heavyFactor: options.heavyFactor, + base64Kb: options.base64Kb, + withNodeModules: true, + }); + console.log(` generated project in ${Date.now() - genStart}ms at ${project.outDir}`); + return project; } /** - * Live-path mode: mirrors the docs-addon per-edit "waves" that drive the #35260 OOM — many - * individual per-component `batchExtract` calls on the ONE shared program, whose type-resolution - * cache accumulates across calls. This is the exact shape the program-recycle fix bounds (the - * recycle check runs between calls), unlike the `--scope all` single-call cold pass which OOMs - * within one call regardless of recycling. - * - * Run under a `--max-old-space-size` cap: - * - recycle on (default): the heap sawtooths as the shared program is recycled, and survives. - * - `--recycle off` (ratio Infinity): the type cache climbs to the cap and the process OOMs — - * the negative control the gate asserts. + * Unlike the `--scope all` single-call cold pass (which OOMs within one call regardless of + * recycling), the recycle check runs between the per-component calls here, so under a heap cap: + * recycle on sawtooths and survives, `--recycle off` climbs to the cap and OOMs (the negative + * control the gate asserts). */ function runLiveMode( manager: ComponentMetaManagerLike, - entries: StoryRef[], + entries: StoryRefLike[], options: HarnessOptions ): void { const recycleEnabled = options.recycleHeapPressureRatio === undefined; @@ -257,157 +234,101 @@ function runLiveMode( } } -async function main() { - const options = parseArgs(process.argv.slice(2)); - const gcAvailable = typeof global.gc === 'function'; +/** + * The cold pass is the *identical* operation a `scope=all` refresh save performs + * (extractPropsFromStories over every entry), so an OOM there is the same OOM every refresh-all + * save would hit. + */ +function refreshEngine( + manager: ComponentMetaManagerLike, + entries: StoryRefLike[], + project: ReturnType, + options: HarnessOptions +): SeriesEngine { + // Track how many extra props each component currently has, so each save grows the type. + const extraByComponent = new Array(options.components).fill(options.props); + const changedIndex = (save: number) => (save - 1) % options.components; + + return { + async cold() { + manager.batchExtract(entries); + return undefined; + }, + async applySave(save) { + const i = changedIndex(save); + const componentPath = project.componentPaths[i]; + // Mutate the component's props interface on disk so the type genuinely changes. + extraByComponent[i] += 1; + fs.writeFileSync( + componentPath, + componentSource(i, extraByComponent[i], { + heavyTypes: options.heavyTypes, + heavyFactor: options.heavyFactor, + base64Kb: options.base64Kb, + }) + ); + // Must run after the write, or the program serves a stale snapshot for this file. + manager.onFilesChanged([{ filePath: componentPath, type: 'changed' }]); + }, + async reextract(save) { + manager.batchExtract(options.scope === 'changed' ? [entries[changedIndex(save)]] : entries); + return undefined; + }, + dispose: () => manager.dispose(), + }; +} + +harnessMain(async () => { + const options = parseOptions(process.argv.slice(2)); console.log('docgen-memory harness'); console.log( ` components=${options.components} variants=${options.variants} props=${options.props} ` + - `saves=${options.saves} mode=${options.mode} scope=${options.scope} forceGc=${options.forceGc && gcAvailable}` + `saves=${options.saves} mode=${options.mode} scope=${options.scope} ` + + `forceGc=${options.forceGc && gcAvailable()}` ); - if (options.forceGc && !gcAvailable) { + if (options.forceGc && !gcAvailable()) { console.log(' (run with `node --expose-gc` to measure retained heap; continuing without it)'); } - const genStart = Date.now(); - let project: ReturnType; - if (options.reuse && fs.existsSync(path.join(path.resolve(options.outDir), 'tsconfig.json'))) { - const outDir = path.resolve(options.outDir); - const componentPaths: string[] = []; - const storyPaths: string[] = []; - for (let i = 0; i < options.components; i++) { - componentPaths.push(path.join(outDir, 'src', `Comp${i}`, `Comp${i}.tsx`)); - storyPaths.push(path.join(outDir, 'src', `Comp${i}`, `Comp${i}.stories.tsx`)); - } - project = { outDir, configPath: path.join(outDir, 'tsconfig.json'), componentPaths, storyPaths }; - console.log(` reusing generated project at ${outDir}`); - } else { - project = generateProject({ - outDir: options.outDir, - components: options.components, - variants: options.variants, - props: options.props, - heavyTypes: options.heavyTypes, - heavyFactor: options.heavyFactor, - base64Kb: options.base64Kb, - withNodeModules: true, - }); - console.log(` generated project in ${Date.now() - genStart}ms at ${project.outDir}`); - } - + const project = resolveProject(options); const ComponentMetaManager = await loadComponentMetaManager(); const manager = new ComponentMetaManager(ts, options.recycleHeapPressureRatio); - const entries = buildEntries(project.componentPaths, project.storyPaths); + const entries = buildStoryRefs(project.componentPaths, project.storyPaths); if (options.mode === 'live') { runLiveMode(manager, entries, options); return; } - // Initial extraction — builds the full TS program once (the cold path). This is the *identical* - // operation a `scope=all` refresh save performs (extractPropsFromStories over every entry), so an - // OOM here is the same OOM every refresh-all save would hit; it simply lands on the first pass when - // the full-extraction working set already exceeds the heap cap. - const initialStart = Date.now(); - console.log(` full extraction over ${entries.length} components (cold pass == refresh-all save)…`); - manager.batchExtract(entries); - console.log(` initial batchExtract: ${Date.now() - initialStart}ms`); + const series = await runSeries(refreshEngine(manager, entries, project, options), { + saves: options.saves, + coldLabel: `${entries.length} components`, + forceGc: options.forceGc, + }); - const baseline = sampleMemory(options.forceGc && gcAvailable); - console.log( - ` baseline: rss=${baseline.rssMb.toFixed(0)}MB heapUsed=${baseline.heapUsedMb.toFixed(0)}MB` + - (baseline.retainedHeapMb !== undefined - ? ` retained=${baseline.retainedHeapMb.toFixed(0)}MB` - : '') - ); - - const samples: Sample[] = []; - // Track how many extra props each component currently has, so each save grows the type. - const extraByComponent = new Array(options.components).fill(options.props); + const rssValues = series.samples.map((s) => s.rssMb); + const peakRss = Math.max(series.baseline.rssMb, ...rssValues); + const finalRss = rssValues.at(-1) ?? series.baseline.rssMb; + const rssSlope = leastSquaresSlope(rssValues); + const { retainedGrowth } = series; - for (let save = 1; save <= options.saves; save++) { - const i = (save - 1) % options.components; - const componentPath = project.componentPaths[i]; - - // Mutate the component's props interface on disk so the type genuinely changes. - extraByComponent[i] += 1; - fs.writeFileSync( - componentPath, - componentSource(i, extraByComponent[i], { - heavyTypes: options.heavyTypes, - heavyFactor: options.heavyFactor, - base64Kb: options.base64Kb, - }) - ); - // Bump project versions so the next extraction re-reads the mutated file (matches the dev - // server's file-watcher → onFilesChanged flow); without this the program serves stale snapshots. - manager.onFilesChanged([{ filePath: componentPath, type: 'changed' }]); - - const saveStart = Date.now(); - const toExtract = options.scope === 'changed' ? [entries[i]] : entries; - manager.batchExtract(toExtract); - const durMs = Date.now() - saveStart; - - const mem = sampleMemory(options.forceGc && gcAvailable); - samples.push({ save, ...mem }); - console.log( - ` save ${String(save).padStart(3)}: ${String(durMs).padStart(5)}ms ` + - `rss=${mem.rssMb.toFixed(0).padStart(5)}MB heapUsed=${mem.heapUsedMb.toFixed(0).padStart(5)}MB` + - (mem.retainedHeapMb !== undefined - ? ` retained=${mem.retainedHeapMb.toFixed(0).padStart(5)}MB` - : '') - ); - } - - manager.dispose(); - - const rssValues = samples.map((s) => s.rssMb); - const peakRss = Math.max(baseline.rssMb, ...rssValues); - const finalRss = rssValues.at(-1) ?? baseline.rssMb; - const rssSlope = slopePerSave(rssValues); - - const retainedValues = samples - .map((s) => s.retainedHeapMb) - .filter((v): v is number => v !== undefined); - const retainedSlope = retainedValues.length ? slopePerSave(retainedValues) : undefined; - const retainedGrowth = - retainedValues.length && baseline.retainedHeapMb !== undefined - ? (retainedValues.at(-1) as number) - baseline.retainedHeapMb - : undefined; - - const transients = samples - .map((s) => (s.retainedHeapMb !== undefined ? s.heapUsedMb - s.retainedHeapMb : undefined)) - .filter((v): v is number => v !== undefined); - const avgTransient = transients.length - ? transients.reduce((a, b) => a + b, 0) / transients.length - : undefined; - - console.log('\nsummary'); + printSeriesSummary(series, options.saves); console.log(` peak rss: ${peakRss.toFixed(0)}MB`); - if (avgTransient !== undefined) { - console.log(` avg transient/save: ${avgTransient.toFixed(0)}MB (pre-GC heapUsed − post-GC heapUsed)`); - } console.log(` final rss: ${finalRss.toFixed(0)}MB`); console.log(` rss slope: ${rssSlope.toFixed(1)}MB/save`); - if (retainedSlope !== undefined) { - console.log(` retained slope: ${retainedSlope.toFixed(2)}MB/save (post-GC heapUsed)`); - console.log(` retained growth: ${retainedGrowth!.toFixed(0)}MB over ${options.saves} saves`); + if (retainedGrowth !== undefined) { console.log( - retainedGrowth! > 5 + retainedGrowth > 5 ? ' → classification: RETAINED leak (memory held between saves)' - : ' → classification: TRANSIENT pressure (post-GC heap flat; OOM is GC-can\'t-keep-up)' + : " → classification: TRANSIENT pressure (post-GC heap flat; OOM is GC-can't-keep-up)" ); } if (options.jsonOut) { fs.writeFileSync( options.jsonOut, - JSON.stringify( - { options, baseline, samples, peakRss, finalRss, rssSlope, retainedSlope, retainedGrowth, avgTransient }, - null, - 2 - ) + JSON.stringify({ options, ...series, peakRss, finalRss, rssSlope }, null, 2) ); console.log(` wrote ${options.jsonOut}`); } @@ -418,9 +339,4 @@ async function main() { ); process.exitCode = 1; } -} - -main().catch((err) => { - console.error(err); - process.exitCode = 1; }); diff --git a/scripts/bench/docgen-perf/aggregate.test.ts b/scripts/bench/docgen-perf/aggregate.test.ts new file mode 100644 index 000000000000..cf488729ba99 --- /dev/null +++ b/scripts/bench/docgen-perf/aggregate.test.ts @@ -0,0 +1,109 @@ +import { describe, expect, it } from 'vitest'; + +import type { MemorySample, SaveSample } from '../docgen-shared/samples.ts'; +import type { SeriesResult } from '../docgen-shared/series.ts'; +import { summarizeSeries } from '../docgen-shared/stats.ts'; +import { designatedRep, oneShotMetrics, seriesMetrics } from './aggregate.ts'; + +function sample(save: number, durMs: number, heapUsedMb = 20, retainedHeapMb = 10): SaveSample { + return { save, durMs, rssMb: 100, heapUsedMb, retainedHeapMb }; +} + +/** Built through summarizeSeries, so the fixture cannot claim figures a real run would not produce. */ +function repOf(coldMs: number, samples: SaveSample[], baseline: MemorySample): SeriesResult { + return { coldMs, baseline, samples, ...summarizeSeries(samples, baseline) }; +} + +function rep(coldMs: number, durations: number[]): SeriesResult { + return repOf( + coldMs, + durations.map((d, i) => sample(i + 1, d)), + { rssMb: 90, heapUsedMb: 12, retainedHeapMb: 10 } + ); +} + +describe('designatedRep', () => { + it('picks the repetition whose cold sample is the median', () => { + const reps = [rep(900, [1]), rep(100, [2]), rep(300, [3]), rep(200, [4]), rep(250, [5])]; + expect(designatedRep(reps).coldMs).toBe(250); + }); + + it('never picks the slow first repetition just because it came first', () => { + const reps = [rep(5000, [999]), rep(100, [10]), rep(110, [11]), rep(120, [12]), rep(130, [13])]; + expect(designatedRep(reps).coldMs).not.toBe(5000); + expect(designatedRep(reps).coldMs).toBe(120); + }); + + it('takes the lower middle for an even count', () => { + const reps = [rep(10, [1]), rep(20, [2]), rep(30, [3]), rep(40, [4])]; + expect(designatedRep(reps).coldMs).toBe(20); + }); + + it('does not mutate its input', () => { + const reps = [rep(30, [1]), rep(10, [2]), rep(20, [3])]; + designatedRep(reps); + expect(reps.map((r) => r.coldMs)).toEqual([30, 10, 20]); + }); +}); + +describe('seriesMetrics', () => { + const reps = [rep(300, [9, 9, 9]), rep(100, [1, 2, 3]), rep(200, [4, 5, 6])]; + + it('medians cold across every repetition', () => { + expect(seriesMetrics(reps, 3).coldExtractionMs).toMatchObject({ + samples: [300, 100, 200], + median: 200, + }); + }); + + it('reads warm from the designated repetition, not the first', () => { + expect(seriesMetrics(reps, 3).warmExtractionMs).toMatchObject({ samples: [4, 5, 6], median: 5 }); + }); + + it('marks whole-project scan as not applicable rather than faking one', () => { + expect(seriesMetrics(reps, 3).wholeProjectScanMs).toEqual({ status: 'n/a' }); + }); + + it('derives transient memory from the same repetition warm came from', () => { + expect(seriesMetrics(reps, 3).peakTransientMb).toMatchObject({ mean: 10 }); + }); + + it('rejects a repetition count below the pinned N', () => { + expect(() => seriesMetrics(reps, 5)).toThrow('recorded 3 repetitions, expected the pinned 5'); + }); + + it('rejects an empty set of repetitions', () => { + expect(() => seriesMetrics([], 5)).toThrow('no completed repetition recorded'); + }); + + it('rejects a run with no retained samples', () => { + const noGc = repOf(100, [{ save: 1, durMs: 5, rssMb: 100, heapUsedMb: 20 }], { + rssMb: 90, + heapUsedMb: 12, + }); + expect(() => seriesMetrics([noGc], 1)).toThrow('retained metrics missing'); + }); +}); + +describe('oneShotMetrics', () => { + const reps = [ + { coldMs: 300, warmMs: 280, peakRssMb: 500 }, + { coldMs: 100, warmMs: 120, peakRssMb: 400 }, + { coldMs: 200, warmMs: 200, peakRssMb: 450 }, + ]; + + it('reports cold and whole-project scan as the same measurement', () => { + const metrics = oneShotMetrics(reps, 3); + expect(metrics.wholeProjectScanMs).toEqual(metrics.coldExtractionMs); + }); + + it('has no retained series to report', () => { + const metrics = oneShotMetrics(reps, 3); + expect(metrics.retainedGrowthMb).toEqual({ status: 'n/a' }); + expect(metrics.retainedSlopeMbPerSave).toEqual({ status: 'n/a' }); + }); + + it('enforces the pinned repetition count', () => { + expect(() => oneShotMetrics(reps, 5)).toThrow('expected the pinned 5'); + }); +}); diff --git a/scripts/bench/docgen-perf/aggregate.ts b/scripts/bench/docgen-perf/aggregate.ts new file mode 100644 index 000000000000..10762817ce50 --- /dev/null +++ b/scripts/bench/docgen-perf/aggregate.ts @@ -0,0 +1,87 @@ +/** + * Turns N repetitions into the five floor metrics (see scripts/bench/PERF-METHODOLOGY.md). Warm + * latency and all three memory metrics read the same designated repetition's save series, so those + * four figures always describe one run; cold latency is a median across every repetition instead. + */ +import type { SeriesResult } from '../docgen-shared/series.ts'; +import { mean, median } from '../docgen-shared/stats.ts'; +import { type EngineMetrics, NOT_APPLICABLE } from './types.ts'; + +/** + * Repetition 1 is systematically the slowest - it pays for a cold module graph and a cold OS page + * cache - so taking it would bias warm latency and all three memory metrics at once. Cold latency + * needs no such protection: it is already a median across every repetition. + */ +export function designatedRep(reps: T[]): T { + const byCold = [...reps].sort((a, b) => a.coldMs - b.coldMs); + return byCold[Math.floor((byCold.length - 1) / 2)]; +} + +/** + * An engine that failed part-way through holds fewer samples than expectedN. Reporting those would + * put numbers taken at an unrecorded N into the results file, which the comparison method rests on + * not happening. + */ +function assertRepetitionCount(reps: unknown[], expectedN: number): void { + if (reps.length === 0) { + throw new Error('no completed repetition recorded'); + } + if (reps.length !== expectedN) { + throw new Error(`recorded ${reps.length} repetitions, expected the pinned ${expectedN}`); + } +} + +export function seriesMetrics(reps: SeriesResult[], expectedN: number): EngineMetrics { + assertRepetitionCount(reps, expectedN); + const designated = designatedRep(reps); + const coldSamples = reps.map((r) => r.coldMs); + const warmSamples = designated.samples.map((s) => s.durMs); + const { transients, avgTransient } = designated; + + if ( + avgTransient === undefined || + designated.retainedGrowth === undefined || + designated.retainedSlope === undefined + ) { + throw new Error('retained metrics missing (child must run under --expose-gc)'); + } + + return { + coldExtractionMs: { status: 'measured', samples: coldSamples, median: median(coldSamples) }, + warmExtractionMs: { status: 'measured', samples: warmSamples, median: median(warmSamples) }, + // Per-component engines have no batch pass; recording one would be a faked equivalent. + wholeProjectScanMs: NOT_APPLICABLE, + peakTransientMb: { status: 'measured', samples: transients, mean: avgTransient }, + retainedGrowthMb: { status: 'measured', value: designated.retainedGrowth }, + retainedSlopeMbPerSave: { status: 'measured', value: designated.retainedSlope }, + }; +} + +export interface OneShotRepetition { + coldMs: number; + warmMs: number; + peakRssMb: number; + coldMembers?: number; + warmMembers?: number; + /** Of the cold pass's members, how many carry a type the engine never resolved. */ + coldOpaqueTypes?: number; +} + +/** + * Metrics for a one-shot CLI engine: a fresh process per run, so cold extraction and the + * whole-project scan are the same full-project measurement, and there is no retained series to read. + */ +export function oneShotMetrics(reps: OneShotRepetition[], expectedN: number): EngineMetrics { + assertRepetitionCount(reps, expectedN); + const coldSamples = reps.map((r) => r.coldMs); + const warmSamples = reps.map((r) => r.warmMs); + const peaks = reps.map((r) => r.peakRssMb); + return { + coldExtractionMs: { status: 'measured', samples: coldSamples, median: median(coldSamples) }, + warmExtractionMs: { status: 'measured', samples: warmSamples, median: median(warmSamples) }, + wholeProjectScanMs: { status: 'measured', samples: coldSamples, median: median(coldSamples) }, + peakTransientMb: { status: 'measured', samples: peaks, mean: mean(peaks) }, + retainedGrowthMb: NOT_APPLICABLE, + retainedSlopeMbPerSave: NOT_APPLICABLE, + }; +} diff --git a/scripts/bench/docgen-perf/cli.ts b/scripts/bench/docgen-perf/cli.ts new file mode 100644 index 000000000000..4fdfdedff700 --- /dev/null +++ b/scripts/bench/docgen-perf/cli.ts @@ -0,0 +1,39 @@ +/** Command line for the docgen perf suite. */ +import * as path from 'node:path'; + +import { z } from 'zod'; + +import { parseHarnessOptions } from '../docgen-shared/args.ts'; +import { ALL_ENGINE_IDS, DEFAULT_ENGINE_IDS } from './registry.ts'; +import type { EngineId } from './types.ts'; + +const OPTIONS = { + quick: { type: 'boolean' }, + engine: { type: 'string', multiple: true }, + json: { type: 'string' }, +} as const; + +export interface CliOptions { + quick: boolean; + engines: EngineId[]; + jsonOut: string; +} + +export function parseCliOptions(argv: string[], workRoot: string): CliOptions { + const schema = z.object({ + quick: z.boolean().default(false), + // No --engine means the default set, which is why this defaults to empty rather than to + // DEFAULT_ENGINE_IDS: an explicit empty list and an absent flag are the same request. + engines: z.array(z.enum(ALL_ENGINE_IDS as [EngineId, ...EngineId[]])).default([]), + jsonOut: z.string().default(path.join(workRoot, 'results.json')), + }); + const options = parseHarnessOptions(argv, OPTIONS, schema, (values) => ({ + ...values, + engines: values.engine, + jsonOut: values.json, + })); + return { + ...options, + engines: options.engines.length ? options.engines : DEFAULT_ENGINE_IDS, + }; +} diff --git a/scripts/bench/docgen-perf/config.ts b/scripts/bench/docgen-perf/config.ts new file mode 100644 index 000000000000..b124ad19edd3 --- /dev/null +++ b/scripts/bench/docgen-perf/config.ts @@ -0,0 +1,118 @@ +/** + * Fixed measurement parameters for the per-engine docgen performance suite. + * + * N is pinned here for every engine and recorded with the results; numbers taken at different N are + * not comparable. The --quick profile exists for smoke runs only and marks its results + * non-comparable. + */ + +/** Fresh-process spawns per cold/scan median. One value for all engines. */ +export const PINNED_N = 5; + +/** Spawns for --quick smoke runs. Never comparable with PINNED_N results. */ +export const QUICK_N = 2; + +/** Sampling interval for the compodoc child's externally-polled peak RSS. */ +export const RSS_POLL_INTERVAL_MS = 100; + +export interface ReactScenarioConfig { + components: number; + variants: number; + props: number; + saves: number; +} + +export interface VueScenarioConfig { + name: 'flat' | 'workspace' | 'base-type-touch'; + packages: number; + componentsPerPackage: number; + chainDepth: number; + fanOut: number; + heavyLib: boolean; + saves: number; +} + +export interface AngularScenarioConfig { + components: number; + props: number; +} + +export interface SuiteProfile { + n: number; + comparable: boolean; + react: ReactScenarioConfig; + vue: VueScenarioConfig[]; + angular: AngularScenarioConfig; +} + +export const DEFAULT_PROFILE: SuiteProfile = { + n: PINNED_N, + comparable: true, + react: { components: 300, variants: 4, props: 10, saves: 20 }, + vue: [ + { + name: 'flat', + packages: 1, + componentsPerPackage: 20, + chainDepth: 1, + fanOut: 4, + heavyLib: false, + saves: 15, + }, + { + name: 'workspace', + packages: 4, + componentsPerPackage: 10, + chainDepth: 3, + fanOut: 4, + heavyLib: true, + saves: 15, + }, + { + name: 'base-type-touch', + packages: 4, + componentsPerPackage: 10, + chainDepth: 3, + fanOut: 4, + heavyLib: true, + saves: 10, + }, + ], + angular: { components: 100, props: 8 }, +}; + +export const QUICK_PROFILE: SuiteProfile = { + n: QUICK_N, + comparable: false, + react: { components: 20, variants: 2, props: 4, saves: 4 }, + vue: [ + { + name: 'flat', + packages: 1, + componentsPerPackage: 5, + chainDepth: 1, + fanOut: 2, + heavyLib: false, + saves: 3, + }, + { + name: 'workspace', + packages: 2, + componentsPerPackage: 3, + chainDepth: 2, + fanOut: 2, + heavyLib: true, + saves: 3, + }, + { + name: 'base-type-touch', + packages: 2, + componentsPerPackage: 3, + chainDepth: 2, + fanOut: 2, + heavyLib: true, + saves: 2, + }, + ], + angular: { components: 10, props: 4 }, +}; diff --git a/scripts/bench/docgen-perf/engine.ts b/scripts/bench/docgen-perf/engine.ts new file mode 100644 index 000000000000..d41d8e46fe22 --- /dev/null +++ b/scripts/bench/docgen-perf/engine.ts @@ -0,0 +1,127 @@ +/** + * What the orchestrator knows about an engine. Engines differ only in how one repetition is + * produced; everything downstream (aggregation, member counts) follows from that choice. + */ +import * as path from 'node:path'; + +import type { SeriesResult } from '../docgen-shared/series.ts'; +import { seriesMetrics } from './aggregate.ts'; +import type { SuiteProfile } from './config.ts'; +import type { SeriesChildSpec } from './spawn.ts'; +import type { EngineId, EngineMetrics } from './types.ts'; + +export interface ScenarioSpec { + name: string; + /** Recorded verbatim in the results so a stored run is self-describing. */ + params: Record; +} + +export interface MeasureContext { + /** Scratch directory for this engine/scenario: generated project and per-repetition JSON. */ + scenarioDir: string; + rssPollIntervalMs: number; + runSeriesChild(spec: SeriesChildSpec, outDir: string, jsonPath: string): SeriesResult; +} + +/** `Sample` is whatever one repetition of this engine produces. */ +export abstract class BenchEngine { + abstract readonly id: EngineId; + + /** Engines outside the default run only measure when named with `--engine`. */ + inDefaultRun = true; + + abstract scenarios(profile: SuiteProfile): ScenarioSpec[]; + + abstract measure(ctx: MeasureContext, scenario: ScenarioSpec, rep: number): Promise; + + abstract aggregate(samples: Sample[], expectedN: number): EngineMetrics; + + /** + * Anything that must resolve before the run starts. A returned string is the skip reason: a + * missing external tool is a partial install, not a regression, so it is reported as skipped + * rather than failed. + */ + preflight(): string | undefined { + return undefined; + } + + /** The resolved version of an externally installed engine, recorded with the results. */ + version(): string | undefined { + return undefined; + } + + /** Members the cold pass documented. Undefined when the engine cannot report a count. */ + coldMembers(_sample: Sample): number | undefined { + return undefined; + } + + /** Members the timed re-extraction documented. */ + warmMembers(_sample: Sample): number | undefined { + return undefined; + } + + /** + * Of the cold pass's members, how many the engine documented under a type name it never resolved. + * Two engines can agree on {@link coldMembers} and still have done entirely different work, so a + * member count alone does not establish like-for-like; this is what separates them. + */ + coldOpaqueTypes(_sample: Sample): number | undefined { + return undefined; + } +} + +export interface SeriesChildConfig { + id: EngineId; + /** Child entry point, relative to this directory. */ + child: string; + scenarios(profile: SuiteProfile): ScenarioSpec[]; + args(scenario: ScenarioSpec): string[]; + /** Only the reused docgen-memory harness runs under the jiti loader. */ + jiti?: boolean; + inDefaultRun?: boolean; +} + +/** + * An engine measured by spawning a series-harness child, one fresh process per repetition. All + * engines of this kind share the series harness, so aggregation and member counts are the same + * for each. + */ +export class SeriesChildEngine extends BenchEngine { + readonly id: EngineId; + readonly #config: SeriesChildConfig; + + constructor(config: SeriesChildConfig) { + super(); + this.id = config.id; + this.inDefaultRun = config.inDefaultRun ?? true; + this.#config = config; + } + + scenarios(profile: SuiteProfile): ScenarioSpec[] { + return this.#config.scenarios(profile); + } + + async measure(ctx: MeasureContext, scenario: ScenarioSpec, rep: number): Promise { + return ctx.runSeriesChild( + { + childPath: path.join(import.meta.dirname, this.#config.child), + args: this.#config.args(scenario), + jiti: this.#config.jiti, + }, + path.join(ctx.scenarioDir, 'project'), + path.join(ctx.scenarioDir, `rep${rep}.json`) + ); + } + + aggregate(samples: SeriesResult[], expectedN: number): EngineMetrics { + return seriesMetrics(samples, expectedN); + } + + coldMembers(sample: SeriesResult): number | undefined { + return sample.coldMembers; + } + + warmMembers(sample: SeriesResult): number | undefined { + return sample.warmMembers; + } +} diff --git a/scripts/bench/docgen-perf/engines/compodoc-doc.test.ts b/scripts/bench/docgen-perf/engines/compodoc-doc.test.ts new file mode 100644 index 000000000000..4397481b7846 --- /dev/null +++ b/scripts/bench/docgen-perf/engines/compodoc-doc.test.ts @@ -0,0 +1,53 @@ +import * as path from 'node:path'; + +import { describe, expect, it } from 'vitest'; + +import { countDocumentation } from './compodoc-doc.ts'; + +/** Real compodoc output, committed as the Angular docgen baselines. */ +const fixture = (name: string) => + countDocumentation( + path.join( + import.meta.dirname, + '../../../../code/lib/docgen-harness/src/angular/__testfixtures__', + name, + 'compodoc-input.json' + ) + ); + +describe('countDocumentation', () => { + it('counts an inherited member once, not once per class that lists it', () => { + // The component declares `heading` and `dismissed` and inherits `dismissible`, which compodoc + // also lists separately under `classes`. Counting `classes` too would report it twice. + expect(fixture('cross-file-inheritance').members).toBe(3); + }); + + it('counts signal inputs and outputs like decorator ones', () => { + expect(fixture('signal-io').members).toBe(6); + }); + + describe('opaque types', () => { + it('separates a union written inline from the same union behind an alias', () => { + // This component's three inputs are `ButtonKind`, `"small" | "large"` and `ToneOption`. The + // inline one describes itself; the two aliases are names compodoc never looked through, and + // an engine that resolved them would document the same three members off far more work. + const counts = fixture('decorator-union-enum'); + expect(counts.members).toBe(3); + expect(counts.opaqueTypes).toBe(2); + }); + + it('counts an unresolved generic parameter', () => { + expect(fixture('decorator-generic')).toEqual({ members: 2, opaqueTypes: 2 }); + }); + + it('counts an output whose payload type was dropped', () => { + // compodoc records `EventEmitter`, losing the emitted shape. + expect(fixture('decorator-io-basics')).toEqual({ members: 5, opaqueTypes: 1 }); + }); + + it('reports none when every member describes itself', () => { + expect(fixture('properties-methods-noise').opaqueTypes).toBe(0); + expect(fixture('jsdoc-tags').opaqueTypes).toBe(0); + }); + }); +}); diff --git a/scripts/bench/docgen-perf/engines/compodoc-doc.ts b/scripts/bench/docgen-perf/engines/compodoc-doc.ts new file mode 100644 index 000000000000..ae0c7e890e42 --- /dev/null +++ b/scripts/bench/docgen-perf/engines/compodoc-doc.ts @@ -0,0 +1,76 @@ +/** + * Reading compodoc's `documentation.json`. + * + * Compodoc records how much it documented and how much of it it actually resolved, which is what + * makes its timings readable: it never expands a named type, so a chain of aliases costs it nothing + * and it still reports every member. Counting members alone would let it look equal to an engine + * that did far more work. + */ +import * as fs from 'node:fs'; + +interface MemberEntry { + type?: string; + subProperties?: unknown[]; +} + +interface MemberHolder { + inputsClass?: MemberEntry[]; + outputsClass?: MemberEntry[]; + propertiesClass?: MemberEntry[]; + methodsClass?: MemberEntry[]; +} + +interface Documentation { + components?: MemberHolder[]; + directives?: MemberHolder[]; +} + +const MEMBER_ARRAYS = ['inputsClass', 'outputsClass', 'propertiesClass', 'methodsClass'] as const; + +/** + * `classes` is deliberately left out. Compodoc copies an ancestor's members into every descendant's + * own arrays, so a base class documented under `classes` would be counted a second time. This also + * matches what Storybook reads, which is a component's own four arrays. + */ +function documentedMembers(doc: Documentation): MemberEntry[] { + return [...(doc.components ?? []), ...(doc.directives ?? [])].flatMap((holder) => + MEMBER_ARRAYS.flatMap((key) => holder[key] ?? []) + ); +} + +/** + * Names that describe themselves, so recording one is not a resolution an engine skipped. Keyword + * spellings are lowercase as compodoc emits them (`function`, not `Function`); both are listed + * because the two spellings reach the same member field. + */ +const RESOLVED_TYPES = new Set([ + 'string', 'number', 'boolean', 'any', 'unknown', 'void', 'never', 'null', 'undefined', + 'object', 'symbol', 'bigint', 'function', 'Date', 'Function', 'Array', 'Object', +]); + +/** + * A member whose type is a bare name the engine never looked through — `Hop19Shape`, not + * `{ x: string }`. Inline object literals, primitives and literal unions are all self-describing, + * so only a lone identifier counts. + * + * This is a floor: wrapped forms such as `Partial` are equally unresolved but do not match, + * so the true share of unresolved types is at least what this reports. + */ +function isOpaque(entry: MemberEntry): boolean { + if (entry.subProperties?.length || !entry.type) { + return false; + } + const named = entry.type.replace(/\[\]$/, '').trim(); + return /^[A-Za-z_$][\w$]*$/.test(named) && !RESOLVED_TYPES.has(named); +} + +export interface DocumentationCounts { + members: number; + opaqueTypes: number; +} + +export function countDocumentation(documentationJsonPath: string): DocumentationCounts { + const doc = JSON.parse(fs.readFileSync(documentationJsonPath, 'utf8')) as Documentation; + const members = documentedMembers(doc); + return { members: members.length, opaqueTypes: members.filter(isOpaque).length }; +} diff --git a/scripts/bench/docgen-perf/engines/compodoc.ts b/scripts/bench/docgen-perf/engines/compodoc.ts new file mode 100644 index 000000000000..4f9d681289f4 --- /dev/null +++ b/scripts/bench/docgen-perf/engines/compodoc.ts @@ -0,0 +1,179 @@ +import { spawn, spawnSync } from 'node:child_process'; +import * as fs from 'node:fs'; +import { createRequire } from 'node:module'; +import * as path from 'node:path'; + +import { outputTail } from '../../docgen-shared/child-output.ts'; +import { type OneShotRepetition, oneShotMetrics } from '../aggregate.ts'; +import { type DocumentationCounts, countDocumentation } from './compodoc-doc.ts'; +import type { AngularScenarioConfig, SuiteProfile } from '../config.ts'; +import { BenchEngine, type MeasureContext, type ScenarioSpec } from '../engine.ts'; +import { angularComponentSource, generateAngularProject } from '../generators/angular.ts'; +import type { EngineId, EngineMetrics } from '../types.ts'; + +const require = createRequire(import.meta.url); + +interface ResolvedCompodoc { + /** The CLI entry point, run as `node ` so no shell shim or exec bit is involved. */ + cli: string; + version: string; +} + +function resolveCompodoc(): ResolvedCompodoc | undefined { + try { + const packagePath = require.resolve('@compodoc/compodoc/package.json'); + const pkg = JSON.parse(fs.readFileSync(packagePath, 'utf8')); + return { + cli: path.join(path.dirname(packagePath), pkg.bin.compodoc), + version: pkg.version, + }; + } catch { + return undefined; + } +} + +interface CompodocRun extends DocumentationCounts { + durMs: number; + peakRssMb: number; +} + +function runCompodocOnce( + cli: string, + projectDir: string, + docsOutDir: string, + pollIntervalMs: number +): Promise { + fs.rmSync(docsOutDir, { recursive: true, force: true }); + const args = [cli, '-p', 'tsconfig.json', '-e', 'json', '-d', docsOutDir, '--silent']; + + return new Promise((resolve, reject) => { + const start = Date.now(); + const child = spawn(process.execPath, args, { cwd: projectDir }); + let peakRssKb = 0; + let output = ''; + child.stdout.on('data', (chunk: Buffer) => (output += chunk.toString())); + child.stderr.on('data', (chunk: Buffer) => (output += chunk.toString())); + + const poll = setInterval(() => { + if (child.pid === undefined) { + return; + } + + const ps = spawnSync('ps', ['-o', 'rss=', '-p', String(child.pid)], { encoding: 'utf8' }); + if (ps.error || typeof ps.stdout !== 'string') { + return; + } + const rssKb = Number(ps.stdout.trim()); + if (Number.isFinite(rssKb) && rssKb > peakRssKb) { + peakRssKb = rssKb; + } + }, pollIntervalMs); + + child.on('error', (err) => { + clearInterval(poll); + reject(err); + }); + child.on('close', (status) => { + clearInterval(poll); + const durMs = Date.now() - start; + if (status !== 0) { + reject(new Error(`compodoc exited with status ${status}:\n${outputTail(output, 8)}`)); + return; + } + const documentationJson = path.join(docsOutDir, 'documentation.json'); + if (!fs.existsSync(documentationJson)) { + reject(new Error('compodoc run produced no documentation.json')); + return; + } + // Counted before the next run overwrites the file, which both runs share. + const counts = countDocumentation(documentationJson); + // The polled peak misses spikes shorter than the interval; the recorded value is a floor. + resolve({ durMs, peakRssMb: peakRssKb / 1024, ...counts }); + }); + }); +} + +/** + * One repetition: fresh project, cold full run, touch one component, warm full run. Both runs are + * fresh compodoc processes, matching the one-sample-per-fresh-process topology. + */ +export async function runCompodocRepetition( + cli: string, + scenario: AngularScenarioConfig, + workDir: string, + pollIntervalMs: number +): Promise { + const projectDir = path.join(workDir, 'project'); + const docsOutDir = path.join(workDir, 'docs'); + const project = generateAngularProject({ + outDir: projectDir, + components: scenario.components, + props: scenario.props, + }); + + const cold = await runCompodocOnce(cli, project.outDir, docsOutDir, pollIntervalMs); + + // Touch one component so the warm run sees a genuinely changed file. + fs.writeFileSync(project.componentPaths[0], angularComponentSource(0, scenario.props + 1)); + const warm = await runCompodocOnce(cli, project.outDir, docsOutDir, pollIntervalMs); + + return { + coldMs: cold.durMs, + warmMs: warm.durMs, + peakRssMb: Math.max(cold.peakRssMb, warm.peakRssMb), + coldMembers: cold.members, + // A second whole-project pass, so this counts the project, not the one file that changed. It is + // not comparable with a series engine's warm count, which covers only the re-extracted member. + warmMembers: warm.members, + coldOpaqueTypes: cold.opaqueTypes, + }; +} + +export class CompodocEngine extends BenchEngine { + readonly id: EngineId = 'compodoc'; + + #resolved: ResolvedCompodoc | undefined; + + scenarios(profile: SuiteProfile): ScenarioSpec[] { + return [{ name: 'default', params: { ...profile.angular } }]; + } + + preflight(): string | undefined { + this.#resolved = resolveCompodoc(); + return this.#resolved + ? undefined + : '@compodoc/compodoc did not resolve; it is pinned in scripts/package.json, so run yarn install'; + } + + version(): string | undefined { + return this.#resolved?.version; + } + + async measure(ctx: MeasureContext, scenario: ScenarioSpec): Promise { + if (!this.#resolved) { + throw new Error('compodoc unresolved; preflight must run first'); + } + return runCompodocRepetition( + this.#resolved.cli, + { components: Number(scenario.params.components), props: Number(scenario.params.props) }, + ctx.scenarioDir, + ctx.rssPollIntervalMs + ); + } + + aggregate(samples: OneShotRepetition[], expectedN: number): EngineMetrics { + return oneShotMetrics(samples, expectedN); + } + + coldMembers(sample: OneShotRepetition): number | undefined { + return sample.coldMembers; + } + + warmMembers(sample: OneShotRepetition): number | undefined { + return sample.warmMembers; + } + + coldOpaqueTypes(sample: OneShotRepetition): number | undefined { + return sample.coldOpaqueTypes; + } +} diff --git a/scripts/bench/docgen-perf/engines/react-legacy.ts b/scripts/bench/docgen-perf/engines/react-legacy.ts new file mode 100644 index 000000000000..9a0415585c82 --- /dev/null +++ b/scripts/bench/docgen-perf/engines/react-legacy.ts @@ -0,0 +1,216 @@ +/** + * Series harness for the legacy React docgen engines: `react-docgen` (the budgeted legacy control) + * and `react-docgen-typescript` (measurable, no budget row). + * + * Both cache per file for the process lifetime and expose only global invalidation, so every save + * must invalidate before re-extracting or the sample is a cache hit. + * + * Do not run this child under jiti - react-docgen's browserslist dependency fails its JSON data + * require under that loader ("jsReleases.map is not a function"). + * + * Run: + * node --expose-gc scripts/bench/docgen-perf/engines/react-legacy.ts \ + * --parser react-docgen --components 300 --saves 20 --json /tmp/result.json + */ +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +import { z } from 'zod'; + +import { + type GeneratedProject, + componentRef, + componentSource, + generateProject, +} from '../../docgen-memory/generate-project.ts'; +import { countOption, parseHarnessOptions } from '../../docgen-shared/args.ts'; +import { SANDBOX_DIRECTORY } from '../../docgen-shared/paths.ts'; +import { + type ComponentRefLike, + loadReactRendererModule, +} from '../../docgen-shared/react-renderer-module.ts'; +import { + type SeriesEngine, + harnessMain, + runSeriesHarness, +} from '../../docgen-shared/series.ts'; + +/** The renderer module surfaces this harness loads, narrowed to what it calls. */ +interface UtilsModule { + invalidateCache(): void; +} + +interface ReactDocgenModule { + getReactDocgen( + path: string, + component: ComponentRefLike + ): { type: 'success' } | { type: 'error'; error: { name: string; message: string } }; +} + +interface ReactDocgenTypescriptModule { + parseWithReactDocgenTypescript(filePath: string): Promise>; + invalidateParser(): void; +} + +const PARSERS = ['react-docgen', 'react-docgen-typescript'] as const; +/** + * Which components to re-extract per save. + * all - re-extract every component, the shape `generator.ts` actually runs. + * changed - re-extract only the component whose file changed. + */ +const SCOPES = ['all', 'changed'] as const; + +const OPTIONS = { + parser: { type: 'string' }, + components: { type: 'string' }, + variants: { type: 'string' }, + props: { type: 'string' }, + saves: { type: 'string' }, + scope: { type: 'string' }, + out: { type: 'string' }, + json: { type: 'string' }, +} as const; + +const SCHEMA = z.object({ + parser: z.enum(PARSERS).default('react-docgen'), + components: countOption(300), + variants: countOption(4), + props: countOption(10), + saves: countOption(20), + scope: z.enum(SCOPES).default('changed'), + outDir: z + .string() + .default(path.join(SANDBOX_DIRECTORY, 'docgen-perf', 'react-legacy', 'project')), + jsonOut: z.string().optional(), +}); + +interface HarnessOptions { + parser: (typeof PARSERS)[number]; + components: number; + variants: number; + props: number; + saves: number; + scope: (typeof SCOPES)[number]; + outDir: string; + jsonOut?: string; +} + +function parseOptions(argv: string[]): HarnessOptions { + return parseHarnessOptions(argv, OPTIONS, SCHEMA, (values) => ({ + ...values, + outDir: values.out, + jsonOut: values.json, + })); +} + +interface Parser { + /** Extract `Comp{i}`. Throws on a failed or empty result, which is fast for the wrong reason. */ + extractOne(i: number): Promise; + /** Global invalidation is the only surface these engines expose. */ + invalidate(): void; +} + +async function loadParser(options: HarnessOptions, project: GeneratedProject): Promise { + const { componentPaths } = project; + const { invalidateCache } = await loadReactRendererModule('utils.ts'); + + if (options.parser === 'react-docgen') { + const { getReactDocgen } = await loadReactRendererModule('reactDocgen.ts'); + return { + async extractOne(i) { + const result = getReactDocgen(componentPaths[i], componentRef(i, componentPaths[i])); + if (result.type === 'error') { + throw new Error( + `react-docgen failed on Comp${i}: ${result.error.name} ${result.error.message}` + ); + } + }, + invalidate: invalidateCache, + }; + } + + const { parseWithReactDocgenTypescript, invalidateParser } = + await loadReactRendererModule('reactDocgenTypescript.ts'); + // The parser resolves its tsconfig from process.cwd(); point it at the generated project. + process.chdir(project.outDir); + return { + async extractOne(i) { + const docs = await parseWithReactDocgenTypescript(componentPaths[i]); + if (docs.length === 0) { + throw new Error(`react-docgen-typescript returned no docs for Comp${i}`); + } + }, + invalidate() { + invalidateParser(); + invalidateCache(); + }, + }; +} + +async function createEngine(options: HarnessOptions): Promise { + const genStart = Date.now(); + const project = generateProject({ + outDir: options.outDir, + components: options.components, + variants: options.variants, + props: options.props, + heavyTypes: false, + heavyFactor: 1, + base64Kb: 0, + withNodeModules: true, + }); + console.log(` generated project in ${Date.now() - genStart}ms at ${project.outDir}`); + + const { extractOne, invalidate } = await loadParser(options, project); + + const extractAll = async () => { + for (let i = 0; i < options.components; i++) { + await extractOne(i); + } + }; + + // Track how many extra props each component currently has, so each save grows its type. + const extraByComponent = new Array(options.components).fill(options.props); + const changedIndex = (save: number) => (save - 1) % options.components; + + return { + async cold() { + await extractAll(); + return undefined; + }, + async applySave(save) { + const i = changedIndex(save); + extraByComponent[i] += 1; + fs.writeFileSync(project.componentPaths[i], componentSource(i, extraByComponent[i])); + // Without this the re-extraction below is a cache hit, not a measurement. + invalidate(); + }, + async reextract(save) { + if (options.scope === 'all') { + await extractAll(); + } else { + await extractOne(changedIndex(save)); + } + return undefined; + }, + }; +} + +harnessMain(async () => { + const options = parseOptions(process.argv.slice(2)); + await runSeriesHarness({ + title: `react-legacy harness (${options.parser})`, + options, + banner: { + components: options.components, + variants: options.variants, + props: options.props, + saves: options.saves, + scope: options.scope, + }, + saves: options.saves, + coldLabel: `${options.components} components`, + jsonOut: options.jsonOut, + setup: () => createEngine(options), + }); +}); diff --git a/scripts/bench/docgen-perf/engines/vue-component-meta.ts b/scripts/bench/docgen-perf/engines/vue-component-meta.ts new file mode 100644 index 000000000000..678e3ffaf110 --- /dev/null +++ b/scripts/bench/docgen-perf/engines/vue-component-meta.ts @@ -0,0 +1,88 @@ +/** + * Series harness for the vue-component-meta engine - Vue's opt-in successor to vue-docgen-api. + * + * The checker never re-stats files on its own; a disk rewrite must be followed by + * `checker.updateFile(path, content)` or the re-extraction measures a stale-cache no-op. Checker + * creation runs inside the cold pass, since it is part of first-extraction cost: `flat` uses + * `createCheckerByJson`, the workspace scenarios use `createChecker` on the deepest package. + * + * Run: + * node --expose-gc scripts/bench/docgen-perf/engines/vue-component-meta.ts \ + * --scenario workspace --packages 4 --components-per-package 10 --heavy-lib --json /tmp/result.json + */ +import * as fs from 'node:fs'; + +import { type MetaCheckerOptions, createChecker, createCheckerByJson } from 'vue-component-meta'; + +import { type SeriesEngine, harnessMain, runSeriesHarness } from '../../docgen-shared/series.ts'; +import { + type VueHarnessOptions, + parseVueOptions, + setUpVueScenario, + vueBanner, +} from './vue-scenario.ts'; + +type Checker = ReturnType; + +/** Mirrors the production Vite plugin's checker options. */ +const CHECKER_OPTIONS: MetaCheckerOptions = { + forceUseTs: true, + noDeclarations: true, + printer: { newLine: 1 }, +}; + +function extractOne(checker: Checker, sfcPath: string): number { + const exportNames = checker.getExportNames(sfcPath); + if (!exportNames.includes('default')) { + throw new Error( + `no default export found in ${sfcPath} (got: ${exportNames.join(', ') || 'none'})` + ); + } + const meta = checker.getComponentMeta(sfcPath, 'default'); + if (meta.props.length === 0) { + throw new Error(`vue-component-meta returned zero props for ${sfcPath}`); + } + return meta.props.length + meta.events.length + meta.slots.length + meta.exposed.length; +} + +function createEngine(options: VueHarnessOptions): SeriesEngine { + const scenario = setUpVueScenario(options); + let checker: Checker | undefined; + let measuredPath = scenario.targetPaths[0]; + + return { + async cold() { + checker = + options.scenario === 'flat' + ? createCheckerByJson(scenario.project.outDir, { include: ['**/*'] }, CHECKER_OPTIONS) + : createChecker(scenario.project.packageConfigPaths[scenario.targetPackage], CHECKER_OPTIONS); + let members = 0; + for (const sfcPath of scenario.targetPaths) { + members += extractOne(checker, sfcPath); + } + return members; + }, + async applySave(save) { + const mutation = scenario.mutationFor(save); + fs.writeFileSync(mutation.filePath, mutation.content); + checker!.updateFile(mutation.filePath, mutation.content); + measuredPath = mutation.measuredPath; + }, + async reextract() { + return extractOne(checker!, measuredPath); + }, + }; +} + +harnessMain(async () => { + const options = parseVueOptions(process.argv.slice(2), 'vue-component-meta'); + await runSeriesHarness({ + title: `vue-component-meta harness (${options.scenario})`, + options, + banner: vueBanner(options), + saves: options.saves, + coldLabel: `${options.componentsPerPackage} components, checker creation included`, + jsonOut: options.jsonOut, + setup: async () => createEngine(options), + }); +}); diff --git a/scripts/bench/docgen-perf/engines/vue-docgen-api.ts b/scripts/bench/docgen-perf/engines/vue-docgen-api.ts new file mode 100644 index 000000000000..255c098a3827 --- /dev/null +++ b/scripts/bench/docgen-perf/engines/vue-docgen-api.ts @@ -0,0 +1,73 @@ +/** + * Series harness for the vue-docgen-api engine - Vue's legacy docgen and still the default + * (`resolveDocgenOptions` in `code/frameworks/vue3-vite/src/preset.ts`), the Vue equivalent of the + * react-docgen control. + * + * The parser reads from disk on every call and keeps no cache, so one save costs one `parse` and + * needs no invalidation step. + * + * Run: + * node --expose-gc scripts/bench/docgen-perf/engines/vue-docgen-api.ts \ + * --scenario workspace --packages 4 --components-per-package 10 --heavy-lib --json /tmp/result.json + */ +import * as fs from 'node:fs'; + +import { parse } from 'vue-docgen-api'; + +import { type SeriesEngine, harnessMain, runSeriesHarness } from '../../docgen-shared/series.ts'; +import { + type VueHarnessOptions, + parseVueOptions, + setUpVueScenario, + vueBanner, +} from './vue-scenario.ts'; + +/** + * Member count is reported alongside timing: a parse that resolved nothing is fast for the wrong + * reason. + */ +async function extractOne(sfcPath: string): Promise { + const doc = await parse(sfcPath); + return ( + Object.keys(doc.props ?? {}).length + + Object.keys(doc.events ?? {}).length + + Object.keys(doc.slots ?? {}).length + + Object.keys(doc.expose ?? {}).length + ); +} + +function createEngine(options: VueHarnessOptions): SeriesEngine { + const scenario = setUpVueScenario(options); + let measuredPath = scenario.targetPaths[0]; + + return { + async cold() { + let members = 0; + for (const sfcPath of scenario.targetPaths) { + members += await extractOne(sfcPath); + } + return members; + }, + async applySave(save) { + const mutation = scenario.mutationFor(save); + fs.writeFileSync(mutation.filePath, mutation.content); + measuredPath = mutation.measuredPath; + }, + async reextract() { + return extractOne(measuredPath); + }, + }; +} + +harnessMain(async () => { + const options = parseVueOptions(process.argv.slice(2), 'vue-docgen-api'); + await runSeriesHarness({ + title: `vue-docgen-api harness (${options.scenario})`, + options, + banner: vueBanner(options), + saves: options.saves, + coldLabel: `${options.componentsPerPackage} components`, + jsonOut: options.jsonOut, + setup: async () => createEngine(options), + }); +}); diff --git a/scripts/bench/docgen-perf/engines/vue-scenario.ts b/scripts/bench/docgen-perf/engines/vue-scenario.ts new file mode 100644 index 000000000000..e1d554c3211c --- /dev/null +++ b/scripts/bench/docgen-perf/engines/vue-scenario.ts @@ -0,0 +1,156 @@ +/** + * The scenario plumbing both Vue engine harnesses share: option parsing, project generation, the + * measured component set, and the per-save mutation. Both engines must see identical projects and + * touch the same files in the same order, or the ratio between their medians measures the generator + * instead of the engines. + * + * Scenarios: + * flat single package; the measured set is every component. + * workspace packages/* layout; the measured set is the deepest package's components, whose + * prop types pull the whole cross-package chain. + * base-type-touch workspace layout, but every save touches the widely-imported base type instead + * of a component, so each sample pays a wide invalidation. + */ +import * as path from 'node:path'; + +import { z } from 'zod'; + +import { countOption, parseHarnessOptions } from '../../docgen-shared/args.ts'; +import { SANDBOX_DIRECTORY } from '../../docgen-shared/paths.ts'; +import { + type GeneratedVueProject, + baseTypesSource, + generateVueProject, + vueComponentSource, +} from '../generators/vue.ts'; + +export const VUE_SCENARIOS = ['flat', 'workspace', 'base-type-touch'] as const; +export type VueScenario = (typeof VUE_SCENARIOS)[number]; + +const OPTIONS = { + scenario: { type: 'string' }, + packages: { type: 'string' }, + 'components-per-package': { type: 'string' }, + 'chain-depth': { type: 'string' }, + 'fan-out': { type: 'string' }, + 'heavy-lib': { type: 'boolean' }, + saves: { type: 'string' }, + out: { type: 'string' }, + json: { type: 'string' }, +} as const; + +/** `engineDirName` only supplies the default scratch directory, so it is bound per harness. */ +const schemaFor = (engineDirName: string) => + z.object({ + scenario: z.enum(VUE_SCENARIOS).default('workspace'), + packages: countOption(4), + componentsPerPackage: countOption(10), + chainDepth: countOption(3), + fanOut: countOption(4), + heavyLib: z.boolean().default(false), + saves: countOption(15), + outDir: z + .string() + .default(path.join(SANDBOX_DIRECTORY, 'docgen-perf', engineDirName, 'project')), + jsonOut: z.string().optional(), + }); + +export interface VueHarnessOptions { + scenario: VueScenario; + packages: number; + componentsPerPackage: number; + chainDepth: number; + fanOut: number; + heavyLib: boolean; + saves: number; + outDir: string; + jsonOut?: string; +} + +export function parseVueOptions(argv: string[], engineDirName: string): VueHarnessOptions { + return parseHarnessOptions(argv, OPTIONS, schemaFor(engineDirName), (values) => ({ + ...values, + outDir: values.out, + jsonOut: values.json, + })); +} + +export function vueBanner(options: VueHarnessOptions): Record { + return { + packages: effectivePackages(options), + componentsPerPackage: options.componentsPerPackage, + chainDepth: options.chainDepth, + fanOut: options.fanOut, + heavyLib: options.heavyLib, + saves: options.saves, + }; +} + +/** The `flat` scenario is a single-package layout whatever `--packages` says. */ +export function effectivePackages(options: VueHarnessOptions): number { + return options.scenario === 'flat' ? 1 : options.packages; +} + +/** One save's disk mutation: which file changes, and what it becomes. */ +export interface VueSaveMutation { + filePath: string; + content: string; + /** The component to re-extract afterwards. */ + measuredPath: string; +} + +export interface VueScenarioSetup { + project: GeneratedVueProject; + /** The components the cold pass extracts, and the saves rotate through. */ + targetPaths: string[]; + targetPackage: number; + /** The disk mutation for save `save`. Deterministic, so both engines apply the same one. */ + mutationFor(save: number): VueSaveMutation; +} + +export function setUpVueScenario(options: VueHarnessOptions): VueScenarioSetup { + const packages = effectivePackages(options); + + const genStart = Date.now(); + const project = generateVueProject({ + outDir: options.outDir, + packages, + componentsPerPackage: options.componentsPerPackage, + chainDepth: options.chainDepth, + fanOut: options.fanOut, + heavyLib: options.heavyLib, + }); + console.log(` generated project in ${Date.now() - genStart}ms at ${project.outDir}`); + + const targetPackage = packages - 1; + const targetPaths = + options.scenario === 'flat' + ? project.componentPaths + : project.componentPaths.slice(-options.componentsPerPackage); + + const extraByTarget = new Array(targetPaths.length).fill(0); + let extraBaseProps = 0; + + return { + project, + targetPaths, + targetPackage, + mutationFor(save) { + if (options.scenario === 'base-type-touch') { + extraBaseProps += 1; + return { + filePath: project.baseTypesPath, + content: baseTypesSource(options.fanOut, extraBaseProps), + measuredPath: targetPaths[0], + }; + } + const t = (save - 1) % targetPaths.length; + extraByTarget[t] += 1; + return { + filePath: targetPaths[t], + content: vueComponentSource(targetPackage, t, extraByTarget[t]), + measuredPath: targetPaths[t], + }; + }, + }; +} diff --git a/scripts/bench/docgen-perf/generators/angular.ts b/scripts/bench/docgen-perf/generators/angular.ts new file mode 100644 index 000000000000..d56351959acd --- /dev/null +++ b/scripts/bench/docgen-perf/generators/angular.ts @@ -0,0 +1,142 @@ +/** + * Generator for a synthetic Angular project consumable by a standalone Compodoc CLI run. + * + * Ships a minimal fake `(at)angular/core` type surface inside its own node_modules so the tree is + * hermetic - no npm install of the real framework is needed for type resolution. + * + * Run directly: + * node scripts/bench/docgen-perf/generators/angular.ts --out ../storybook-sandboxes/docgen-perf-angular --components 100 + */ +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { pathToFileURL } from 'node:url'; + +import { z } from 'zod'; + +import { countOption, parseHarnessOptions } from '../../docgen-shared/args.ts'; + +export interface AngularGenerateOptions { + outDir: string; + components: number; + /** Extra `@Input()` members per component on top of the fixed baseline set. */ + props: number; +} + +export interface GeneratedAngularProject { + outDir: string; + /** Absolute component file paths, in component order. */ + componentPaths: string[]; +} + +const FAKE_ANGULAR_CORE = `export declare function Component(metadata: { + selector?: string; + template?: string; + standalone?: boolean; +}): ClassDecorator; +export declare function Input(bindingPropertyName?: string): PropertyDecorator; +export declare function Output(bindingPropertyName?: string): PropertyDecorator; +export declare class EventEmitter { + emit(value?: T): void; + subscribe(next: (value: T) => void): { unsubscribe(): void }; +} +`; + +function emitFakeAngularCore(projectDir: string): void { + const dir = path.join(projectDir, 'node_modules', '@angular', 'core'); + fs.mkdirSync(dir, { recursive: true }); + fs.writeFileSync( + path.join(dir, 'package.json'), + JSON.stringify({ name: '@angular/core', version: '0.0.0-bench', types: 'index.d.ts' }, null, 2) + ); + fs.writeFileSync(path.join(dir, 'index.d.ts'), FAKE_ANGULAR_CORE); +} + +/** + * `extraProps` grows the input surface by one per warm-run touch, so the second Compodoc run sees + * a genuinely changed file. + */ +export function angularComponentSource(i: number, extraProps: number): string { + const extras = Array.from( + { length: extraProps }, + (_, p) => ` /** Extra input ${p} for component ${i}. */ + @Input() extra${p}?: ${p % 3 === 0 ? `'a' | 'b' | 'c'` : p % 3 === 1 ? 'number' : 'string'};` + ).join('\n'); + + return `import { Component, EventEmitter, Input, Output } from '@angular/core'; + +/** + * Comp${i} - generated bench component. + */ +@Component({ + selector: 'bench-comp${i}', + template: '
{{ label }}
', +}) +export class Comp${i}Component { + /** Primary label shown to the user. */ + @Input() label = ''; + /** Numeric size token. */ + @Input() size?: number; + /** Visual variant. */ + @Input() variant?: 'primary' | 'secondary' | 'tertiary'; + /** Disable interaction. */ + @Input() disabled = false; +${extras ? `${extras}\n` : ''} /** Emits when the user acts on the component. */ + @Output() action = new EventEmitter<{ id: string; value: number }>(); +} +`; +} + +const TSCONFIG = { + compilerOptions: { + target: 'ES2020', + module: 'ESNext', + moduleResolution: 'Bundler', + strict: true, + skipLibCheck: true, + experimentalDecorators: true, + emitDecoratorMetadata: false, + }, + include: ['src/**/*.ts'], +}; + +export function generateAngularProject(options: AngularGenerateOptions): GeneratedAngularProject { + const outDir = path.resolve(options.outDir); + fs.rmSync(outDir, { recursive: true, force: true }); + fs.mkdirSync(path.join(outDir, 'src', 'app'), { recursive: true }); + + emitFakeAngularCore(outDir); + + // The compodoc adapter passes `-p tsconfig.json` with the project dir as cwd. + fs.writeFileSync(path.join(outDir, 'tsconfig.json'), JSON.stringify(TSCONFIG, null, 2)); + + const componentPaths: string[] = []; + for (let i = 0; i < options.components; i++) { + const componentPath = path.join(outDir, 'src', 'app', `comp${i}.component.ts`); + fs.writeFileSync(componentPath, angularComponentSource(i, options.props)); + componentPaths.push(componentPath); + } + + return { outDir, componentPaths }; +} + +function parseOptions(argv: string[]): AngularGenerateOptions { + return parseHarnessOptions( + argv, + { out: { type: 'string' }, components: { type: 'string' }, props: { type: 'string' } } as const, + z.object({ + outDir: z.string().default('../storybook-sandboxes/docgen-perf-angular'), + components: countOption(100), + props: countOption(8), + }), + (values) => ({ ...values, outDir: values.out }) + ); +} + +if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) { + const options = parseOptions(process.argv.slice(2)); + const start = Date.now(); + const result = generateAngularProject(options); + console.log( + `Generated ${options.components} Angular components into ${result.outDir} in ${Date.now() - start}ms` + ); +} diff --git a/scripts/bench/docgen-perf/generators/vue.ts b/scripts/bench/docgen-perf/generators/vue.ts new file mode 100644 index 000000000000..730e6bae6017 --- /dev/null +++ b/scripts/bench/docgen-perf/generators/vue.ts @@ -0,0 +1,299 @@ +/** + * Generator for a synthetic Vue 3 project shaped like the monorepos where vue-component-meta + * struggles: a `packages/*` workspace with per-package tsconfigs (`extends` + `paths` aliases, root + * `references`), cross-package prop-type import chains, and an optional fake heavy `.d.ts` library + * inside the generated tree's own node_modules (hermetic - no npm install). + * + * Structure levers: + * - packages: workspace width. Each `packages/pkg{p}` extends the previous package's props type, + * so the type chain across packages has depth = packages. `packages/types` holds the shared + * base type every package extends - the type the base-type-touch scenario mutates. + * - chainDepth: intra-package type-alias hops between a package's props interface and its parent. + * - fanOut: auxiliary types from `packages/types` referenced by every package's props. + * - heavyLib: emits a fake `(at)bench/heavy-ui` package with a large `.d.ts` surface; package + * props reference it when enabled. + * + * Run directly: + * node scripts/bench/docgen-perf/generators/vue.ts --out ../storybook-sandboxes/docgen-perf-vue --packages 4 --components-per-package 10 + */ +import { createRequire } from 'node:module'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { pathToFileURL } from 'node:url'; + +import { z } from 'zod'; + +import { countOption, parseHarnessOptions } from '../../docgen-shared/args.ts'; + +const require = createRequire(import.meta.url); + +export interface VueGenerateOptions { + outDir: string; + /** Workspace packages (excluding the shared `types` package). 1 = flat single-package layout. */ + packages: number; + componentsPerPackage: number; + /** Type-alias hops between each package's props interface and the type it extends. */ + chainDepth: number; + /** Auxiliary types from the shared types package referenced by every package's props. */ + fanOut: number; + /** Emit and reference the fake heavy `.d.ts` library. */ + heavyLib: boolean; +} + +export interface GeneratedVueProject { + outDir: string; + /** Per-package tsconfig paths, in package order. */ + packageConfigPaths: string[]; + /** Path of the shared base-type module (the base-type-touch scenario's save target). */ + baseTypesPath: string; + /** Absolute SFC paths, in (package, component) order. */ + componentPaths: string[]; +} + +/** Components in the fake heavy library; each carries a fat literal-union prop surface. */ +const HEAVY_LIB_COMPONENTS = 40; +const HEAVY_LIB_PROPS = 20; +const HEAVY_LIB_UNION_MEMBERS = 30; + +/** Copy the minimal node_modules packages the checker needs for Vue type resolution. */ +function copyVueNodeModules(projectDir: string): void { + const dest = path.join(projectDir, 'node_modules'); + const rootNodeModules = path.resolve(require.resolve('vue/package.json'), '../..'); + fs.mkdirSync(path.join(dest, '@vue'), { recursive: true }); + + for (const pkg of ['vue', 'csstype']) { + fs.cpSync(path.join(rootNodeModules, pkg), path.join(dest, pkg), { + recursive: true, + dereference: true, + }); + } + for (const pkg of ['runtime-core', 'runtime-dom', 'shared', 'reactivity', 'compiler-dom']) { + fs.cpSync(path.join(rootNodeModules, '@vue', pkg), path.join(dest, '@vue', pkg), { + recursive: true, + dereference: true, + }); + } +} + +function heavyLibSource(): string { + const components: string[] = []; + for (let c = 0; c < HEAVY_LIB_COMPONENTS; c++) { + const props: string[] = []; + for (let p = 0; p < HEAVY_LIB_PROPS; p++) { + const members = Array.from( + { length: HEAVY_LIB_UNION_MEMBERS }, + (_, m) => `'hc${c}_p${p}_v${m}'` + ).join(' | '); + props.push(` prop${p}?: ${members};`); + } + components.push( + `export interface HeavyComponent${c}Props {\n${props.join('\n')}\n}\n` + + `export declare const HeavyComponent${c}: (props: HeavyComponent${c}Props) => unknown;` + ); + } + return `${components.join('\n\n')}\n`; +} + +function emitHeavyLib(projectDir: string): void { + const libDir = path.join(projectDir, 'node_modules', '@bench', 'heavy-ui'); + fs.mkdirSync(libDir, { recursive: true }); + fs.writeFileSync( + path.join(libDir, 'package.json'), + JSON.stringify({ name: '@bench/heavy-ui', version: '1.0.0', types: 'index.d.ts' }, null, 2) + ); + fs.writeFileSync(path.join(libDir, 'index.d.ts'), heavyLibSource()); +} + +/** + * The shared types package's module source. `extraBaseProps` grows `BaseProps` by one optional + * prop per base-type-touch save, so every dependent package's props type genuinely changes. + */ +export function baseTypesSource(fanOut: number, extraBaseProps: number): string { + const extras = Array.from( + { length: extraBaseProps }, + (_, k) => ` /** Extra base prop ${k}. */\n extraBase${k}?: string;` + ).join('\n'); + const auxTypes = Array.from( + { length: fanOut }, + (_, k) => `export interface Aux${k} {\n aux${k}Value: string;\n aux${k}Count?: number;\n}` + ).join('\n\n'); + return `export interface BaseProps { + /** Stable identifier. */ + id: string; + /** Semantic kind token. */ + kind?: 'alpha' | 'beta' | 'gamma'; + /** Arbitrary metadata bag. */ + meta?: Record; +${extras ? `${extras}\n` : ''}} + +${auxTypes} +`; +} + +function packageTypesSource(p: number, options: VueGenerateOptions): string { + const auxNames = Array.from({ length: options.fanOut }, (_, k) => `Aux${k}`); + const parentImport = + p === 0 + ? `import type { BaseProps, ${auxNames.join(', ')} } from '@bench/types';` + : `import type { Pkg${p - 1}Props } from '@bench/pkg${p - 1}';\nimport type { ${auxNames.join(', ')} } from '@bench/types';`; + const heavyImport = options.heavyLib + ? `import type { HeavyComponent${p % HEAVY_LIB_COMPONENTS}Props } from '@bench/heavy-ui';\n` + : ''; + const parent = p === 0 ? 'BaseProps' : `Pkg${p - 1}Props`; + + const hops: string[] = []; + let current = parent; + for (let h = 0; h < options.chainDepth - 1; h++) { + hops.push(`type Hop${h} = ${current};`); + current = `Hop${h}`; + } + + const fanProps = auxNames + .map((aux, k) => ` /** Fan-out reference ${k}. */\n pkg${p}Fan${k}?: ${aux};`) + .join('\n'); + const heavyProp = options.heavyLib + ? `\n /** Heavy library surface reference. */\n pkg${p}Heavy?: HeavyComponent${p % HEAVY_LIB_COMPONENTS}Props;` + : ''; + + return `${parentImport} +${heavyImport} +${hops.length ? `${hops.join('\n')}\n` : ''} +export interface Pkg${p}Props extends ${current} { +${fanProps}${heavyProp} +} +`; +} + +/** + * An SFC's source. `extraProps` grows the intersection type by one optional prop per save, so a + * simulated save genuinely changes the component's type. + */ +export function vueComponentSource(p: number, i: number, extraProps: number): string { + const extras = Array.from({ length: extraProps }, (_, k) => ` extra${k}?: string;`).join(''); + const propsType = extras ? `Pkg${p}Props & {${extras} }` : `Pkg${p}Props`; + return ` + + +`; +} + +const BASE_COMPILER_OPTIONS = { + target: 'ESNext', + module: 'ESNext', + moduleResolution: 'Bundler', + strict: true, + skipLibCheck: true, + jsx: 'preserve', + lib: ['ESNext', 'DOM'], +}; + +export function generateVueProject(options: VueGenerateOptions): GeneratedVueProject { + const outDir = path.resolve(options.outDir); + fs.rmSync(outDir, { recursive: true, force: true }); + fs.mkdirSync(outDir, { recursive: true }); + + copyVueNodeModules(outDir); + if (options.heavyLib) { + emitHeavyLib(outDir); + } + + const typesDir = path.join(outDir, 'packages', 'types'); + fs.mkdirSync(typesDir, { recursive: true }); + const baseTypesPath = path.join(typesDir, 'index.ts'); + fs.writeFileSync(baseTypesPath, baseTypesSource(options.fanOut, 0)); + + const paths: Record = { '@bench/types': ['packages/types/index.ts'] }; + for (let p = 0; p < options.packages; p++) { + paths[`@bench/pkg${p}`] = [`packages/pkg${p}/types.ts`]; + } + fs.writeFileSync( + path.join(outDir, 'tsconfig.base.json'), + JSON.stringify({ compilerOptions: { ...BASE_COMPILER_OPTIONS, baseUrl: '.', paths } }, null, 2) + ); + + const componentPaths: string[] = []; + const packageConfigPaths: string[] = []; + + for (let p = 0; p < options.packages; p++) { + const pkgDir = path.join(outDir, 'packages', `pkg${p}`); + fs.mkdirSync(path.join(pkgDir, 'src'), { recursive: true }); + fs.writeFileSync(path.join(pkgDir, 'types.ts'), packageTypesSource(p, options)); + + const pkgConfigPath = path.join(pkgDir, 'tsconfig.json'); + fs.writeFileSync( + pkgConfigPath, + JSON.stringify( + { + extends: '../../tsconfig.base.json', + include: ['types.ts', 'src/**/*.ts', 'src/**/*.vue'], + }, + null, + 2 + ) + ); + packageConfigPaths.push(pkgConfigPath); + + for (let i = 0; i < options.componentsPerPackage; i++) { + const componentPath = path.join(pkgDir, 'src', `Comp${p}x${i}.vue`); + fs.writeFileSync(componentPath, vueComponentSource(p, i, 0)); + componentPaths.push(componentPath); + } + } + + // The root config declares `references` but no package sets `composite`, so `tsc -b` would reject + // this tree. It is here because the production Vite plugin branches on `references` being present. + fs.writeFileSync( + path.join(outDir, 'tsconfig.json'), + JSON.stringify( + { + files: [], + references: Array.from({ length: options.packages }, (_, p) => ({ + path: `./packages/pkg${p}`, + })), + }, + null, + 2 + ) + ); + + return { outDir, packageConfigPaths, baseTypesPath, componentPaths }; +} + +function parseOptions(argv: string[]): VueGenerateOptions { + return parseHarnessOptions( + argv, + { + out: { type: 'string' }, + packages: { type: 'string' }, + 'components-per-package': { type: 'string' }, + 'chain-depth': { type: 'string' }, + 'fan-out': { type: 'string' }, + 'heavy-lib': { type: 'boolean' }, + } as const, + z.object({ + outDir: z.string().default('../storybook-sandboxes/docgen-perf-vue'), + packages: countOption(4), + componentsPerPackage: countOption(10), + chainDepth: countOption(3), + fanOut: countOption(4), + heavyLib: z.boolean().default(false), + }), + (values) => ({ ...values, outDir: values.out }) + ); +} + +if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) { + const options = parseOptions(process.argv.slice(2)); + const start = Date.now(); + const result = generateVueProject(options); + console.log( + `Generated ${options.packages}×${options.componentsPerPackage} Vue components ` + + `into ${result.outDir} in ${Date.now() - start}ms` + ); +} diff --git a/scripts/bench/docgen-perf/ratios.test.ts b/scripts/bench/docgen-perf/ratios.test.ts new file mode 100644 index 000000000000..0b1d3106bb8c --- /dev/null +++ b/scripts/bench/docgen-perf/ratios.test.ts @@ -0,0 +1,132 @@ +import { describe, expect, it } from 'vitest'; + +import { CONTROL_PAIRS, computeRatios, engineOrderForRep } from './ratios.ts'; +import { type EngineId, type EngineResult, NOT_APPLICABLE, type ScenarioResult } from './types.ts'; + +function scenario(cold: number, warm: number, members?: [number, number]): ScenarioResult { + return { + params: {}, + coldMembers: members?.[0], + warmMembers: members?.[1], + metrics: { + coldExtractionMs: { status: 'measured', samples: [cold], median: cold }, + warmExtractionMs: { status: 'measured', samples: [warm], median: warm }, + wholeProjectScanMs: NOT_APPLICABLE, + peakTransientMb: NOT_APPLICABLE, + retainedGrowthMb: NOT_APPLICABLE, + retainedSlopeMbPerSave: NOT_APPLICABLE, + }, + }; +} + +function measured(scenarios: Record): EngineResult { + return { status: 'measured', scenarios }; +} + +describe('engineOrderForRep', () => { + const engines: EngineId[] = ['react-legacy', 'react-osa', 'vue-docgen-api', 'vue-component-meta']; + + it('keeps the listed order on odd repetitions', () => { + expect(engineOrderForRep(engines, 1)).toEqual(engines); + expect(engineOrderForRep(engines, 3)).toEqual(engines); + }); + + it('swaps every control pair on even repetitions', () => { + expect(engineOrderForRep(engines, 2)).toEqual([ + 'react-osa', + 'react-legacy', + 'vue-component-meta', + 'vue-docgen-api', + ]); + }); + + it('leaves a pair alone when only one of its sides is running', () => { + expect(engineOrderForRep(['react-legacy', 'compodoc'], 2)).toEqual(['react-legacy', 'compodoc']); + }); + + it('does not mutate its input', () => { + const input: EngineId[] = ['react-legacy', 'react-osa']; + engineOrderForRep(input, 2); + expect(input).toEqual(['react-legacy', 'react-osa']); + }); + + it('gives each side of a pair the first slot at least once across the pinned five', () => { + const firsts = [1, 2, 3, 4, 5].map((rep) => engineOrderForRep(engines, rep)[0]); + expect(new Set(firsts)).toEqual(new Set(['react-legacy', 'react-osa'])); + }); +}); + +describe('computeRatios', () => { + it('divides the legacy median by the new engine median', () => { + const ratios = computeRatios({ + 'react-legacy': measured({ default: scenario(400, 40) }), + 'react-osa': measured({ default: scenario(100, 20) }), + }); + expect(ratios.react.default.cold).toBe(4); + expect(ratios.react.default.warm).toBe(2); + }); + + it('produces nothing when one side did not measure', () => { + expect( + computeRatios({ + 'react-legacy': measured({ default: scenario(400, 40) }), + 'react-osa': { status: 'failed', reason: 'boom' }, + }) + ).toEqual({}); + }); + + it('produces nothing when one side was skipped', () => { + expect( + computeRatios({ + 'vue-docgen-api': measured({ flat: scenario(20, 2) }), + 'vue-component-meta': { status: 'skipped', reason: 'not installed' }, + }) + ).toEqual({}); + }); + + it('pairs scenarios by name and ignores ones only one side ran', () => { + const ratios = computeRatios({ + 'vue-docgen-api': measured({ flat: scenario(20, 2), workspace: scenario(30, 3) }), + 'vue-component-meta': measured({ flat: scenario(400, 40) }), + }); + expect(Object.keys(ratios.vue)).toEqual(['flat']); + }); + + it('marks a pair like-for-like when both documented the same members', () => { + const ratios = computeRatios({ + 'vue-docgen-api': measured({ flat: scenario(20, 2, [50, 5]) }), + 'vue-component-meta': measured({ flat: scenario(400, 40, [50, 5]) }), + }); + expect(ratios.vue.flat.likeForLike).toBe(true); + }); + + it('marks a pair not like-for-like when the member counts disagree', () => { + // vue-docgen-api does not resolve tsconfig `paths` aliases, so it documents a fraction of what + // vue-component-meta does and is fast for the wrong reason. + const ratios = computeRatios({ + 'vue-docgen-api': measured({ flat: scenario(23, 1, [6, 0]) }), + 'vue-component-meta': measured({ flat: scenario(425, 40, [320, 32]) }), + }); + expect(ratios.vue.flat).toMatchObject({ + likeForLike: false, + legacyColdMembers: 6, + nextColdMembers: 320, + legacyWarmMembers: 0, + nextWarmMembers: 32, + }); + }); + + it('leaves like-for-like unknown when the engines report no member counts', () => { + // Unknown is not the same claim as unequal; only an explicit false earns a warning. + const ratios = computeRatios({ + 'react-legacy': measured({ default: scenario(400, 40) }), + 'react-osa': measured({ default: scenario(100, 20) }), + }); + expect(ratios.react.default.likeForLike).toBeUndefined(); + }); + + it('names every pair it knows uniquely', () => { + const names = CONTROL_PAIRS.map((p) => p.name); + expect(new Set(names).size).toBe(names.length); + }); +}); diff --git a/scripts/bench/docgen-perf/ratios.ts b/scripts/bench/docgen-perf/ratios.ts new file mode 100644 index 000000000000..5b461eefe5de --- /dev/null +++ b/scripts/bench/docgen-perf/ratios.ts @@ -0,0 +1,106 @@ +/** + * The calibration references: each framework's legacy-engine median divided by its new-engine + * median, both measured in the same invocation. Ratios stand in for absolute milliseconds because + * wall-clock on shared CI executors is too noisy to gate (PERF-METHODOLOGY.md, "Budget shape"). + */ +import type { + EngineId, + EngineResult, + LatencyMetric, + NotApplicable, + RatioEntry, + Ratios, + ScenarioResult, +} from './types.ts'; + +interface ControlPair { + /** Key under which this pair's ratios appear in the results. */ + name: string; + legacy: EngineId; + next: EngineId; +} + +export const CONTROL_PAIRS: ControlPair[] = [ + { name: 'react', legacy: 'react-legacy', next: 'react-osa' }, + { name: 'vue', legacy: 'vue-docgen-api', next: 'vue-component-meta' }, +]; + +/** + * Each control pair swaps sides on even repetitions so cache warming and thermal drift do not + * consistently favour whichever engine is listed first. + */ +export function engineOrderForRep(engines: EngineId[], rep: number): EngineId[] { + const order = [...engines]; + if (rep % 2 !== 0) { + return order; + } + for (const { legacy, next } of CONTROL_PAIRS) { + const legacyIdx = order.indexOf(legacy); + const nextIdx = order.indexOf(next); + if (legacyIdx >= 0 && nextIdx >= 0) { + order[legacyIdx] = next; + order[nextIdx] = legacy; + } + } + return order; +} + +/** Undefined unless both sides measured: dividing by a skipped or failed side is not a comparison. */ +function medianRatio(legacy: LatencyMetric | NotApplicable, next: LatencyMetric | NotApplicable) { + return legacy.status === 'measured' && next.status === 'measured' + ? legacy.median / next.median + : undefined; +} + +/** + * Undefined, not false, when an engine reports no member counts - see renderRatios for why that + * distinction matters. + */ +function likeForLike(legacy: ScenarioResult, next: ScenarioResult): boolean | undefined { + if (legacy.coldMembers === undefined || next.coldMembers === undefined) { + return undefined; + } + const warmAgrees = + legacy.warmMembers === undefined || + next.warmMembers === undefined || + legacy.warmMembers === next.warmMembers; + return legacy.coldMembers === next.coldMembers && warmAgrees; +} + +function ratioFor(legacy: ScenarioResult, next: ScenarioResult): RatioEntry { + return { + cold: medianRatio(legacy.metrics.coldExtractionMs, next.metrics.coldExtractionMs), + warm: medianRatio(legacy.metrics.warmExtractionMs, next.metrics.warmExtractionMs), + legacyColdMembers: legacy.coldMembers, + nextColdMembers: next.coldMembers, + legacyWarmMembers: legacy.warmMembers, + nextWarmMembers: next.warmMembers, + likeForLike: likeForLike(legacy, next), + }; +} + +/** + * Ratios for every control pair whose two engines both measured in this invocation. A pair with one + * failed or skipped side yields nothing: dividing a fresh median by a stale one is not a comparison. + */ +export function computeRatios(results: Partial>): Ratios { + const ratios: Ratios = {}; + + for (const pair of CONTROL_PAIRS) { + const legacy = results[pair.legacy]; + const next = results[pair.next]; + if (legacy?.status !== 'measured' || next?.status !== 'measured') { + continue; + } + for (const [scenarioName, legacyScenario] of Object.entries(legacy.scenarios)) { + const nextScenario = next.scenarios[scenarioName]; + if (!nextScenario) { + continue; + } + ratios[pair.name] ??= {}; + ratios[pair.name][scenarioName] = ratioFor(legacyScenario, nextScenario); + } + } + + return ratios; +} diff --git a/scripts/bench/docgen-perf/registry.test.ts b/scripts/bench/docgen-perf/registry.test.ts new file mode 100644 index 000000000000..e46092d1e095 --- /dev/null +++ b/scripts/bench/docgen-perf/registry.test.ts @@ -0,0 +1,123 @@ +import { describe, expect, it } from 'vitest'; + +import type { SeriesResult } from '../docgen-shared/series.ts'; +import { QUICK_PROFILE } from './config.ts'; +import type { MeasureContext, ScenarioSpec } from './engine.ts'; +import { CONTROL_PAIRS } from './ratios.ts'; +import { ALL_ENGINE_IDS, DEFAULT_ENGINE_IDS, ENGINES, engineById } from './registry.ts'; +import type { EngineId } from './types.ts'; + +/** Captures what an engine would have spawned, without spawning it. */ +function captureSpawn() { + const calls: Array<{ spec: Parameters[0]; jsonPath: string }> = + []; + const ctx: MeasureContext = { + scenarioDir: '/tmp/scenario', + rssPollIntervalMs: 100, + runSeriesChild(spec, _outDir, jsonPath) { + calls.push({ spec, jsonPath }); + return {} as SeriesResult; + }, + }; + return { ctx, calls }; +} + +async function specFor(id: EngineId, scenario: ScenarioSpec, rep = 1) { + const { ctx, calls } = captureSpawn(); + await engineById(id).measure(ctx, scenario, rep); + return calls[0]; +} + +const reactScenario = engineById('react-legacy').scenarios(QUICK_PROFILE)[0]; + +describe('the engine table', () => { + it('registers each id exactly once', () => { + expect(new Set(ALL_ENGINE_IDS).size).toBe(ALL_ENGINE_IDS.length); + }); + + it('keeps the unbudgeted react-docgen-typescript engine out of the default run', () => { + expect(DEFAULT_ENGINE_IDS).not.toContain('react-legacy-rdt'); + expect(ALL_ENGINE_IDS).toContain('react-legacy-rdt'); + }); + + it('names an engine that is not registered', () => { + expect(() => engineById('svelte' as EngineId)).toThrow('no engine registered for "svelte"'); + }); + + it('gives every engine at least one scenario, with unique names', () => { + for (const engine of ENGINES) { + const names = engine.scenarios(QUICK_PROFILE).map((s) => s.name); + expect(names.length, engine.id).toBeGreaterThan(0); + expect(new Set(names).size, engine.id).toBe(names.length); + } + }); + + it('registers both sides of every control pair', () => { + // A pair naming an unregistered engine would silently never produce a ratio. + for (const pair of CONTROL_PAIRS) { + expect(ALL_ENGINE_IDS, pair.name).toContain(pair.legacy); + expect(ALL_ENGINE_IDS, pair.name).toContain(pair.next); + } + }); + + it('runs both sides of every control pair by default', () => { + // A ratio needs both sides measured in the same invocation. + for (const pair of CONTROL_PAIRS) { + expect(DEFAULT_ENGINE_IDS, pair.name).toContain(pair.legacy); + expect(DEFAULT_ENGINE_IDS, pair.name).toContain(pair.next); + } + }); +}); + +describe('what the series engines spawn', () => { + it('runs only the reused memory harness under jiti', async () => { + // Under jiti, react-docgen's browserslist dependency fails its JSON data require, so the + // legacy child must run on native type stripping. + expect((await specFor('react-osa', reactScenario)).spec.jiti).toBe(true); + expect((await specFor('react-legacy', reactScenario)).spec.jiti).toBeFalsy(); + expect((await specFor('react-legacy-rdt', reactScenario)).spec.jiti).toBeFalsy(); + }); + + it('sends the two React legacy engines to one child with different parsers', async () => { + const legacy = await specFor('react-legacy', reactScenario); + const rdt = await specFor('react-legacy-rdt', reactScenario); + expect(rdt.spec.childPath).toBe(legacy.spec.childPath); + expect(legacy.spec.args).toContain('react-docgen'); + expect(rdt.spec.args).toContain('react-docgen-typescript'); + }); + + it('measures both React engines on one changed component per save', async () => { + // Equal work on both sides is what makes the React ratio an engine comparison. + for (const id of ['react-legacy', 'react-osa'] as EngineId[]) { + const { spec } = await specFor(id, reactScenario); + expect(spec.args.slice(spec.args.indexOf('--scope')), id).toEqual(['--scope', 'changed']); + } + }); + + it('passes the scenario size through to the child', async () => { + const { spec } = await specFor('react-legacy', reactScenario); + const components = spec.args[spec.args.indexOf('--components') + 1]; + expect(components).toBe(String(QUICK_PROFILE.react.components)); + }); + + it('passes --heavy-lib only for scenarios that ask for it', async () => { + const scenarios = engineById('vue-component-meta').scenarios(QUICK_PROFILE); + const flat = scenarios.find((s) => s.name === 'flat')!; + const workspace = scenarios.find((s) => s.name === 'workspace')!; + expect((await specFor('vue-component-meta', flat)).spec.args).not.toContain('--heavy-lib'); + expect((await specFor('vue-component-meta', workspace)).spec.args).toContain('--heavy-lib'); + }); + + it('gives the two Vue engines identical flags, so the ratio compares engines not projects', async () => { + for (const scenario of engineById('vue-docgen-api').scenarios(QUICK_PROFILE)) { + const legacy = await specFor('vue-docgen-api', scenario); + const meta = await specFor('vue-component-meta', scenario); + expect(meta.spec.args, scenario.name).toEqual(legacy.spec.args); + expect(meta.spec.childPath).not.toBe(legacy.spec.childPath); + } + }); + + it('writes each repetition to its own result file', async () => { + expect((await specFor('react-legacy', reactScenario, 3)).jsonPath).toMatch(/rep3\.json$/); + }); +}); diff --git a/scripts/bench/docgen-perf/registry.ts b/scripts/bench/docgen-perf/registry.ts new file mode 100644 index 000000000000..b41c8e53ed49 --- /dev/null +++ b/scripts/bench/docgen-perf/registry.ts @@ -0,0 +1,87 @@ +/** + * The engine table. Adding an engine means adding one entry here; the orchestrator has no + * per-engine branches. Everything below is data: which child to spawn, with which flags, over + * which scenarios. + */ +import type { SuiteProfile } from './config.ts'; +import { type BenchEngine, type ScenarioSpec, SeriesChildEngine } from './engine.ts'; +import { CompodocEngine } from './engines/compodoc.ts'; +import type { EngineId } from './types.ts'; + +const reactScenarios = (profile: SuiteProfile): ScenarioSpec[] => [ + { name: 'default', params: { ...profile.react } }, +]; + +const vueScenarios = (profile: SuiteProfile): ScenarioSpec[] => + profile.vue.map((scenario) => ({ name: scenario.name, params: { ...scenario } })); + +const reactArgs = ({ params }: ScenarioSpec): string[] => [ + '--components', String(params.components), + '--variants', String(params.variants), + '--props', String(params.props), + '--saves', String(params.saves), +]; + +const vueArgs = ({ params }: ScenarioSpec): string[] => [ + '--scenario', String(params.name), + '--packages', String(params.packages), + '--components-per-package', String(params.componentsPerPackage), + '--chain-depth', String(params.chainDepth), + '--fan-out', String(params.fanOut), + ...(params.heavyLib ? ['--heavy-lib'] : []), + '--saves', String(params.saves), +]; + +export const ENGINES: BenchEngine[] = [ + // --scope changed on both sides of the React pair compares the engines on equal work; that is + // not what a real legacy save costs - see the calibration caveat in PERF-METHODOLOGY.md. + new SeriesChildEngine({ + id: 'react-legacy', + child: 'engines/react-legacy.ts', + scenarios: reactScenarios, + args: (scenario) => [...reactArgs(scenario), '--parser', 'react-docgen', '--scope', 'changed'], + }), + // Measurable through the same child, but it carries no budget row, so it stays out by default. + new SeriesChildEngine({ + id: 'react-legacy-rdt', + child: 'engines/react-legacy.ts', + scenarios: reactScenarios, + inDefaultRun: false, + args: (scenario) => [ + ...reactArgs(scenario), + '--parser', 'react-docgen-typescript', + '--scope', 'changed', + ], + }), + new SeriesChildEngine({ + id: 'react-osa', + child: '../docgen-memory/memory-harness.ts', + scenarios: reactScenarios, + jiti: true, + args: (scenario) => [...reactArgs(scenario), '--mode', 'refresh', '--scope', 'changed'], + }), + new SeriesChildEngine({ + id: 'vue-docgen-api', + child: 'engines/vue-docgen-api.ts', + scenarios: vueScenarios, + args: vueArgs, + }), + new SeriesChildEngine({ + id: 'vue-component-meta', + child: 'engines/vue-component-meta.ts', + scenarios: vueScenarios, + args: vueArgs, + }), + new CompodocEngine(), +]; + +export const ALL_ENGINE_IDS: EngineId[] = ENGINES.map((engine) => engine.id); +export const DEFAULT_ENGINE_IDS: EngineId[] = ENGINES.filter((e) => e.inDefaultRun).map((e) => e.id); + +export function engineById(id: EngineId): BenchEngine { + const engine = ENGINES.find((candidate) => candidate.id === id); + if (!engine) { + throw new Error(`no engine registered for "${id}"`); + } + return engine; +} diff --git a/scripts/bench/docgen-perf/report.test.ts b/scripts/bench/docgen-perf/report.test.ts new file mode 100644 index 000000000000..29666ef568f6 --- /dev/null +++ b/scripts/bench/docgen-perf/report.test.ts @@ -0,0 +1,97 @@ +import { describe, expect, it } from 'vitest'; + +import { renderRatios, renderResults } from './report.ts'; +import { NOT_APPLICABLE, type Ratios } from './types.ts'; + +describe('renderRatios', () => { + const notLikeForLike: Ratios = { + vue: { + flat: { + cold: 18.48, + warm: 22.5, + legacyColdMembers: 6, + nextColdMembers: 320, + legacyWarmMembers: 0, + nextWarmMembers: 32, + likeForLike: false, + }, + }, + }; + + it('warns on the cold line when the engines documented different members', () => { + const [cold] = renderRatios(notLikeForLike); + expect(cold).toContain('documented members 6 vs 320'); + expect(cold).toContain('NOT like-for-like'); + }); + + it('warns on the warm line too', () => { + // The warm ratio is the more misleading of the two: the legacy Vue parser documented nothing + // on the save it was timed on, so a bare ratio reads as a 22x speed win over identical work. + const warm = renderRatios(notLikeForLike).find((line) => line.includes('warm')); + expect(warm).toContain('documented members 0 vs 32'); + expect(warm).toContain('NOT like-for-like'); + }); + + it('prints no warning when the engines did the same work', () => { + const lines = renderRatios({ + vue: { + flat: { cold: 1.2, warm: 1.1, legacyColdMembers: 50, nextColdMembers: 50, likeForLike: true }, + }, + }); + expect(lines.every((line) => !line.includes('NOT like-for-like'))).toBe(true); + expect(lines[0]).toContain('documented members 50 vs 50'); + }); + + it('prints a bare ratio when neither engine reports member counts', () => { + const lines = renderRatios({ react: { default: { cold: 4, warm: 2 } } }); + expect(lines).toEqual([ + ' ratio cold legacy/new (react/default): 4.00', + ' ratio warm legacy/new (react/default): 2.00', + ]); + }); + + it('says so when there is no ratio at all', () => { + expect(renderRatios({})).toEqual([ + ' no calibration ratio: it needs both sides of a control pair measured in one run', + ]); + }); + + it('omits a half of the pair that did not measure', () => { + const lines = renderRatios({ react: { default: { cold: 4 } } }); + expect(lines).toHaveLength(1); + expect(lines[0]).toContain('cold'); + }); +}); + +describe('renderResults', () => { + it('lists a failed engine as a status line rather than a table row', () => { + const { table, statusLines } = renderResults(['compodoc'], { + compodoc: { status: 'failed', reason: 'child exited with status 1:\nsecond line' }, + }); + expect(table).toHaveLength(1); // header only + expect(statusLines).toEqual([' compodoc: FAILED - child exited with status 1:']); + }); + + it('renders n/a metrics as n/a rather than zero', () => { + const { table } = renderResults(['react-osa'], { + 'react-osa': { + status: 'measured', + scenarios: { + default: { + params: {}, + metrics: { + coldExtractionMs: { status: 'measured', samples: [100], median: 100 }, + warmExtractionMs: { status: 'measured', samples: [10], median: 10 }, + wholeProjectScanMs: NOT_APPLICABLE, + peakTransientMb: { status: 'measured', samples: [5], mean: 5 }, + retainedGrowthMb: { status: 'measured', value: 3 }, + retainedSlopeMbPerSave: { status: 'measured', value: 0.25 }, + }, + }, + }, + }, + }); + expect(table[1]).toContain('n/a'); + expect(table[1]).toContain('0.25MB/save'); + }); +}); diff --git a/scripts/bench/docgen-perf/report.ts b/scripts/bench/docgen-perf/report.ts new file mode 100644 index 000000000000..3a6773e960d9 --- /dev/null +++ b/scripts/bench/docgen-perf/report.ts @@ -0,0 +1,110 @@ +/** Renders the suite's terminal output. Pure string building, so it is testable without a runner. */ +import type { EngineId, EngineMetrics, EngineResult, Ratios } from './types.ts'; + +const HEADER = ['engine/scenario', 'cold', 'warm', 'scan', 'peak', 'ret-growth', 'ret-slope']; + +export function formatCell(metric: EngineMetrics[keyof EngineMetrics], unit: 'ms' | 'MB'): string { + if (metric.status === 'n/a') { + return 'n/a'; + } + if ('median' in metric) { + return `${metric.median.toFixed(0)}${unit}`; + } + if ('mean' in metric) { + return `${metric.mean.toFixed(0)}${unit}`; + } + return `${metric.value.toFixed(unit === 'MB' ? 1 : 0)}${unit}`; +} + +export interface RenderedResults { + /** Already padded into aligned lines. */ + table: string[]; + /** One line per engine that skipped or failed, in place of a table row. */ + statusLines: string[]; +} + +export function renderResults( + engineIds: EngineId[], + results: Partial> +): RenderedResults { + const rows: string[][] = [HEADER]; + const statusLines: string[] = []; + + for (const engineId of engineIds) { + const result = results[engineId]; + if (!result) { + continue; + } + if (result.status !== 'measured') { + statusLines.push( + ` ${engineId}: ${result.status.toUpperCase()} - ${result.reason.split('\n')[0]}` + ); + continue; + } + for (const [scenarioName, scenario] of Object.entries(result.scenarios)) { + const m = scenario.metrics; + rows.push([ + `${engineId}/${scenarioName}`, + formatCell(m.coldExtractionMs, 'ms'), + formatCell(m.warmExtractionMs, 'ms'), + formatCell(m.wholeProjectScanMs, 'ms'), + formatCell(m.peakTransientMb, 'MB'), + formatCell(m.retainedGrowthMb, 'MB'), + m.retainedSlopeMbPerSave.status === 'measured' + ? `${m.retainedSlopeMbPerSave.value.toFixed(2)}MB/save` + : 'n/a', + ]); + } + } + + const widths = HEADER.map((_, col) => Math.max(...rows.map((row) => row[col].length))); + const table = rows.map((row) => ` ${row.map((cell, col) => cell.padEnd(widths[col])).join(' ')}`); + return { table, statusLines }; +} + +/** + * A ratio between engines that documented different numbers of members is not a speed comparison - + * the faster side was fast because it resolved less. Printing such a ratio bare reads as a clean + * win, so the member counts and the warning travel with it, on the warm line as much as the cold + * one. The warm line needs it more: the legacy Vue parser documents zero members on the save it is + * timed on, so a bare warm ratio would read as a clean win over identical work. + */ +export function renderRatios(ratios: Ratios): string[] { + const lines: string[] = []; + + for (const [pairName, scenarios] of Object.entries(ratios)) { + for (const [scenarioName, entry] of Object.entries(scenarios)) { + const notLikeForLike = entry.likeForLike === false; + const label = `${pairName}/${scenarioName}`; + + if (entry.cold !== undefined) { + lines.push( + ` ratio cold legacy/new (${label}): ${entry.cold.toFixed(2)}` + + memberNote(entry.legacyColdMembers, entry.nextColdMembers, notLikeForLike) + ); + } + if (entry.warm !== undefined) { + lines.push( + ` ratio warm legacy/new (${label}): ${entry.warm.toFixed(2)}` + + memberNote(entry.legacyWarmMembers, entry.nextWarmMembers, notLikeForLike) + ); + } + } + } + + if (lines.length === 0) { + lines.push(' no calibration ratio: it needs both sides of a control pair measured in one run'); + } + return lines; +} + +function memberNote( + legacy: number | undefined, + next: number | undefined, + notLikeForLike: boolean +): string { + if (legacy === undefined || next === undefined) { + return notLikeForLike ? ' [NOT like-for-like]' : ''; + } + return ` [documented members ${legacy} vs ${next}${notLikeForLike ? ' - NOT like-for-like' : ''}]`; +} diff --git a/scripts/bench/docgen-perf/run.ts b/scripts/bench/docgen-perf/run.ts new file mode 100644 index 000000000000..6366f4e97653 --- /dev/null +++ b/scripts/bench/docgen-perf/run.ts @@ -0,0 +1,188 @@ +/** + * Orchestrator for the per-engine docgen performance suite. See scripts/bench/PERF-METHODOLOGY.md + * for the measurement contract. + * + * Run: + * yarn bench:docgen-perf # from scripts/, full profile + * yarn bench:docgen-perf --quick # smoke profile; results marked non-comparable + * yarn bench:docgen-perf --engine react-legacy --engine react-osa + */ +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +import { SANDBOX_DIRECTORY } from '../docgen-shared/paths.ts'; +import { designatedRep } from './aggregate.ts'; +import { parseCliOptions } from './cli.ts'; +import { DEFAULT_PROFILE, QUICK_PROFILE, RSS_POLL_INTERVAL_MS, type SuiteProfile } from './config.ts'; +import type { ScenarioSpec } from './engine.ts'; +import { computeRatios, engineOrderForRep } from './ratios.ts'; +import { engineById } from './registry.ts'; +import { renderRatios, renderResults } from './report.ts'; +import { runSeriesChild } from './spawn.ts'; +import type { EngineId, EngineResult, ScenarioResult, SuiteResults } from './types.ts'; + +const WORK_ROOT = path.join(SANDBOX_DIRECTORY, 'docgen-perf'); + +/** Raw repetition samples, keyed by `engine/scenario`. */ +type SampleStore = Map; + +function scenarioKey(engineId: EngineId, scenarioName: string): string { + return `${engineId}/${scenarioName}`; +} + +/** One repetition of one engine, across every scenario that engine runs. */ +async function measureEngine( + engineId: EngineId, + profile: SuiteProfile, + rep: number, + store: SampleStore +): Promise { + const engine = engineById(engineId); + for (const scenario of engine.scenarios(profile)) { + const key = scenarioKey(engineId, scenario.name); + console.log(` ${key} (rep ${rep}/${profile.n})…`); + const sample = await engine.measure( + { + scenarioDir: path.join(WORK_ROOT, engineId, scenario.name), + rssPollIntervalMs: RSS_POLL_INTERVAL_MS, + runSeriesChild, + }, + scenario, + rep + ); + store.set(key, [...(store.get(key) ?? []), sample]); + } +} + +function assembleScenario( + engineId: EngineId, + scenario: ScenarioSpec, + profile: SuiteProfile, + store: SampleStore +): ScenarioResult { + const engine = engineById(engineId); + const samples = store.get(scenarioKey(engineId, scenario.name)) ?? []; + const metrics = engine.aggregate(samples, profile.n); + // Member counts come from the same repetition as the warm and memory metrics, so every figure + // reported for a scenario describes one run. + const designated = designatedRep(samples as Array<{ coldMs: number }>); + return { + params: scenario.params, + metrics, + coldMembers: engine.coldMembers(designated), + warmMembers: engine.warmMembers(designated), + coldOpaqueTypes: engine.coldOpaqueTypes(designated), + }; +} + +async function main() { + const options = parseCliOptions(process.argv.slice(2), WORK_ROOT); + const profile: SuiteProfile = options.quick ? QUICK_PROFILE : DEFAULT_PROFILE; + + console.log('docgen-perf suite'); + console.log(` engines=${options.engines.join(',')} n=${profile.n} comparable=${profile.comparable}`); + if (!profile.comparable) { + console.log(' QUICK PROFILE: results are non-comparable smoke numbers'); + } + + const store: SampleStore = new Map(); + const failed = new Map(); + const skipped = new Map(); + + for (const engineId of options.engines) { + const reason = engineById(engineId).preflight(); + if (reason) { + skipped.set(engineId, reason); + console.log(` ${engineId}: SKIPPED - ${reason}`); + } + } + + for (let rep = 1; rep <= profile.n; rep++) { + console.log(`\n=== repetition ${rep}/${profile.n} ===`); + for (const engineId of engineOrderForRep(options.engines, rep)) { + if (failed.has(engineId) || skipped.has(engineId)) { + continue; + } + try { + await measureEngine(engineId, profile, rep, store); + } catch (err) { + const reason = err instanceof Error ? err.message : String(err); + failed.set(engineId, reason); + console.error(` ${engineId} FAILED: ${reason}`); + } + } + } + + const engineResults: SuiteResults['engines'] = {}; + const engineVersions: SuiteResults['engineVersions'] = {}; + + for (const engineId of options.engines) { + const engine = engineById(engineId); + const version = engine.version(); + if (version) { + engineVersions[engineId] = version; + } + + const skipReason = skipped.get(engineId); + if (skipReason) { + engineResults[engineId] = { status: 'skipped', reason: skipReason }; + continue; + } + // An engine that failed part-way holds fewer samples than the pinned N, so it stays failed + // rather than being reported as measured at an unrecorded N. + const failReason = failed.get(engineId); + if (failReason) { + engineResults[engineId] = { status: 'failed', reason: failReason }; + continue; + } + try { + const scenarios: Record = {}; + for (const scenario of engine.scenarios(profile)) { + scenarios[scenario.name] = assembleScenario(engineId, scenario, profile, store); + } + engineResults[engineId] = { status: 'measured', scenarios } satisfies EngineResult; + } catch (err) { + const reason = err instanceof Error ? err.message : String(err); + failed.set(engineId, reason); + engineResults[engineId] = { status: 'failed', reason }; + } + } + + const results: SuiteResults = { + generatedAt: new Date().toISOString(), + nodeVersion: process.version, + pinnedN: profile.n, + comparable: profile.comparable, + rssPollIntervalMs: RSS_POLL_INTERVAL_MS, + engineVersions, + engines: engineResults, + ratios: computeRatios(engineResults), + }; + + console.log('\nresults'); + const { table, statusLines } = renderResults(options.engines, engineResults); + for (const line of [...table, ...statusLines, ...renderRatios(results.ratios)]) { + console.log(line); + } + if (!profile.comparable) { + console.log(' QUICK PROFILE: results are non-comparable smoke numbers'); + } + + fs.mkdirSync(path.dirname(options.jsonOut), { recursive: true }); + fs.writeFileSync(options.jsonOut, JSON.stringify(results, null, 2)); + console.log(` wrote ${options.jsonOut}`); + + if (failed.size > 0) { + console.error('\ndocgen-perf suite FAILED:'); + for (const [engineId, reason] of failed) { + console.error(` - ${engineId}: ${reason}`); + } + process.exit(1); + } + console.log('\ndocgen-perf suite completed.'); +} + +main().catch((err) => { + console.error(err); + process.exitCode = 1; +}); diff --git a/scripts/bench/docgen-perf/spawn.ts b/scripts/bench/docgen-perf/spawn.ts new file mode 100644 index 000000000000..b9e376957005 --- /dev/null +++ b/scripts/bench/docgen-perf/spawn.ts @@ -0,0 +1,59 @@ +/** + * Spawning one measured child process and reading its result back. + * + * Every measurement runs in a fresh process so it starts from a clean heap; an engine measured after + * another engine's garbage would report that garbage as its own. + */ +import { spawnSync } from 'node:child_process'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +import { outputTail } from '../docgen-shared/child-output.ts'; +import type { SeriesResult } from '../docgen-shared/series.ts'; + +export interface SeriesChildSpec { + childPath: string; + args: string[]; + /** + * Run the child under the jiti loader. Only the reused docgen-memory harness needs it: under + * jiti, react-docgen's browserslist dependency fails on its JSON data require + * ("jsReleases.map is not a function"), so the legacy child must run on native type stripping. + */ + jiti?: boolean; +} + +/** + * Run one series-harness child and parse its result JSON. A non-zero exit, or a missing or + * unreadable result, is a failure - never a silently empty measurement. + */ +export function runSeriesChild( + spec: SeriesChildSpec, + outDir: string, + jsonPath: string +): SeriesResult { + fs.mkdirSync(path.dirname(jsonPath), { recursive: true }); + // Remove any stale result so a crashed run cannot be mistaken for a prior success. + fs.rmSync(jsonPath, { force: true }); + + const nodeArgs = [ + '--expose-gc', + ...(spec.jiti ? ['--import', 'jiti/register'] : []), + spec.childPath, + ...spec.args, + '--out', + outDir, + '--json', + jsonPath, + ]; + + const proc = spawnSync(process.execPath, nodeArgs, { encoding: 'utf8' }); + const output = `${proc.stdout ?? ''}${proc.stderr ?? ''}`; + + if (proc.status !== 0) { + throw new Error(`child exited with status ${proc.status}:\n${outputTail(output, 4)}`); + } + if (!fs.existsSync(jsonPath)) { + throw new Error(`child wrote no result JSON at ${jsonPath}:\n${outputTail(output, 4)}`); + } + return JSON.parse(fs.readFileSync(jsonPath, 'utf8')) as SeriesResult; +} diff --git a/scripts/bench/docgen-perf/types.ts b/scripts/bench/docgen-perf/types.ts new file mode 100644 index 000000000000..a8a7bf6366dc --- /dev/null +++ b/scripts/bench/docgen-perf/types.ts @@ -0,0 +1,103 @@ +/** + * Result shapes for the per-engine docgen performance suite. The measurement contract behind these + * shapes is scripts/bench/PERF-METHODOLOGY.md. + */ +import type { EngineId } from '../docgen-shared/engine-ids.ts'; + +export type { EngineId }; + +/** A latency metric: median of repeated samples (fresh process each, for cold/scan). */ +export interface LatencyMetric { + status: 'measured'; + samples: number[]; + median: number; +} + +/** A memory metric aggregated as the mean of a per-save (or per-run) series. */ +export interface SeriesMeanMetric { + status: 'measured'; + samples: number[]; + mean: number; +} + +/** A single-valued metric read from one run's series (retained growth, retained slope). */ +export interface ValueMetric { + status: 'measured'; + value: number; +} + +/** The explicit marker for a metric that does not apply to an engine; never a faked equivalent. */ +export interface NotApplicable { + status: 'n/a'; +} + +export const NOT_APPLICABLE: NotApplicable = { status: 'n/a' }; + +export interface EngineMetrics { + coldExtractionMs: LatencyMetric | NotApplicable; + warmExtractionMs: LatencyMetric | NotApplicable; + wholeProjectScanMs: LatencyMetric | NotApplicable; + peakTransientMb: SeriesMeanMetric | NotApplicable; + retainedGrowthMb: ValueMetric | NotApplicable; + retainedSlopeMbPerSave: ValueMetric | NotApplicable; +} + +export interface ScenarioResult { + params: Record; + metrics: EngineMetrics; + /** + * Members the cold pass documented, when the engine reports it. Two engines over the same project + * can differ by an order of magnitude here, and a timing ratio between them means nothing without + * it. + */ + coldMembers?: number; + /** Members the timed re-extraction documented. Warm ratios need this for the same reason. */ + warmMembers?: number; + /** + * Of {@link coldMembers}, how many the engine documented under a type name it never resolved. + * An engine that prints `Hop19Shape` and one that expands it into its fields report the same + * member count off very different work, so equal counts alone do not make a ratio like-for-like. + */ + coldOpaqueTypes?: number; +} + +export type EngineResult = + | { status: 'measured'; scenarios: Record } + | { status: 'skipped'; reason: string } + | { status: 'failed'; reason: string }; + +/** + * One control pair's comparison for one scenario: the legacy engine's median over the new engine's, + * both measured in the same invocation. + * + * `likeForLike` is false when the two engines documented different numbers of members, which means + * they did not do the same work and the ratio measures resolution depth as much as speed. A ratio + * whose `likeForLike` is false must not become a budget. + */ +export interface RatioEntry { + cold?: number; + warm?: number; + legacyColdMembers?: number; + nextColdMembers?: number; + legacyWarmMembers?: number; + nextWarmMembers?: number; + likeForLike?: boolean; +} + +/** Keyed by control-pair name, then by scenario name. */ +export type Ratios = Record>; + +export interface SuiteResults { + generatedAt: string; + nodeVersion: string; + /** The one pinned N; numbers taken at different N are not comparable. */ + pinnedN: number; + /** False for --quick smoke runs, whose numbers must never be compared against real runs. */ + comparable: boolean; + /** Sampling interval for the compodoc child's externally-polled peak RSS. */ + rssPollIntervalMs: number; + /** Resolved versions of externally-installed engines, when they ran. */ + engineVersions: Partial>; + engines: Partial>; + ratios: Ratios; +} diff --git a/scripts/bench/docgen-shared/args.test.ts b/scripts/bench/docgen-shared/args.test.ts new file mode 100644 index 000000000000..21694d3779c9 --- /dev/null +++ b/scripts/bench/docgen-shared/args.test.ts @@ -0,0 +1,102 @@ +import { describe, expect, it } from 'vitest'; +import { z } from 'zod'; + +import { countOption, parseHarnessOptions } from './args.ts'; + +const OPTIONS = { + components: { type: 'string' }, + scope: { type: 'string' }, + heavy: { type: 'boolean' }, + out: { type: 'string' }, + 'components-per-package': { type: 'string' }, + 'max-retained-growth': { type: 'string' }, +} as const; + +const SCHEMA = z.object({ + components: countOption(300), + scope: z.enum(['all', 'changed']).default('changed'), + heavy: z.boolean().default(false), + outDir: z.string().default('/default'), + componentsPerPackage: countOption(10), + maxRetainedGrowthMb: countOption(400), +}); + +interface Options { + components: number; + scope: 'all' | 'changed'; + heavy: boolean; + outDir: string; + componentsPerPackage: number; + maxRetainedGrowthMb: number; +} + +const parse = (argv: string[]) => + parseHarnessOptions(argv, OPTIONS, SCHEMA, (values) => ({ + ...values, + outDir: values.out, + maxRetainedGrowthMb: values.maxRetainedGrowth, + })); + +describe('parseHarnessOptions', () => { + it('applies defaults when nothing is passed', () => { + expect(parse([])).toEqual({ + components: 300, + scope: 'changed', + heavy: false, + outDir: '/default', + componentsPerPackage: 10, + maxRetainedGrowthMb: 400, + }); + }); + + it('coerces numeric flags', () => { + expect(parse(['--components', '20']).components).toBe(20); + }); + + it('maps kebab-case flags onto the schema shape', () => { + expect(parse(['--components-per-package', '5']).componentsPerPackage).toBe(5); + }); + + it('carries a flag whose schema key is not its camelCase', () => { + // `--max-retained-growth` camel-cases to `maxRetainedGrowth`, but the schema names the unit. + // Without the rename in `toInput` the flag is accepted and then silently ignored, which is how + // the memory harness lost its threshold flag once already. + expect(parse(['--max-retained-growth', '50']).maxRetainedGrowthMb).toBe(50); + }); + + it('reads boolean flags', () => { + expect(parse(['--heavy']).heavy).toBe(true); + }); + + it('rejects an unknown flag', () => { + // The reason for strict mode: `--component 5` would otherwise be dropped silently and the + // harness would benchmark 300 components while reporting a run someone asked for at 5. + expect(() => parse(['--component', '5'])).toThrow(/Unknown option/); + }); + + it('rejects a flag whose value is another flag', () => { + expect(() => parse(['--components', '--out', 'x'])).toThrow(/ambiguous/); + }); + + it('rejects a flag with no value at all', () => { + expect(() => parse(['--components'])).toThrow(/argument missing/); + }); + + it('rejects a value outside an enum, naming the flag', () => { + expect(() => parse(['--scope', 'some'])).toThrow(/--scope/); + }); + + describe('countOption', () => { + it('rejects a non-numeric value', () => { + expect(() => parse(['--components', 'lots'])).toThrow(/--components/); + }); + + it('rejects a fraction', () => { + expect(() => parse(['--components', '2.5'])).toThrow(/--components/); + }); + + it('rejects a negative', () => { + expect(() => parse(['--components', '-3'])).toThrow(/--components/); + }); + }); +}); diff --git a/scripts/bench/docgen-shared/args.ts b/scripts/bench/docgen-shared/args.ts new file mode 100644 index 000000000000..ad3bae5bd831 --- /dev/null +++ b/scripts/bench/docgen-shared/args.ts @@ -0,0 +1,38 @@ +import { type ParseArgsConfig, parseArgs } from 'node:util'; + +import { z } from 'zod'; + +type OptionsConfig = NonNullable; + +/** parseArgs yields strings, so numeric flags are coerced here */ +export const countOption = (fallback: number) => + z.coerce.number().int().nonnegative().default(fallback); + +/** + * parseArgs keys flags as written, so `--fan-out` arrives as `fan-out`; schema keys are camelCase. + * A schema key that is not exactly its flag's camelCase — `maxRetainedGrowthMb` for + * `--max-retained-growth` — must still be renamed in `toInput`, or Zod silently uses its default. + */ +function toCamelCase(flag: string): string { + return flag.replace(/-([a-z])/g, (_, letter: string) => letter.toUpperCase()); +} + +export function parseHarnessOptions( + argv: string[], + options: OptionsConfig, + schema: z.ZodTypeAny, + toInput?: (values: Record) => Record +): Out { + const { values } = parseArgs({ args: argv, options, strict: true }); + const flags = Object.fromEntries( + Object.entries(values).map(([flag, value]) => [toCamelCase(flag), value]) + ); + const result = schema.safeParse(toInput ? toInput(flags) : flags); + if (!result.success) { + const issues = result.error.issues + .map((issue) => ` --${issue.path.join('.')}: ${issue.message}`) + .join('\n'); + throw new Error(`invalid options:\n${issues}`); + } + return result.data as Out; +} diff --git a/scripts/bench/docgen-shared/budgets.ts b/scripts/bench/docgen-shared/budgets.ts new file mode 100644 index 000000000000..7526d68307a6 --- /dev/null +++ b/scripts/bench/docgen-shared/budgets.ts @@ -0,0 +1,22 @@ +import type { EngineId } from './engine-ids.ts'; + +export interface MemoryBudgets { + /** Max allowed post-GC retained growth (MB) across the run. */ + maxRetainedGrowthMb: number; + /** Max allowed average transient working set added per save (MB). */ + maxTransientMb: number; + /** Max allowed post-GC retained-heap slope (MB/save). */ + maxRetainedSlopeMb: number; +} + +const MEMORY_BUDGETS: Partial> = { + 'react-osa': { maxRetainedGrowthMb: 60, maxTransientMb: 90, maxRetainedSlopeMb: 3 }, +}; + +export function memoryBudgetsFor(engine: EngineId): MemoryBudgets { + const budgets = MEMORY_BUDGETS[engine]; + if (!budgets) { + throw new Error(`no memory budgets recorded for "${engine}"`); + } + return budgets; +} diff --git a/scripts/bench/docgen-shared/child-output.ts b/scripts/bench/docgen-shared/child-output.ts new file mode 100644 index 000000000000..c9778ba246ea --- /dev/null +++ b/scripts/bench/docgen-shared/child-output.ts @@ -0,0 +1,8 @@ +export function outputTail(output: string, count: number): string { + return output + .trim() + .split('\n') + .slice(-count) + .map((line) => ` ${line}`) + .join('\n'); +} diff --git a/scripts/bench/docgen-shared/engine-ids.ts b/scripts/bench/docgen-shared/engine-ids.ts new file mode 100644 index 000000000000..49934a8fa1e7 --- /dev/null +++ b/scripts/bench/docgen-shared/engine-ids.ts @@ -0,0 +1,10 @@ +/** + * The docgen engines under measurement. + */ +export type EngineId = + | 'react-legacy' + | 'react-legacy-rdt' + | 'react-osa' + | 'vue-docgen-api' + | 'vue-component-meta' + | 'compodoc'; diff --git a/scripts/bench/docgen-shared/paths.ts b/scripts/bench/docgen-shared/paths.ts new file mode 100644 index 000000000000..717094dd89ca --- /dev/null +++ b/scripts/bench/docgen-shared/paths.ts @@ -0,0 +1,13 @@ +/** + * Mirrors SANDBOX_DIRECTORY from scripts/utils/constants.ts. That module deliberately stays on the + * CJS-only `__dirname` global for Playwright compatibility, so the native-Node harness entry points + * here cannot import it. + */ +import * as path from 'node:path'; + +const ROOT_DIRECTORY = path.join(import.meta.dirname, '..', '..', '..'); + +export const SANDBOX_DIRECTORY = + process.env.STORYBOOK_SANDBOX_ROOT && path.isAbsolute(process.env.STORYBOOK_SANDBOX_ROOT) + ? process.env.STORYBOOK_SANDBOX_ROOT + : path.join(ROOT_DIRECTORY, process.env.STORYBOOK_SANDBOX_ROOT || '../storybook-sandboxes'); diff --git a/scripts/bench/docgen-shared/react-renderer-module.ts b/scripts/bench/docgen-shared/react-renderer-module.ts new file mode 100644 index 000000000000..84b902192e67 --- /dev/null +++ b/scripts/bench/docgen-shared/react-renderer-module.ts @@ -0,0 +1,20 @@ +const COMPONENT_MANIFEST = '../../../code/renderers/react/src/componentManifest/'; + +export async function loadReactRendererModule(relativePath: string): Promise { + const url = new URL(`${COMPONENT_MANIFEST}${relativePath}`, import.meta.url).href; + return (await import(url)) as T; +} + +export interface ComponentRefLike { + componentName: string; + importName: string; + localImportName: string; + importId: string; + isPackage: boolean; + path: string; +} + +export interface StoryRefLike { + storyPath: string; + component: ComponentRefLike; +} diff --git a/scripts/bench/docgen-shared/samples.ts b/scripts/bench/docgen-shared/samples.ts new file mode 100644 index 000000000000..84cbe5a8254e --- /dev/null +++ b/scripts/bench/docgen-shared/samples.ts @@ -0,0 +1,15 @@ +export interface MemorySample { + /** Resident Set Size, the total memory allocated for the process. */ + rssMb: number; + /** The V8 heap used by the process. */ + heapUsedMb: number; + /** Post-GC heap. Present only when the child runs under `node --expose-gc`. */ + retainedHeapMb?: number; +} + +export interface SaveSample extends MemorySample { + /** The save number, counting from 1. */ + save: number; + /** The duration of the save's re-extraction, in milliseconds. */ + durMs: number; +} diff --git a/scripts/bench/docgen-shared/sampling.ts b/scripts/bench/docgen-shared/sampling.ts new file mode 100644 index 000000000000..f49f851d4643 --- /dev/null +++ b/scripts/bench/docgen-shared/sampling.ts @@ -0,0 +1,39 @@ +/** + * Memory sampling shared by every docgen bench harness: pre-GC rss/heapUsed is the transient-pressure + * signal, post-GC heapUsed (only under `node --expose-gc`) is the retained signal. + */ +import type { MemorySample } from './samples.ts'; + +export const MB = 1024 * 1024; + +export function gc(): void { + if (typeof global.gc === 'function') { + global.gc(); + global.gc(); + } +} + +export function gcAvailable(): boolean { + return typeof global.gc === 'function'; +} + +export function sampleMemory(forceGc: boolean): MemorySample { + const pre = process.memoryUsage(); + let retainedHeapMb: number | undefined; + if (forceGc && gcAvailable()) { + gc(); + retainedHeapMb = process.memoryUsage().heapUsed / MB; + } + return { rssMb: pre.rss / MB, heapUsedMb: pre.heapUsed / MB, retainedHeapMb }; +} + +/** The one per-save log line every harness prints, so their output stays comparable by eye. */ +export function formatSampleLine(save: number, durMs: number, mem: MemorySample): string { + return ( + ` save ${String(save).padStart(3)}: ${String(durMs).padStart(5)}ms ` + + `rss=${mem.rssMb.toFixed(0).padStart(5)}MB heapUsed=${mem.heapUsedMb.toFixed(0).padStart(5)}MB` + + (mem.retainedHeapMb !== undefined + ? ` retained=${mem.retainedHeapMb.toFixed(0).padStart(5)}MB` + : '') + ); +} diff --git a/scripts/bench/docgen-shared/series.ts b/scripts/bench/docgen-shared/series.ts new file mode 100644 index 000000000000..4a5733f40c99 --- /dev/null +++ b/scripts/bench/docgen-shared/series.ts @@ -0,0 +1,140 @@ +/** + * The save-series harness every docgen engine child runs: + * one timed cold pass, then K simulated saves that mutate the project on disk, invalidate the + * engine's caches, and re-extract, with memory sampled around forced GC after every save. Engines + * differ only in what those steps mean, so they implement {@link SeriesEngine} and this module owns + * the timing. + */ +import * as fs from 'node:fs'; + +import type { MemorySample, SaveSample } from './samples.ts'; +import { formatSampleLine, gcAvailable, sampleMemory } from './sampling.ts'; +import { type SeriesSummary, summarizeSeries } from './stats.ts'; + +export interface SeriesEngine { + /** + * One full extraction over the measured set, from a cold start. Returns how many members it + * documented, when the engine can report that; two engines over one project can differ by an order + * of magnitude, and a timing ratio between them means nothing without it. + */ + cold(): Promise; + /** Mutate the project on disk for save `save` and invalidate the engine's caches. Never timed. */ + applySave(save: number): Promise; + /** Re-extract after save `save`. This call, and only this call, is the warm sample. */ + reextract(save: number): Promise; + dispose?(): void; +} + +export interface SeriesResult extends SeriesSummary { + coldMs: number; + coldMembers?: number; + /** Members documented by the last timed re-extraction. */ + warmMembers?: number; + baseline: MemorySample; + samples: SaveSample[]; +} + +export interface SeriesOptions { + saves: number; + /** Describes the measured set for the cold-pass log line, e.g. "300 components". */ + coldLabel: string; + /** Off only for the memory harness's uncapped pressure configs, which measure pre-GC rss. */ + forceGc?: boolean; +} + +export async function runSeries( + engine: SeriesEngine, + options: SeriesOptions +): Promise { + const forceGc = options.forceGc ?? true; + + console.log(` full extraction over ${options.coldLabel} (cold pass)…`); + const coldStart = Date.now(); + const coldMembers = await engine.cold(); + const coldMs = Date.now() - coldStart; + console.log( + ` cold pass: ${coldMs}ms` + (coldMembers !== undefined ? ` (${coldMembers} documented members)` : '') + ); + + const baseline = sampleMemory(forceGc); + const samples: SaveSample[] = []; + let warmMembers: number | undefined; + + for (let save = 1; save <= options.saves; save++) { + await engine.applySave(save); + + const saveStart = Date.now(); + warmMembers = await engine.reextract(save); + const durMs = Date.now() - saveStart; + + const mem = sampleMemory(forceGc); + samples.push({ save, durMs, ...mem }); + console.log(formatSampleLine(save, durMs, mem)); + } + + engine.dispose?.(); + + return { coldMs, coldMembers, warmMembers, baseline, samples, ...summarizeSeries(samples, baseline) }; +} + +export function printSeriesSummary(result: SeriesResult, saves: number): void { + console.log('\nsummary'); + console.log(` cold pass: ${result.coldMs}ms`); + if (result.coldMembers !== undefined) { + console.log( + ` documented members: ${result.coldMembers} cold, ${result.warmMembers ?? 0} on the last save` + ); + } + if (result.avgTransient !== undefined) { + console.log(` avg transient/save: ${result.avgTransient.toFixed(0)}MB`); + } + if (result.retainedSlope !== undefined && result.retainedGrowth !== undefined) { + console.log(` retained slope: ${result.retainedSlope.toFixed(2)}MB/save`); + console.log(` retained growth: ${result.retainedGrowth.toFixed(0)}MB over ${saves} saves`); + } +} + +export interface SeriesHarnessSpec extends SeriesOptions { + /** Banner line, e.g. `vue-component-meta harness (workspace)`. */ + title: string; + /** The resolved options, recorded in the result JSON so a stored run is self-describing. */ + options: object; + /** The subset echoed under the banner. Defaults to {@link options}. */ + banner?: Record; + jsonOut?: string; + /** Build the project and the engine. Runs before the cold pass, so its cost is not measured. */ + setup(): Promise; +} + +/** + * Children own only their {@link SeriesEngine}; everything the orchestrator reads back is produced + * here so the children cannot drift apart in what they report. + */ +export async function runSeriesHarness(spec: SeriesHarnessSpec): Promise { + console.log(spec.title); + console.log( + ` ${Object.entries(spec.banner ?? spec.options) + .map(([key, value]) => `${key}=${String(value)}`) + .join(' ')}` + ); + if ((spec.forceGc ?? true) && !gcAvailable()) { + console.log(' (run with `node --expose-gc` to measure retained heap; continuing without it)'); + } + + const engine = await spec.setup(); + const result = await runSeries(engine, spec); + printSeriesSummary(result, spec.saves); + + if (spec.jsonOut) { + fs.writeFileSync(spec.jsonOut, JSON.stringify({ options: spec.options, ...result }, null, 2)); + console.log(` wrote ${spec.jsonOut}`); + } +} + +/** Entry-point wrapper: any throw becomes a non-zero exit, which is how the orchestrator sees it. */ +export function harnessMain(run: () => Promise): void { + run().catch((err) => { + console.error(err); + process.exitCode = 1; + }); +} diff --git a/scripts/bench/docgen-shared/stats.test.ts b/scripts/bench/docgen-shared/stats.test.ts new file mode 100644 index 000000000000..f06ccd2cd5ed --- /dev/null +++ b/scripts/bench/docgen-shared/stats.test.ts @@ -0,0 +1,56 @@ +import { describe, expect, it } from 'vitest'; + +import { leastSquaresSlope, mean, median } from './stats.ts'; + +describe('median', () => { + it('returns the middle value for odd-length input', () => { + expect(median([3, 1, 2])).toBe(2); + }); + + it('averages the two middle values for even-length input', () => { + expect(median([4, 1, 3, 2])).toBe(2.5); + }); + + it('returns the value itself for a single sample', () => { + expect(median([7])).toBe(7); + }); + + it('does not mutate its input', () => { + const values = [3, 1, 2]; + median(values); + expect(values).toEqual([3, 1, 2]); + }); + + it('throws on empty input', () => { + expect(() => median([])).toThrow('median() requires at least one value'); + }); +}); + +describe('mean', () => { + it('averages the values', () => { + expect(mean([1, 2, 3, 4])).toBe(2.5); + }); + + it('throws on empty input', () => { + expect(() => mean([])).toThrow('mean() requires at least one value'); + }); +}); + +describe('leastSquaresSlope', () => { + it('returns 0 for fewer than two points', () => { + expect(leastSquaresSlope([])).toBe(0); + expect(leastSquaresSlope([5])).toBe(0); + }); + + it('recovers the slope of a perfect line', () => { + expect(leastSquaresSlope([1, 3, 5, 7])).toBe(2); + }); + + it('returns 0 for a flat series', () => { + expect(leastSquaresSlope([4, 4, 4])).toBe(0); + }); + + it('fits noisy data to the least-squares line', () => { + expect(leastSquaresSlope([0, 2, 1, 3])).toBeCloseTo(0.8, 5); + }); +}); diff --git a/scripts/bench/docgen-shared/stats.ts b/scripts/bench/docgen-shared/stats.ts new file mode 100644 index 000000000000..1229e38de4fb --- /dev/null +++ b/scripts/bench/docgen-shared/stats.ts @@ -0,0 +1,73 @@ +/** Statistics helpers shared by the docgen bench harnesses. */ +import type { MemorySample, SaveSample } from './samples.ts'; + +/** Throws on an empty input so a missing series fails loudly. */ +export function median(values: number[]): number { + if (values.length === 0) { + throw new Error('median() requires at least one value'); + } + const sorted = [...values].sort((a, b) => a - b); + const mid = Math.floor(sorted.length / 2); + return sorted.length % 2 === 1 ? sorted[mid] : (sorted[mid - 1] + sorted[mid]) / 2; +} + +/** Throws on an empty input so a missing series fails loudly. */ +export function mean(values: number[]): number { + if (values.length === 0) { + throw new Error('mean() requires at least one value'); + } + return values.reduce((a, b) => a + b, 0) / values.length; +} + +/** Least-squares slope of `values` vs index, in units-per-step. 0 for fewer than two points. */ +export function leastSquaresSlope(values: number[]): number { + const n = values.length; + if (n < 2) { + return 0; + } + const meanX = (n - 1) / 2; + const meanY = values.reduce((a, b) => a + b, 0) / n; + let num = 0; + let den = 0; + for (let i = 0; i < n; i++) { + num += (i - meanX) * (values[i] - meanY); + den += (i - meanX) ** 2; + } + return den === 0 ? 0 : num / den; +} + +/** The retained- and transient-memory figures every series harness derives from its save series. */ +export interface SeriesSummary { + /** Least-squares slope of retained heap over the save series, MB per save. */ + retainedSlope?: number; + /** Final retained sample minus the pre-series baseline, MB. */ + retainedGrowth?: number; + /** Per-save allocation spike above the retained baseline, MB. */ + transients: number[]; + /** Mean of {@link transients}. */ + avgTransient?: number; +} + +/** + * Derive the retained/transient series figures shared by every series harness. Kept here so the + * memory harness and the per-engine harnesses cannot drift apart in how they compute them. + */ +export function summarizeSeries(samples: SaveSample[], baseline: MemorySample): SeriesSummary { + // Samples taken without --expose-gc carry no retained heap; dropping them keeps a missing reading + // from being averaged in as a zero. + const gcSampled = samples.filter( + (s): s is SaveSample & { retainedHeapMb: number } => s.retainedHeapMb !== undefined + ); + const retained = gcSampled.map((s) => s.retainedHeapMb); + const transients = gcSampled.map((s) => s.heapUsedMb - s.retainedHeapMb); + const last = retained.at(-1); + return { + retainedSlope: retained.length ? leastSquaresSlope(retained) : undefined, + retainedGrowth: + last !== undefined && baseline.retainedHeapMb !== undefined + ? last - baseline.retainedHeapMb + : undefined, + transients, + avgTransient: transients.length ? mean(transients) : undefined, + }; +} diff --git a/scripts/package.json b/scripts/package.json index 685753fdef20..51ca20d9b19e 100644 --- a/scripts/package.json +++ b/scripts/package.json @@ -6,6 +6,7 @@ "scripts": { "bench-packages": "jiti ./bench/bench-packages.ts", "bench:docgen-memory": "node --import jiti/register ./bench/docgen-memory/gate.ts", + "bench:docgen-perf": "node ./bench/docgen-perf/run.ts", "build-package": "jiti ./build-package.ts", "check": "node ./check/check-package.ts", "check-package": "jiti ./check-package.ts", @@ -49,6 +50,7 @@ "dependencies": { "@actions/core": "^1.11.1", "@anthropic-ai/claude-agent-sdk": "^0.2.85", + "@compodoc/compodoc": "2.0.0", "@fal-works/esbuild-plugin-global-externals": "^2.1.2", "@google-cloud/bigquery": "^6.2.1", "@octokit/graphql": "^5.0.6", @@ -153,6 +155,9 @@ "typescript-native": "npm:typescript@^7.0.2", "uuid": "^9.0.1", "vitest": "^4.1.5", + "vue": "^3.2.47", + "vue-component-meta": "^3.2.7", + "vue-docgen-api": "^4.75.1", "wait-on": "^8.0.3", "window-size": "^1.1.1", "yaml": "^2.8.1", diff --git a/yarn.lock b/yarn.lock index 4da66d7b7fdd..2022fd24cb71 100644 --- a/yarn.lock +++ b/yarn.lock @@ -48,6 +48,13 @@ __metadata: languageName: node linkType: hard +"@aduh95/viz.js@npm:3.4.0": + version: 3.4.0 + resolution: "@aduh95/viz.js@npm:3.4.0" + checksum: 10c0/6acb40e881dc0b7ac7a6800a1508e8c420e1f6b09bd42e3289aa2199345729af8aec6ffc52dc72cfe678476fcc341f4a83bc2c3d59cd223232e8495f3a414ab0 + languageName: node + linkType: hard + "@ampproject/remapping@npm:2.3.0, @ampproject/remapping@npm:^2.2.0, @ampproject/remapping@npm:^2.3.0": version: 2.3.0 resolution: "@ampproject/remapping@npm:2.3.0" @@ -434,6 +441,38 @@ __metadata: languageName: node linkType: hard +"@angular-devkit/core@npm:22.0.4": + version: 22.0.4 + resolution: "@angular-devkit/core@npm:22.0.4" + dependencies: + ajv: "npm:8.20.0" + ajv-formats: "npm:3.0.1" + jsonc-parser: "npm:3.3.1" + picomatch: "npm:4.0.4" + rxjs: "npm:7.8.2" + source-map: "npm:0.7.6" + peerDependencies: + chokidar: ^5.0.0 + peerDependenciesMeta: + chokidar: + optional: true + checksum: 10c0/0d84969ad31113a48202c644038386f7dd120af950490125d7b8d56c4ebc9ca362cff56be70f90eb3c3be1c15cbcec11aad648e46caebabba82efc91e75dff20 + languageName: node + linkType: hard + +"@angular-devkit/schematics@npm:22.0.4": + version: 22.0.4 + resolution: "@angular-devkit/schematics@npm:22.0.4" + dependencies: + "@angular-devkit/core": "npm:22.0.4" + jsonc-parser: "npm:3.3.1" + magic-string: "npm:0.30.21" + ora: "npm:9.4.0" + rxjs: "npm:7.8.2" + checksum: 10c0/02288bf8f636d3517d03e97942ac1fab4ec8e4448d0d918abd96b06b1a8d0e043482993b51fc69e6c2fe43a3459fb4bbf8c3cccc803f17649ebc2f61c656d94d + languageName: node + linkType: hard + "@angular/animations@npm:^19.1.1": version: 19.2.15 resolution: "@angular/animations@npm:19.2.15" @@ -855,6 +894,13 @@ __metadata: languageName: node linkType: hard +"@arr/every@npm:^1.0.0": + version: 1.0.1 + resolution: "@arr/every@npm:1.0.1" + checksum: 10c0/f9769bc97eca37d1c0dedfda4c17506c0e1269d95c15a7478a49c4d84d43512a3f350b71c95e045cf4113d3398ebd6d298f2eea09196a1ed61c311bc4e6964e7 + languageName: node + linkType: hard + "@aw-web-design/x-default-browser@npm:1.4.126": version: 1.4.126 resolution: "@aw-web-design/x-default-browser@npm:1.4.126" @@ -2644,6 +2690,69 @@ __metadata: languageName: node linkType: hard +"@compodoc/compodoc@npm:2.0.0": + version: 2.0.0 + resolution: "@compodoc/compodoc@npm:2.0.0" + dependencies: + "@angular-devkit/schematics": "npm:22.0.4" + "@compodoc/ngd-transformer": "npm:^2.1.3" + body-parser: "npm:^2.3.0" + bootstrap.native: "npm:^5.1.10" + cheerio: "npm:1.2.0" + chokidar: "npm:^5.0.0" + commander: "npm:^15.0.0" + cosmiconfig: "npm:^9.0.2" + es6-shim: "npm:^0.35.8" + fs-extra: "npm:^11.3.5" + handlebars: "npm:^4.7.9" + html-entities: "npm:^2.6.0" + i18next: "npm:26.3.3" + json5: "npm:^2.2.3" + loglevel: "npm:^1.9.2" + loglevel-plugin-prefix: "npm:^0.8.4" + lunr: "npm:^2.3.9" + marked: "npm:18.0.5" + neotraverse: "npm:^1.0.1" + picocolors: "npm:^1.1.1" + polka: "npm:^0.5.2" + prismjs: "npm:^1.30.0" + semver: "npm:^7.8.5" + sirv: "npm:^3.0.2" + svg-pan-zoom: "npm:^3.6.2" + tablesort: "npm:^5.7.1" + tinyglobby: "npm:^0.2.17" + ts-morph: "npm:^28.0.0" + uuid: "npm:14.0.1" + vis-network: "npm:^10.1.0" + bin: + compodoc: bin/index-cli.js + checksum: 10c0/327e6b7d173f75be1996b59b89d3832467599782530f1e2b1ff8bcc73e977833e69e35e5ab57bfd79ddfb02e31ea58e5ee9a58201829751b330601f74af0efc8 + languageName: node + linkType: hard + +"@compodoc/ngd-core@npm:~2.1.1": + version: 2.1.1 + resolution: "@compodoc/ngd-core@npm:2.1.1" + dependencies: + ansi-colors: "npm:^4.1.3" + fancy-log: "npm:^2.0.0" + typescript: "npm:^5.0.4" + checksum: 10c0/f78f4231e39d96e397bb39614cfbe8dfc6b985eb9b2795cd5188a40f65eb60eb096dc29de61baa6d914fbc86ec7f21b5a2d02e6000b26d82ae3cd682aea79275 + languageName: node + linkType: hard + +"@compodoc/ngd-transformer@npm:^2.1.3": + version: 2.1.3 + resolution: "@compodoc/ngd-transformer@npm:2.1.3" + dependencies: + "@aduh95/viz.js": "npm:3.4.0" + "@compodoc/ngd-core": "npm:~2.1.1" + dot: "npm:^2.0.0-beta.1" + fs-extra: "npm:^11.1.1" + checksum: 10c0/fb7b13860fc022cda284ca0bae55f8c6610621a060c943740838b5fa2f6be7957c99c2750024cf949b2c8f1c5f41bc64cd8e28ab333f985627f5991bcaf5cdcf + languageName: node + linkType: hard + "@cypress/request@npm:3.0.6": version: 3.0.6 resolution: "@cypress/request@npm:3.0.6" @@ -6487,6 +6596,13 @@ __metadata: languageName: node linkType: hard +"@polka/url@npm:^0.5.0": + version: 0.5.0 + resolution: "@polka/url@npm:0.5.0" + checksum: 10c0/76324126e1608f4cdc5c6b11b618ca67612c37d45401f3fbcce4967294093761b7568cc6ae74ade8afbe744547e72e9d50bc38f690ddea0904a1d66a6b8a0093 + languageName: node + linkType: hard + "@polka/url@npm:^1.0.0-next.21, @polka/url@npm:^1.0.0-next.24": version: 1.0.0-next.29 resolution: "@polka/url@npm:1.0.0-next.29" @@ -10404,6 +10520,7 @@ __metadata: dependencies: "@actions/core": "npm:^1.11.1" "@anthropic-ai/claude-agent-sdk": "npm:^0.2.85" + "@compodoc/compodoc": "npm:2.0.0" "@fal-works/esbuild-plugin-global-externals": "npm:^2.1.2" "@google-cloud/bigquery": "npm:^6.2.1" "@octokit/graphql": "npm:^5.0.6" @@ -10512,6 +10629,9 @@ __metadata: verdaccio: "npm:^5.33.0" verdaccio-auth-memory: "npm:^10.3.1" vitest: "npm:^4.1.5" + vue: "npm:^3.2.47" + vue-component-meta: "npm:^3.2.7" + vue-docgen-api: "npm:^4.75.1" wait-on: "npm:^8.0.3" window-size: "npm:^1.1.1" yaml: "npm:^2.8.1" @@ -11195,6 +11315,29 @@ __metadata: languageName: node linkType: hard +"@thednp/event-listener@npm:^2.0.15": + version: 2.0.15 + resolution: "@thednp/event-listener@npm:2.0.15" + checksum: 10c0/02ce4402985f72af44049b6557443dd96a47f37301ea2dbc82f009aa56f22d9d5f06b45e4cb07f6c3bc5ff30c0faadd118095212f908b40b9521348aee231c39 + languageName: node + linkType: hard + +"@thednp/position-observer@npm:^1.1.3": + version: 1.1.3 + resolution: "@thednp/position-observer@npm:1.1.3" + dependencies: + "@thednp/shorty": "npm:^2.0.14" + checksum: 10c0/1fc95a4d8a67e5accab98a6aac571053190bd491702d09759242357d35d3b7d035998470d86abe6f113304b1a5ea44722e5caa1832ef51c36d8654e3890d1951 + languageName: node + linkType: hard + +"@thednp/shorty@npm:^2.0.14": + version: 2.0.14 + resolution: "@thednp/shorty@npm:2.0.14" + checksum: 10c0/72a4ecdbdbeb05953649721f412c1301f950450906fe5914cf4225b83e49f18dc0573f3526aa6efaaff2e442753b5e5b897ed91f5be0792ad4766c3462a0277d + languageName: node + linkType: hard + "@tmcp/adapter-valibot@npm:^0.1.5": version: 0.1.6 resolution: "@tmcp/adapter-valibot@npm:0.1.6" @@ -11260,6 +11403,17 @@ __metadata: languageName: node linkType: hard +"@ts-morph/common@npm:~0.29.0": + version: 0.29.0 + resolution: "@ts-morph/common@npm:0.29.0" + dependencies: + minimatch: "npm:^10.0.1" + path-browserify: "npm:^1.0.1" + tinyglobby: "npm:^0.2.14" + checksum: 10c0/97ab8ca66558b817e5475a8893ee1476ab760a3ac71fa9868413d95ddbaaf909eda81716e3a8d14291cbf449e624af66de81bcddd3ec2e36525728b4edf5e8ee + languageName: node + linkType: hard + "@tybys/wasm-util@npm:^0.10.3": version: 0.10.3 resolution: "@tybys/wasm-util@npm:0.10.3" @@ -13675,6 +13829,18 @@ __metadata: languageName: node linkType: hard +"ajv@npm:8.20.0": + version: 8.20.0 + resolution: "ajv@npm:8.20.0" + dependencies: + fast-deep-equal: "npm:^3.1.3" + fast-uri: "npm:^3.0.1" + json-schema-traverse: "npm:^1.0.0" + require-from-string: "npm:^2.0.2" + checksum: 10c0/5df9a1c8f83863cde1bd3a9ddb426f599718f88e3dc9153616c79fb28e0be455335830d7f21d745576519f057b371352daa31047b6a33d7036fe08777d60cf2a + languageName: node + linkType: hard + "ajv@npm:^6.10.0, ajv@npm:^6.12.3, ajv@npm:^6.12.4, ajv@npm:^6.12.5, ajv@npm:^6.12.6": version: 6.12.6 resolution: "ajv@npm:6.12.6" @@ -13711,7 +13877,7 @@ __metadata: languageName: node linkType: hard -"ansi-colors@npm:4.1.3, ansi-colors@npm:^4.1.1": +"ansi-colors@npm:4.1.3, ansi-colors@npm:^4.1.1, ansi-colors@npm:^4.1.3": version: 4.1.3 resolution: "ansi-colors@npm:4.1.3" checksum: 10c0/ec87a2f59902f74e61eada7f6e6fe20094a628dab765cfdbd03c3477599368768cffccdb5d3bb19a1b6c99126783a143b1fee31aab729b31ffe5836c7e5e28b9 @@ -14819,6 +14985,23 @@ __metadata: languageName: node linkType: hard +"body-parser@npm:^2.3.0": + version: 2.3.0 + resolution: "body-parser@npm:2.3.0" + dependencies: + bytes: "npm:^3.1.2" + content-type: "npm:^2.0.0" + debug: "npm:^4.4.3" + http-errors: "npm:^2.0.1" + iconv-lite: "npm:^0.7.2" + on-finished: "npm:^2.4.1" + qs: "npm:^6.15.2" + raw-body: "npm:^3.0.2" + type-is: "npm:^2.1.0" + checksum: 10c0/2a8fbbdc471b588338555a3e1a597d1eb0ad0c21cf20fdc3bac5d3f8d9c3a4b19b4163575ab852a43a5dfc0df7770ced5284f979e561f1a24c2f851ce89e695a + languageName: node + linkType: hard + "body-parser@npm:~1.20.5": version: 1.20.5 resolution: "body-parser@npm:1.20.5" @@ -14856,6 +15039,17 @@ __metadata: languageName: node linkType: hard +"bootstrap.native@npm:^5.1.10": + version: 5.1.10 + resolution: "bootstrap.native@npm:5.1.10" + dependencies: + "@thednp/event-listener": "npm:^2.0.15" + "@thednp/position-observer": "npm:^1.1.3" + "@thednp/shorty": "npm:^2.0.14" + checksum: 10c0/6484411fc4ba5b7d1bdd9da039d07efc32c30b5b216b1442cea4ef235b6398b22ccf721b7216258113f8e9cca86558bf6155abb98ebe7ff55a51e4345130e997 + languageName: node + linkType: hard + "bplist-parser@npm:^0.2.0": version: 0.2.0 resolution: "bplist-parser@npm:0.2.0" @@ -15315,7 +15509,7 @@ __metadata: languageName: node linkType: hard -"bytes@npm:3.1.2, bytes@npm:~3.1.2": +"bytes@npm:3.1.2, bytes@npm:^3.1.2, bytes@npm:~3.1.2": version: 3.1.2 resolution: "bytes@npm:3.1.2" checksum: 10c0/76d1c43cbd602794ad8ad2ae94095cddeb1de78c5dddaa7005c51af10b0176c69971a6d88e805a90c2b6550d76636e43c40d8427a808b8645ede885de4a0358e @@ -15635,7 +15829,7 @@ __metadata: languageName: node linkType: hard -"cheerio@npm:^1.0.0": +"cheerio@npm:1.2.0, cheerio@npm:^1.0.0": version: 1.2.0 resolution: "cheerio@npm:1.2.0" dependencies: @@ -15991,6 +16185,13 @@ __metadata: languageName: node linkType: hard +"code-block-writer@npm:^13.0.3": + version: 13.0.3 + resolution: "code-block-writer@npm:13.0.3" + checksum: 10c0/87db97b37583f71cfd7eced8bf3f0a0a0ca53af912751a734372b36c08cd27f3e8a4878ec05591c0cd9ae11bea8add1423e132d660edd86aab952656dd41fd66 + languageName: node + linkType: hard + "code-point-at@npm:^1.0.0": version: 1.1.0 resolution: "code-point-at@npm:1.1.0" @@ -16112,6 +16313,13 @@ __metadata: languageName: node linkType: hard +"commander@npm:^15.0.0": + version: 15.0.0 + resolution: "commander@npm:15.0.0" + checksum: 10c0/539229c171914ea1ccd45ee5f10d924289a12a684ea3a7a44147abe54003c35ed6de9ae4ad198d88c29fdf403dca0428451a388234e6dc01b6faa978fb207206 + languageName: node + linkType: hard + "commander@npm:^2.18.0, commander@npm:^2.19.0, commander@npm:^2.20.0, commander@npm:^2.8.1": version: 2.20.3 resolution: "commander@npm:2.20.3" @@ -16280,6 +16488,13 @@ __metadata: languageName: node linkType: hard +"content-type@npm:^2.0.0": + version: 2.0.0 + resolution: "content-type@npm:2.0.0" + checksum: 10c0/491539fff707d7594b0ca4fabcc084bef2a31ffa754ff0a4f80c4377e3963cff0394317f9271c24087596c97fa675bc123d61fa34ffe65b4904e7d3d3098de72 + languageName: node + linkType: hard + "content-type@npm:~1.0.4, content-type@npm:~1.0.5": version: 1.0.5 resolution: "content-type@npm:1.0.5" @@ -16507,6 +16722,23 @@ __metadata: languageName: node linkType: hard +"cosmiconfig@npm:^9.0.2": + version: 9.0.2 + resolution: "cosmiconfig@npm:9.0.2" + dependencies: + env-paths: "npm:^2.2.1" + import-fresh: "npm:^3.3.0" + js-yaml: "npm:^4.1.0" + parse-json: "npm:^5.2.0" + peerDependencies: + typescript: ">=4.9.5" + peerDependenciesMeta: + typescript: + optional: true + checksum: 10c0/132d32863c0b3d9ace7010a10011e09089532b36e3b11e02004d671dcbd8b8f2f16293b82d510d9c15890f30358baf8722127c7b1c011449c90b6a178f9c6594 + languageName: node + linkType: hard + "create-ecdh@npm:^4.0.4": version: 4.0.4 resolution: "create-ecdh@npm:4.0.4" @@ -17511,6 +17743,13 @@ __metadata: languageName: node linkType: hard +"dot@npm:^2.0.0-beta.1": + version: 2.0.0-beta.1 + resolution: "dot@npm:2.0.0-beta.1" + checksum: 10c0/28876e214576210384fd49bcf2f3c4afc6478cec7de18e923c07a085dd1f3c565ff162c53911040267fc9ff96fa732dde8676c503ffd60d5a522101bc1ad1e21 + languageName: node + linkType: hard + "dotenv-expand@npm:^10.0.0": version: 10.0.0 resolution: "dotenv-expand@npm:10.0.0" @@ -18311,6 +18550,13 @@ __metadata: languageName: node linkType: hard +"es6-shim@npm:^0.35.8": + version: 0.35.8 + resolution: "es6-shim@npm:0.35.8" + checksum: 10c0/e54e147677b9cd57c96bbc2332c3096ab861040b22daa27728204cacf9aa656d6a3eeb5367a037b276bfbe442ec866938a95b4612accf0cd24d01c209e48eaf9 + languageName: node + linkType: hard + "esast-util-from-estree@npm:^2.0.0": version: 2.0.0 resolution: "esast-util-from-estree@npm:2.0.0" @@ -19309,6 +19555,15 @@ __metadata: languageName: node linkType: hard +"fancy-log@npm:^2.0.0": + version: 2.0.0 + resolution: "fancy-log@npm:2.0.0" + dependencies: + color-support: "npm:^1.1.3" + checksum: 10c0/a6e116f3346756a7363eea343b551c1375d2cd2afc2a92d224feb78589b6b3cff85db9dc5d5df89792a0f7c1e17f731f52cb3d2807302f0516422be6269b95a8 + languageName: node + linkType: hard + "fast-content-type-parse@npm:^2.0.0": version: 2.0.1 resolution: "fast-content-type-parse@npm:2.0.1" @@ -19966,6 +20221,17 @@ __metadata: languageName: node linkType: hard +"fs-extra@npm:^11.1.1, fs-extra@npm:^11.3.5": + version: 11.4.0 + resolution: "fs-extra@npm:11.4.0" + dependencies: + graceful-fs: "npm:^4.2.0" + jsonfile: "npm:^6.0.1" + universalify: "npm:^2.0.0" + checksum: 10c0/1e311eca820a12d3d795f2043dcd5628d017f4d21e634f3f9ef314ae68bee9979758262fb9cb35ffa6f26454c3fffe8b15558733e91e8fc729b07b10bb98d8b6 + languageName: node + linkType: hard + "fs-extra@npm:^5.0.0": version: 5.0.0 resolution: "fs-extra@npm:5.0.0" @@ -20710,6 +20976,24 @@ __metadata: languageName: node linkType: hard +"handlebars@npm:^4.7.9": + version: 4.7.9 + resolution: "handlebars@npm:4.7.9" + dependencies: + minimist: "npm:^1.2.5" + neo-async: "npm:^2.6.2" + source-map: "npm:^0.6.1" + uglify-js: "npm:^3.1.4" + wordwrap: "npm:^1.0.0" + dependenciesMeta: + uglify-js: + optional: true + bin: + handlebars: bin/handlebars + checksum: 10c0/22f8105a7e68e81aff2662bb434edf05f757d21d850731d71cec886d69c10cd33d3c43e34b2892968ec62de8241611851d3d0674c8ef324ea3e01dc66262faa9 + languageName: node + linkType: hard + "happy-dom@npm:^17.6.3": version: 17.6.3 resolution: "happy-dom@npm:17.6.3" @@ -21095,7 +21379,7 @@ __metadata: languageName: node linkType: hard -"html-entities@npm:^2.1.0": +"html-entities@npm:^2.1.0, html-entities@npm:^2.6.0": version: 2.6.0 resolution: "html-entities@npm:2.6.0" checksum: 10c0/7c8b15d9ea0cd00dc9279f61bab002ba6ca8a7a0f3c36ed2db3530a67a9621c017830d1d2c1c65beb9b8e3436ea663e9cf8b230472e0e413359399413b27c8b7 @@ -21217,19 +21501,7 @@ __metadata: languageName: node linkType: hard -"http-errors@npm:~1.6.2": - version: 1.6.3 - resolution: "http-errors@npm:1.6.3" - dependencies: - depd: "npm:~1.1.2" - inherits: "npm:2.0.3" - setprototypeof: "npm:1.1.0" - statuses: "npm:>= 1.4.0 < 2" - checksum: 10c0/17ec4046ee974477778bfdd525936c254b872054703ec2caa4d6f099566b8adade636ae6aeeacb39302c5cd6e28fb407ebd937f500f5010d0b6850750414ff78 - languageName: node - linkType: hard - -"http-errors@npm:~2.0.0, http-errors@npm:~2.0.1": +"http-errors@npm:^2.0.1, http-errors@npm:~2.0.0, http-errors@npm:~2.0.1": version: 2.0.1 resolution: "http-errors@npm:2.0.1" dependencies: @@ -21242,6 +21514,18 @@ __metadata: languageName: node linkType: hard +"http-errors@npm:~1.6.2": + version: 1.6.3 + resolution: "http-errors@npm:1.6.3" + dependencies: + depd: "npm:~1.1.2" + inherits: "npm:2.0.3" + setprototypeof: "npm:1.1.0" + statuses: "npm:>= 1.4.0 < 2" + checksum: 10c0/17ec4046ee974477778bfdd525936c254b872054703ec2caa4d6f099566b8adade636ae6aeeacb39302c5cd6e28fb407ebd937f500f5010d0b6850750414ff78 + languageName: node + linkType: hard + "http-parser-js@npm:>=0.5.1": version: 0.5.10 resolution: "http-parser-js@npm:0.5.10" @@ -21440,6 +21724,18 @@ __metadata: languageName: node linkType: hard +"i18next@npm:26.3.3": + version: 26.3.3 + resolution: "i18next@npm:26.3.3" + peerDependencies: + typescript: ^5 || ^6 + peerDependenciesMeta: + typescript: + optional: true + checksum: 10c0/7a6676a8bade1683d23c3187b92d571528eeadf9f3d6776b4ef1bd27789d2ee7594a9aa7ca05394d8e11c931c8c8f224d85b2b5059a5613cb79b04b8d46a291e + languageName: node + linkType: hard + "iconv-lite@npm:0.4.24, iconv-lite@npm:~0.4.24": version: 0.4.24 resolution: "iconv-lite@npm:0.4.24" @@ -21458,6 +21754,15 @@ __metadata: languageName: node linkType: hard +"iconv-lite@npm:^0.7.2, iconv-lite@npm:~0.7.0": + version: 0.7.3 + resolution: "iconv-lite@npm:0.7.3" + dependencies: + safer-buffer: "npm:>= 2.1.2 < 3.0.0" + checksum: 10c0/be2fd2414f7e94be3a63063fa0ad5919c16bc7b6e00c77eea0765ced6db405739f38dd51016583058fba25cc1d0106ba30b83fbb7a7e32f824d4db76f23d8d33 + languageName: node + linkType: hard + "icss-utils@npm:^5.0.0, icss-utils@npm:^5.1.0": version: 5.1.0 resolution: "icss-utils@npm:5.1.0" @@ -23852,6 +24157,20 @@ __metadata: languageName: node linkType: hard +"loglevel-plugin-prefix@npm:^0.8.4": + version: 0.8.4 + resolution: "loglevel-plugin-prefix@npm:0.8.4" + checksum: 10c0/357524eec4c165ff823b5bbf72e8373ff529e5cb95c1f4b20749847bd5b5b16ab328d6d33d1a9019f1a2dc52e28fca5d595e52f2ee20e24986182a6f9552a9ec + languageName: node + linkType: hard + +"loglevel@npm:^1.9.2": + version: 1.9.2 + resolution: "loglevel@npm:1.9.2" + checksum: 10c0/1e317fa4648fe0b4a4cffef6de037340592cee8547b07d4ce97a487abe9153e704b98451100c799b032c72bb89c9366d71c9fb8192ada8703269263ae77acdc7 + languageName: node + linkType: hard + "longest-streak@npm:^3.0.0": version: 3.1.0 resolution: "longest-streak@npm:3.1.0" @@ -23955,6 +24274,13 @@ __metadata: languageName: node linkType: hard +"lunr@npm:^2.3.9": + version: 2.3.9 + resolution: "lunr@npm:2.3.9" + checksum: 10c0/77d7dbb4fbd602aac161e2b50887d8eda28c0fa3b799159cee380fbb311f1e614219126ecbbd2c3a9c685f1720a8109b3c1ca85cc893c39b6c9cc6a62a1d8a8b + languageName: node + linkType: hard + "lz-string@npm:^1.5.0": version: 1.5.0 resolution: "lz-string@npm:1.5.0" @@ -24121,6 +24447,15 @@ __metadata: languageName: node linkType: hard +"marked@npm:18.0.5": + version: 18.0.5 + resolution: "marked@npm:18.0.5" + bin: + marked: bin/marked.js + checksum: 10c0/22763935a0a243c41851b010a086ea85a09951e379948b6aa629b6cecdcd66e3a5e8c4acac2f6b29c183c20609d3e5102495270207dfd5f4c46ddd773a4ddb9b + languageName: node + linkType: hard + "matcher-collection@npm:^1.0.0, matcher-collection@npm:^1.1.1": version: 1.1.2 resolution: "matcher-collection@npm:1.1.2" @@ -24140,6 +24475,15 @@ __metadata: languageName: node linkType: hard +"matchit@npm:^1.0.0": + version: 1.1.0 + resolution: "matchit@npm:1.1.0" + dependencies: + "@arr/every": "npm:^1.0.0" + checksum: 10c0/f2cd8702ea37dd9859cf1df762581c822180eb72372b9240397a80320cb4ffb4828b3d8ed0beff2429daf500d1ec746c2963dbb3c6386f1b5b9eb969e0f1caaa + languageName: node + linkType: hard + "math-intrinsics@npm:^1.1.0": version: 1.1.0 resolution: "math-intrinsics@npm:1.1.0" @@ -24299,6 +24643,13 @@ __metadata: languageName: node linkType: hard +"media-typer@npm:^1.1.0": + version: 1.1.1 + resolution: "media-typer@npm:1.1.1" + checksum: 10c0/924644de220c3107fc53ec0299b7c1488503470265d32f1492535bd798cc369ba7ceaf9c2a4533361590dcf7208d1cc7b7032dbc1a5918773f5f5d912cab5211 + languageName: node + linkType: hard + "memfs-or-file-map-to-github-branch@npm:^1.3.0": version: 1.3.0 resolution: "memfs-or-file-map-to-github-branch@npm:1.3.0" @@ -24776,6 +25127,15 @@ __metadata: languageName: node linkType: hard +"mime-types@npm:^3.0.0": + version: 3.0.2 + resolution: "mime-types@npm:3.0.2" + dependencies: + mime-db: "npm:^1.54.0" + checksum: 10c0/35a0dd1035d14d185664f346efcdb72e93ef7a9b6e9ae808bd1f6358227010267fab52657b37562c80fc888ff76becb2b2938deb5e730818b7983bf8bd359767 + languageName: node + linkType: hard + "mime-types@npm:^3.0.1": version: 3.0.1 resolution: "mime-types@npm:3.0.1" @@ -25334,6 +25694,13 @@ __metadata: languageName: node linkType: hard +"neotraverse@npm:^1.0.1": + version: 1.0.1 + resolution: "neotraverse@npm:1.0.1" + checksum: 10c0/98e5d2d1974f91d527851a9dcfceb62701a1d31970b324b9f35fbdd244d173edc7a379ea1773f64771a9dc282486e3d3b693b24e14b1e969291612b2ff6b5054 + languageName: node + linkType: hard + "next@npm:^15.2.3": version: 15.5.6 resolution: "next@npm:15.5.6" @@ -26163,6 +26530,22 @@ __metadata: languageName: node linkType: hard +"ora@npm:9.4.0": + version: 9.4.0 + resolution: "ora@npm:9.4.0" + dependencies: + chalk: "npm:^5.6.2" + cli-cursor: "npm:^5.0.0" + cli-spinners: "npm:^3.2.0" + is-interactive: "npm:^2.0.0" + is-unicode-supported: "npm:^2.1.0" + log-symbols: "npm:^7.0.1" + stdin-discarder: "npm:^0.3.2" + string-width: "npm:^8.1.0" + checksum: 10c0/8f2d7a8869cd68607797ac0ffe9f5cdceeb6009437672510d9920aea794cdd055164e3fe804248624c4940a71b22f94f1ffd94ce8fecf0746baef97a5c121a91 + languageName: node + linkType: hard + "ordered-binary@npm:^1.5.3": version: 1.6.0 resolution: "ordered-binary@npm:1.6.0" @@ -27489,6 +27872,16 @@ __metadata: languageName: node linkType: hard +"polka@npm:^0.5.2": + version: 0.5.2 + resolution: "polka@npm:0.5.2" + dependencies: + "@polka/url": "npm:^0.5.0" + trouter: "npm:^2.0.1" + checksum: 10c0/113005fe146e1ee67bc6e1a5438c71e1b59ac05a8bc435d593be210e21f712e6b9ba4d163072e3c941f8efbb7516df9c55bf18934948f91623d03dc9657a1410 + languageName: node + linkType: hard + "polka@npm:^1.0.0-next.28": version: 1.0.0-next.28 resolution: "polka@npm:1.0.0-next.28" @@ -28243,6 +28636,16 @@ __metadata: languageName: node linkType: hard +"qs@npm:^6.15.2": + version: 6.15.3 + resolution: "qs@npm:6.15.3" + dependencies: + es-define-property: "npm:^1.0.1" + side-channel: "npm:^1.1.1" + checksum: 10c0/8f3f6e45ece255347d57696628401cde29e9ec649fff698b53bd3150dea7cefdf33036e1bc1826b9f110bfa7cb0ec4ab9f5297eca628ce216c55af82c304e08e + languageName: node + linkType: hard + "qs@npm:~6.5.2": version: 6.5.3 resolution: "qs@npm:6.5.3" @@ -28327,6 +28730,18 @@ __metadata: languageName: node linkType: hard +"raw-body@npm:^3.0.2": + version: 3.0.2 + resolution: "raw-body@npm:3.0.2" + dependencies: + bytes: "npm:~3.1.2" + http-errors: "npm:~2.0.1" + iconv-lite: "npm:~0.7.0" + unpipe: "npm:~1.0.0" + checksum: 10c0/d266678d08e1e7abea62c0ce5864344e980fa81c64f6b481e9842c5beaed2cdcf975f658a3ccd67ad35fc919c1f6664ccc106067801850286a6cbe101de89f29 + languageName: node + linkType: hard + "raw-body@npm:~2.5.3": version: 2.5.3 resolution: "raw-body@npm:2.5.3" @@ -30366,6 +30781,15 @@ __metadata: languageName: node linkType: hard +"semver@npm:^7.8.5": + version: 7.8.5 + resolution: "semver@npm:7.8.5" + bin: + semver: bin/semver.js + checksum: 10c0/b1f3127a5be8125a94f37188b361c212466c292c6910adce3ec106cff5dc211ccaedc4739c11bb70fda59d6fc1f040a9bca289f4e093451521a2372e5231fe0c + languageName: node + linkType: hard + "send@npm:0.19.0": version: 0.19.0 resolution: "send@npm:0.19.0" @@ -30720,6 +31144,16 @@ __metadata: languageName: node linkType: hard +"side-channel-list@npm:^1.0.1": + version: 1.0.1 + resolution: "side-channel-list@npm:1.0.1" + dependencies: + es-errors: "npm:^1.3.0" + object-inspect: "npm:^1.13.4" + checksum: 10c0/d346c787fd2f9f1c2fdea14f00e8250118db0e7596d85a6cb9faa75f105d31a73a8f7a341c93d7df2a2429098c3d37a77bd3be9e88c37094b8c01807bc77c7a2 + languageName: node + linkType: hard + "side-channel-map@npm:^1.0.1": version: 1.0.1 resolution: "side-channel-map@npm:1.0.1" @@ -30758,6 +31192,19 @@ __metadata: languageName: node linkType: hard +"side-channel@npm:^1.1.1": + version: 1.1.1 + resolution: "side-channel@npm:1.1.1" + dependencies: + es-errors: "npm:^1.3.0" + object-inspect: "npm:^1.13.4" + side-channel-list: "npm:^1.0.1" + side-channel-map: "npm:^1.0.1" + side-channel-weakmap: "npm:^1.0.2" + checksum: 10c0/dc0ab81d67f61bda9247d053ce93f41c3fd8ad2bdcb9cf9d8d2f8540d488f26d87a5e99ebfc07eea49ec025867b2452b705442d974b1478f0395e69f6bfb3270 + languageName: node + linkType: hard + "siginfo@npm:^2.0.0": version: 2.0.0 resolution: "siginfo@npm:2.0.0" @@ -31278,7 +31725,7 @@ __metadata: languageName: node linkType: hard -"stdin-discarder@npm:^0.3.1": +"stdin-discarder@npm:^0.3.1, stdin-discarder@npm:^0.3.2": version: 0.3.2 resolution: "stdin-discarder@npm:0.3.2" checksum: 10c0/5dbaba9efbcb447a4450d5ae19794641ea9166abe96dc4b5547a109db1bb6e8bdb17bbe1029e02ca8d9d8ee996b7c7cbcce12b12c18c121871cd4f574292381a @@ -32007,6 +32454,13 @@ __metadata: languageName: node linkType: hard +"svg-pan-zoom@npm:^3.6.2": + version: 3.6.2 + resolution: "svg-pan-zoom@npm:3.6.2" + checksum: 10c0/dc05030bd6545d523de71e38d79d6830f780a74416a609269cc378fe8778796e794cb3e27f0f5b20bd7ee930597820e5b4463e4e4e0708127bed81e020339ec0 + languageName: node + linkType: hard + "symlink-or-copy@npm:^1.0.0, symlink-or-copy@npm:^1.0.1, symlink-or-copy@npm:^1.1.8, symlink-or-copy@npm:^1.2.0, symlink-or-copy@npm:^1.3.1": version: 1.3.1 resolution: "symlink-or-copy@npm:1.3.1" @@ -32036,6 +32490,13 @@ __metadata: languageName: node linkType: hard +"tablesort@npm:^5.7.1": + version: 5.7.1 + resolution: "tablesort@npm:5.7.1" + checksum: 10c0/be69ec398438695b87589de702861d4ca126dba0c4a8c1f4bbc28ffa4ea257a623ee764abd769ec88d7f8347dc36b5157b9d36d282c181a8c43b19413cd88854 + languageName: node + linkType: hard + "tagged-tag@npm:^1.0.0": version: 1.0.0 resolution: "tagged-tag@npm:1.0.0" @@ -32649,6 +33110,15 @@ __metadata: languageName: node linkType: hard +"trouter@npm:^2.0.1": + version: 2.0.1 + resolution: "trouter@npm:2.0.1" + dependencies: + matchit: "npm:^1.0.0" + checksum: 10c0/9d294dbe74d45649d929d3abf0d4cb8e508985a2ea0f8be41b3b097a985352b14f65dd76b68ae651731128995eceddba375b15719356c419bbfe754c5fb5673c + languageName: node + linkType: hard + "trouter@npm:^4.0.0": version: 4.0.0 resolution: "trouter@npm:4.0.0" @@ -32698,6 +33168,16 @@ __metadata: languageName: node linkType: hard +"ts-morph@npm:^28.0.0": + version: 28.0.0 + resolution: "ts-morph@npm:28.0.0" + dependencies: + "@ts-morph/common": "npm:~0.29.0" + code-block-writer: "npm:^13.0.3" + checksum: 10c0/969570a5e983b4193735fff43878ce7b78787b4860bf3d23f330c86d7de707286969e5a82d486dce333eea27bf9d373a46f9de58a80c0bf86d2a462620563a55 + languageName: node + linkType: hard + "ts-pnp@npm:^1.1.6": version: 1.2.0 resolution: "ts-pnp@npm:1.2.0" @@ -32867,6 +33347,17 @@ __metadata: languageName: node linkType: hard +"type-is@npm:^2.1.0": + version: 2.1.0 + resolution: "type-is@npm:2.1.0" + dependencies: + content-type: "npm:^2.0.0" + media-typer: "npm:^1.1.0" + mime-types: "npm:^3.0.0" + checksum: 10c0/a6018f8f509de48f2c7429305e3a920e73b374fa93127dd0877ae1c2df65a5d33907caac8afb0c37a9b9fc7c49f29e3f55d668963dc845d966930b667c07f50e + languageName: node + linkType: hard + "type-is@npm:~1.6.18": version: 1.6.18 resolution: "type-is@npm:1.6.18" @@ -33539,6 +34030,15 @@ __metadata: languageName: node linkType: hard +"uuid@npm:14.0.1": + version: 14.0.1 + resolution: "uuid@npm:14.0.1" + bin: + uuid: dist-node/bin/uuid + checksum: 10c0/2d961289097cb68c37d93d1da0b5c3aafc1eb9948a455cca8d0ae53a099b2fe583b8933254a84097f700e6bac2e23033364904df0467c22551d5d9611febcaf6 + languageName: node + linkType: hard + "uuid@npm:^11.1.0": version: 11.1.0 resolution: "uuid@npm:11.1.0" @@ -33756,6 +34256,20 @@ __metadata: languageName: node linkType: hard +"vis-network@npm:^10.1.0": + version: 10.1.0 + resolution: "vis-network@npm:10.1.0" + peerDependencies: + "@egjs/hammerjs": ^2.0.0 + component-emitter: ^1.3.0 || ^2.0.0 + keycharm: ^0.2.0 || ^0.3.0 || ^0.4.0 + uuid: ^3.4.0 || ^7.0.0 || ^8.0.0 || ^9.0.0 || ^10.0.0 || ^11.0.0 || ^13.0.0 || ^14.0.0 + vis-data: ">=8.0.0" + vis-util: ">=6.0.0" + checksum: 10c0/c3f553f03f80e096c7f8cea4fb98011906dde67c65e7d1e8ca0ed5811dad0dc9f648dde93a6a61ce3ac24f153a7a4eb96e931104c8ad1801701a50db687f5bb6 + languageName: node + linkType: hard + "vite-dev-rpc@npm:^1.1.0": version: 1.1.0 resolution: "vite-dev-rpc@npm:1.1.0"