[Security Solution] Make kbn-openapi-generator producing lazy loaded Zod schemas

[maximpn]

Summary

Reduces the baseline heap footprint of the many generated Zod schemas (OpenAPI *.gen.ts files) by deferring their construction until first use, and letting the GC reclaim materialized schemas once no consumer is holding on to them. Generated schemas are declared at module-load time but only a subset is actually exercised at runtime, so materializing all of them eagerly wastes memory — and pinning them for the process lifetime wastes memory even for schemas used once.

Changes

• Add lazySchema(factory) in @kbn/zod/v4 — a Zod-typed wrapper around lazyGCableObject used by generated schemas. Unused schemas stay as a single Proxy + closure; transiently-used schemas are collectible after their last reference is dropped.
• Update the zod_operation_schema.handlebars template in @kbn/openapi-generator so generated request/response/params/body schemas and named component schemas are wrapped in lazySchema(() => …).

Caveat documented on the helper: instanceof z.ZodType on a lazy schema is false because the Proxy target is an empty object. Zod internals and typical consumers use structural _zod / .def checks rather than instanceof, so this is safe in practice.

Feature flag

Lazy loaded Zod schemas have been smoke tested and there aren't noticeable performance degradations revealed. Anyway disableLazyZodSchemas feature flag has been added to have an ability to disable this optimization. After applying the changes Kibana restart is required. To make the optimization disabled add the following to the Kibana config

feature_flags:
  overrides:
    disableLazyZodSchemas: true

Identify risks

• Consumers that rely on instanceof z.ZodType / instanceof z.ZodObject against a generated schema will see false. Mitigation: Zod and common helpers use structural checks (_zod, .def) rather than instanceof, and this is called out in the helper JSDoc. Reviewers should flag any instanceof usage against generated schemas.
• First-access and rebuild cost: the first property access on a previously unused schema materializes it. Because the cache is a WeakRef, if the materialized schema is reclaimed between uses (only possible once the caller has dropped its reference), the next access rebuilds it from the factory — so repeatedly re-materializing the same schema across GC cycles adds overhead. Hot paths should retain a reference (e.g. const s = Schema; s.parse(...)) for the duration of their work.
• Large surface change in regenerated *.gen.ts files — regeneration is mechanical from the updated handlebars template.

Part of #264170

[sdesalas]

Local testing: Does this improve heap memory.

Seems like there is a visible improvement by lazy loading in my local --dev environment while testing installion of detection rules, though it might just be postponed until the user tries out all the features in kibana.

Note that idle memory consumption after installing prebuilt rules with kibana-puppetter-scripts is almost 100mb lower than in main. Peak memory is reduced but not as much, around 25MB lower, although GC makes it hard to estimate exactly how much this would mean in an instance with lower memory limits. Could be lower even (better) since GC appears less aggressive when more memory is available to the process.

• 📦 Raw data: main vs lazy-loading.zip

Two datasets × three runs. Metric: heap_used from kibana-memory-profile-*.csv.

1400 in folder names: runs used NODE_OPTIONS="--max_old_space_size=1400" (V8 old-space limit 1400 MiB).

Method

• Time window: For each run, the first sample’s timestamp_ms is t₀. Only rows with elapsed ≥ 30 s after t₀ are included (startup/warmup excluded).
• Per run: peak = max heap_used in that window; average = arithmetic mean of heap_used over included samples.
• Logs: A quick scan of paired kibana.output.*.txt files found no ERROR / FATAL / OOM-style lines; comparison below is CSV-only.

Dataset: main-1400-round* (baseline)

Run folder	Peak heap_used (after 30 s)	Avg heap_used (after 30 s)	Samples (included / total)
main-1400-round1	1235.66 MiB	1121.35 MiB	555 / 600
main-1400-round2	1257.13 MiB	1100.33 MiB	555 / 600
main-1400-round3	1242.56 MiB	1120.50 MiB	560 / 600

Across 3 runs (mean of per-run values): peak ≈ 1245.1 MiB, average ≈ 1114.1 MiB.

Dataset: lazy-loading-1400-round* (current work)

Run folder	Peak heap_used (after 30 s)	Avg heap_used (after 30 s)	Samples (included / total)
lazy-loading-1400-round1	1219.63 MiB	1060.34 MiB	554 / 600
lazy-loading-1400-round2	1218.85 MiB	1068.25 MiB	555 / 600
lazy-loading-1400-round3	1219.34 MiB	1070.83 MiB	492 / 537

Across 3 runs (mean of per-run values): peak ≈ 1219.3 MiB, average ≈ 1066.5 MiB.

Which set is more memory efficient?

lazy-loading-1400-round* (current work) is more memory efficient on heap_used after the 30 s cutoff:

• Peak: ~25.8 MiB lower mean peak than main-1400-round* (~2.1% lower).
• Average: ~47.6 MiB lower mean of per-run averages than main (~4.3% lower).

All six runs show the same ordering: lazy-loading peaks are clustered ~1219 MiB vs main ~1236–1257 MiB; lazy-loading averages are lower in every run.

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: e8cdbef
• Storybooks Preview

Failed CI Steps

• Build Cloud Image
• Post-Build

Metrics [docs]

‼️ ERROR: no builds found for mergeBase sha [7c23f84]

History

• 💔 Build #432360 failed a78fc75
• 💔 Build #432331 failed 2b971ea
• 💔 Build #432284 failed fce6939
• 💔 Build #432211 failed 1b28c20
• 💔 Build #431901 failed af24903

cc @maximpn

[rudolf]

We should remove the unused readme but can be done in a follow-up PR too

[kibanamachine]

Starting backport for target branches: 9.4

https://github.com/elastic/kibana/actions/runs/24797125575

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	9.4	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 264125

Questions ?

Please refer to the Backport tool documentation

[maximpn]

💚 All backports created successfully

Status	Branch	Result
✅	9.4

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

[rylnd]

Looks like this was force-merged for expedience. I was mid-review, so: here are the outstanding comments for if/when we do some followup.

[rylnd]

I noticed that we don't have coverage for .merge() (which is used by allOf), nor for the actual materialization mechanism for lazySchema that would catch regressions to its implementation. Here are some examples of what I'm thinking:

Suggested change

	it('forwards .merge() on the underlying object schema', () => {
	const Base = lazySchema(() => z.object({ id: z.string() }));
	const Merged = Base.merge(z.object({ name: z.string() }));
	expect(Merged.parse({ id: 'a', name: 'b' })).toEqual({ id: 'a', name: 'b' });
	expect(Merged.safeParse({ id: 'a' }).success).toBe(false);
	});
	describe('materialization', () => {
	it('does not invoke factories for unused schemas', () => {
	const factories = Array.from({ length: 20 }, (_, i) =>
	jest.fn(() => z.object({ id: z.literal(i) }))
	);
	const schemas = factories.map((f) => lazySchema(f));
	for (const f of factories) {
	expect(f).not.toHaveBeenCalled();
	}
	schemas[3].parse({ id: 3 });
	schemas[7].parse({ id: 7 });
	expect(factories[3]).toHaveBeenCalledTimes(1);
	expect(factories[7]).toHaveBeenCalledTimes(1);
	const untouched = factories.filter((_, i) => i !== 3 && i !== 7);
	for (const f of untouched) {
	expect(f).not.toHaveBeenCalled();
	}
	});
	it('does not materialize a lazy schema used only as a nested reference until the outer schema parses', () => {
	const innerFactory = jest.fn(() => z.object({ value: z.number() }));
	const Inner = lazySchema(innerFactory);
	const Outer = lazySchema(() => z.object({ inner: Inner, tag: z.string() }));
	expect(innerFactory).not.toHaveBeenCalled();
	Outer.parse({ inner: { value: 1 }, tag: 't' });
	expect(innerFactory).toHaveBeenCalledTimes(1);
	});
	it('does not materialize a lazy schema used in .merge() until the merged schema parses', () => {
	const baseFactory = jest.fn(() => z.object({ id: z.string() }));
	const Base = lazySchema(baseFactory);
	const Merged = lazySchema(() => Base.merge(z.object({ name: z.string() })));
	expect(baseFactory).not.toHaveBeenCalled();
	Merged.parse({ id: 'a', name: 'b' });
	expect(baseFactory).toHaveBeenCalledTimes(1);
	});
	});

[rylnd]

This new limit far exceeds the current count as per the latest build, which is 153.5KB. Should we choose a more conservative value here, or perhaps even keep the current limit as the change doesn't seem needed?