Add meta IDs to fleet output schemas for named OAS components

[TinaHeiligers]

Summary

Adds meta: { id } annotations to Fleet output schema variants so the OAS generator produces named $ref components instead of inline schemas. Also introduces NewOutputSchema for the POST create endpoint so that create-request and response component names are distinct.

Relates to: https://github.com/elastic/kibana-team/issues/2787

Details:

In output.ts:

• OutputSchema variants get meta: { id: 'output_elasticsearch' } etc. — used for GET responses
• New NewOutputSchema with meta: { id: 'new_output_elasticsearch' } etc. — used for POST create requests
• UpdateOutputSchema is unchanged (future work)
• Kafka schema.conditional fields are unchanged — runtime validation behavior is preserved

In rest_spec/output.ts:

• PostOutputRequestSchema.body changed from OutputSchema to NewOutputSchema

In allowlist.json:

• Allowlist entries for 5 structural breaking changes (inline schemas → $ref components)

New NewOutputSchema

PostOutputRequestSchema and OutputResponseSchema both used OutputSchema. When variants get meta: { id }, both reference the same named components. Downstream code generators (e.g., oapi-codegen) produce one type per component name, collapsing create-request and response types. NewOutputSchema gives the POST path its own component names, following the same pattern as the existing UpdateOutputSchema.

OAS components produced

Schema	Components
OutputSchema (responses)	output_elasticsearch, output_remote_elasticsearch, output_logstash, output_kafka
NewOutputSchema (POST request)	new_output_elasticsearch, new_output_remote_elasticsearch, new_output_logstash, new_output_kafka
UpdateOutputSchema (PUT request)	unchanged — inline (future work)

How to test this

1. Run Fleet tests:

   yarn test:jest --config x-pack/platform/plugins/shared/fleet/server/jest.config.js
   

2. Verify OAS output:

   node scripts/capture_oas_snapshot --no-serverless --include-path /api/fleet/outputs
   

   Inspect oas_docs/bundle.json:
   • POST /api/fleet/outputs request body should have anyOf with $ref to new_output_* components
   • GET /api/fleet/outputs response items should have anyOf with $ref to output_* components
   • All 8 components should exist under components/schemas

Checklist

• Any text added follows EUI's writing guidelines — N/A, no UI text
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios — 3,919 Fleet tests pass
• This was checked for breaking HTTP API changes — 5 structural changes allowlisted (inline → $ref, no data shape change)
• Flaky Test Runner was used on any tests changed — N/A, no test files changed
• The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

Identify risks

Risk	Severity	Mitigation
Structural OAS change breaks unknown consumers	Low	anyOf with $ref is structurally equivalent to inline anyOf. Any consumer following $ref sees identical schemas.
NewOutputSchema diverges from OutputSchema over time	Low	Both spread the same variant schemas (ElasticSearchSchema, etc.). Changes to variants propagate to both.

Release note

skip

Co-Authored by Claude Opus 4.6 (low effort)

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[jloleysens]

I'm a bit concerned about us handling the fix here... The config-schema uses schema.conditional which is known to not work in OAS

https://docs.elastic.dev/kibana-dev-docs/genereating-oas-for-http-apis#some-good-practices-to-consider

Is it possible for us to fix this in the route definition to avoid creating a script/code owned by Core that is tightly coupled to fleets APIs?

[jloleysens]

I know you call this out in your risks... can we find a way to address these issues in the defined schemas?

[TinaHeiligers]

Thanks for the review — you're right, and this is a better direction.

I've looked into schema.conditional (which the docs actually discourage for OAS generation) and found that schema.discriminatedUnion() is the correct primitive here. It natively produces:

• oneOf with $ref entries (named components via meta: { id: '...' })
• discriminator with propertyName mapping

So the fix would be changing Fleet's OutputSchema from schema.oneOf([...]) to schema.discriminatedUnion('type', [...]) with IDs on each variant — no post-processing script needed 😄

The kafka schema.conditional fields (compression_level, connection_type, password, username) also need fixing at the schema level since conditionals produce broken OAS. Those should become explicit field definitions.

I'll rework this PR to make the changes in Fleet's schema definitions directly instead of the componentization script approach. That keeps ownership where it belongs and eliminates the coupling concern.

[theletterf]

Approving to unblock.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[TinaHeiligers]

Update: Schema-level fix investigation

I looked into what a schema-level fix would be found the following:

1. schema.discriminatedUnion('type', [...]) — Natively produces oneOf with $ref components + discriminator in OAS (implementation, OAS post-processor). Works for OutputSchema (POST + responses). Cannot work for UpdateOutputSchema because its type field is optional (schema.maybe(schema.literal(...))) and discriminatedUnion requires a literal discriminator.

2. schema.oneOf([...]) with meta: { id } — Adding meta: { id: 'output_elasticsearch' } to each variant produces anyOf with $ref to named components. Structurally equivalent to the inline schemas but properly componentized. No discriminator (added downstream by consumers).

3. Kafka schema.conditional replacement — Replacing the 4 conditional fields (compression_level, connection_type, username, password) with explicit optional types fixes the broken OAS they produce (anyOf/oneOf/not:{} patterns). However, this changes Fleet's runtime validation behavior (e.g., compression_level would be accepted even when compression != gzip). I abandoned this approach because it's too risky without the Fleet team's input. (The OAS docs explicitly discourage schema.conditional for this reason.)

4. OAS overlays — Can add discriminators, remove properties, and update field types, but can't extract inline schemas into $ref components. Useful as a complement to schema changes, not a standalone solution.

5. schema.allOf composition — Could share base schemas across CRUD variants, but adds complexity to Fleet's schema hierarchy and allOf can confuse some codegen tools.

The problem that needs fixing is that Fleet's PostOutputRequestSchema uses OutputSchema directly. This is the same schema used for GET responses via OutputResponseSchema = schema.object({ item: OutputSchema.extendsDeep(...) }).

Adding meta: { id: 'output_elasticsearch' } to the variants, results in both POST request and GET response referencing the same named components. Downstream code generators (e.g., the Terraform provider's oapi-codegen) produce one Go type per component name, it's just the way Go works. This means NewOutputElasticsearch and OutputElasticsearch collapse into a single OutputElasticsearch type — breaking the provider's build, which expects separate types for create vs. read operations. That's what led to the transforms in the first place.

The problem isn't a Kibana bug, it's a consequence of reusing the schema between request and response and only shows up when components get named.

What could work:

Option A: Add a separate NewOutputSchema in Fleet

Define a new schema in Fleet's output.ts with its own meta: { id: 'new_output_*' } IDs, mirroring how UpdateOutputSchema already exists separately. This is ~10 lines of new code.

• Pro: Cleanly separates create and response component names, downstream codegen works correctly
• Pro: Follows the existing pattern (UpdateOutputSchema already exists for the same reason)
• Con: Adds code to Fleet's plugin that exists primarily to benefit downstream OAS consumers
• Con: Creates a third copy of the output variant schemas (Output, NewOutput, UpdateOutput)

Option B: Revert all schema changes

Remove all meta: { id } annotations. Fleet OAS stays fully inline. Downstream consumers continue handling extraction themselves.

• Pro: Zero risk, zero Fleet code changes
• Con: No OAS quality improvement
• Con: Doesn't address the original feedback about fixing at the schema level

Option C: Keep meta: { id } on OutputSchema only, consumers adapt

Keep named components for response/POST schemas. Downstream consumers that need separate create vs. read types handle the mapping themselves.

• Pro: Kibana OAS improves (named components, discriminator can be added via overlay later)
• Pro: Minimal Fleet code change (just meta: { id } on existing variants)
• Con: Consumers that generate separate request/response types need workarounds
• Con: Shifts complexity from Kibana to every downstream consumer

---

I'll update the PR code once we align on a direction. The current code on the branch reflects an earlier iteration and will change.

I've pushed changes to reflect Option A, adding a separate NewOutputSchema in Fleet.

@jloleysens — would appreciate your input on if my choice is appropriate from a platform perspective. I'm trying to find someone from @elastic/fleet to weigh in.

[TinaHeiligers]

blocked pending a decision on the best option forward.
I'm leaning towards Option A since it already mirrors the existing UpdateOutputSchema with minimal code changes and the most benefit. (Also reflected in the code ATM)
Option A accepted by core.

[TinaHeiligers]

Self-review: proactively addressing anticipated reviewer questions on design decisions.

[TinaHeiligers]

Self-review

• The PUT endpoint will be addressed in a follow-up PR (adding meta.id to UpdateOutputSchema) after validating the POST/GET pattern with downstream consumers (terraform-provider).

• Response schemas work as expected — extendsDeep preserves meta.id on inner variants, and the OAS snapshot produces the $ref components as expected (confirmed with running node scripts/capture_oas_snapshot locally).

• models/index.ts re-exports NewOutputSchema (not shown in this diff), so the import resolves fine.

[TinaHeiligers]

NewOutputSchema is intentionally a separate declaration despite identical field structures. The only difference is meta.id values (new_output_* vs output_*), which controls OAS component naming.

schema.oneOf doesn't support remapping meta.id after construction, so we can't clone-and-rename. The field-spread lists reference shared constants (ElasticSearchSchema, etc.), so the risk of structural drift is low — field changes happen in the shared constants, not here.

Code generators like oapi-codegen produce one type per component name. Without separate names, POST request and GET response types would collapse into one, losing the create-vs-read distinction needed by the Terraform provider.

[jloleysens]

It's a minor cost to have this live closer to the code owners' code. My only nit would be maybe we can name this ResponseOutputSchema instead of NewOutputSchema + add a code comment. Up to the fleet team reviewers though!

[TinaHeiligers]

Good call on naming — NewOutputSchema is a bit ambiguous.

One issue with ResponseOutputSchema is that this schema is used for the POST request body (see PostOutputRequestSchema.body), not for responses. There's already an OutputResponseSchema (line 355) that wraps the GET/POST response — so ResponseOutputSchema would be confusingly close to that.

Some options that better convey the intent:

• CreateOutputSchema — matches the POST/create semantics
• PostOutputSchema — matches the HTTP method

@elastic/fleet — any preference? Happy to rename and add a clarifying comment.

[juliaElastic]

I think NewOutputSchema sounds good, we have similar names in other types, e.g. NewAgentPolicySchema.
A comment about why we need a separate schema object would be good.

[TinaHeiligers]

@juliaElastic comment added in 8eea242

[TinaHeiligers]

We're adding meta.id on the schema.object wrappers, not on inner fields. That leaves kafka's schema.conditional fields as they are, keeping runtime validation behavior the same.

[jloleysens]

Nice work @TinaHeiligers ! Thank you for addressing my feedback. Happy to chat more with Fleet if they do not like this or have some other feedback.

[jloleysens]

It's a minor cost to have this live closer to the code owners' code. My only nit would be maybe we can name this ResponseOutputSchema instead of NewOutputSchema + add a code comment. Up to the fleet team reviewers though!

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 0a2d202

Failed CI Steps

• FTR Configs #174

Test Failures

• [job] [logs] FTR Configs #174 / lens app - group 5 lens formula should duplicate a moving average formula and be a valid table with conditional coloring

Metrics [docs]

✅ unchanged

History

• 💔 Build #416551 failed 8f50555
• 💔 Build #415059 failed af3dce3
• 💔 Build #414288 failed e57e3d2

[TinaHeiligers]

These changes came in from main. The commit base for this PR is older than when these changes were merged, hence seeing a diff

[tobio]

Do we expect a discriminator to be included in the spec here?

[TinaHeiligers]

Each variant already has a unique type enum value so consumers can distinguish them, but there's no explicit discriminator in the spec ATM. The schemas use schema.oneOf which doesn't emit one.

Switching to schema.discriminatedUnion('type', [...]) would automatically emit an explicit discriminator. OTOH, anyOf is perfectly valid as-is, so switching would be an improvement for codegen consumers. Happy to add it as a follow-up if you think it's worth doing.