[Lens] Improve Visualizations API documentation

[florent-leborgne]

Closes #236265

Summary

Improve the Visualizations API documentation ahead of its public release in 9.4. This PR (Kibana) and a companion docs-content PR work together to create a cohesive documentation experience across the API reference and the narrative chart documentation.

Related: #260969 (Dashboards API docs, same pattern)

Documentation strategy

Users discovering the Visualizations API will follow one of three paths. Our changes ensure each path leads to success:

Path 1: Start from the API reference
A user lands on the API docs and wants to create a visualization. The tag intro orients them: what chart types exist, how data sources work (DSL vs ES|QL), and what the current limitations are. They pick an endpoint, see a concise description, copy a code sample, and it works out of the box with sample data. If they need a more complex example, the chart types table links directly to tested payload examples on each chart page.

Path 2: Start from the chart documentation
A user is reading about how to build a bar chart or a metric in the UI. At the bottom of each example and advanced scenario, a collapsible dropdown shows the equivalent API payload. They can copy it, modify it, and use it in their automation. The dropdown links back to the API reference for the full schema.

Path 3: Explore the schema
A user browses the request body schema to understand all available options. Every parameter they encounter has a clear, actionable description explaining what it does, its valid values, defaults, and relationships to other fields.

Changes in this PR (Kibana)

API reference intro (overlay)

• Getting started section with chart types table, each linking to documentation and tested API payload examples
• DSL vs ES|QL data source modes with code snippets
• ES|QL limitation note, sample dataset note, links to auth and spaces docs
• "Adapting examples to your own data" guidance
• Error handling section explaining how to read oneOf validation errors

Endpoint descriptions (route files)

• Concise, consistent third-person voice across all 5 endpoints
• Non-obvious behaviors surfaced: ES|QL limitation on create/update, no partial updates, dashboard impact on delete
• since: '9.4.0' on all availability configs
• 401/404 response codes documented

Schema property descriptions (~320 fields across config_builder and route schemas)

Replaced tautological descriptions with actionable text across shared and chart-specific schema files. Every description now ends with a period, includes defaults and valid values where available, and provides context beyond restating the field name.

Files improved (config_builder):

• shared.ts — label, title, description, collapseBy, axis title, legend truncation, panel filters
• metric_ops.ts — reduced_time_range, time_shift, time_scale, formula, field, empty_as_null, static value, column (ES|QL), time_field, percentile, window
• bucket_ops.ts — date histogram, terms (fields, limit, increase_accuracy, includes/excludes, other_bucket, rank_by with titles), histogram, range
• format.ts — decimals, suffix, compact, from/to (duration), pattern
• filter.ts — expression, filter label
• color.ts — color by value, static color, apply color to
• charts/xy.ts — ~90 descriptions: layer types, legend, axis config, domain, scale, fitting, curve, overlays, decorations, annotations, reference lines
• charts/datatable.ts — ~26 descriptions: density, pagination, sorting, alignment, summary rows, one-click filter
• charts/metric.ts — subtitle, compare palette/icon/value, styling (labels, values, icon, secondary)
• charts/pie.ts — legend, slice labels, donut hole, values display
• charts/gauge.ts — shapes (bullet/circular), title, subtitle, ticks
• charts/heatmap.ts — axes, cells, legend, labels
• charts/tagcloud.ts — orientation, font sizes, caption
• charts/treemap.ts — legend, labels, metrics, breakdowns
• charts/waffle.ts — legend, metrics, breakdowns
• charts/mosaic.ts — legend, breakdowns, group breakdowns
• charts/legacy_metric.ts — labels, values, font size
• charts/partition_shared.ts — value display, nested legend
• charts/region_map.ts — EMS boundaries, join field
• charts/shared.ts — legend visibility

Files improved (route schemas):

• common.ts — inline lensItemMetaSchema with 7 described meta fields
• create.ts — id param, overwrite query param
• search.ts — fields, search_fields, query, page, per_page, total

Lens term de-emphasis

• "Lens documentation" → "visualizations documentation" in overlay
• Code comment updated in schema
• Saved object type lens left as-is (it's the actual value)

data_source rename alignment

• All YAML examples and overlay inline code updated: dataset → data_source, type: index → type: data_view_spec, index → index_pattern, type: dataView → type: data_view_reference, id → ref_id (per [Lens as Code] Use as code dataView schema #260914)

Response and request parameter descriptions (route schema files)

• Response meta: created_at, updated_at, created_by, updated_by, managed, origin_id, type, id
• Search request query: fields, search_fields, query, page, per_page
• Search response meta: page, per_page, total
• Create: optional id path param, overwrite query param

Examples and code samples (overlay + YAML files)

• 10 YAML example files for 4 chart types, all validated against a live 9.4 instance
• cURL and Console code samples on all 5 endpoints

⚠️ Note for reviewers: inlined route schemas

The route schema files (common.ts, create.ts, search.ts) define response meta and request query fields inline rather than referencing shared schemas from kbn-content-management-utils. This is intentional:

Why: ExtendsDeepOptions in kbn-config-schema does not include a meta field — it only supports unknowns. Calling .extendsDeep({ meta: { description: '...' } }) compiles without error (TypeScript doesn't enforce excess property checking at these call sites), but the meta value is silently ignored and never propagated. Descriptions added this way are absent from the generated OpenAPI spec, resulting in undocumented fields in the published API reference. See item 9 in the findings gist for details.

Trade-off: These inline definitions are decoupled from the shared savedObjectSchema and searchOptionsSchemas. If the shared schemas change their types, the Lens route schemas won't automatically follow. The types are identical today and are unlikely to change (stable saved object fields), but this is a conscious trade-off.

Alternatives:

1. Extend ExtendsDeepOptions to support meta in kbn-config-schema — this would let API teams add descriptions to shared schemas without forking them, and is the cleanest platform-level fix.
2. Add meta: { description } to the shared schemas in kbn-content-management-utils/src/schema.ts — all content management APIs would inherit descriptions. Lens-specific descriptions (e.g., "Always lens" for type) could then use inline overrides only where needed.

If the team prefers either alternative, these inline definitions can be reverted. Each affected file has a code comment explaining the rationale and pointing to common.ts for the full context.

Changes in companion PR (elastic/docs-content#5645)

50 inline API payload dropdowns across all 13 chart type pages, placed after each individual example and advanced scenario:

• 27 chart examples (metric, bar, line, area, pie, gauge, table, heatmap, tag cloud, treemap, waffle, mosaic, region map)
• 13 advanced scenarios (time shifts, reference lines, stacked charts, nested breakdowns, formulas, pivot tables)
• All 50 payloads pass integration tests against a 9.4 snapshot

Lens overview page: new "Create visualizations with the API" section linking to both stateful and serverless API docs

Integration test findings

All examples validated against a live 9.4 instance. Findings and testing scripts:
https://gist.github.com/florent-leborgne/21996938a5583965e3f0de59da6dedd5

Key items for the team:

1. metricNoESQL schema missing type: "primary", labels, icon (ES|QL variant is correct)
2. Null values rejected for optional fields (must omit, not null)
3. other_bucket runtime expects object, not boolean
4. Optional id path param missing from POST spec
5. Filter schema names need human-readable titles
6. lensPanelFilters description duplicated at array and items level
7. .extendsDeep() does not support meta — descriptions silently dropped (see item 9 in gist)

Version badges

Added since: '9.4.0' to the availability config on all 5 route definitions so the API docs render "Technical Preview; added in 9.4.0" badges.

Preview

Latest combined preview (Dashboards + Visualizations): https://bump.sh/preview/111e0488-c656-40cb-8379-4d50cf4b1554

[elasticmachine]

🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

• Click to trigger kibana-pull-request for this PR!
• Click to trigger kibana-deploy-project-from-pr for this PR!
• Click to trigger kibana-deploy-cloud-from-pr for this PR!
• Click to trigger kibana-entity-store-performance-from-pr for this PR!
• Click to trigger kibana-storybooks-from-pr for this PR!

[macroscopeapp]

🔴 Critical oas_docs/makefile:25

Running make api-docs fails with ENOENT when the api-docs-postprocess target executes. The Makefile paths oas_docs/output/kibana.yaml resolve to oas_docs/oas_docs/output/kibana.yaml relative to the working directory, but the scripts read files without resolving against REPO_ROOT. This matches the pattern used by api-docs-overlay and api-docs-lint, which expect to run from the oas_docs/ directory. Consider changing paths to output/kibana.yaml to match the convention used in sibling targets, or modify the scripts to resolve paths using REPO_ROOT.

-	@node scripts/strip_additional_properties.js oas_docs/output/kibana.yaml
-	@node scripts/strip_discriminators.js oas_docs/output/kibana.yaml
-	@node scripts/strip_additional_properties.js oas_docs/output/kibana.serverless.yaml
-	@node scripts/strip_discriminators.js oas_docs/output/kibana.serverless.yaml

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file oas_docs/makefile around lines 25-28:

Running `make api-docs` fails with `ENOENT` when the `api-docs-postprocess` target executes. The Makefile paths `oas_docs/output/kibana.yaml` resolve to `oas_docs/oas_docs/output/kibana.yaml` relative to the working directory, but the scripts read files without resolving against `REPO_ROOT`. This matches the pattern used by `api-docs-overlay` and `api-docs-lint`, which expect to run from the `oas_docs/` directory. Consider changing paths to `output/kibana.yaml` to match the convention used in sibling targets, or modify the scripts to resolve paths using `REPO_ROOT`.

[florent-leborgne]

Superseded by a series of smaller PRs