[Lens] Update Lens CM and APIs by nickofthyme · Pull Request #229534 · elastic/kibana

nickofthyme · 2025-07-26T22:22:37Z

Summary

Lens transforms Phase 1 - Foundation for Lens by-ref transforms.

Major Changes:

Overhauls the Lens Content Management to use schema-driven types
- Defines rigid schema and types to be used inside CM.
- Implement a working item transform based on the lens item version.
  - Applied runtime migrations (v0 to v1) on all necessary CRUD operations.
- Attempts to isolate lens SO types to CM.
All Lens /public CRUD operations pass through API, no more CM public client. One exception is the mSearch used from visualizations plugin.
Refactors Lens API
- Allow new API structure (i.e. data & meta) as to align with the dashboard API (see [Dashboards as code] Return meta information in API response #196609).
- Stub out usage of different formats between API request/response types and the Lens CM Item.
- Stubs out the Config Builder flow to convert the new API format to Lens item, and vise versa.
- Use CM schemas to build out Lens route request and response schema
Refactor lens /public client to use new CRUD API
- Need to patch a basic lens client for visualizations plugin to use for updating basic attributes.
- Refactor lens client to new client consuming the Lens API instead of CM client.
- Patch Lens item to conform to the LensDocument, the new client still expect the old LensDocument but converts the API format to pass to/from the server. The client does all the conversions.
- Updates all usages of lens client to new client.
Rely on shared/common schema definitions whenever possible.
Change the visualize library client to use a basic client with only get and update methods.

Closes #225367

Checklist

Unit or functional tests were updated or added to match the most common scenarios
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.

…c type

- make accessible from /common - fix reference.name schema misalignment - add `createdBy` and `updatedBy` to the SO schema - use generics on common schema functions to pass type correctly - add schema for search result

- The `getContentClientFactory` required request but never uses it. - Add `request` to `StorageContext` to pass to CM storage handlers

- This never checked if `mSearch` was available when creating the ctx. - Check for `mSearch` before adding to ctx, log server warning if missing - Add `logger` to `mSearch` factory

- We should allow transforms to run on unversioned items - These version are not targets, they are only pulled up to the first version. - allow using `kbn/object-versioning` in common.

- Vis type alias client only needs basic methods (i.e. `get` and `update`). - Create a slim `BasicVisualizationClient` - The basic client can use `cm` or `http`.

- isolates the Lens SO to CM

- update constants and use route schema - update handling of API formats in and out of CM client

- This add CM transforms to convert existing lens items to v1 - Adds version number to lens SO attributes. - Duplicates and tweaks runtime migrations for legend and color mappings.

- Refactor lens client to use new CRUD api instead of CM - Create basic lens client using CM to use new API types - Convert lens type alias client to use new basic client - Update calls to lens client dependency to `http` - Update lens doc service to use new lens client - Update lens attribute service to use updated doc service

…apping'

…te --fix'

…ns-transforms-phase-1

markov00

As previously discussed, you need to remove the isNewApiFormat before saving the config into the SO. Currently this is not done and that property is currently going to be stored into the SO.

## Summary Changes to the `down` method type from elastic#153304, missed passing the generic input type. Also fixes types on schema utility functions from `@kbn/content-management-utils`. I created elastic#231191 and a child issues for CO teams to track and fix. _Pulling this out of a larger PR, elastic#229534, to ease CO review._ ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

markov00

Changes looks good to me, addressed all my concerns

AlexGPlay

data discovery changes lgtm

- change version index to false - set dynamic to explicit default of strict

…ns-transforms-phase-1

Copilot

Pull Request Overview

This PR implements Phase 1 of the Lens by-ref transforms foundation, introducing a comprehensive overhaul of the Lens Content Management system to use schema-driven types and establishing a new API structure aligned with the dashboard API.

Key changes include:

Complete restructuring of Lens Content Management to use strict schema-driven types with runtime migrations from v0 to v1
Introduction of new Lens CRUD API with data/meta structure format replacing direct CM client usage
Implementation of Config Builder stubs for API format transformations
Addition of comprehensive schema definitions and versioning system

Reviewed Changes

Copilot reviewed 141 out of 141 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
`x-pack/platform/plugins/shared/lens/server/`	Complete overhaul of server-side CM with new v1 schemas, transforms, and API routes
`x-pack/platform/plugins/shared/lens/public/persistence/`	Refactored client-side persistence layer with new LensClient and attribute service
`x-pack/platform/test/api_integration/apis/lens/`	Updated API integration tests to use new endpoint paths and response formats
`x-pack/solutions/security/plugins/security_solution/public/`	Removed deprecated `type`, `updated_at`, and `version` properties from Lens attributes
`x-pack/platform/plugins/shared/lens/common/`	Added new transform system, constants, and content management schemas

x-pack/platform/plugins/shared/lens/public/visualizations/xy/persistence.ts

x-pack/platform/plugins/shared/lens/server/content_management/v1/schema/common.ts

x-pack/platform/plugins/shared/lens/server/api/routes/visualizations/create.ts

x-pack/platform/plugins/shared/lens/server/content_management/v1/schema/common.ts

TinaHeiligers

Saved objects changes LGTM.

packages/kbn-check-saved-objects-cli/current_mappings.json

x-pack/platform/plugins/shared/lens/server/saved_objects.ts

elasticmachine · 2025-09-05T19:34:57Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: db701f4

Failed CI Steps

FTR Configs #94

Test Failures

[job] [logs] FTR Configs #94 / Observability AI Assistant Functional tests knowledge_base_management/index.spec.ts Knowledge management tab Bulk import knowledge base entries successfully imports multiple entries from a NDJSON file

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`lens`	1410	1414	+4

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/content-management-utils`	124	139	+15
`lens`	573	598	+25
total			+40

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`dashboard`	623.0KB	623.0KB	+10.0B
`discover`	1.1MB	1.1MB	+10.0B
`eventAnnotationListing`	204.5KB	204.6KB	+10.0B
`lens`	1.5MB	1.5MB	+177.0B
`observabilityAIAssistantApp`	177.7KB	177.7KB	+10.0B
`onechat`	841.5KB	841.6KB	+10.0B
`securitySolution`	10.3MB	10.3MB	-284.0B
`visualizations`	351.4KB	351.5KB	+27.0B
total			-30.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`lens`	58.9KB	60.9KB	+2.0KB

Unknown metric groups

API count

id	before	after	diff
`@kbn/content-management-utils`	191	206	+15
`lens`	672	697	+25
`visualizations`	869	870	+1
total			+41

ESLint disabled line counts

id	before	after	diff
`lens`	24	23	-1

Total ESLint disabled count

id	before	after	diff
`lens`	24	23	-1

History

💛 Build #336115 was flaky 59d6a3b
💚 Build #336066 succeeded 3bcea70
💔 Build #336042 failed 7508682
💚 Build #333396 succeeded 16b2429
💔 Build #333371 failed 1718047
💔 Build #333361 failed 0642315

## Summary **Lens transforms Phase 1** - Foundation for Lens by-ref transforms. Major Changes: - Overhauls the Lens Content Management to use schema-driven types - Defines rigid schema and types to be used inside CM. - Implement a working item transform based on the lens item version. - Applied runtime migrations (`v0` to `v1`) on all necessary CRUD operations. - Attempts to isolate lens SO types to CM. - All Lens /public CRUD operations pass through API, no more CM public client. One exception is the `mSearch` used from visualizations plugin. - Refactors Lens API - Allow new API structure (i.e. `data` & `meta`) as to align with the dashboard API (see elastic#196609). - Stub out usage of different formats between API request/response types and the Lens CM Item. - Stubs out the Config Builder flow to convert the new API format to Lens item, and vise versa. - Use CM schemas to build out Lens route request and response schema - Refactor lens /public client to use new CRUD API - Need to patch a basic lens client for visualizations plugin to use for updating basic attributes. - Refactor lens client to new client consuming the Lens API instead of CM client. - Patch Lens item to conform to the `LensDocument`, the new client still expect the old `LensDocument` but converts the API format to pass to/from the server. The client does all the conversions. - Updates all usages of lens client to new client. - Rely on shared/common schema definitions whenever possible. - Change the visualize library client to use a basic client with only `get` and `update` methods. Closes elastic#225367 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [x] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Marco Vettorello <vettorello.marco@gmail.com>

## Summary **Lens transforms Phase 1** - Foundation for Lens by-ref transforms. Major Changes: - Overhauls the Lens Content Management to use schema-driven types - Defines rigid schema and types to be used inside CM. - Implement a working item transform based on the lens item version. - Applied runtime migrations (`v0` to `v1`) on all necessary CRUD operations. - Attempts to isolate lens SO types to CM. - All Lens /public CRUD operations pass through API, no more CM public client. One exception is the `mSearch` used from visualizations plugin. - Refactors Lens API - Allow new API structure (i.e. `data` & `meta`) as to align with the dashboard API (see #196609). - Stub out usage of different formats between API request/response types and the Lens CM Item. - Stubs out the Config Builder flow to convert the new API format to Lens item, and vise versa. - Use CM schemas to build out Lens route request and response schema - Refactor lens /public client to use new CRUD API - Need to patch a basic lens client for visualizations plugin to use for updating basic attributes. - Refactor lens client to new client consuming the Lens API instead of CM client. - Patch Lens item to conform to the `LensDocument`, the new client still expect the old `LensDocument` but converts the API format to pass to/from the server. The client does all the conversions. - Updates all usages of lens client to new client. - Rely on shared/common schema definitions whenever possible. - Change the visualize library client to use a basic client with only `get` and `update` methods. Closes #225367 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [x] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Marco Vettorello <vettorello.marco@gmail.com>

## Summary Lens transforms Phase 2 ### Changes #### Integrate the Config Builder This integrates the `LensConfigBuilder` into the full end-to-end flow of all aspects of Lens including: - Lens APIs (transforms config to pass over the wire, convert to SO for saving) - Lens public client (transform api format to old Lens state) - Lens transforms (transforms Lens by-value panels for dashboards) #### Update Lens CM and API schema The schemas following changes from #229534, did not match the integration perfectly and stubbed out a few things. So there was some cleanup needed to these schema to make them clearer and easier to understand. These also needed to be changed to integrate the new `lensApiStateSchema` that will include all New API Formats. #### Registers Lens embeddable transforms Transforms: - `dashboard/server` - Used to transform by-ref Lens dashboard panels. - `dashboard/public` - Used to transform panels from url state. - `lens/public` - Used to transform standalone Lens embeddables (see example [here](https://github.com/elastic/kibana/blob/ed9f709ef56b182116ebcd788245fdecfdacfd8b/x-pack/platform/plugins/shared/cases/public/components/visualizations/lens_renderer.tsx#L40)). These transforms are meant to transform both: - **Item transforms** - Current `v0` Lens state/attributes with old configurations to the latest state/attributes. Applying all existing runtime migrations and pins version to latest Lens item `v1`. - Only for `lens/public` transforms, we are not saving standalone Lens embeddable state in new API format. - **API Format transforms** - Transforms the current `v1` Lens state to the new API format and vis-versa. - Needed to save Lens as `v1` as SO but pass the New API Format over the wire. - Needed to save Lens state in dashboard by-value panels in the New API Format. Transforms are behind a features flag, when enabled it will transform only supported chart types for all lens visualizations including by-ref and by-value. When disabled (default) it will pass around the only Lens SO state as it does today. ```yml # kibana.dev.yml feature_flags.overrides: lens.apiFormat: true ``` ### Issues Closes #229543 Closes #229546 Closed #237192 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) --------- Co-authored-by: Peter Pisljar <peter.pisljar@elastic.co> Co-authored-by: dej611 <dej611@gmail.com> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Marco Liberati <dej611@users.noreply.github.com> Co-authored-by: Marco Vettorello <marco.vettorello@elastic.co>

) ## Summary Lens transforms Phase 2 ### Changes #### Integrate the Config Builder This integrates the `LensConfigBuilder` into the full end-to-end flow of all aspects of Lens including: - Lens APIs (transforms config to pass over the wire, convert to SO for saving) - Lens public client (transform api format to old Lens state) - Lens transforms (transforms Lens by-value panels for dashboards) #### Update Lens CM and API schema The schemas following changes from elastic#229534, did not match the integration perfectly and stubbed out a few things. So there was some cleanup needed to these schema to make them clearer and easier to understand. These also needed to be changed to integrate the new `lensApiStateSchema` that will include all New API Formats. #### Registers Lens embeddable transforms Transforms: - `dashboard/server` - Used to transform by-ref Lens dashboard panels. - `dashboard/public` - Used to transform panels from url state. - `lens/public` - Used to transform standalone Lens embeddables (see example [here](https://github.com/elastic/kibana/blob/ed9f709ef56b182116ebcd788245fdecfdacfd8b/x-pack/platform/plugins/shared/cases/public/components/visualizations/lens_renderer.tsx#L40)). These transforms are meant to transform both: - **Item transforms** - Current `v0` Lens state/attributes with old configurations to the latest state/attributes. Applying all existing runtime migrations and pins version to latest Lens item `v1`. - Only for `lens/public` transforms, we are not saving standalone Lens embeddable state in new API format. - **API Format transforms** - Transforms the current `v1` Lens state to the new API format and vis-versa. - Needed to save Lens as `v1` as SO but pass the New API Format over the wire. - Needed to save Lens state in dashboard by-value panels in the New API Format. Transforms are behind a features flag, when enabled it will transform only supported chart types for all lens visualizations including by-ref and by-value. When disabled (default) it will pass around the only Lens SO state as it does today. ```yml # kibana.dev.yml feature_flags.overrides: lens.apiFormat: true ``` ### Issues Closes elastic#229543 Closes elastic#229546 Closed elastic#237192 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) --------- Co-authored-by: Peter Pisljar <peter.pisljar@elastic.co> Co-authored-by: dej611 <dej611@gmail.com> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Marco Liberati <dej611@users.noreply.github.com> Co-authored-by: Marco Vettorello <marco.vettorello@elastic.co>

nickofthyme added 21 commits July 26, 2025 15:49

improve CM class to be easier to extend and override defaults

c38e0b8

improve CM rpc types to satisfy generic type instead of be the generi…

a0df8cb

…c type

fix spelling errors

d2af505

move and add constants

dd1d576

update CM common schema

9dee933

- make accessible from /common - fix reference.name schema misalignment - add `createdBy` and `updatedBy` to the SO schema - use generics on common schema functions to pass type correctly - add schema for search result

update storage context params

06ac9d2

- The `getContentClientFactory` required request but never uses it. - Add `request` to `StorageContext` to pass to CM storage handlers

cleanup getMSearchClientFactory

0e1d797

- This never checked if `mSearch` was available when creating the ctx. - Check for `mSearch` before adding to ctx, log server warning if missing - Add `logger` to `mSearch` factory

use api path constants

ff9d7bd

allow initial v0 for down transforms

5ce1467

- We should allow transforms to run on unversioned items - These version are not targets, they are only pulled up to the first version. - allow using `kbn/object-versioning` in common.

refactor visualizations plugin clients

ab47bfd

- Vis type alias client only needs basic methods (i.e. `get` and `update`). - Create a slim `BasicVisualizationClient` - The basic client can use `cm` or `http`.

add content management schema

31dc07f

add lens vis route schema

08313f2

refactor types to use schema definitions

6628892

update imports, constants and types

52615b0

update lens SO mappings to add new version property

b25d312

implement all lens CM CRUD operations with latest CM schema

aa7bd3f

- isolates the Lens SO to CM

update lens CRUD apis

5b80de2

- update constants and use route schema - update handling of API formats in and out of CM client

update CM service with v0 Item transforms

091a840

- This add CM transforms to convert existing lens items to v1 - Adds version number to lens SO attributes. - Duplicates and tweaks runtime migrations for legend and color mappings.

add config builder stub

f2be004

cleanup public constants and types

ce9493f

nickofthyme added Team:Visualizations Team label for Lens, elastic-charts, Graph, legacy editors (TSVB, Visualize, Timelion) t// release_note:skip Skip the PR/issue when compiling release notes backport:skip This PR does not require backporting labels Jul 26, 2025

more type cleanup

77e51b7

nickofthyme mentioned this pull request Jul 27, 2025

[Lens] Add embeddable transforms #229545

Closed

5 tasks

elastic deleted a comment from elasticmachine Jul 27, 2025

kibanamachine and others added 3 commits July 27, 2025 17:49

[CI] Auto-commit changed files from 'node scripts/styled_components_m…

dd1ed49

…apping'

[CI] Auto-commit changed files from 'node scripts/check_mappings_upda…

099929a

…te --fix'

Merge branch 'main' into lens-transforms-phase-1

fe7241d

Merge remote-tracking branch 'origin/lens-transforms-phase-1' into le…

16b2429

…ns-transforms-phase-1

markov00 requested changes Sep 2, 2025

View reviewed changes

nickofthyme added 2 commits September 3, 2025 09:26

Merge branch 'main' into lens-transforms-phase-1

de4125f

chore: remove isNewApiFormat from saved state

edbeaa3

nickofthyme requested a review from markov00 September 3, 2025 16:14

markov00 approved these changes Sep 3, 2025

View reviewed changes

nickofthyme requested a review from TinaHeiligers September 3, 2025 16:52

AlexGPlay approved these changes Sep 4, 2025

View reviewed changes

nickofthyme added 5 commits September 4, 2025 13:57

chore: update lens mappings

7bd7988

- change version index to false - set dynamic to explicit default of strict

Merge branch 'main' into lens-transforms-phase-1

7508682

chore: update lens SO mappings

43a5207

Merge remote-tracking branch 'origin/lens-transforms-phase-1' into le…

3bcea70

…ns-transforms-phase-1

chore: update lens SO mappings to not set index to false

59d6a3b

Copilot AI review requested due to automatic review settings September 4, 2025 23:56

Copilot AI reviewed Sep 4, 2025

View reviewed changes

nickofthyme added 2 commits September 5, 2025 12:46

chore: update lens so mappings

e06d7c7

Merge branch 'main' into lens-transforms-phase-1

db701f4

TinaHeiligers approved these changes Sep 5, 2025

View reviewed changes

packages/kbn-check-saved-objects-cli/current_mappings.json Outdated Show resolved Hide resolved

packages/kbn-check-saved-objects-cli/current_mappings.json Show resolved Hide resolved

x-pack/platform/plugins/shared/lens/server/saved_objects.ts Show resolved Hide resolved

nickofthyme merged commit 74c64ba into elastic:main Sep 6, 2025
12 checks passed

kibanamachine added the v9.2.0 label Sep 6, 2025

nickofthyme mentioned this pull request Sep 16, 2025

[Lens] Integrate Config Builder into APIs and transforms #235277

Merged

2 tasks

nickofthyme mentioned this pull request Sep 29, 2025

[Lens] Add telemetry to track obsolete Lens runtime migrations #236848

Draft

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Lens] Update Lens CM and APIs#229534

[Lens] Update Lens CM and APIs#229534
nickofthyme merged 112 commits intoelastic:mainfrom
nickofthyme:lens-transforms-phase-1

nickofthyme commented Jul 26, 2025 •

edited

Loading

Uh oh!

markov00 left a comment

Uh oh!

markov00 left a comment

Uh oh!

AlexGPlay left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TinaHeiligers left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Sep 5, 2025

API count

ESLint disabled line counts

Total ESLint disabled count

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

Conversation

nickofthyme commented Jul 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

Uh oh!

markov00 left a comment

Choose a reason for hiding this comment

Uh oh!

markov00 left a comment

Choose a reason for hiding this comment

Uh oh!

AlexGPlay left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TinaHeiligers left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Sep 5, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Page load bundle

API count

ESLint disabled line counts

Total ESLint disabled count

History

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

nickofthyme commented Jul 26, 2025 •

edited

Loading