Removed switchToModelVersionAt from SO API definition by TinaHeiligers · Pull Request #219029 · elastic/kibana

TinaHeiligers · 2025-04-24T02:09:40Z

Summary

Part of #201807

Strategy:
This PR assumes that switchToModelVersionAt has done it’s job and all SO type owners now use modelVersions instead of migrations.
It takes the approach of trusting that this is fine™

Changes in this PR:

switchToModelVersionAt removed from all registered saved objects, so their mapping hash had to be updated.
Implements globalSwitchToModelVersionAt directly in version_map
Updates logic to prioritize modelVersions over migrations in getLatestMappingsModelVersion, that previously relied on switchToModelVersionAt being set
Updates unit tests to match the refactored logic
Updates snapshots
Adapts SO types registered in the cases, SLO and encrypted saved objects plugins.
Updates docs

Risks:
- Plugin developers disregard the deprecated status of migrations and introduce new versions, which will not be BWC. Saved object type hash changes will notify the core team for a code owner review. The core team needs to investigate the related changes and provide feedback.

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

…rectly

…ject type registration definitions

…ModelVersionAt from the public SO type

TinaHeiligers · 2025-04-24T16:00:00Z

@elasticmachine merge upstream

elasticmachine · 2025-04-24T22:36:58Z

Pinging @elastic/kibana-core (Team:Core)

… src/core/server/integration_tests/ci_checks'

elasticmachine · 2025-04-24T22:57:11Z

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

TinaHeiligers · 2025-04-24T23:39:31Z

src/core/packages/saved-objects/base-server-internal/src/model_version/version_map.ts

 */
 export const getCurrentVirtualVersion = (type: SavedObjectsType): string => {
-  if (type.switchToModelVersionAt) {
+  if (type.modelVersions) {


One of the main changes in this PR is to assume we can relax the enforced switch to modelVersions. There is a risk that type owners continue to add migrations even though they are deprecated.

TinaHeiligers · 2025-04-24T23:41:44Z

...ges/saved-objects/migration-server-internal/src/document_migrator/validate_migration.test.ts

+          validate({ type, kibanaVersion: '8.11.0' })
        ).toThrowErrorMatchingInlineSnapshot(
-          `"Migration for type foo for version 8.10.0 registered after switchToModelVersionAt (8.9.0)"`
+          `"Migration for type foo for version 8.11.0 registered after globalSwitchToModelVersionAt (8.10.0)"`


Validation still throws an error, so we should catch situations where new migrations are added.

TinaHeiligers

self review

kdelemme

ux logs management code change lgtm!

elasticmachine · 2025-04-25T00:23:53Z

💚 Build Succeeded

Buildkite Build
Commit: 733eac2

Metrics [docs]

Unknown metric groups

API count

id	before	after	diff
`@kbn/core-saved-objects-server`	574	573	-1

References to deprecated APIs

id	before	after	diff
`@kbn/core`	638	636	-2

History

💔 Build #295992 failed 082f809
💛 Build #295923 was flaky 082f809
💔 Build #295602 failed 27e63d7

kibanamachine · 2025-04-26T15:00:26Z

Starting backport for target branches: 9.0

https://github.com/elastic/kibana/actions/runs/14682376211

kibanamachine · 2025-04-26T15:06:11Z

💔 All backports failed

Status	Branch	Result
❌	9.0	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 219029

Questions ?

Please refer to the Backport tool documentation

TinaHeiligers · 2025-04-26T17:50:25Z

💚 All backports created successfully

Status	Branch	Result
✅	9.0

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

## Summary Part of elastic#201807 **Strategy**: This PR assumes that `switchToModelVersionAt` has done it’s job and all SO type owners now use `modelVersions` instead of `migrations`. It takes the approach of trusting that this is fine™ **Changes in this PR:** - `switchToModelVersionAt` removed from all registered saved objects, so their mapping hash had to be updated. - Implements `globalSwitchToModelVersionAt` directly in `version_map` - Updates logic to prioritize `modelVersions` over `migrations` in `getLatestMappingsModelVersion`, that previously relied on `switchToModelVersionAt` being set - Updates unit tests to match the refactored logic - Updates snapshots - Adapts SO types registered in the **cases**, **SLO** and **encrypted saved objects** plugins. - Updates docs **Risks**: - Plugin developers disregard the deprecated status of `migrations` and introduce new versions, which will not be BWC. Saved object type hash changes will notify the core team for a code owner review. The core team needs to investigate the related changes and provide feedback. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [x] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

kibanamachine · 2025-04-28T15:48:49Z

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.
cc: @TinaHeiligers

TinaHeiligers · 2025-04-28T21:29:39Z

backport PR: #219329

…#219329) # Backport This will backport the following commits from `main` to `9.0`: - [Removed switchToModelVersionAt from SO API definition (#219029)](#219029)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

…c#219029)" This reverts commit 19c017a.

…" (#220063) This reverts commit 19c017a. ## Summary Reverts #219029 as it is causing SO Migration crashes in our QA environment. It can also be tested locally on `main`: 1. On a previous version (`git checkout deploy@1745820726`) 2. Start ES: `yarn es serverless --projectType es` 3. Start Kibana: `yarn start --serverless=es` 4. Wait for migrations to complete 5. Stop Kibana 6. Move back to `main` 7. Start Kibana: `yarn start --serverless=es` 8. Wait for the error On `main`, we can see that Kibana fails to start with the fatal error: ``` FATAL Error: Unable to complete saved object migrations for the [.kibana_alerting_cases] index: Model version conflict: inconsistent higher/lower versions ``` In this branch, it starts successfully.

…" (#220059) Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

…c#219029)" (elastic#220059) Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 75c0f95) # Conflicts: # src/core/server/integration_tests/ci_checks/saved_objects/check_registered_types.test.ts

…219029)" (#220059) (#220259) # Backport This will backport the following commits from `main` to `9.0`: - [Revert "Removed switchToModelVersionAt from SO API definition (#219029)" (#220059)](#220059)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

## Summary Part of elastic#201807 **Strategy**: This PR assumes that `switchToModelVersionAt` has done it’s job and all SO type owners now use `modelVersions` instead of `migrations`. It takes the approach of trusting that this is fine™ **Changes in this PR:** - `switchToModelVersionAt` removed from all registered saved objects, so their mapping hash had to be updated. - Implements `globalSwitchToModelVersionAt` directly in `version_map` - Updates logic to prioritize `modelVersions` over `migrations` in `getLatestMappingsModelVersion`, that previously relied on `switchToModelVersionAt` being set - Updates unit tests to match the refactored logic - Updates snapshots - Adapts SO types registered in the **cases**, **SLO** and **encrypted saved objects** plugins. - Updates docs **Risks**: - Plugin developers disregard the deprecated status of `migrations` and introduce new versions, which will not be BWC. Saved object type hash changes will notify the core team for a code owner review. The core team needs to investigate the related changes and provide feedback. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [x] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

…c#219029)" (elastic#220059) Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

…20985) ## Summary * Address #217145 - Put in place a check to ensure we're upgrading from Kibana 8.18.0 or newer. * Address #220521 - New attempt at removing the `switchToModelVersionAt` property, inspired on #219029. The previous attempt caused a regression: index meta information started storing _modelVersions_ that were older than the previously stored ones, which were defaulting to 10.0.0 for SO types that did NOT define `modelVersions`. This was due to the removal of the applyTypeDefaults, which was ensuring all SOs had the `switchToModelVersionAt` property set. This flag was then used by `src/core/packages/saved-objects/base-server-internal/src/model_version/version_map.ts` to determine whether to use `modelVersions` or the legacy `migrations` property in order to determine the latest model version for a given type. When removing the `switchToModelVersionAt` flag (and its default backfill), the logic started defaulting to the latest `migrations` version for those SO types that were not defining any `modelVersion`, resulting in older versions that those stored in the SO indices. This caused incident https://elasticco.atlassian.net/browse/INC-3818. This regression has been shipped in 9.0.0 (the PR was [backported](#219329)), so in top of the cleanup, we now need to address #220521 to ensure a smooth transition _OnPrem => Serverless_. --------- Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

…astic#220985) ## Summary * Address elastic#217145 - Put in place a check to ensure we're upgrading from Kibana 8.18.0 or newer. * Address elastic#220521 - New attempt at removing the `switchToModelVersionAt` property, inspired on elastic#219029. The previous attempt caused a regression: index meta information started storing _modelVersions_ that were older than the previously stored ones, which were defaulting to 10.0.0 for SO types that did NOT define `modelVersions`. This was due to the removal of the applyTypeDefaults, which was ensuring all SOs had the `switchToModelVersionAt` property set. This flag was then used by `src/core/packages/saved-objects/base-server-internal/src/model_version/version_map.ts` to determine whether to use `modelVersions` or the legacy `migrations` property in order to determine the latest model version for a given type. When removing the `switchToModelVersionAt` flag (and its default backfill), the logic started defaulting to the latest `migrations` version for those SO types that were not defining any `modelVersion`, resulting in older versions that those stored in the SO indices. This caused incident https://elasticco.atlassian.net/browse/INC-3818. This regression has been shipped in 9.0.0 (the PR was [backported](elastic#219329)), so in top of the cleanup, we now need to address elastic#220521 to ensure a smooth transition _OnPrem => Serverless_. --------- Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

TinaHeiligers added 6 commits April 23, 2025 16:25

Removed switchToModelVersionAt from SO API definition

7d4d915

Refactors validation logic implementing globalSwitchToModelVersion di…

57f9dd5

…rectly

Removes switchToModelVersionAt from cases, slo and encrypted saved ob…

24d50a4

…ject type registration definitions

Cleanup docs

1ef91b2

clean up integration tests

f083f42

Updates all saved object type hashes resulting from removing switchTo…

27e63d7

…ModelVersionAt from the public SO type

elasticmachine and others added 2 commits April 24, 2025 18:00

Merge branch 'main' into kbn201807-remove-switchToModelVersionAt

f7548ad

update test-helper

082f809

TinaHeiligers mentioned this pull request Apr 24, 2025

Remove switchToModelVersionAt #218978

Closed

Merge branch 'main' into kbn201807-remove-switchToModelVersionAt

3750413

TinaHeiligers marked this pull request as ready for review April 24, 2025 22:36

TinaHeiligers requested review from a team as code owners April 24, 2025 22:36

[CI] Auto-commit changed files from 'node scripts/jest_integration -u…

733eac2

… src/core/server/integration_tests/ci_checks'

botelastic bot added the Team:actionable-obs Formerly "obs-ux-management", responsible for SLO, o11y alerting, significant events, & synthetics. label Apr 24, 2025

TinaHeiligers commented Apr 24, 2025

View reviewed changes

kdelemme approved these changes Apr 25, 2025

View reviewed changes

jeramysoucy approved these changes Apr 25, 2025

View reviewed changes

TinaHeiligers merged commit 19c017a into elastic:main Apr 26, 2025
11 checks passed

TinaHeiligers deleted the kbn201807-remove-switchToModelVersionAt branch April 26, 2025 15:00

kibanamachine added the v9.1.0 label Apr 26, 2025

TinaHeiligers mentioned this pull request Apr 26, 2025

[9.0] Removed switchToModelVersionAt from SO API definition (#219029) #219329

Merged

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Apr 28, 2025

kibanamachine mentioned this pull request Apr 29, 2025

[ResponseOps][Cases]Allow dashes in host names in Observables #219038

Merged

kibanamachine added v9.0.1 and removed backport missing Added to PRs automatically when the are determined to be missing a backport. labels Apr 29, 2025

afharo mentioned this pull request May 5, 2025

Revert "Removed switchToModelVersionAt from SO API definition (#219029)" #220059

Merged

afharo added a commit to afharo/kibana that referenced this pull request May 5, 2025

Revert "Removed switchToModelVersionAt from SO API definition (elasti…

3b26555

…c#219029)" This reverts commit 19c017a.

afharo mentioned this pull request May 5, 2025

Revert "Removed switchToModelVersionAt from SO API definition (#219029)" #220063

Merged

afharo added a commit that referenced this pull request May 5, 2025

Revert "Removed switchToModelVersionAt from SO API definition (#219029)…

75c0f95

…" (#220059) Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

gsoldevila mentioned this pull request May 20, 2025

APEX-72 Enforce Kibana >= 8.18.0; Remove switchToModelVersionAt #220985

Merged

rudolf mentioned this pull request Jul 23, 2025

Remove legacy migration definitions and switchToModelVersionAt from SavedObjectType interface #201807

Closed

Conversation

TinaHeiligers commented Apr 24, 2025 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

Uh oh!

TinaHeiligers commented Apr 24, 2025

Uh oh!

elasticmachine commented Apr 24, 2025

Uh oh!

elasticmachine commented Apr 24, 2025

Uh oh!

TinaHeiligers Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TinaHeiligers Apr 24, 2025

Choose a reason for hiding this comment

Uh oh!

TinaHeiligers left a comment

Choose a reason for hiding this comment

Uh oh!

kdelemme left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Apr 25, 2025

💚 Build Succeeded

Metrics [docs]

API count

References to deprecated APIs

History

Uh oh!

Uh oh!

kibanamachine commented Apr 26, 2025

Uh oh!

kibanamachine commented Apr 26, 2025

💔 All backports failed

Manual backport

Questions ?

Uh oh!

TinaHeiligers commented Apr 26, 2025

💚 All backports created successfully

Questions ?

Uh oh!

kibanamachine commented Apr 28, 2025

Uh oh!

TinaHeiligers commented Apr 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

TinaHeiligers commented Apr 24, 2025 •

edited by kibanamachine

Loading

TinaHeiligers Apr 24, 2025 •

edited

Loading