Skip to content

[8.7] Introduce CLEANUP_UNKNOWN_AND_EXCLUDED step (#149931)#152216

Merged
gsoldevila merged 2 commits intoelastic:8.7from
gsoldevila:backport/8.7/pr-149931
Feb 27, 2023
Merged

[8.7] Introduce CLEANUP_UNKNOWN_AND_EXCLUDED step (#149931)#152216
gsoldevila merged 2 commits intoelastic:8.7from
gsoldevila:backport/8.7/pr-149931

Conversation

@gsoldevila
Copy link
Copy Markdown
Member

Backport

This will backport the following commits from main to 8.7:

Questions ?

Please refer to the Backport tool documentation

In the context of migrations,
elastic#147371 avoids reindexing during
an upgrade, provided that `diffMappings === false`.

This _alternative path_ skips some key steps that are performed before
reindexing:
* `CHECK_UNKNOWN_DOCUMENTS`
* `CALCULATE_EXCLUDE_FILTERS`

These steps enrich a search query that is used during reindexing,
effectively filtering out undesired documents.

If the mappings [match](elastic#147371)
(or they are
[compatible](elastic#149326)) and we _no
longer reindex_, this cleanup operation does not happen, leaving
undesired documents in our system indices.

The goal of this PR is to add an extra step in the state machine
(`CLEANUP_UNKNOWN_AND_EXCLUDED`), which will actively cleanup a system
index if we're going the _skip reindexing_ path.

![image](https://user-images.githubusercontent.com/25349407/216979691-fef40638-f990-4850-bac8-ee3e58330a7f.png)

Fixes elastic#150299

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
(cherry picked from commit 754e868)
@gsoldevila gsoldevila added the backport This PR is a backport of another PR label Feb 27, 2023
@gsoldevila gsoldevila enabled auto-merge (squash) February 27, 2023 13:31
@kibana-ci
Copy link
Copy Markdown

kibana-ci commented Feb 27, 2023

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] Rules, Alerts and Exceptions ResponseOps Cypress Tests on Security Solution / Rules selection should correctly update the selection label when rules are bulk selected and then unselected via the table select all checkbox

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
@kbn/core-root-server-internal 22 24 +2
@kbn/core-saved-objects-migration-server-internal 79 81 +2
total +4

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id before after diff
@kbn/core-saved-objects-migration-server-internal 45 46 +1
Unknown metric groups

API count

id before after diff
@kbn/core-root-server-internal 23 25 +2
@kbn/core-saved-objects-migration-server-internal 112 116 +4
total +6

ESLint disabled in files

id before after diff
core 0 1 +1

ESLint disabled line counts

id before after diff
core 7 8 +1

Total ESLint disabled count

id before after diff
core 7 9 +2

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@gsoldevila gsoldevila merged commit 5bbaa55 into elastic:8.7 Feb 27, 2023
@rudolf rudolf added Epic:ScaleMigrations Scale upgrade migrations to millions of saved objects and removed Epic:ScaleMigrations Scale upgrade migrations to millions of saved objects labels Jun 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport This PR is a backport of another PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants