[upgrade assistant] Stop rollup jobs before reindexing by mattkime · Pull Request #212815 · elastic/kibana

mattkime · 2025-02-28T21:11:55Z

This PR improve support for rollup indices. Rollup indices can be handled like normal indices but jobs should be stopped before reindexing begins or index is marked read only. Also handles case where the rollup job is already stopped.

To review: Mark the following read only and make sure rollup jobs are handled as appropriate: Rollup index with and without job running, normal index.

Follow up to #212592 and #214656

Closes: #211850

…reindexing

x-pack/platform/plugins/private/upgrade_assistant/server/lib/reindexing/reindex_service.ts

…e/kibana into upgrade_assistant_rollup_support

x-pack/platform/plugins/private/upgrade_assistant/server/lib/reindexing/reindex_service.ts

sabarasaba

tested with the shared data folder and it seems to be working correctly

left few nits, let me know what you think

sabarasaba · 2025-03-19T09:49:51Z

x-pack/platform/plugins/private/upgrade_assistant/server/lib/reindexing/reindex_actions.ts

+      let rollupJob;
+      if (rollupIndices.length > 0) {
+        // there should only be one job
+        rollupJob = rollupCaps[rollupIndices[0]].rollup_jobs[0].job_id;


nit: this seems a bit brittle, any chance we can make it more resilient?

x-pack/platform/plugins/private/upgrade_assistant/server/lib/update_index/index.ts

mattkime · 2025-03-19T17:40:49Z

/ci

…e/kibana into upgrade_assistant_rollup_support

mattkime · 2025-03-20T13:46:52Z

/ci

mattkime · 2025-03-20T16:26:31Z

/ci

mattkime · 2025-03-20T17:07:28Z

/ci

elasticmachine · 2025-03-21T05:04:15Z

Pinging @elastic/kibana-management (Team:Kibana Management)

sabarasaba

left few comments, let me know what you think

x-pack/platform/plugins/private/upgrade_assistant/server/lib/rollup_job.ts

…ollup_job.ts Co-authored-by: Ignacio Rivas <rivasign@gmail.com>

…e/kibana into upgrade_assistant_rollup_support

…llup_support

jloleysens

Left a few qs, not blockers but I think it would be nice to add an E2E test. Maybe in x-pack/test/upgrade_assistant_integration/upgrade_assistant/reindexing.ts.

x-pack/platform/plugins/private/upgrade_assistant/server/lib/reindexing/reindex_service.ts

jloleysens · 2025-03-20T10:38:16Z

x-pack/platform/plugins/private/upgrade_assistant/server/lib/update_index/index.ts

+        // stop related rollup job if it exists
+        const rollupJob = await getRollupJobByIndexName(esClient, index);
+        if (rollupJob) {
+          await esClient.rollup.stopJob({ id: rollupJob });


Should we also wait_for_completion: true here? Or might it take too long?

jloleysens · 2025-04-03T11:44:29Z

x-pack/platform/plugins/private/upgrade_assistant/server/lib/reindexing/reindex_service.ts

+    const { indexName, rollupJob } = reindexOp.attributes;
+
+    if (rollupJob) {
+      await esClient.rollup.stopJob({ id: rollupJob, wait_for_completion: true });


What happens when it fails to stop the job? Do we provide users with actionable information about the issue like a manual workaround?

I think this gets into the expectations surrounding this code. If any other aspect of reindexing fails, do we tell the users about a manual work around?

There's not much reason to expect this to fail. Actually, I'm not sure I'd have much advice for the user if this failed since I'm unfamiliar with what could cause it to fail.

Yeah generally we return the error message. My question was more about what the content of a failure message is and is it something users can self-service? Generally UA reindexing error messages are self-servicable but I'm not familiar with common failure conditions here so I thought I'd ask.

It might be the case that these are, for example, retryable or something we can handle.

I checked with @martijnvg and he stated that there's no reason to anticipate any error states when stopping rollups so I think I'll leave the code as is.

elasticmachine · 2025-04-13T15:49:27Z

💚 Build Succeeded

Buildkite Build
Commit: f9fb598

Metrics [docs]

✅ unchanged

History

💛 Build #291583 was flaky 60e4752
💚 Build #290122 succeeded 627a56a
💔 Build #289742 failed 2db3f54
💚 Build #286215 succeeded e72ca9d
💔 Build #286194 failed 1ab8bfb
💔 Build #286127 failed 25f1694

cc @mattkime

…to 9.1 (#218049) ## Summary forward port of #212815 --- This PR improve support for rollup indices. Rollup indices can be handled like normal indices but jobs should be stopped before reindexing begins or index is marked read only. Also handles case where the rollup job is already stopped. To review: Mark the following read only and make sure rollup jobs are handled as appropriate: Rollup index with and without job running, normal index. Follow up to #212592 and #214656 Closes: #211850

…to 9.1 (elastic#218049) ## Summary forward port of elastic#212815 --- This PR improve support for rollup indices. Rollup indices can be handled like normal indices but jobs should be stopped before reindexing begins or index is marked read only. Also handles case where the rollup job is already stopped. To review: Mark the following read only and make sure rollup jobs are handled as appropriate: Rollup index with and without job running, normal index. Follow up to elastic#212592 and elastic#214656 Closes: elastic#211850

support rollup indices as regular indices. Disable rollup job before …

d35c4a8

…reindexing

mattkime changed the title ~~support rollup indices as regular indices. Disable rollup job before …~~ [upgrade assistant] Stop rollup jobs before reindexing Mar 18, 2025

mattkime added release_note:skip Skip the PR/issue when compiling release notes backport:skip This PR does not require backporting labels Mar 18, 2025

Merge branch '8.x' into upgrade_assistant_rollup_support

e9ff4f5

mattkime commented Mar 18, 2025

View reviewed changes

x-pack/platform/plugins/private/upgrade_assistant/server/lib/reindexing/reindex_service.ts Outdated Show resolved Hide resolved

mattkime added 2 commits March 19, 2025 01:02

fix stopping rollup jobs before marking indices read only

7d57062

Merge branch 'upgrade_assistant_rollup_support' of github.com:mattkim…

890ed48

…e/kibana into upgrade_assistant_rollup_support

mattkime commented Mar 19, 2025

View reviewed changes

x-pack/platform/plugins/private/upgrade_assistant/server/lib/reindexing/reindex_service.ts Show resolved Hide resolved

mattkime marked this pull request as ready for review March 19, 2025 06:04

mattkime requested a review from kibanamachine as a code owner March 19, 2025 06:04

sabarasaba self-requested a review March 19, 2025 09:07

sabarasaba approved these changes Mar 19, 2025

View reviewed changes

mattkime added 2 commits March 19, 2025 07:06

Merge branch '8.x' into upgrade_assistant_rollup_support

e3f985a

refactor / cleanup

8f51cce

mattkime added 3 commits March 19, 2025 12:41

Merge branch '8.x' into upgrade_assistant_rollup_support

0e1f79c

add debugging comment for functional test

7865008

Merge branch 'upgrade_assistant_rollup_support' of github.com:mattkim…

1354f35

…e/kibana into upgrade_assistant_rollup_support

add comment

e7121ab

add comment

690b90a

mattkime added 2 commits March 20, 2025 21:53

add try catch around rollup calls

f25812c

Merge branch '8.x' into upgrade_assistant_rollup_support

217549e

mattkime self-assigned this Mar 21, 2025

mattkime added the Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more t// label Mar 21, 2025

mattkime added the Feature:Upgrade Assistant label Mar 21, 2025

sabarasaba reviewed Mar 21, 2025

View reviewed changes

mattkime and others added 6 commits March 21, 2025 05:32

Update x-pack/platform/plugins/private/upgrade_assistant/server/lib/r…

50fe3de

…ollup_job.ts Co-authored-by: Ignacio Rivas <rivasign@gmail.com>

better logging

e00d935

Merge branch 'upgrade_assistant_rollup_support' of github.com:mattkim…

25f1694

…e/kibana into upgrade_assistant_rollup_support

supply logging where its needed

1ab8bfb

add log dependency

e72ca9d

Merge remote-tracking branch 'upstream/8.x' into upgrade_assistant_ro…

2db3f54

…llup_support

jloleysens approved these changes Apr 3, 2025

View reviewed changes

mattkime added 3 commits April 3, 2025 21:07

fix jest tests, add wait_for_completion in one spot

627a56a

Merge branch '8.x' into upgrade_assistant_rollup_support

b2d65e9

Merge branch '8.x' into upgrade_assistant_rollup_support

60e4752

mattkime added backport:skip This PR does not require backporting and removed backport:skip This PR does not require backporting labels Apr 9, 2025

Merge branch '8.x' into upgrade_assistant_rollup_support

f9fb598

mattkime merged commit c776bc4 into elastic:8.x Apr 14, 2025
8 checks passed

mattkime mentioned this pull request Apr 14, 2025

[upgrade assistant] Stop rollup jobs before reindexing - forwardport to 9.1 #218049

Merged

Conversation

mattkime commented Feb 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sabarasaba left a comment

Choose a reason for hiding this comment

Uh oh!

sabarasaba Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mattkime commented Mar 19, 2025

Uh oh!

mattkime commented Mar 20, 2025

Uh oh!

mattkime commented Mar 20, 2025

Uh oh!

mattkime commented Mar 20, 2025

Uh oh!

elasticmachine commented Mar 21, 2025

Uh oh!

sabarasaba left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jloleysens left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jloleysens Mar 20, 2025

Choose a reason for hiding this comment

Uh oh!

jloleysens Apr 3, 2025

Choose a reason for hiding this comment

Uh oh!

mattkime Apr 3, 2025

Choose a reason for hiding this comment

Uh oh!

jloleysens Apr 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattkime Apr 8, 2025

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Apr 13, 2025

💚 Build Succeeded

Metrics [docs]

History

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mattkime commented Feb 28, 2025 •

edited

Loading

jloleysens Apr 7, 2025 •

edited

Loading