Retry ES API calls that fail with 410/Gone to prevent kibana from crashing at startup

[rudolf]

Summary

For Saved Object migrations, ignore ES API calls that fail with 410/Gone status code. This prevents Kibana from crashing on startup under the following conditions:

1. Kibana passes the ES version check by successfully querying ES nodes.info.
2. Shortly after, just as Kibana attempts to start it's migrations, the ES proxy becomes unhealthy and starts returning 410/Gone status codes.

Checklist

Delete any items that are not applicable to this PR.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• This was checked for keyboard-only and screenreader accessibility
• This renders correctly on smaller devices using a responsive layout. (You can test this in your browser
• This was checked for cross-browser compatibility, including a check against IE11

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[bhavyarm]

@rudolf cc @LeeDr can you please let us know how you found this issue? We have a smoke test for it but I can't reproduce it. Thanks!

[rudolf]

@bhavyarm It was by coincidence that we noticed it because an ES Proxy was incorrectly configured which would cause it to always return 410 status codes to any requests. The only way to test this would be to point Kibana to an proxy which allows us to control the response status codes. I've used a tool like MITMProxy to do something like this before https://docs.mitmproxy.org/stable/addons-scripting/ if we don't have any existing tools let me know if I can help set something up.

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request
• Commit: 95b7bf7

---

Test Failures

Kibana Pipeline / x-pack-intake-agent / X-Pack Jest Tests.x-pack/legacy/plugins/uptime/server/lib/helper/__test__.getHistogramIntervalFormatted specifies the interval necessary to divide a given timespan into equal buckets, rounded to the nearest integer, expressed in ms

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

Stack Trace

Error: expect(received).toBeTruthy()

Received: false
    at Object.it (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/x-pack/legacy/plugins/uptime/server/lib/helper/__test__/get_histogram_interval_formatted.test.ts:18:50)
    at Object.asyncJestTest (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-jasmine2/build/jasmineAsyncInstall.js:102:37)
    at resolve (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-jasmine2/build/queueRunner.js:43:12)
    at new Promise (<anonymous>)
    at mapper (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-jasmine2/build/queueRunner.js:26:19)
    at promise.then (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-jasmine2/build/queueRunner.js:73:41)
    at process._tickCallback (internal/process/next_tick.js:68:7)

---

History

• 💚 Build #24850 succeeded 95b7bf7

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[rudolf]

@elasticmachine merge upstream

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 570fa17

History

• 💔 Build #25107 failed 95b7bf7
• 💚 Build #24850 succeeded 95b7bf7

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream