[Fleet] Add retries w/ backoff to Fleet setup on Kibana boot

[joshdover]

Summary

Closes #165971

This adds retry logic w/ backoff to Fleet's setup process that runs on Kibana boot. For now, this is behind a xpack.fleet.internal.retrySetupOnBoot feature flag that is only enabled on Serverless projects.

In ESS, we depend on the Integrations Server's boot process to implicitly perform the retry logic, since it depends on Fleet setup to be completed before it can successfully boot. This behavior does not exist on Serverless so we need built-in logic to perform this. The objects that are setup on serverless are:

• Elasticsearch output
• Fleet Server URL config
• (in some Security projects) A preconfigured Agent policy for hosted CSPM.

This change does not block Kibana startup, so that Kibana can still be used while Fleet's setup process is retrying to completion. Fleet will report the current status of it's setup on the Kibana-wide /api/status API, including the number of retry attempts and the last error encountered.

This change does not include any explicit tests, but instead is relying on the existing test coverage of Serverless end-to-end tests, Fleet setup FTR tests, and the exponential-backoff library's unit tests.

Checklist

Delete any items that are not applicable to this PR.

• Unit or functional tests were updated or added to match the most common scenarios

[ghost]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• /oblt-deploy-serverless : Deploy a serverless Kibana instance using the Observability test environments.
• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[joshdover]

Kibana already has this dependency as a transitive dep, thought it made sense to just use it again here.

[jlind23]

Changed the keyword in the description to Closes as stated here

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[juliaElastic]

LGTM

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 9f06bf3

Failed CI Steps

• FTR Configs #15
• FTR Configs #60
• FTR Configs #67
• Defend Workflows Cypress Tests #4

Test Failures

• [job] [logs] Defend Workflows Cypress Tests #4 / Artifact pages Trusted applications should update Endpoint Policy on Endpoint when adding Trusted application name should update Endpoint Policy on Endpoint when adding Trusted application name
• [job] [logs] FTR Configs #67 / Observability Log Explorer DatasetSelection initialization and update when the "index" query param exists should fallback to the "All logs" selection and notify the user of an invalid encoded index
• [job] [logs] FTR Configs #60 / serverless observability UI navigation navigate observability sidenav & breadcrumbs
• [job] [logs] FTR Configs #15 / serverless security UI landing page has serverless side nav

Metrics [docs]

✅ unchanged

History

• 💔 Build #163282 failed 28d5710
• 💔 Build #162964 failed 790328a
• 💔 Build #162901 failed 5612b99
• 💔 Build #162855 failed c4dc631
• 💔 Build #162787 failed 234248c

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream