Handle bulkGet errors on package retrieval from ES storage#111114
Merged
joshdover merged 1 commit intoelastic:masterfrom Sep 3, 2021
Merged
Handle bulkGet errors on package retrieval from ES storage#111114joshdover merged 1 commit intoelastic:masterfrom
joshdover merged 1 commit intoelastic:masterfrom
Conversation
joshdover
added
v8.0.0
release_note:skip
Contributor
|
Pinging @elastic/fleet (Team:Fleet) |
kpollich
approved these changes
Sep 3, 2021
Member
kpollich
left a comment
kpollich
left a comment
There was a problem hiding this comment.
Changes LGTM and I'm okay landing this before tests are sorted as we're working with a bit of an unknown bug 🚀
Contributor
💚 Build SucceededMetrics [docs]
To update your PR or re-run it, just comment with: |
joshdover
added
the
auto-backport
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Sep 3, 2021
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Sep 3, 2021
Contributor
kibanamachine
added a commit
that referenced
this pull request
Sep 3, 2021
kibanamachine
added a commit
that referenced
this pull request
Sep 3, 2021
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds error handling and logging for the case where not all package assets can be successfully retrieved from storage. Now in this case, we'll fallback to attempting to fetch the package from the registry when we can't successfully retrieve all the assets from ES for some reason.
This came up on the edge-oblt cluster on the latest 8.0.0 snapshot. In my analysis of that cluster, I'm not seeing any assets that are actually missing, so I currently suspect a different bug that may be produced by something in the SOC. In particular, this recent change seems relevant: #109967
This additional logging and fallback behavior should help us nail down the root cause.
@nchaulet any recommendations on where to add tests for this? The best place would be unit tests on this archive storage code, but it seems we don't have this yet. Outside of that option I don't see any good options since I'm not yet sure of the root cause here to be able to reproduce.