Always retry provisioning operations on failure (continue)#610
Merged
metal3-io-bot merged 1 commit intometal3-io:masterfrom Sep 17, 2020
Merged
Always retry provisioning operations on failure (continue)#610metal3-io-bot merged 1 commit intometal3-io:masterfrom
metal3-io-bot merged 1 commit intometal3-io:masterfrom
Conversation
metal3-io-bot
added
do-not-merge/work-in-progress
andfasano
force-pushed
the
keep-retrying-ext
branch
2 times, most recently
from
f8fa6dc to
e7e1dd7
Compare
andfasano
changed the title
metal3-io-bot
removed
the
do-not-merge/work-in-progress
Member
Author
|
/test-integration |
This was referenced Jul 31, 2020
dhellmann
reviewed
Aug 5, 2020
zaneb
reviewed
Aug 5, 2020
| return backOffDuration | ||
| } | ||
|
|
||
| func (r actionFailed) Result() (result reconcile.Result, err error) { |
Member
There was a problem hiding this comment.
FWIW it would be fine to change the signature of Result() to pass the error count if it were more convenient for us to not calculate it until later.
metal3-io-bot
added
size/L
zaneb
reviewed
Aug 6, 2020
andfasano
force-pushed
the
keep-retrying-ext
branch
from
63258fa to
c2a937b
Compare
andfasano
force-pushed
the
keep-retrying-ext
branch
2 times, most recently
from
7e4bdfc to
d5bea5a
Compare
Member
Author
|
/test-integration |
dhellmann
reviewed
Sep 8, 2020
dhellmann
reviewed
Sep 9, 2020
metal3-io-bot
added
the
approved
Member
|
would you mind squashing the commits please ? |
andfasano
force-pushed
the
keep-retrying-ext
branch
from
f460be4 to
9d8411e
Compare
Member
Author
|
/test-integration |
andfasano
force-pushed
the
keep-retrying-ext
branch
from
9d8411e to
e5d6d9b
Compare
Member
Author
|
/test govet |
Member
Author
|
/test unit |
Member
Author
|
/test-integration |
Member
|
/approve |
zaneb
reviewed
Sep 15, 2020
Member
zaneb
left a comment
zaneb
left a comment
There was a problem hiding this comment.
Refactoring in a separate patch is good, but please squash stuff like "Fix imports".
/approve
andfasano
force-pushed
the
keep-retrying-ext
branch
from
e5d6d9b to
0e97bf4
Compare
Member
Author
|
/test-integration |
Improve the reconciliation loop whenever an action failure is detected (or credential error) by applying a retry pattern with exponentional backoff with jitter to avoid service overloading - currently affected statuses are: Deprovisioning, Externally Provisioned, Inspecting, Provisioned, Provisioning and Registering. A new `ErrorCount` field has been added in the `BareMetalHostStatus` to support such behavior: it gets incremented every time an action failure is recorded, and it is cleared out when the action completes successfully.
andfasano
force-pushed
the
keep-retrying-ext
branch
from
0e97bf4 to
b1b31f7
Compare
Member
Author
|
/test-integration |
Member
|
/approve |
Contributor
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: andfasano, dhellmann, zaneb The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Member
|
/lgtm |
Member
|
/test-integration |
honza
pushed a commit
to honza/baremetal-operator
that referenced
this pull request
Sep 23, 2020
Always retry provisioning operations on failure (continue)
honza
pushed a commit
to honza/baremetal-operator
that referenced
this pull request
Sep 29, 2020
Always retry provisioning operations on failure (continue)
honza
pushed a commit
to honza/baremetal-operator
that referenced
this pull request
Sep 29, 2020
Always retry provisioning operations on failure (continue)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR replaces the one started from @honza on PR #584.
Reconciliation loop now retries the operation whenever an action failure is detected, with a backoff. The backoff calculation has been reviewed and some jitter added, and the error count is zeroed when the action succeeds.