HDDS-10626. [LeaseRecovery] OM shuts down with 'SecretKey client must have been initialized already' #6467
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… have been initialized already'
jojochuang
reviewed
Apr 2, 2024
|
|
||
| OMClientResponse omClientResponse = validateAndUpdateCache(); | ||
| OMResponse omResponse = omClientResponse.getOMResponse(); | ||
| assertEquals(OzoneManagerProtocolProtos.Status.OK, omResponse.getStatus()); |
Contributor
There was a problem hiding this comment.
is it okay to just let it return OK? Or is it better to return an error? Ideally the client should keep retrying until OM is running.
Contributor
Author
There was a problem hiding this comment.
This path is only for ratis raft log reapply, so skip the token is OK. But I find a more clean way to handle this. I will upload a new patch.
Contributor
Author
There was a problem hiding this comment.
The new patch is tested by Pratyush, and it works, the issue doesn't happen again(It's reproducible) . @jojochuang
Contributor
ashishkumar50
left a comment
ashishkumar50
left a comment
There was a problem hiding this comment.
Thanks @ChenSammi for the patch, Change LGTM +1.
Contributor
Author
|
Thanks @jojochuang and @ashishkumar50 for the review. |
chungen0126
pushed a commit
to chungen0126/ozone
that referenced
this pull request
May 3, 2024
… have been initialized already' (apache#6467)
chungen0126
pushed a commit
to chungen0126/ozone
that referenced
this pull request
May 3, 2024
… have been initialized already' (apache#6467)
jojochuang
pushed a commit
to jojochuang/ozone
that referenced
this pull request
May 29, 2024
… have been initialized already' (apache#6467) (cherry picked from commit 6cfe9cf)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
OzoneManager crashed during reapply Ratis transactions during startup. During recoverLease reapply, the request wants to generate the block token which depends on SecretKey service, while SecretKey client is not instantiated at that time because OzoneManager is reapply txs, and all the other services are not yet initialized.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-10626
How was this patch tested?
Tested in the cluster where the issue is found and can repeatedly reproduced.