HDFS-16484. [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread#4032
Merged
tasanuma merged 1 commit intoapache:trunkfrom Apr 13, 2022
Merged
HDFS-16484. [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread#4032tasanuma merged 1 commit intoapache:trunkfrom
tasanuma merged 1 commit intoapache:trunkfrom
Conversation
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
Contributor
Author
|
@ayushtkn Could you please take a look at this. Thanks a lot. |
Contributor
Author
|
@tasanuma Could you please take a look at this. Thanks a lot. |
tasanuma
reviewed
Apr 8, 2022
Member
There was a problem hiding this comment.
How about cathing and ignoring the exception here?
Suggested change
| ctxt.removeSPSHint(startINode); | |
| try { | |
| ctxt.removeSPSHint(startINode); | |
| } catch (FileNotFoundException e) { | |
| // ignore if the file doesn't already exist | |
| startINode = null; | |
| } |
Member
There was a problem hiding this comment.
@liubingxing I still prefer to catch the FileNotFoundException here. What do you think?
Contributor
Author
There was a problem hiding this comment.
@tasanuma Thanks for your suggestion.
It is also a good way to solve this problem by catching the FileNotFoundException here.
I updated the code according to your suggestion.
Member
There was a problem hiding this comment.
@liubingxing Thank you for updating the PR. I think we can keep the retry logic implemented before while catching the FileNotFoundException here.
Contributor
Author
There was a problem hiding this comment.
@liubingxing I still prefer to catch the FileNotFoundException here. What do you think?
Sorry @tasanuma, I misunderstand your meaning here. I updated the code, please take a look.
tasanuma
reviewed
Apr 8, 2022
Member
There was a problem hiding this comment.
Do we still need to implement the retry logic after catching the FileNotFoundException?
Contributor
Author
There was a problem hiding this comment.
@tasanuma Thanks for your reviews.
I'm not sure if there are any other exceptions may cause this thread work incorrectly.
And the retry logic may help to avoid this thread falling into an infinite loop if the 'startINode' fail to be set to null.
|
🎊 +1 overall
This message was automatically generated. |
tasanuma
reviewed
Apr 12, 2022
Comment on lines
266
to
269
Member
There was a problem hiding this comment.
- Let's define the constant of the max retry count (
private static final int MAX_RETRY_COUNT = 3;) inSPSPathIdProcessor. - How about logging a message when skipping the inode?
Suggested change
| retryCount++; | |
| if (retryCount >= 3) { | |
| startINode = null; | |
| } | |
| retryCount++; | |
| if (retryCount >= MAX_RETRY_COUNT) { | |
| LOG.warn("Skipping this inode {} due to too many retries.", startINode); | |
| startINode = null; | |
| } |
- And I think it's better to move the retry logic to the end of the catch block.
Contributor
Author
There was a problem hiding this comment.
@tasanuma Thanks for your advices. I updated the code.
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
Member
|
@liubingxing Thanks for your contribution! |
Contributor
Author
|
@tasanuma Thanks for your review and merge. |
HarshitGupta11
pushed a commit
to HarshitGupta11/hadoop
that referenced
this pull request
Nov 28, 2022
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
JIRA: HDFS-16484
Currently, we ran SPS in our cluster and found this log. The SPSPathIdProcessor thread enters an infinite loop and prints the same log all the time.
The reason is that #ctxt.getNextSPSPath() get a inodeId which path does not exist. The inodeId will not be set to null, causing the thread hold this inodeId forever.