[HBASE-25536] Remove 0 length wal file from logQueue if it belongs to… #2908
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
🎊 +1 overall
This message was automatically generated. |
wchevreuil
reviewed
Jan 28, 2021
Contributor
wchevreuil
left a comment
wchevreuil
left a comment
There was a problem hiding this comment.
Makes sense to me. How difficult is it to add UTs testing this?
bharathv
reviewed
Jan 28, 2021
| private void handleEofException(IOException e) { | ||
| if ((e instanceof EOFException || e.getCause() instanceof EOFException) && | ||
| logQueue.size() > 1 && this.eofAutoRecovery) { | ||
| (source.isRecovered() || logQueue.size() > 1) && this.eofAutoRecovery) { |
Contributor
There was a problem hiding this comment.
nice find, mind adding a quick comment since this is a subtle behavior?
Contributor
There was a problem hiding this comment.
On a related note, should we add a source and global metric counter to track the number of 0 size files dequeued? Seems more common than we thought, easy to monitor may be..
Contributor
Author
There was a problem hiding this comment.
On a related note, should we add a source and global metric counter to track the number of 0 size files dequeued? Seems more common than we thought, easy to monitor may be..
@bharathv Good idea. Could you please create a separate jira to track this. Thank you !
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
Contributor
Author
Will add in next commit. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
Contributor
Author
|
This patch applies cleanly for branch-2 |
bharathv
approved these changes
Jan 29, 2021
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
virajjasani
reviewed
Jan 29, 2021
Contributor
virajjasani
left a comment
virajjasani
left a comment
There was a problem hiding this comment.
Left small nit, looks good otherwise
| when(source.isPeerEnabled()).thenReturn(true); | ||
| ReplicationSourceWALReader reader = | ||
| new ReplicationSourceWALReader(fs, conf, queue, 0, getDummyFilter(), source); | ||
| reader.run(); |
virajjasani
approved these changes
Jan 29, 2021
| reader.run(); | ||
| // ReplicationSourceWALReaderThread#handleEofException method will | ||
| // remove empty log from logQueue. | ||
| assertEquals(0, queue.size()); |
Contributor
There was a problem hiding this comment.
I see, If we use thread.start(), this could be flaky if we reach here before reader thread removes entry from queue so we will need to add some wait here based on this exit condition (queue.isEmpty()). It's a trade off with starting thread separately as opposed to synchronously executing it with thread.run().
It's upto you, this is anyways trivial. thread.run() is also fine.
virajjasani
pushed a commit
that referenced
this pull request
Jan 29, 2021
…ld sources (#2908) Signed-off-by: Wellington Chevreuil <[email protected]> Signed-off-by: Geoffrey Jacoby <[email protected]> Signed-off-by: Bharath Vissapragada <[email protected]> Signed-off-by: Viraj Jasani <[email protected]>
virajjasani
pushed a commit
that referenced
this pull request
Jan 29, 2021
…ld sources (#2908) Signed-off-by: Wellington Chevreuil <[email protected]> Signed-off-by: Geoffrey Jacoby <[email protected]> Signed-off-by: Bharath Vissapragada <[email protected]> Signed-off-by: Viraj Jasani <[email protected]>
virajjasani
pushed a commit
that referenced
this pull request
Jan 29, 2021
…ld sources (#2908) Signed-off-by: Wellington Chevreuil <[email protected]> Signed-off-by: Geoffrey Jacoby <[email protected]> Signed-off-by: Bharath Vissapragada <[email protected]> Signed-off-by: Viraj Jasani <[email protected]>
virajjasani
pushed a commit
that referenced
this pull request
Jan 29, 2021
…ld sources (#2908) Signed-off-by: Wellington Chevreuil <[email protected]> Signed-off-by: Geoffrey Jacoby <[email protected]> Signed-off-by: Bharath Vissapragada <[email protected]> Signed-off-by: Viraj Jasani <[email protected]>
virajjasani
pushed a commit
that referenced
this pull request
Jan 29, 2021
…ld sources (#2912) (#2908) Signed-off-by: Wellington Chevreuil <[email protected]> Signed-off-by: Geoffrey Jacoby <[email protected]> Signed-off-by: Bharath Vissapragada <[email protected]> Signed-off-by: Viraj Jasani <[email protected]>
wchevreuil
pushed a commit
to wchevreuil/hbase
that referenced
this pull request
May 24, 2021
…ld sources (apache#2908) Signed-off-by: Wellington Chevreuil <[email protected]> Signed-off-by: Geoffrey Jacoby <[email protected]> Signed-off-by: Bharath Vissapragada <[email protected]> Signed-off-by: Viraj Jasani <[email protected]> (cherry picked from commit ea1a9e9) Change-Id: Ifb5a34850ff75f4353fc1f0aa6f534897bd5458a
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
… old sources.