HDDS-1207. Refactor Container Report Processing logic and plugin new Replication Manager. #662
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…Replication Manager.
|
🎊 +1 overall
This message was automatically generated. |
arp7
reviewed
Apr 2, 2019
| LOG.info("Starting Replication Monitor Thread."); | ||
| running = true; | ||
| replicationMonitor.start(); | ||
| CompletableFuture.runAsync(() -> { |
Contributor
There was a problem hiding this comment.
Don't use the forkJoin commonPool. It has very few threads and can be easily exhausted. We saw this to be a frequent issue in unit tests.
Instead use the overload that accepts an Executor.
| " in {} milliseconds.", delay); | ||
| Thread.sleep(delay); | ||
| } catch (InterruptedException ignored) { | ||
| // InterruptedException is ignored. |
| } | ||
| } | ||
| final Set<ContainerID> missingReplicas = new HashSet<>(containersInSCM); | ||
| missingReplicas.removeAll(containersInDn); |
Contributor
There was a problem hiding this comment.
What are the last few lines are doing? Didn't quite get it.
Contributor
Author
There was a problem hiding this comment.
Here we have to identify the replicas which are missing from the reported datanode.
SCM (NodeManager) has the list of container replicas that are expected in a given datanode, we identify the missing replicas by finding the delta between the list maintained in NodeManager and the list of replicas reported by the datanode.
Once we identify the missing replicas, we go and update the replica map in ContainerManager.
| * @param replicas List of ContainerReplicaProto | ||
| * @param publisher EventPublisher reference | ||
| */ | ||
| private void updateDeleteTransaction(final DatanodeDetails datanodeDetails, |
Contributor
There was a problem hiding this comment.
Had a few questions on this function. I am not familiar with this part.
Contributor
Author
There was a problem hiding this comment.
This is for updating the block deletion transaction Id reported by the container replica. If the container replica is lagging behind in block deletion, we trigger PENDING_DELETE_STATUS event so that SCMBlockDeletingService can resend block deletion commands to the datanode.
|
🎊 +1 overall
This message was automatically generated. |
shanthoosh
pushed a commit
to shanthoosh/hadoop
that referenced
this pull request
Oct 15, 2019
This Request is a copy of apache#647(got garbled). This PR already addresses all the comments brought up in the other request. Author: Boris S <[email protected]> Author: Boris S <[email protected]> Author: Boris Shkolnik <[email protected]> Reviewers: Shanthoosh Venkatraman <[email protected]>, Prateek Maheshwari <[email protected]> Closes apache#662 from sborya/NewConsumerAdmin2
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR #620 brings in new ReplicationManager, this change is to refactor ContainerReportProcessing logic in SCM so that it complements ReplicationManager and plugin the new ReplicationManager code.