HDDS-8504. ReplicationManager: Pass used and excluded node separately for Under and Mis-Replication #4694
Conversation
adoroszlai
left a comment
adoroszlai
left a comment
There was a problem hiding this comment.
Thanks @sodonnel for the patch, LGTM, with one question.
...rc/main/java/org/apache/hadoop/hdds/scm/container/replication/ECUnderReplicationHandler.java
Show resolved
Hide resolved
ashishkumar50
left a comment
ashishkumar50
left a comment
There was a problem hiding this comment.
Hi @sodonnel, Thanks for working on this, PR LGTM.
I have one question "favoredNodes" is not used in container placement scenario or do we have any plan to use?
So far as I know favoredNodes is not used anywhere, which is a same as it makes the topology code significantly more complex. Perhaps there was a plan to use it once that never got implemented, but it was before my time on the project. |
|
I am going to commit this as it is. We can consider in the future if it makes sense to add the replicationManager excluded nodes into the master exclude list for simple replication. Right now its only used for EC under replication. |
* master: (78 commits) HDDS-8575. Intermittent failure in TestCloseContainerEventHandler.testCloseContainerWithDelayByLeaseManager (apache#4688) HDDS-7241. EC: Reconstruction could fail with orphan blocks. (apache#4718) HDDS-8577. [Snapshot] Disable compaction log when loading metadata for snapshot (apache#4697) HDDS-7080. EC: Offline reconstruction needs better logging (apache#4719) HDDS-8626. Config thread pool in ReplicationServer (apache#4715) HDDS-8616. Underreplication not fixed if all replicas start decommissioning (apache#4711) HDDS-8254. Close containers when volume reaches utilisation threshold (apache#4583) HDDS-8254. Close containers when volume reaches utilisation threshold (apache#4583) HDDS-8615. Explicitly show EC block type in 'ozone debug chunkinfo' command output (apache#4706) HDDS-8623. Delete duplicate getBucketInfo in OMKeyCommitRequest (apache#4712) HDDS-8339. Recon Show the number of keys marked for Deletion in Recon UI. (apache#4519) HDDS-8572. Support CodecBuffer for protobuf v3 codecs. (apache#4693) HDDS-8010. Improve DN warning message when getBlock does not find the block. (apache#4698) HDDS-8621. IOException is never thrown in SCMRatisServer.getRatisRoles(). (apache#4710) HDDS-8463. S3 key uniqueness in deletedTable (apache#4660) HDDS-8584. Hadoop client write slowly when stream enabled (apache#4703) HDDS-7732. EC: Verify block deletion from missing EC containers (apache#4705) HDDS-8581. Avoid random ports in integration tests (apache#4699) HDDS-8504. ReplicationManager: Pass used and excluded node separately for Under and Mis-Replication (apache#4694) HDDS-8576. Close RocksDB instance in RDBStore if RDBStore's initialization fails after RocksDB instance creation (apache#4692) ...
3 participants
What changes were proposed in this pull request?
With HDDS-7226 merged, the RackAwarePlacementPolicy now supports passing used nodes and excluded nodes separately. This allows it to select the racks correctly when there are one or two used nodes.
We need to change the Ratis under and mis-replication handlers to pass the used nodes and excluded nodes as two separate lists now.
Also the EC under-replication handler should be fixed to pass used nodes separately too.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-8504
How was this patch tested?
New unit and changed tests