HDDS-1569 Support creating multiple pipelines with same datanode #28
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ChenSammi
reviewed
Oct 16, 2019
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java
Outdated
Show resolved
Hide resolved
...g/apache/hadoop/hdds/scm/container/placement/algorithms/ContainerPlacementPolicyFactory.java
Outdated
Show resolved
Hide resolved
...hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineActionHandler.java
Outdated
Show resolved
Hide resolved
...op-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
Outdated
Show resolved
Hide resolved
...hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java
Outdated
Show resolved
Hide resolved
...hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java
Outdated
Show resolved
Hide resolved
...est/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestRatisPipelineCreateAndDestroy.java
Outdated
Show resolved
Hide resolved
...est/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestRatisPipelineCreateAndDestroy.java
Outdated
Show resolved
Hide resolved
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneCluster.java
Outdated
Show resolved
Hide resolved
timmylicheng
force-pushed
the
HDDS-1569
branch
from
48f46d3 to
fa8fa7b
Compare
Contributor
Author
|
/retest |
timmylicheng
force-pushed
the
HDDS-1569
branch
from
fa8fa7b to
b5c8818
Compare
Contributor
Author
|
/retest |
timmylicheng
force-pushed
the
HDDS-1569
branch
from
b5c8818 to
005588d
Compare
Contributor
Author
|
/retest |
timmylicheng
force-pushed
the
HDDS-1569
branch
2 times, most recently
from
7dc00e8 to
dffbd1b
Compare
Contributor
Author
|
/retest |
xiaoyuyao
reviewed
Oct 22, 2019
Contributor
There was a problem hiding this comment.
why do we need to add the dn->pipelineId in 75 since line 73 already add it?
Contributor
Author
There was a problem hiding this comment.
Line 73 is adding v to new keyset when key is absent and line 75 is adding v to the existing keyset. Reason why we don't have to have extra if here is computeIfAbsent and computeIfPresent are both judging if key is present or not. So we don't have to see if Map.get(k) is null.
Contributor
There was a problem hiding this comment.
Correct me if I'm wrong, dn2ObjectMap.computeIfAbsent(dnId, k -> ConcurrentHashMap.newKeySet()) ensures for a dnId we will have a KeySet().
Then the returned value (KeySet) allows line 73 to continue the operation to add pipeline id with a .add(pipeline.getId).
JDK document can be found here: https://docs.oracle.com/javase/8/docs/api/java/util/Map.html#computeIfAbsent-K-java.util.function.Function-
Contributor
Author
There was a problem hiding this comment.
Yea. Line 75 is for when we already one existing keyset. For multi pipeline case, the keyset may be created by previous pipeline allocation and the new one would just be added into the existing keyset.
Contributor
There was a problem hiding this comment.
The document says existing value(keyset) will be returned. So the line 74-75 is not necessary.
Returns:
the current (existing or computed) value associated with the specified key,
A good example can be found here: https://stackoverflow.com/a/39268896
Contributor
Author
There was a problem hiding this comment.
I see. Thanks for the tipping. Based on the example here, Shall I update the removePipeline method? I'm making a change and please take a look.
xiaoyuyao
reviewed
Oct 22, 2019
xiaoyuyao
reviewed
Oct 22, 2019
...ds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelinePlacementPolicy.java
Outdated
Show resolved
Hide resolved
xiaoyuyao
reviewed
Oct 23, 2019
...op-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
Outdated
Show resolved
Hide resolved
xiaoyuyao
reviewed
Oct 23, 2019
...op-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
Outdated
Show resolved
Hide resolved
xiaoyuyao
reviewed
Oct 23, 2019
...op-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
Outdated
Show resolved
Hide resolved
xiaoyuyao
reviewed
Oct 23, 2019
...op-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
Outdated
Show resolved
Hide resolved
xiaoyuyao
reviewed
Oct 23, 2019
Contributor
xiaoyuyao
left a comment
xiaoyuyao
left a comment
There was a problem hiding this comment.
LGTM overall. A few comments added inline...
xiaoyuyao
reviewed
Oct 23, 2019
...est/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestRatisPipelineCreateAndDestroy.java
Outdated
Show resolved
Hide resolved
xiaoyuyao
reviewed
Oct 23, 2019
...tegration-test/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestSCMPipelineManager.java
Outdated
Show resolved
Hide resolved
xiaoyuyao
reviewed
Oct 23, 2019
...test/src/test/java/org/apache/hadoop/hdds/scm/safemode/TestSCMSafeModeWithPipelineRules.java
Outdated
Show resolved
Hide resolved
xiaoyuyao
reviewed
Oct 23, 2019
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneClusterImpl.java
Outdated
Show resolved
Hide resolved
xiaoyuyao
reviewed
Oct 23, 2019
...-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestContainerReplicationEndToEnd.java
Outdated
Show resolved
Hide resolved
xiaoyuyao
reviewed
Oct 23, 2019
...ne/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java
Outdated
Show resolved
Hide resolved
xiaoyuyao
reviewed
Oct 23, 2019
.../hadoop/ozone/container/common/statemachine/commandhandler/TestCloseContainerByPipeline.java
Outdated
Show resolved
Hide resolved
xiaoyuyao
reviewed
Oct 23, 2019
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneCluster.java
Outdated
Show resolved
Hide resolved
timmylicheng
force-pushed
the
HDDS-1569
branch
2 times, most recently
from
bb51565 to
8600ad9
Compare
timmylicheng
force-pushed
the
HDDS-1569
branch
from
8600ad9 to
36cf803
Compare
Contributor
|
+1, pending CI. Thanks for the update @timmylicheng |
ChenSammi
pushed a commit
that referenced
this pull request
Nov 4, 2019
…tributed by Li Cheng. This closes #28
timmylicheng
added a commit
to timmylicheng/hadoop-ozone
that referenced
this pull request
Nov 25, 2019
…tributed by Li Cheng. This closes apache#28
ChenSammi
pushed a commit
that referenced
this pull request
Dec 3, 2019
…tributed by Li Cheng. This closes #28
timmylicheng
added a commit
to timmylicheng/hadoop-ozone
that referenced
this pull request
Dec 23, 2019
…tributed by Li Cheng. This closes apache#28
timmylicheng
added a commit
to timmylicheng/hadoop-ozone
that referenced
this pull request
Jan 7, 2020
…tributed by Li Cheng. This closes apache#28
timmylicheng
added a commit
to timmylicheng/hadoop-ozone
that referenced
this pull request
Feb 10, 2020
…tributed by Li Cheng. This closes apache#28
timmylicheng
added a commit
to timmylicheng/hadoop-ozone
that referenced
this pull request
Feb 12, 2020
…tributed by Li Cheng. This closes apache#28
anuengineer
pushed a commit
that referenced
this pull request
Feb 19, 2020
* HDDS-1577. Add default pipeline placement policy implementation. (#1366) (cherry picked from commit b640a5f6d53830aee4b9c2a7d17bf57c987962cd) * HDDS-1571. Create an interface for pipeline placement policy to support network topologies. (#1395) (cherry picked from commit 753fc6703a39154ed6013e44dbae572391748906) * HDDS-2089: Add createPipeline CLI. (#1418) (cherry picked from commit 326b5acd4a63fe46821919322867f5daff30750c) * HDDS-1569 Support creating multiple pipelines with same datanode. Contributed by Li Cheng. This closes #28 * HDDS-1572 Implement a Pipeline scrubber to clean up non-OPEN pipeline. (#237) * Rebase Fix * HDDS-2650 Fix createPipeline CLI. (#340) * HDDS-2035 Implement datanode level CLI to reveal pipeline relation. (#348) * Revert "HDDS-2650 Fix createPipeline CLI. (#340)" This reverts commit 7c71710. * HDDS-2650 Fix createPipeline CLI and make it message based. (#370) * HDDS-1574 Average out pipeline allocation on datanodes and add metrcs/test (#291) * Resolve rebase conflict. * HDDS-2756. Handle pipeline creation failure in different way when it exceeds pipeline limit Closes #401 * HDDS-2115 Add acceptance test for createPipeline CLI and datanode list CLI (#375) * HDDS-2115 Add acceptance test for createPipeline CLI and datanode list CLI. * HDDS-2772 Better management for pipeline creation limitation. (#410) * HDDS-2913 Update config names and CLI for multi-raft feature. (#462) * HDDS-2924. Fix Pipeline#nodeIdsHash collision issue. (#478) * HDDS-2923 Add fall-back protection for rack awareness in pipeline creation. (#516) * HDDS-3007 Fix CI test failure for TestSCMNodeManager. (#550) Co-authored-by: Sammi Chen <[email protected]> Co-authored-by: Xiaoyu Yao <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
NOTICE
https://issues.apache.org/jira/browse/HDDS-1569
related PR: apache/hadoop#1431
Changes: