Skip to content

HDDS-12915. Intermittent failure in testCreatePipelineThrowErrorWithDataNodeLimit #8359

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
adoroszlai merged 1 commit into from
Apr 30, 2025
Merged

HDDS-12915. Intermittent failure in testCreatePipelineThrowErrorWithDataNodeLimit #8359

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 10 additions & 8 deletions ...rver-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestRatisPipelineProvider.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,6 @@
import org.apache.hadoop.hdds.utils.db.DBStore;
import org.apache.hadoop.hdds.utils.db.DBStoreBuilder;
import org.apache.hadoop.ozone.ClientVersion;
import org.apache.ozone.test.tag.Flaky;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Assumptions;
import org.junit.jupiter.api.Test;
Expand All @@ -82,6 +81,7 @@ public class TestRatisPipelineProvider {
@TempDir
private File testDir;
private DBStore dbStore;
private int nodeCount = 10;

public void init(int maxPipelinePerNode) throws Exception {
init(maxPipelinePerNode, new OzoneConfiguration());
Expand All @@ -95,7 +95,7 @@ public void init(int maxPipelinePerNode, OzoneConfiguration conf)
public void init(int maxPipelinePerNode, OzoneConfiguration conf, File dir) throws Exception {
conf.set(HddsConfigKeys.OZONE_METADATA_DIRS, dir.getAbsolutePath());
dbStore = DBStoreBuilder.createDBStore(conf, SCMDBDefinition.get());
nodeManager = new MockNodeManager(true, 10);
nodeManager = new MockNodeManager(true, nodeCount);
nodeManager.setNumPipelinePerDatanode(maxPipelinePerNode);
SCMHAManager scmhaManager = SCMHAManagerStub.getInstance(true);
conf.setInt(ScmConfigKeys.OZONE_DATANODE_PIPELINE_LIMIT,
Expand All @@ -112,6 +112,7 @@ public void init(int maxPipelinePerNode, OzoneConfiguration conf, File dir) thro

@AfterEach
void cleanup() throws Exception {
nodeCount = 10;
if (dbStore != null) {
dbStore.close();
}
Expand Down Expand Up @@ -167,9 +168,9 @@ public void testCreatePipelineWithFactorOne() throws Exception {
createPipelineAndAssertions(HddsProtos.ReplicationFactor.ONE);
}

private List<DatanodeDetails> createListOfNodes(int nodeCount) {
private List<DatanodeDetails> createListOfNodes(int count) {
List<DatanodeDetails> nodes = new ArrayList<>();
for (int i = 0; i < nodeCount; i++) {
for (int i = 0; i < count; i++) {
nodes.add(MockDatanodeDetails.randomDatanodeDetails());
}
return nodes;
Expand Down Expand Up @@ -365,12 +366,13 @@ public void testCreatePipelinesWhenNotEnoughSpace(@TempDir File tempDir) throws
}

@ParameterizedTest
@CsvSource({ "1, 2", "2, 5" })
@Flaky("HDDS-12915")
@CsvSource({ "1, 3", "2, 6"})
public void testCreatePipelineThrowErrorWithDataNodeLimit(int limit, int pipelineCount) throws Exception {
init(limit);
// increasing node count to avoid intermittent failures due to unhealthy nodes.
nodeCount = 13;
Copy link
Contributor

@ashishkumar50 ashishkumar50 Apr 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why 13 is required and why sometime 10 is working?

Copy link
Contributor Author

@priyeshkaratha priyeshkaratha Apr 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the mocked setup, we have 10 nodes in total, out of which only 8 are healthy. When the maximum pipeline limit per node is set to 2, ideally, we expect a maximum of 5 pipelines to be allocated. However, due to the filtering logic considering multiple conditions (such as node health, pipeline limits, and other constraints), the allocation can sometimes fail.

To consistently ensure that at least 9 healthy nodes are available for pipeline creation, the nodeCount should be set to 13. This ensures a sufficient pool of healthy nodes even after filtering, allowing the expected pipeline allocation to succeed reliably.

init(limit, new OzoneConfiguration(), testDir);

// Create pipelines up to the limit (2 for limit=1, 5 for limit=2).
// Create pipelines up to the limit (3 for limit=1, 6 for limit=2).
for (int i = 0; i < pipelineCount; i++) {
stateManager.addPipeline(
provider.create(RatisReplicationConfig.getInstance(ReplicationFactor.THREE),
Expand Down