Skip to content

HDDS-12378. Change default hdds.scm.safemode.min.datanode to 3 #8331

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nandakumar131 merged 7 commits into from
May 21, 2025
Merged

HDDS-12378. Change default hdds.scm.safemode.min.datanode to 3 #8331

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
96d71a8
Change default hdds.scm.safemode.min.datanode to 3
kostacie Apr 24, 2025
d23c395
Add warn log, fix doc
kostacie Apr 25, 2025
f4b3218
Fix doc
kostacie Apr 25, 2025
0be4df6
Fix tests and config
kostacie Apr 28, 2025
f683f37
Delete warn message, revert changes
kostacie May 21, 2025
3efcf26
Clean up
kostacie May 21, 2025
e53e08d
Delete unused test, revert changes
kostacie May 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ public final class HddsConfigKeys {
public static final boolean HDDS_SCM_SAFEMODE_ENABLED_DEFAULT = true;
public static final String HDDS_SCM_SAFEMODE_MIN_DATANODE =
"hdds.scm.safemode.min.datanode";
public static final int HDDS_SCM_SAFEMODE_MIN_DATANODE_DEFAULT = 1;
public static final int HDDS_SCM_SAFEMODE_MIN_DATANODE_DEFAULT = 3;

public static final String
HDDS_SCM_WAIT_TIME_AFTER_SAFE_MODE_EXIT =
Expand Down
2 changes: 1 addition & 1 deletion hadoop-hdds/common/src/main/resources/ozone-default.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1637,7 +1637,7 @@

<property>
<name>hdds.scm.safemode.min.datanode</name>
<value>1</value>
<value>3</value>
<tag>HDDS,SCM,OPERATION</tag>
<description>Minimum DataNodes which should be registered to get SCM out of
safe mode.
Expand Down
4 changes: 2 additions & 2 deletions hadoop-hdds/docs/content/concept/StorageContainerManager.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,6 @@ key | default | description
----|---------|------------
ozone.scm.container.size | 5GB | Default container size used by Ozone
ozone.scm.block.size | 256MB | The default size of a data block.
hdds.scm.safemode.min.datanode | 1 | Minimum number of datanodes to start the real work.
hdds.scm.safemode.min.datanode | 3 | Minimum number of datanodes to start the real work.

@myskov myskov Apr 25, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's also a Chinese version of the doc. Could you please make changes there too?

@kostacie kostacie Apr 25, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, sure.

ozone.scm.http-address | 0.0.0.0:9876 | HTTP address of the SCM server
ozone.metadata.dirs | none | Directory to store persisted data (RocksDB).
ozone.metadata.dirs | none | Directory to store persisted data (RocksDB).

@myskov myskov Apr 25, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please revert irrelevant change

@kostacie kostacie Apr 25, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For some reason I can't revert that as it changes itself and adds an extra line.

2 changes: 1 addition & 1 deletion hadoop-hdds/docs/content/concept/StorageContainerManager.zh.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,6 @@ SCM 负责创建 Ozone 集群。当通过 `init` 命令启动 SCM 时，SCM 将
----|---------|------------
ozone.scm.container.size | 5GB | Ozone 使用的默认容器的大小
ozone.scm.block.size | 256MB | 数据块的默认大小
hdds.scm.safemode.min.datanode | 1 | 能够启动实际工作所需的最小数据节点数
hdds.scm.safemode.min.datanode | 3 | 能够启动实际工作所需的最小数据节点数
ozone.scm.http-address | 0.0.0.0:9876 | SCM 服务端使用的 HTTP 地址
ozone.metadata.dirs | none | 存储持久化数据的目录（RocksDB）
15 changes: 14 additions & 1 deletion ...ds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/DataNodeSafeModeRule.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@
import org.apache.hadoop.hdds.scm.server.SCMDatanodeProtocolServer.NodeRegistrationContainerReport;
import org.apache.hadoop.hdds.server.events.EventQueue;
import org.apache.hadoop.hdds.server.events.TypedEvent;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

/**
* Class defining Safe mode exit criteria according to number of DataNodes
Expand All @@ -36,6 +38,7 @@ public class DataNodeSafeModeRule extends
SafeModeExitRule<NodeRegistrationContainerReport> {

private static final String NAME = "DataNodeSafeModeRule";
private static final Logger LOG = LoggerFactory.getLogger(DataNodeSafeModeRule.class);

// Min DataNodes required to exit safe mode.
private int requiredDns;
Expand Down Expand Up @@ -66,7 +69,17 @@ protected boolean validate() {
if (validateBasedOnReportProcessing()) {
return registeredDns >= requiredDns;
}
return nodeManager.getNodes(NodeStatus.inServiceHealthy()).size() >= requiredDns;

int healthyCount = nodeManager.getNodes(NodeStatus.inServiceHealthy()).size();
int healthyReadOnlyCount = nodeManager.getNodes(NodeStatus.inServiceHealthyReadOnly()).size();
int staleCount = nodeManager.getNodes(NodeStatus.inServiceStale()).size();

if (healthyCount + healthyReadOnlyCount + staleCount == 1) {
LOG.warn("Only one Datanode is available in the cluster. " +
"Consider setting 'hdds.scm.safemode.min.datanode=1' in the configuration.");
}

@nandakumar131 nandakumar131 Apr 30, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SCM doesn't remember Datanode list across SCM restart. So during SCM restart, initially the datanode list in SCM will be empty and at some point the first Datanode will register making (healthyCount + healthyReadOnlyCount + staleCount == 1) to true and print the warn log message.

We will end up printing this warn log message during every SCM start-up, which is incorrect.

@kostacie kostacie May 5, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the review @nandakumar131. Should we add a flag that would indicate that the log message has already been printed? Would it be a good solution?
Or would you recommend anything to avoid such problem?

@adoroszlai adoroszlai May 20, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we can remove the warning, unless @nandakumar131 has some other suggestion.

@nandakumar131 nandakumar131 May 21, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need any logging for this.


return healthyCount >= requiredDns;
}

@Override
Expand Down
21 changes: 21 additions & 0 deletions ...erver-scm/src/test/java/org/apache/hadoop/hdds/scm/safemode/TestDataNodeSafeModeRule.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,8 @@ public void testDataNodeSafeModeRuleWithNodeManager() throws Exception {
rule.setValidateBasedOnReportProcessing(false);

when(nodeManager.getNodes(NodeStatus.inServiceHealthy())).thenReturn(new ArrayList<>());
when(nodeManager.getNodes(NodeStatus.inServiceHealthyReadOnly())).thenReturn(new ArrayList<>());
when(nodeManager.getNodes(NodeStatus.inServiceStale())).thenReturn(new ArrayList<>());

assertFalse(rule.validate());

Expand All @@ -169,4 +171,23 @@ public void testDataNodeSafeModeRuleWithNodeManager() throws Exception {

verify(nodeManager, times(2)).getNodes(NodeStatus.inServiceHealthy());
}

@Test
public void testDataNodeSafeModeRuleWithSingleDataNode() throws Exception {
int requiredDns = 2;
setup(requiredDns);

rule.setValidateBasedOnReportProcessing(false);

List<DatanodeDetails> singleHealthy = new ArrayList<>();
singleHealthy.add(MockDatanodeDetails.randomDatanodeDetails());

when(nodeManager.getNodes(NodeStatus.inServiceHealthy())).thenReturn(singleHealthy);
when(nodeManager.getNodes(NodeStatus.inServiceHealthyReadOnly())).thenReturn(new ArrayList<>());
when(nodeManager.getNodes(NodeStatus.inServiceStale())).thenReturn(new ArrayList<>());

assertFalse(rule.validate());

verify(nodeManager, times(1)).getNodes(NodeStatus.inServiceHealthy());
}
}
1 change: 1 addition & 0 deletions ...cm/src/test/java/org/apache/hadoop/hdds/scm/safemode/TestHealthyPipelineSafeModeRule.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ public void testHealthyPipelineSafeModeRuleWithNoPipelines()
ContainerManager containerManager = mock(ContainerManager.class);
when(containerManager.getContainers()).thenReturn(containers);
config.set(HddsConfigKeys.OZONE_METADATA_DIRS, tempFile.getPath());
config.setInt(HddsConfigKeys.HDDS_SCM_SAFEMODE_MIN_DATANODE, 0);
// enable pipeline check
config.setBoolean(
HddsConfigKeys.HDDS_SCM_SAFEMODE_PIPELINE_CREATION, false);
Expand Down
1 change: 1 addition & 0 deletions .../server-scm/src/test/java/org/apache/hadoop/hdds/scm/safemode/TestSCMSafeModeManager.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,7 @@ public void setUp() throws IOException {
config.setBoolean(HddsConfigKeys.HDDS_SCM_SAFEMODE_PIPELINE_CREATION,
false);
config.set(HddsConfigKeys.OZONE_METADATA_DIRS, tempDir.getAbsolutePath());
config.setInt(HddsConfigKeys.HDDS_SCM_SAFEMODE_MIN_DATANODE, 1);
scmMetadataStore = new SCMMetadataStoreImpl(config);
}

Expand Down
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/compose/ozonescripts/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ OZONE-SITE.XML_ozone.scm.block.client.address=scm
OZONE-SITE.XML_ozone.metadata.dirs=/data/metadata
OZONE-SITE.XML_ozone.scm.client.address=scm
OZONE-SITE.XML_ozone.server.default.replication=1
OZONE-SITE.XML_hdds.scm.safemode.min.datanode=1
OZONE-SITE.XML_hdds.datanode.dir=/data/hdds
OZONE-SITE.XML_hdds.datanode.volume.min.free.space=100MB
OZONE-SITE.XML_hdds.scmclient.max.retry.timeout=30s
Expand Down