From 3742ef855dda93bd6e7f1de99bad252a09d857ac Mon Sep 17 00:00:00 2001 From: zhangkangwei Date: Fri, 20 Sep 2024 15:52:09 +0800 Subject: [PATCH 1/6] HDDS-11477. [doc] Add configuration description for datanode docs --- hadoop-hdds/docs/content/concept/Datanodes.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/hadoop-hdds/docs/content/concept/Datanodes.md b/hadoop-hdds/docs/content/concept/Datanodes.md index 47c09a798fc4..a1570aa40f9d 100644 --- a/hadoop-hdds/docs/content/concept/Datanodes.md +++ b/hadoop-hdds/docs/content/concept/Datanodes.md @@ -76,3 +76,15 @@ blocks that get reported. That is a 40x reduction in the block reports. This extra indirection helps tremendously with scaling Ozone. SCM has far less block data to process and the namespace service (Ozone Manager) as a different service are critical to scaling Ozone. + + +## Notable configurations + +key | default |
description
+----|---------|------------ +dfs.container.ratis.datanode.storage.dir | none | This directory is used for storing Ratis metadata like logs. +ozone.scm.datanode.id.dir | none | The path that datanodes will use to store the datanode ID. +hdds.datanode.dir | none | Determines where on the local filesystem HDDS data will be stored. +hdds.datanode.dir.du.reserved | none | Reserved space in bytes per volume. Always leave this much space free for non dfs use. +ozone.metadata.dirs | none | Directory to store persisted data (RocksDB). +ozone.recon.address | 0.0.0.0:9891 | RPC address of the Recon. From d5f813a013c9babc86d2f9d4bffee199ac388c6c Mon Sep 17 00:00:00 2001 From: zhangkangwei Date: Fri, 20 Sep 2024 17:13:04 +0800 Subject: [PATCH 2/6] HDDS-11477. [doc] Add Recon description for Datanode docs. --- hadoop-hdds/docs/content/concept/Datanodes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hadoop-hdds/docs/content/concept/Datanodes.md b/hadoop-hdds/docs/content/concept/Datanodes.md index a1570aa40f9d..3d472731d211 100644 --- a/hadoop-hdds/docs/content/concept/Datanodes.md +++ b/hadoop-hdds/docs/content/concept/Datanodes.md @@ -87,4 +87,4 @@ ozone.scm.datanode.id.dir | none | The path that datanodes will use to store the hdds.datanode.dir | none | Determines where on the local filesystem HDDS data will be stored. hdds.datanode.dir.du.reserved | none | Reserved space in bytes per volume. Always leave this much space free for non dfs use. ozone.metadata.dirs | none | Directory to store persisted data (RocksDB). -ozone.recon.address | 0.0.0.0:9891 | RPC address of the Recon. +ozone.recon.address | 0.0.0.0:9891 | RPC address of the Recon. Use to connect Recon. From 63ef1bd6846d2f27342f466374eae9222851823e Mon Sep 17 00:00:00 2001 From: zhangkangwei Date: Fri, 20 Sep 2024 17:35:35 +0800 Subject: [PATCH 3/6] HDDS-11477. [doc] Add zh for Datanode docs. --- hadoop-hdds/docs/content/concept/Datanodes.zh.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/hadoop-hdds/docs/content/concept/Datanodes.zh.md b/hadoop-hdds/docs/content/concept/Datanodes.zh.md index 8f129df7b9be..abbf581ea562 100644 --- a/hadoop-hdds/docs/content/concept/Datanodes.zh.md +++ b/hadoop-hdds/docs/content/concept/Datanodes.zh.md @@ -49,3 +49,16 @@ Ozone 的存储容器是一个自包含的超级块，容器中包含一系列 SCM 如何获得容器的位置？这一点和现有的 HDFS 十分相似。数据节点会定期发送类似于块报告的容器报告，容器报告比块报告的内容简洁的多，比如，对于一个存储容量为 196 TB 的集群，Ozone 大概会拥有四万个容器，相比于 HDFS 的一百五十万个块，块报告数量缩减为四十分之一。 这种间接管理的方式大大地提高了 Ozone 的扩展性，因为 SCM 需要处理的块数据大大减少，且命名服务（OM）作为一个独特的服务主体对于扩展 Ozone 具有重要意义。 + + + +## 需要关注的配置项 + +配置项 |默认值 |
描述
+----|---------|------------ +dfs.container.ratis.datanode.storage.dir | none | 该目录用于存储 Ratis 元数据，如日志。 +ozone.scm.datanode.id.dir | none | 数据节点上用于存储数据节点 ID 的路径。 +hdds.datanode.dir | none | 此配置决定数据节点上的数据将存储在本地文件系统的哪个位置。 +hdds.datanode.dir.du.reserved | none | 每个卷保留的存储空间（以字节为单位）。始终为非DFS用途保留这么多空闲空间。 +ozone.metadata.dirs | none | 用于存储持久化数据（RocksDB）的目录。 +ozone.recon.address | 0.0.0.0:9891 | Recon的RPC地址. 使用 连接到Recon. \ No newline at end of file From 3dc774dac96081d93c8d372198e99cd1ca814589 Mon Sep 17 00:00:00 2001 From: zhangkangwei Date: Mon, 23 Sep 2024 11:09:35 +0800 Subject: [PATCH 4/6] HDDS-11477. [doc] update zh to fix punctuation. --- hadoop-hdds/docs/content/concept/Datanodes.zh.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hadoop-hdds/docs/content/concept/Datanodes.zh.md b/hadoop-hdds/docs/content/concept/Datanodes.zh.md index abbf581ea562..3010a6bd009e 100644 --- a/hadoop-hdds/docs/content/concept/Datanodes.zh.md +++ b/hadoop-hdds/docs/content/concept/Datanodes.zh.md @@ -61,4 +61,4 @@ ozone.scm.datanode.id.dir | none | 数据节点上用于存储数据节点 ID hdds.datanode.dir | none | 此配置决定数据节点上的数据将存储在本地文件系统的哪个位置。 hdds.datanode.dir.du.reserved | none | 每个卷保留的存储空间（以字节为单位）。始终为非DFS用途保留这么多空闲空间。 ozone.metadata.dirs | none | 用于存储持久化数据（RocksDB）的目录。 -ozone.recon.address | 0.0.0.0:9891 | Recon的RPC地址. 使用 连接到Recon. \ No newline at end of file +ozone.recon.address | 0.0.0.0:9891 | Recon的RPC地址. 使用 连接到Recon。 \ No newline at end of file From f007846218fb5644dd55c677ff25334d3e00649a Mon Sep 17 00:00:00 2001 From: zhangkangwei Date: Mon, 23 Sep 2024 11:22:25 +0800 Subject: [PATCH 5/6] HDDS-11477. [doc] Update zh to fix punctuation again. --- hadoop-hdds/docs/content/concept/Datanodes.zh.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/hadoop-hdds/docs/content/concept/Datanodes.zh.md b/hadoop-hdds/docs/content/concept/Datanodes.zh.md index 3010a6bd009e..32071c9e51e4 100644 --- a/hadoop-hdds/docs/content/concept/Datanodes.zh.md +++ b/hadoop-hdds/docs/content/concept/Datanodes.zh.md @@ -51,7 +51,6 @@ SCM 如何获得容器的位置？这一点和现有的 HDFS 十分相似。数 这种间接管理的方式大大地提高了 Ozone 的扩展性，因为 SCM 需要处理的块数据大大减少，且命名服务（OM）作为一个独特的服务主体对于扩展 Ozone 具有重要意义。 - ## 需要关注的配置项 配置项 |默认值 |
描述
@@ -61,4 +60,4 @@ ozone.scm.datanode.id.dir | none | 数据节点上用于存储数据节点 ID hdds.datanode.dir | none | 此配置决定数据节点上的数据将存储在本地文件系统的哪个位置。 hdds.datanode.dir.du.reserved | none | 每个卷保留的存储空间（以字节为单位）。始终为非DFS用途保留这么多空闲空间。 ozone.metadata.dirs | none | 用于存储持久化数据（RocksDB）的目录。 -ozone.recon.address | 0.0.0.0:9891 | Recon的RPC地址. 使用 连接到Recon。 \ No newline at end of file +ozone.recon.address | 0.0.0.0:9891 | Recon的RPC地址。 使用 连接到Recon。 \ No newline at end of file From d6aaeee98a1535fc818815bce7a835848ad4e301 Mon Sep 17 00:00:00 2001 From: zhangkangwei Date: Mon, 23 Sep 2024 16:57:52 +0800 Subject: [PATCH 6/6] HDDS-11477. [doc] Update datanodes.md to hdds.datanode.dir. --- hadoop-hdds/docs/content/concept/Datanodes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hadoop-hdds/docs/content/concept/Datanodes.md b/hadoop-hdds/docs/content/concept/Datanodes.md index 3d472731d211..cf246712f68f 100644 --- a/hadoop-hdds/docs/content/concept/Datanodes.md +++ b/hadoop-hdds/docs/content/concept/Datanodes.md @@ -84,7 +84,7 @@ key | default |
description
----|---------|------------ dfs.container.ratis.datanode.storage.dir | none | This directory is used for storing Ratis metadata like logs. ozone.scm.datanode.id.dir | none | The path that datanodes will use to store the datanode ID. -hdds.datanode.dir | none | Determines where on the local filesystem HDDS data will be stored. +hdds.datanode.dir | none | Determines where HDDS data will be stored on the local filesystem. hdds.datanode.dir.du.reserved | none | Reserved space in bytes per volume. Always leave this much space free for non dfs use. ozone.metadata.dirs | none | Directory to store persisted data (RocksDB). ozone.recon.address | 0.0.0.0:9891 | RPC address of the Recon. Use to connect Recon.