Skip to content

HDFS-17254. DataNode httpServer has too many worker threads #6307

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
2005hithlj wants to merge 1 commit into from
Closed

HDFS-17254. DataNode httpServer has too many worker threads #6307

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -975,6 +975,9 @@ public class DFSConfigKeys extends CommonConfigurationKeys {
public static final String DFS_DATANODE_HTTP_ADDRESS_DEFAULT = "0.0.0.0:" + DFS_DATANODE_HTTP_DEFAULT_PORT;
public static final String DFS_DATANODE_HTTP_INTERNAL_PROXY_PORT =
"dfs.datanode.http.internal-proxy.port";
public static final String DFS_DATANODE_NETTY_WORKER_NUM_THREADS_KEY =
"dfs.datanode.netty.worker.threads";
public static final int DFS_DATANODE_NETTY_WORKER_NUM_THREADS_DEFAULT = 0;
Copy link
Contributor

@slfan1989 slfan1989 Nov 30, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution, but is it reasonable to default to 0 for the number of threads?

Copy link
Contributor Author

@2005hithlj 2005hithlj Dec 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@slfan1989 Thank you for your review, the default value should not continue to be 0, and I plan to change the default value to 10. What do you think, sir?

Copy link
Contributor Author

@2005hithlj 2005hithlj Dec 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@slfan1989 I have changed the default value to 10, and it works normally in our HDFS cluster.

public static final String DFS_DATANODE_MAX_RECEIVER_THREADS_KEY =
HdfsClientConfigKeys.DeprecatedKeys.DFS_DATANODE_MAX_RECEIVER_THREADS_KEY;
public static final int DFS_DATANODE_MAX_RECEIVER_THREADS_DEFAULT = 4096;
Expand Down
13 changes: 12 additions & 1 deletion ...oop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,12 +65,15 @@
import java.security.GeneralSecurityException;
import java.util.Enumeration;
import java.util.Map;
import java.util.concurrent.Executors;

import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_ADMIN;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_HTTPS_ADDRESS_DEFAULT;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_HTTPS_ADDRESS_KEY;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_HTTP_ADDRESS_KEY;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_HTTP_INTERNAL_PROXY_PORT;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_NETTY_WORKER_NUM_THREADS_KEY;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_NETTY_WORKER_NUM_THREADS_DEFAULT;
Copy link
Contributor

@tomscut tomscut Dec 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add this configuration into hdfs-default.xml. Thanks.


/**
* Data node HTTP Server Class.
Expand Down Expand Up @@ -144,7 +147,15 @@ public DatanodeHttpServer(final Configuration conf,
confForCreate.set(FsPermission.UMASK_LABEL, "000");

this.bossGroup = new NioEventLoopGroup();
this.workerGroup = new NioEventLoopGroup();
int workerCount = conf.getInt(DFS_DATANODE_NETTY_WORKER_NUM_THREADS_KEY,
DFS_DATANODE_NETTY_WORKER_NUM_THREADS_DEFAULT);
if (workerCount < 0) {
LOG.warn("The value of {} is less than 0, will use default value: {}",
DFS_DATANODE_NETTY_WORKER_NUM_THREADS_KEY, DFS_DATANODE_NETTY_WORKER_NUM_THREADS_DEFAULT);
workerCount = DFS_DATANODE_NETTY_WORKER_NUM_THREADS_DEFAULT;
}
this.workerGroup = new NioEventLoopGroup(workerCount, Executors.newCachedThreadPool());

this.externalHttpChannel = externalHttpChannel;
HttpConfig.Policy policy = DFSUtil.getHttpPolicy(conf);
final ChannelHandler[] handlers = getFilterHandlers(conf);
Expand Down
8 changes: 8 additions & 0 deletions hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,14 @@
</description>
</property>

<property>
<name>dfs.datanode.netty.worker.threads</name>
<value>0</value>
<description>
The number of Datanode http server worker threads.
</description>
</property>

<property>
<name>dfs.datanode.handler.count</name>
<value>10</value>
Expand Down