Skip to content

HDFS-16504. Add parameter for NameNode to process getBloks request #4068

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Neilxzn wants to merge 5 commits into from
Closed

HDFS-16504. Add parameter for NameNode to process getBloks request #4068

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f69a1c8
add `dfs.namenode.get-blocks.check.operation` to enable or disable ch…
Mar 14, 2022
aff3d57
unify code
Mar 14, 2022
10821d0
fix `dfs.namenode.get-blocks.check.operation`'s deecription and Unify…
Mar 17, 2022
92b0436
fix unit test
Mar 18, 2022
54b4ff8
fix unit test 2
Mar 18, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -746,6 +746,10 @@ public class DFSConfigKeys extends CommonConfigurationKeys {
*/
public static final String DFS_NAMENODE_GETBLOCKS_MAX_QPS_KEY = "dfs.namenode.get-blocks.max-qps";
public static final int DFS_NAMENODE_GETBLOCKS_MAX_QPS_DEFAULT = 20;
public static final String DFS_NAMENODE_GETBLOCKS_CHECK_OPERATION_KEY
= "dfs.namenode.get-blocks.check.operation";
public static final boolean DFS_NAMENODE_GETBLOCKS_CHECK_OPERATION_DEFAULT
= true;

public static final String DFS_BALANCER_MOVEDWINWIDTH_KEY = "dfs.balancer.movedWinWidth";
public static final long DFS_BALANCER_MOVEDWINWIDTH_DEFAULT = 5400*1000L;
Expand Down
17 changes: 15 additions & 2 deletions ...roject/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -505,6 +505,12 @@ private boolean isClientPortInfoAbsent(CallerContext ctx){
private final int snapshotDiffReportLimit;
private final int blockDeletionIncrement;

/**
* Whether enable checkOperation when call getBlocks.
* It is enabled by default.
*/
private final boolean isGetBlocksCheckOperationEnabled;

/** Interval between each check of lease to release. */
private final long leaseRecheckIntervalMs;
/** Maximum time the lock is hold to release lease. */
Expand Down Expand Up @@ -1066,6 +1072,10 @@ static FSNamesystem loadFromDisk(Configuration conf) throws IOException {
Preconditions.checkArgument(blockDeletionIncrement > 0,
DFSConfigKeys.DFS_NAMENODE_BLOCK_DELETION_INCREMENT_KEY +
" must be a positive integer.");
this.isGetBlocksCheckOperationEnabled = conf.getBoolean(
DFSConfigKeys.DFS_NAMENODE_GETBLOCKS_CHECK_OPERATION_KEY,
DFSConfigKeys.DFS_NAMENODE_GETBLOCKS_CHECK_OPERATION_DEFAULT);

} catch(IOException e) {
LOG.error(getClass().getSimpleName() + " initialization failed.", e);
close();
Expand Down Expand Up @@ -1938,10 +1948,13 @@ public boolean isInStandbyState() {
*/
public BlocksWithLocations getBlocks(DatanodeID datanode, long size, long
minimumBlockSize, long timeInterval) throws IOException {
checkOperation(OperationCategory.READ);
OperationCategory checkOp =
isGetBlocksCheckOperationEnabled ? OperationCategory.READ :
OperationCategory.UNCHECKED;
checkOperation(checkOp);
readLock();
try {
checkOperation(OperationCategory.READ);
checkOperation(checkOp);
return getBlockManager().getBlocksWithLocations(datanode, size,
minimumBlockSize, timeInterval);
} finally {
Expand Down
8 changes: 8 additions & 0 deletions hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4119,6 +4119,14 @@
Mover, and StoragePolicySatisfier.
</description>
</property>
<property>
<name>dfs.namenode.get-blocks.check.operation</name>
<value>true</value>
<description>
Whether enable checkOperation when call getBlocks.
It is enabled (true) by default.
Comment thread
Neilxzn marked this conversation as resolved.
Outdated
</description>
</property>
<property>
<name>dfs.balancer.dispatcherThreads</name>
<value>200</value>
Expand Down