HDFS-15945. DataNodes with zero capacity and zero blocks should be decommissioned immediately. #2854
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…commissioned immediately.
|
💔 -1 overall
This message was automatically generated. |
Member
Author
|
Seems the failure of |
|
💔 -1 overall
This message was automatically generated. |
virajjasani
reviewed
Apr 5, 2021
Contributor
virajjasani
left a comment
virajjasani
left a comment
There was a problem hiding this comment.
One minor comment, else looks good.
...ct/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
Show resolved
Hide resolved
Member
Author
|
Based on a discussion with @virajjasani (#2854 (comment)), I realized it doesn't matter what the capacity is. If a datanode which doesn't have any blocks, it could be decommissioned safely. Updated the PR based on that. |
This reverts commit 0aa3649.
Member
Author
|
On second thought, the last commit has a problem. Just after restarting NameNode, NameNode hasn't received any block reports from any DataNode, so NameNode recognizes all DataNodes as zero blocks. Therefore, when restarting NameNode while decommissioning a DataNode, the DataNode becomes decommissioned imediately before replicating its blocks. Actually After all, I think we need to consider if the DataNode has zero capacity or not. If the capacity is zero, it means the DataNode has a problem with its storage, and we can decommission it safely. |
Contributor
|
Oh I see, yeah this is a possibility. I agree that we should bring back zero capacity check.
Nice |
Member
Author
|
@virajjasani Thanks for your confirmation. Reverted the last commit and added more comment. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
Member
Author
|
The failed tests succeeded locally. |
jojochuang
reviewed
Apr 7, 2021
| if (!node.checkBlockReportReceived()) { | ||
| LOG.info("Node {} hasn't sent its first block report.", node); | ||
| return false; | ||
| if (node.getCapacity() == 0 && node.getNumBlocks() == 0) { |
Contributor
There was a problem hiding this comment.
DatanodeDescriptor#getNumBlocks() returns the variable numBlocks.
However, it is only set during initialization.
Instead, I suspect we want to DatanodeDescriptor#use numBlocks() where the number is computed, aggregated from all existing storage volumes.
jojochuang
reviewed
Apr 7, 2021
| capacities[i][j] = 0; | ||
| } | ||
| } | ||
| getCluster().startDataNodes(getConf(), 1, null, true, null, null, null, |
Contributor
There was a problem hiding this comment.
IMO a more complete repro of the scenario should include:
- start DN with volumes, update config to tolerate volume failures.
- intentionally corrupt the volumes (delete VERSION file, for example)
- trigger volume scanner, wait for the DN to drop the volume
Maybe we don't need a very faithful repro, but I am worried this test doesn't cover the real scenario.
Member
Author
There was a problem hiding this comment.
Thanks for your detailed reviews, @jojochuang. I will try to reproduce it by a unit test.
Member
Author
|
@jojochuang Sorry for being very late. We found the root cause of this problem. There is a bug in hadoop-3.3.0 that DataNode doesn't shutdown even if the number of the failed volumes is greater than |
Member
Author
|
As I said in the last comment, this is not a problem anymore after HDFS-15963. I'm closing this PR. |
Contributor
|
Great! Glad to find out. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
JIRA: https://issues.apache.org/jira/browse/HDFS-15945