HDFS-16303. Improve handling of datanode lost while decommissioning #3921

KevinWikant · 2022-01-24T19:32:04Z

Description of PR

Backport of this trunk commit to Hadoop 3.x

How was this patch tested?

All "TestDecommission", "TestDecommissionWithBackoffMonitor", & "DatanodeAdminManager" tests pass when run locally

Same manual testing from #3675 applies.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[n/a ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[n/a] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[n/a] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hadoop-yetus · 2022-01-25T01:03:50Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	10m 5s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	30m 54s		branch-3.3 passed
+1 💚	compile	1m 15s		branch-3.3 passed
+1 💚	checkstyle	0m 50s		branch-3.3 passed
+1 💚	mvnsite	1m 23s		branch-3.3 passed
+1 💚	javadoc	1m 31s		branch-3.3 passed
+1 💚	spotbugs	3m 4s		branch-3.3 passed
+1 💚	shadedclient	25m 34s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 13s		the patch passed
+1 💚	compile	1m 8s		the patch passed
+1 💚	javac	1m 8s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 41s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 45 unchanged - 1 fixed = 47 total (was 46)
+1 💚	mvnsite	1m 12s		the patch passed
+1 💚	javadoc	1m 20s		the patch passed
+1 💚	spotbugs	3m 5s		the patch passed
+1 💚	shadedclient	25m 10s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	222m 12s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 45s		The patch does not generate ASF License warnings.
		328m 54s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
	hadoop.hdfs.server.datanode.TestBPOfferService
	hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3921/2/artifact/out/Dockerfile
GITHUB PR	#3921
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 371ae086db04 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `99a7dc1`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3921/2/testReport/
Max. process+thread count	3187 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3921/2/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

KevinWikant · 2022-01-26T18:25:31Z

[ERROR] Tests run: 17, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 75.795 s <<< FAILURE! - in org.apache.hadoop.hdfs.server.datanode.TestBPOfferService
[ERROR] testMissBlocksWhenReregister(org.apache.hadoop.hdfs.server.datanode.TestBPOfferService)  Time elapsed: 15.474 s  <<< FAILURE!
java.lang.AssertionError
    at org.junit.Assert.fail(Assert.java:87)
    at org.junit.Assert.assertTrue(Assert.java:42)
    at org.junit.Assert.assertTrue(Assert.java:53)
    at org.apache.hadoop.hdfs.server.datanode.TestBPOfferService.testMissBlocksWhenReregister(TestBPOfferService.java:350)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

This is a flaky test:

"TestBPOfferService#testMissBlocksWhenReregister fails intermittently"
https://issues.apache.org/jira/browse/HDFS-15646

hadoop-yetus · 2022-01-26T23:26:27Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 39s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	32m 20s		branch-3.3 passed
+1 💚	compile	1m 16s		branch-3.3 passed
+1 💚	checkstyle	0m 50s		branch-3.3 passed
+1 💚	mvnsite	1m 26s		branch-3.3 passed
+1 💚	javadoc	1m 34s		branch-3.3 passed
+1 💚	spotbugs	3m 22s		branch-3.3 passed
+1 💚	shadedclient	25m 56s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 14s		the patch passed
+1 💚	compile	1m 8s		the patch passed
+1 💚	javac	1m 8s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 41s		hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 45 unchanged - 1 fixed = 45 total (was 46)
+1 💚	mvnsite	1m 14s		the patch passed
+1 💚	javadoc	1m 28s		the patch passed
+1 💚	spotbugs	3m 17s		the patch passed
+1 💚	shadedclient	25m 27s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	195m 58s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 47s		The patch does not generate ASF License warnings.
		295m 47s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.TestBPOfferService
	hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3921/3/artifact/out/Dockerfile
GITHUB PR	#3921
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 0fa3513b7b56 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `3641938`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3921/3/testReport/
Max. process+thread count	3833 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3921/3/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

KevinWikant · 2022-01-27T21:41:54Z

Unit tests failures are related to flaky tests

[ERROR] Failures: 
[ERROR]   TestBPOfferService.testMissBlocksWhenReregister:350
[ERROR] Errors: 
[ERROR]   TestUnderReplicatedBlocks.testSetRepIncWithUnderReplicatedBlocks:80 Â» TestTimedOut
[INFO] 
[ERROR] Tests run: 6738, Failures: 1, Errors: 1, Skipped: 22

TestBPOfferService.testMissBlocksWhenReregister:

https://issues.apache.org/jira/browse/HDFS-15646
"TestBPOfferService#testMissBlocksWhenReregister fails on trunk"
"TestBPOfferService#testMissBlocksWhenReregister fails intermittently"
"TestBPOfferService#testMissBlocksWhenReregister is flaky"

TestUnderReplicatedBlocks.testSetRepIncWithUnderReplicatedBlocks:

https://issues.apache.org/jira/browse/HDFS-15646
"TestUnderReplicatedBlocks#testSetrepIncWithUnderReplicatedBlocks test timeout"

KevinWikant · 2022-01-27T21:50:19Z

@aajisaka please review this PR to backport https://issues.apache.org/jira/browse/HDFS-16303

aajisaka · 2022-01-28T06:41:36Z

This PR contains the change in #3923.

Would you address #3923 (comment)? You'll need to do the followings:

Create a separate jira to track the small fix
Remove the fix from this PR

Then I will merge this PR and backport the separate fix. Thanks.

hadoop-yetus · 2022-01-28T20:52:19Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	10m 26s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	30m 13s		branch-3.3 passed
+1 💚	compile	1m 13s		branch-3.3 passed
+1 💚	checkstyle	0m 48s		branch-3.3 passed
+1 💚	mvnsite	1m 25s		branch-3.3 passed
+1 💚	javadoc	1m 33s		branch-3.3 passed
+1 💚	spotbugs	3m 7s		branch-3.3 passed
+1 💚	shadedclient	25m 2s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 17s		the patch passed
+1 💚	compile	1m 9s		the patch passed
+1 💚	javac	1m 9s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 38s		hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 45 unchanged - 1 fixed = 45 total (was 46)
+1 💚	mvnsite	1m 13s		the patch passed
+1 💚	javadoc	1m 20s		the patch passed
+1 💚	spotbugs	3m 16s		the patch passed
+1 💚	shadedclient	24m 37s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	232m 11s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 46s		The patch does not generate ASF License warnings.
		337m 30s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestRollingUpgrade

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3921/4/artifact/out/Dockerfile
GITHUB PR	#3921
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 97325c2bd100 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `4e8ecd5`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3921/4/testReport/
Max. process+thread count	2709 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3921/4/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

aajisaka · 2022-01-31T04:18:53Z

Thank you @KevinWikant for backporting this!

…pache#3921) Signed-off-by: Akira Ajisaka <[email protected]>

Kevin Wikant added 2 commits January 24, 2022 14:29

HDFS-16303. Improve handling of datanode lost while decommissioning

7f5d477

HDFS-16303. Improve handling of datanode lost while decommissioning v2

99a7dc1

This comment has been minimized.

Sign in to view

HDFS-16303. Improve handling of datanode lost while decommissioning v3

3641938

HDFS-16303. Remove 1 line of code that is part of HDFS-16443

4e8ecd5

aajisaka approved these changes Jan 31, 2022

View reviewed changes

aajisaka merged commit 5e2eac6 into apache:branch-3.3 Jan 31, 2022

bogthe pushed a commit to bogthe/hadoop that referenced this pull request Feb 2, 2022

HDFS-16303. Improve handling of datanode lost while decommissioning (a…

174eb8b

…pache#3921) Signed-off-by: Akira Ajisaka <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-16303. Improve handling of datanode lost while decommissioning #3921

HDFS-16303. Improve handling of datanode lost while decommissioning #3921

Uh oh!

KevinWikant commented Jan 24, 2022 •

edited

Loading

Uh oh!

This comment has been minimized.

hadoop-yetus commented Jan 25, 2022

Uh oh!

KevinWikant commented Jan 26, 2022

Uh oh!

hadoop-yetus commented Jan 26, 2022

Uh oh!

KevinWikant commented Jan 27, 2022

Uh oh!

KevinWikant commented Jan 27, 2022

Uh oh!

aajisaka commented Jan 28, 2022

Uh oh!

hadoop-yetus commented Jan 28, 2022

Uh oh!

aajisaka commented Jan 31, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HDFS-16303. Improve handling of datanode lost while decommissioning #3921

HDFS-16303. Improve handling of datanode lost while decommissioning #3921

Uh oh!

Conversation

KevinWikant commented Jan 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

Uh oh!

This comment has been minimized.

hadoop-yetus commented Jan 25, 2022

Uh oh!

KevinWikant commented Jan 26, 2022

Uh oh!

hadoop-yetus commented Jan 26, 2022

Uh oh!

KevinWikant commented Jan 27, 2022

Uh oh!

KevinWikant commented Jan 27, 2022

Uh oh!

aajisaka commented Jan 28, 2022

Uh oh!

hadoop-yetus commented Jan 28, 2022

Uh oh!

aajisaka commented Jan 31, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

KevinWikant commented Jan 24, 2022 •

edited

Loading