HDFS-16303. Improve handling of datanode lost while decommissioning #3920

KevinWikant · 2022-01-24T17:47:17Z

Description of PR

Backport of this trunk commit to Hadoop 2.x

How was this patch tested?

All "TestDecommission" & "DatanodeAdminManager" tests pass when run locally

Same manual testing from #3675 applies.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[n/a ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[n/a] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[n/a] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hadoop-yetus · 2022-01-24T22:39:53Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	11m 45s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ branch-3.2 Compile Tests _
+1 💚	mvninstall	29m 11s		branch-3.2 passed
+1 💚	compile	1m 1s		branch-3.2 passed
+1 💚	checkstyle	0m 45s		branch-3.2 passed
+1 💚	mvnsite	1m 7s		branch-3.2 passed
+1 💚	javadoc	0m 55s		branch-3.2 passed
+1 💚	spotbugs	2m 54s		branch-3.2 passed
+1 💚	shadedclient	16m 59s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 8s		the patch passed
+1 💚	compile	0m 57s		the patch passed
+1 💚	javac	0m 57s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 40s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 41 unchanged - 0 fixed = 43 total (was 41)
+1 💚	mvnsite	1m 2s		the patch passed
+1 💚	javadoc	0m 46s		the patch passed
+1 💚	spotbugs	2m 57s		the patch passed
+1 💚	shadedclient	18m 4s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	203m 4s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		291m 35s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDatanodeRegistration

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3920/1/artifact/out/Dockerfile
GITHUB PR	#3920
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 71b7a710e8d6 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.2 / `305443c`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3920/1/testReport/
Max. process+thread count	2095 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3920/1/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

KevinWikant · 2022-01-26T18:25:19Z

[ERROR] Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 23.773 s <<< FAILURE! - in org.apache.hadoop.hdfs.TestDatanodeRegistration
[ERROR] testForcedRegistration(org.apache.hadoop.hdfs.TestDatanodeRegistration)  Time elapsed: 11.562 s  <<< FAILURE!
java.lang.AssertionError
    at org.junit.Assert.fail(Assert.java:87)
    at org.junit.Assert.assertTrue(Assert.java:42)
    at org.junit.Assert.assertTrue(Assert.java:53)
    at org.apache.hadoop.hdfs.TestDatanodeRegistration.testForcedRegistration(TestDatanodeRegistration.java:385)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

Unclear if this is a flaky test: https://issues.apache.org/jira/browse/HDFS-15646

Heres the failed assertion:

hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeRegistration.java

Line 385 in 03cfc85

assertTrue(dnd.isRegistered());

hadoop-yetus · 2022-01-26T23:12:51Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 55s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ branch-3.2 Compile Tests _
+1 💚	mvninstall	29m 29s		branch-3.2 passed
+1 💚	compile	1m 1s		branch-3.2 passed
+1 💚	checkstyle	0m 44s		branch-3.2 passed
+1 💚	mvnsite	1m 7s		branch-3.2 passed
+1 💚	javadoc	0m 54s		branch-3.2 passed
+1 💚	spotbugs	2m 56s		branch-3.2 passed
+1 💚	shadedclient	16m 52s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 5s		the patch passed
+1 💚	compile	0m 55s		the patch passed
+1 💚	javac	0m 55s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 40s		the patch passed
+1 💚	mvnsite	1m 2s		the patch passed
+1 💚	javadoc	0m 54s		the patch passed
+1 💚	spotbugs	3m 15s		the patch passed
+1 💚	shadedclient	19m 18s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	202m 21s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 48s		The patch does not generate ASF License warnings.
		281m 24s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.blockmanagement.TestBlockTokenWithShortCircuitRead
	hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3920/2/artifact/out/Dockerfile
GITHUB PR	#3920
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux a19e6f7ebca1 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.2 / `d53e2d6`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3920/2/testReport/
Max. process+thread count	1943 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3920/2/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

KevinWikant · 2022-01-27T21:44:43Z

[ERROR] Failures: 
[ERROR]   TestBlockTokenWithShortCircuitRead.testShortCircuitReadWithInvalidToken:153->checkSlotsAfterSSRWithTokenExpiration:178->checkShmAndSlots:184 expected:<1> but was:<2>
[ERROR]   TestDirectoryScanner.testThrottling:727 Throttle is too permissive
[INFO] 
[ERROR] Tests run: 6208, Failures: 2, Errors: 0, Skipped: 22

TestDirectoryScanner.testThrottling is related to a flaky test:

https://issues.apache.org/jira/browse/HDFS-15646
"TestDirectoryScanner#testThrottling fails: Throttle is too permissive"

TestBlockTokenWithShortCircuitRead.testShortCircuitReadWithInvalidToken is not a reported flaky test:

https://issues.apache.org/jira/browse/HDFS-15646

KevinWikant · 2022-01-27T21:50:06Z

TestBlockTokenWithShortCircuitRead.testShortCircuitReadWithInvalidToken is testing code entirely unrelated to this change, I have added this failure to https://issues.apache.org/jira/browse/HDFS-15646 for tracking

KevinWikant · 2022-01-27T21:50:24Z

@aajisaka please review this PR to backport https://issues.apache.org/jira/browse/HDFS-16303

hadoop-yetus · 2022-01-28T19:46:10Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ branch-3.2 Compile Tests _
+1 💚	mvninstall	32m 28s		branch-3.2 passed
+1 💚	compile	1m 12s		branch-3.2 passed
+1 💚	checkstyle	0m 54s		branch-3.2 passed
+1 💚	mvnsite	1m 21s		branch-3.2 passed
+1 💚	javadoc	1m 4s		branch-3.2 passed
+1 💚	spotbugs	3m 34s		branch-3.2 passed
+1 💚	shadedclient	19m 10s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 18s		the patch passed
+1 💚	compile	1m 4s		the patch passed
+1 💚	javac	1m 4s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 48s		the patch passed
+1 💚	mvnsite	1m 16s		the patch passed
+1 💚	javadoc	1m 1s		the patch passed
+1 💚	spotbugs	3m 31s		the patch passed
+1 💚	shadedclient	21m 23s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	184m 27s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 43s		The patch does not generate ASF License warnings.
		272m 46s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestBlockStoragePolicy
	hadoop.hdfs.server.namenode.TestFsck

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3920/3/artifact/out/Dockerfile
GITHUB PR	#3920
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux e4fe0391ff8f 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.2 / `5064cde`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3920/3/testReport/
Max. process+thread count	3075 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3920/3/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

aajisaka

LGTM. Thank you.

…missioning (apache#3920) Co-authored-by: Kevin Wikant <[email protected]> Signed-off-by: Akira Ajisaka <[email protected]> (cherry picked from commit 543ff58) Change-Id: I5b4c220fd30c859cd8ceca5cac23719f8b65f0b9

HDFS-16303. Improve handling of datanode lost while decommissioning

305443c

HDFS-16303. Improve handling of datanode lost while decommissioning v2

d53e2d6

HDFS-16303. Remove 1 line of code that is part of HDFS-16443

5064cde

aajisaka approved these changes Jan 31, 2022

View reviewed changes

aajisaka merged commit 543ff58 into apache:branch-3.2 Jan 31, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-16303. Improve handling of datanode lost while decommissioning #3920

HDFS-16303. Improve handling of datanode lost while decommissioning #3920

Uh oh!

KevinWikant commented Jan 24, 2022 •

edited

Loading

Uh oh!

hadoop-yetus commented Jan 24, 2022

Uh oh!

KevinWikant commented Jan 26, 2022

Uh oh!

hadoop-yetus commented Jan 26, 2022

Uh oh!

KevinWikant commented Jan 27, 2022

Uh oh!

KevinWikant commented Jan 27, 2022

Uh oh!

KevinWikant commented Jan 27, 2022

Uh oh!

hadoop-yetus commented Jan 28, 2022

Uh oh!

aajisaka left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HDFS-16303. Improve handling of datanode lost while decommissioning #3920

HDFS-16303. Improve handling of datanode lost while decommissioning #3920

Uh oh!

Conversation

KevinWikant commented Jan 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

Uh oh!

hadoop-yetus commented Jan 24, 2022

Uh oh!

KevinWikant commented Jan 26, 2022

Uh oh!

hadoop-yetus commented Jan 26, 2022

Uh oh!

KevinWikant commented Jan 27, 2022

Uh oh!

KevinWikant commented Jan 27, 2022

Uh oh!

KevinWikant commented Jan 27, 2022

Uh oh!

hadoop-yetus commented Jan 28, 2022

Uh oh!

aajisaka left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

KevinWikant commented Jan 24, 2022 •

edited

Loading