HDFS-15386 ReplicaNotFoundException keeps happening in DN after remov…

[brfrn169]

…ing multiple DN's data directories

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 19s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ branch-2.10 Compile Tests _
-1 ❌	mvninstall	0m 16s	root in branch-2.10 failed.
-1 ❌	compile	0m 11s	hadoop-hdfs in branch-2.10 failed.
-0 ⚠️	checkstyle	0m 10s	The patch fails to run checkstyle in hadoop-hdfs
-1 ❌	mvnsite	0m 12s	hadoop-hdfs in branch-2.10 failed.
-1 ❌	javadoc	0m 12s	hadoop-hdfs in branch-2.10 failed.
+0 🆗	spotbugs	0m 49s	Used deprecated FindBugs config; considering switching to SpotBugs.
-1 ❌	findbugs	0m 11s	hadoop-hdfs in branch-2.10 failed.
		_ Patch Compile Tests _
-1 ❌	mvninstall	0m 11s	hadoop-hdfs in the patch failed.
-1 ❌	compile	0m 11s	hadoop-hdfs in the patch failed.
-1 ❌	javac	0m 11s	hadoop-hdfs in the patch failed.
-0 ⚠️	checkstyle	0m 10s	The patch fails to run checkstyle in hadoop-hdfs
-1 ❌	mvnsite	0m 11s	hadoop-hdfs in the patch failed.
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
-1 ❌	javadoc	0m 11s	hadoop-hdfs in the patch failed.
-1 ❌	findbugs	0m 11s	hadoop-hdfs in the patch failed.
		_ Other Tests _
-1 ❌	unit	0m 11s	hadoop-hdfs in the patch failed.
+0 🆗	asflicense	0m 14s	ASF License check generated no output?
		6m 31s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/Dockerfile
GITHUB PR	#2054
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux c155e3a4b1b0 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	branch-2.10 / b411fce
Default Java	Oracle Corporation-1.7.0_95-b00
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/branch-mvninstall-root.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
mvnsite	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/buildtool-patch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
mvnsite	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/testReport/
Max. process+thread count	31 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/1/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	22m 28s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ branch-2.10 Compile Tests _
-1 ❌	mvninstall	0m 15s	root in branch-2.10 failed.
-1 ❌	compile	0m 12s	hadoop-hdfs in branch-2.10 failed.
-0 ⚠️	checkstyle	0m 12s	The patch fails to run checkstyle in hadoop-hdfs
-1 ❌	mvnsite	0m 14s	hadoop-hdfs in branch-2.10 failed.
-1 ❌	javadoc	0m 13s	hadoop-hdfs in branch-2.10 failed.
+0 🆗	spotbugs	1m 0s	Used deprecated FindBugs config; considering switching to SpotBugs.
-1 ❌	findbugs	0m 12s	hadoop-hdfs in branch-2.10 failed.
		_ Patch Compile Tests _
-1 ❌	mvninstall	0m 13s	hadoop-hdfs in the patch failed.
-1 ❌	compile	0m 14s	hadoop-hdfs in the patch failed.
-1 ❌	javac	0m 14s	hadoop-hdfs in the patch failed.
-0 ⚠️	checkstyle	0m 11s	The patch fails to run checkstyle in hadoop-hdfs
-1 ❌	mvnsite	0m 14s	hadoop-hdfs in the patch failed.
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
-1 ❌	javadoc	0m 13s	hadoop-hdfs in the patch failed.
-1 ❌	findbugs	0m 12s	hadoop-hdfs in the patch failed.
		_ Other Tests _
-1 ❌	unit	0m 15s	hadoop-hdfs in the patch failed.
+1 💚	asflicense	0m 35s	The patch does not generate ASF License warnings.
		28m 45s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/Dockerfile
GITHUB PR	#2054
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux aec30aeb67b6 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	branch-2.10 / b411fce
Default Java	Oracle Corporation-1.7.0_95-b00
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/branch-mvninstall-root.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
mvnsite	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/buildtool-patch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
mvnsite	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/testReport/
Max. process+thread count	32 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/3/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 37s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ branch-2.10 Compile Tests _
-1 ❌	mvninstall	0m 15s	root in branch-2.10 failed.
-1 ❌	compile	0m 13s	hadoop-hdfs in branch-2.10 failed.
-0 ⚠️	checkstyle	0m 10s	The patch fails to run checkstyle in hadoop-hdfs
-1 ❌	mvnsite	0m 11s	hadoop-hdfs in branch-2.10 failed.
-1 ❌	javadoc	0m 13s	hadoop-hdfs in branch-2.10 failed.
+0 🆗	spotbugs	0m 49s	Used deprecated FindBugs config; considering switching to SpotBugs.
-1 ❌	findbugs	0m 11s	hadoop-hdfs in branch-2.10 failed.
		_ Patch Compile Tests _
-1 ❌	mvninstall	0m 11s	hadoop-hdfs in the patch failed.
-1 ❌	compile	0m 11s	hadoop-hdfs in the patch failed.
-1 ❌	javac	0m 11s	hadoop-hdfs in the patch failed.
-0 ⚠️	checkstyle	0m 11s	The patch fails to run checkstyle in hadoop-hdfs
-1 ❌	mvnsite	0m 12s	hadoop-hdfs in the patch failed.
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
-1 ❌	javadoc	0m 12s	hadoop-hdfs in the patch failed.
-1 ❌	findbugs	0m 11s	hadoop-hdfs in the patch failed.
		_ Other Tests _
-1 ❌	unit	0m 11s	hadoop-hdfs in the patch failed.
+1 💚	asflicense	0m 31s	The patch does not generate ASF License warnings.
		6m 58s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/Dockerfile
GITHUB PR	#2054
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux e4789a3a9e79 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	branch-2.10 / b411fce
Default Java	Oracle Corporation-1.7.0_95-b00
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/branch-mvninstall-root.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
mvnsite	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/buildtool-patch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
mvnsite	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/testReport/
Max. process+thread count	32 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/4/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[brfrn169]

It looks like QA for branch-2.10 is broken? @sodonnel Can you please help me?

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	11m 7s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ branch-2.10 Compile Tests _
-1 ❌	mvninstall	0m 16s	root in branch-2.10 failed.
-1 ❌	compile	0m 12s	hadoop-hdfs in branch-2.10 failed.
-0 ⚠️	checkstyle	0m 12s	The patch fails to run checkstyle in hadoop-hdfs
-1 ❌	mvnsite	0m 11s	hadoop-hdfs in branch-2.10 failed.
-1 ❌	javadoc	0m 11s	hadoop-hdfs in branch-2.10 failed.
+0 🆗	spotbugs	0m 52s	Used deprecated FindBugs config; considering switching to SpotBugs.
-1 ❌	findbugs	0m 12s	hadoop-hdfs in branch-2.10 failed.
		_ Patch Compile Tests _
-1 ❌	mvninstall	0m 11s	hadoop-hdfs in the patch failed.
-1 ❌	compile	0m 12s	hadoop-hdfs in the patch failed.
-1 ❌	javac	0m 12s	hadoop-hdfs in the patch failed.
-0 ⚠️	checkstyle	0m 11s	The patch fails to run checkstyle in hadoop-hdfs
-1 ❌	mvnsite	0m 12s	hadoop-hdfs in the patch failed.
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
-1 ❌	javadoc	0m 13s	hadoop-hdfs in the patch failed.
-1 ❌	findbugs	0m 12s	hadoop-hdfs in the patch failed.
		_ Other Tests _
-1 ❌	unit	0m 12s	hadoop-hdfs in the patch failed.
+0 🆗	asflicense	0m 16s	ASF License check generated no output?
		16m 38s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/Dockerfile
GITHUB PR	#2054
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 406910e1b801 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	branch-2.10 / 14ff617
Default Java	Oracle Corporation-1.7.0_95-b00
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/branch-mvninstall-root.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
mvnsite	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/buildtool-patch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
mvnsite	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/testReport/
Max. process+thread count	31 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2054/5/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[sodonnel]

The PR branch compiles fine locally and the new tests pass OK (all tests in TestFsDataSetImpl pass). We also know the CI run was clean on trunk.

Let me check with some people if we are good to just push this change in. We don't run CI on all the 3.x branchs after backporting from trunk, and the change here from 3.x is fairly minor.

[sodonnel]

This change looks good, and even though the CI isn't running, its a minor change compared to what was fine on trunk. I will go ahead and merge it.