HDFS-15532: listFiles on root/InternalDir will fail if fallback root has file. #2298

umamaheswararao · 2020-09-11T03:00:40Z

https://issues.apache.org/jira/browse/HDFS-15532

…has file

hadoop-yetus · 2020-09-11T09:57:15Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 30s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+0 🆗	mvndep	3m 34s	Maven dependency ordering for branch
+1 💚	mvninstall	33m 48s	trunk passed
+1 💚	compile	30m 27s	trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	compile	28m 41s	trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚	checkstyle	4m 25s	trunk passed
+1 💚	mvnsite	4m 28s	trunk passed
+1 💚	shadedclient	29m 14s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	2m 6s	trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	javadoc	4m 17s	trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+0 🆗	spotbugs	4m 24s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	7m 49s	trunk passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 31s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 49s	the patch passed
+1 💚	compile	28m 16s	the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	javac	28m 16s	the patch passed
+1 💚	compile	23m 51s	the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚	javac	23m 51s	the patch passed
-0 ⚠️	checkstyle	3m 33s	root: The patch generated 1 new + 44 unchanged - 0 fixed = 45 total (was 44)
+1 💚	mvnsite	3m 48s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	17m 38s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	2m 31s	the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	javadoc	4m 8s	the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚	findbugs	8m 49s	the patch passed
		_ Other Tests _
+1 💚	unit	14m 50s	hadoop-common in the patch passed.
-1 ❌	unit	154m 35s	hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 51s	The patch does not generate ASF License warnings.
		414m 40s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestFileChecksumCompositeCrc
	hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
	hadoop.hdfs.TestFileChecksum
	hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier
	hadoop.hdfs.TestMultipleNNPortQOP
	hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics
	hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2298/1/artifact/out/Dockerfile
GITHUB PR	#2298
JIRA Issue	HDFS-15532
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux b92e771b2da8 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `9960c01`
Default Java	Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
checkstyle	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2298/1/artifact/out/diff-checkstyle-root.txt
unit	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2298/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2298/1/testReport/
Max. process+thread count	3208 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2298/1/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2020-09-11T23:34:07Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 13s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+0 🆗	mvndep	3m 20s	Maven dependency ordering for branch
+1 💚	mvninstall	29m 13s	trunk passed
+1 💚	compile	22m 44s	trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	compile	23m 1s	trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚	checkstyle	3m 27s	trunk passed
+1 💚	mvnsite	3m 28s	trunk passed
+1 💚	shadedclient	25m 50s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 42s	trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	javadoc	3m 26s	trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+0 🆗	spotbugs	3m 56s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	6m 52s	trunk passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 25s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 33s	the patch passed
+1 💚	compile	26m 25s	the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	javac	26m 25s	the patch passed
+1 💚	compile	21m 39s	the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚	javac	21m 39s	the patch passed
+1 💚	checkstyle	3m 27s	the patch passed
+1 💚	mvnsite	3m 22s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	19m 6s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 39s	the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	javadoc	3m 31s	the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚	findbugs	7m 28s	the patch passed
		_ Other Tests _
+1 💚	unit	11m 20s	hadoop-common in the patch passed.
-1 ❌	unit	120m 30s	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 52s	The patch does not generate ASF License warnings.
		344m 34s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks
	hadoop.fs.viewfs.TestViewFsHdfs
	hadoop.hdfs.TestFileChecksumCompositeCrc
	hadoop.hdfs.server.datanode.TestBPOfferService
	hadoop.hdfs.TestSafeModeWithStripedFileWithRandomECPolicy
	hadoop.hdfs.TestFileChecksum
	hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics
	hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier
	hadoop.hdfs.TestMultipleNNPortQOP
	hadoop.hdfs.TestStripedFileAppend

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2298/2/artifact/out/Dockerfile
GITHUB PR	#2298
JIRA Issue	HDFS-15532
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 6599a7f3ff4a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `958cab8`
Default Java	Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
unit	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2298/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2298/2/testReport/
Max. process+thread count	3166 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2298/2/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

ayushtkn · 2020-09-12T04:58:38Z

...p-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java

+        String leafChild = (InodeTree.SlashPath.equals(fs.getPath())) ?
+            InodeTree.SlashPath.toString() :
+            fs.getPath().getName();


(InodeTree.SlashPath.equals(fs.getPath()))
In which case this would be true? Provided there is a check in the if condition

Thanks @ayushtkn for review! I have removed this redundant condition.
I realized that, this bug exist in ViewFs as well. I fixed and added a test case for it. Thanks

hadoop-yetus · 2020-09-12T13:04:16Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 21s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
+0 🆗	mvndep	3m 30s	Maven dependency ordering for branch
+1 💚	mvninstall	33m 58s	trunk passed
+1 💚	compile	29m 32s	trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	compile	23m 55s	trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚	checkstyle	3m 35s	trunk passed
+1 💚	mvnsite	3m 39s	trunk passed
+1 💚	shadedclient	24m 48s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 48s	trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	javadoc	3m 32s	trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+0 🆗	spotbugs	4m 10s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	7m 3s	trunk passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 30s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 45s	the patch passed
+1 💚	compile	31m 23s	the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	javac	31m 23s	the patch passed
+1 💚	compile	27m 5s	the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚	javac	27m 5s	the patch passed
+1 💚	checkstyle	3m 52s	the patch passed
+1 💚	mvnsite	3m 58s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	19m 10s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 53s	the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	javadoc	3m 44s	the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚	findbugs	8m 0s	the patch passed
		_ Other Tests _
+1 💚	unit	13m 4s	hadoop-common in the patch passed.
-1 ❌	unit	144m 16s	hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 31s	The patch does not generate ASF License warnings.
		396m 13s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestFileChecksumCompositeCrc
	hadoop.hdfs.TestFileChecksum
	hadoop.hdfs.server.datanode.TestBPOfferService
	hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier
	hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics
	hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR
	hadoop.hdfs.TestSafeModeWithStripedFileWithRandomECPolicy

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2298/3/artifact/out/Dockerfile
GITHUB PR	#2298
JIRA Issue	HDFS-15532
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 4037b09c1838 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `958cab8`
Default Java	Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
unit	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2298/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2298/3/testReport/
Max. process+thread count	2820 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2298/3/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

ayushtkn

Thanx @umamaheswararao for the update. Changes LGTM
+1

…has file. (#2298). Contributed by Uma Maheswara Rao G. (cherry picked from commit d2779de)

…has file. (apache#2298). Contributed by Uma Maheswara Rao G. (cherry picked from commit d2779de) Change-Id: Ia4e67ec4a77f11d5250aabcb117db37a6c699818 (cherry picked from commit c04653a)

HDFS-15532: listFiles on root/InternalDir will fail if fallback root …

af0b261

…has file

umamaheswararao requested a review from ayushtkn September 11, 2020 06:00

umamaheswararao assigned virajith and rakeshadr and unassigned virajith and rakeshadr Sep 11, 2020

umamaheswararao requested review from rakeshadr and virajith September 11, 2020 06:01

Fixed a checkstyle warning.

cdd0932

ayushtkn reviewed Sep 12, 2020

View reviewed changes

Fixed a review comment and also fixed the issue for ViewFs flow as well

271b8b5

ayushtkn approved these changes Sep 12, 2020

View reviewed changes

umamaheswararao merged commit d2779de into apache:trunk Sep 13, 2020

umamaheswararao deleted the HDFS-15532 branch September 13, 2020 00:07

umamaheswararao added a commit that referenced this pull request Sep 13, 2020

HDFS-15532: listFiles on root/InternalDir will fail if fallback root …

bfa145d

…has file. (#2298). Contributed by Uma Maheswara Rao G. (cherry picked from commit d2779de)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-15532: listFiles on root/InternalDir will fail if fallback root has file. #2298

HDFS-15532: listFiles on root/InternalDir will fail if fallback root has file. #2298

Uh oh!

umamaheswararao commented Sep 11, 2020

Uh oh!

hadoop-yetus commented Sep 11, 2020

Uh oh!

hadoop-yetus commented Sep 11, 2020

Uh oh!

ayushtkn Sep 12, 2020

Uh oh!

umamaheswararao Sep 12, 2020 •

edited

Loading

Uh oh!

hadoop-yetus commented Sep 12, 2020

Uh oh!

ayushtkn left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

HDFS-15532: listFiles on root/InternalDir will fail if fallback root has file. #2298

HDFS-15532: listFiles on root/InternalDir will fail if fallback root has file. #2298

Uh oh!

Conversation

umamaheswararao commented Sep 11, 2020

Uh oh!

hadoop-yetus commented Sep 11, 2020

Uh oh!

hadoop-yetus commented Sep 11, 2020

Uh oh!

ayushtkn Sep 12, 2020

Choose a reason for hiding this comment

Uh oh!

umamaheswararao Sep 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Sep 12, 2020

Uh oh!

ayushtkn left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

umamaheswararao Sep 12, 2020 •

edited

Loading