HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from fallback FS #4094

xinglin · 2022-03-22T17:54:45Z

Description of PR

How was this patch tested?

mvn test -Dtest="TestViewFs*" in both hadoop-common-project and hadoop-hdfs-project.

…en from the fallback FS

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java

mccormickt12

Could you add a pointer to where/how the corresponding logic lives in the non-abstract filesystem object, just to easily verify the logic is the same

xinglin · 2022-03-22T20:23:13Z

Could you add a pointer to where/how the corresponding logic lives in the non-abstract filesystem object, just to easily verify the logic is the same

The logic is in DelegationTokenIssuer.java for FileSystem.java/ViewFileSystem.java. collectDelegationTokens() in DelegationTokenIssuer will collect token for the current issuer and all AdditionalTokenIssuers. In FileSystem.java, we overwrite getAdditionalTokenIssuers() to call getChildFileSystem() and ViewFileSystem overwrites getChildFileSystem(), to include the fallback FS as well. So, ViewFileSystem.java is good here.

hadoop-yetus · 2022-03-23T02:17:20Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	12m 22s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 41s		Maven dependency ordering for branch
+1 💚	mvninstall	23m 25s		trunk passed
+1 💚	compile	24m 12s		trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	21m 22s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	3m 46s		trunk passed
+1 💚	mvnsite	3m 23s		trunk passed
+1 💚	javadoc	2m 22s		trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	3m 30s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	6m 15s		trunk passed
+1 💚	shadedclient	24m 56s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 29s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 22s		the patch passed
+1 💚	compile	23m 42s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	23m 42s		the patch passed
+1 💚	compile	22m 0s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	22m 0s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	3m 35s	/results-checkstyle-root.txt	root: The patch generated 1 new + 41 unchanged - 0 fixed = 42 total (was 41)
+1 💚	mvnsite	3m 20s		the patch passed
+1 💚	javadoc	2m 20s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	3m 23s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	7m 4s		the patch passed
+1 💚	shadedclient	25m 24s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 18s		hadoop-common in the patch passed.
+1 💚	unit	242m 39s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 9s		The patch does not generate ASF License warnings.
		492m 7s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/1/artifact/out/Dockerfile
GITHUB PR	#4094
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 716989d0f40f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `f5adca1`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/1/testReport/
Max. process+thread count	3534 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-03-23T02:51:37Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 47s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 30s		Maven dependency ordering for branch
+1 💚	mvninstall	25m 28s		trunk passed
+1 💚	compile	24m 37s		trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	20m 46s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	3m 29s		trunk passed
+1 💚	mvnsite	3m 27s		trunk passed
+1 💚	javadoc	2m 19s		trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	3m 32s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	6m 17s		trunk passed
+1 💚	shadedclient	24m 48s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 29s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 27s		the patch passed
+1 💚	compile	23m 56s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	23m 56s		the patch passed
+1 💚	compile	20m 58s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	20m 58s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	3m 37s	/results-checkstyle-root.txt	root: The patch generated 1 new + 41 unchanged - 0 fixed = 42 total (was 41)
+1 💚	mvnsite	3m 18s		the patch passed
+1 💚	javadoc	2m 28s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	3m 22s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	6m 51s		the patch passed
+1 💚	shadedclient	27m 17s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 49s		hadoop-common in the patch passed.
+1 💚	unit	237m 45s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 13s		The patch does not generate ASF License warnings.
		479m 26s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/2/artifact/out/Dockerfile
GITHUB PR	#4094
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 7ea0c80d39e9 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `4996eed`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/2/testReport/
Max. process+thread count	3227 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-03-23T04:26:15Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 52s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 23s		Maven dependency ordering for branch
+1 💚	mvninstall	25m 58s		trunk passed
+1 💚	compile	24m 44s		trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	21m 20s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	3m 50s		trunk passed
+1 💚	mvnsite	3m 17s		trunk passed
+1 💚	javadoc	2m 17s		trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	3m 17s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 59s		trunk passed
+1 💚	shadedclient	26m 8s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 25s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 26s		the patch passed
+1 💚	compile	23m 54s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	23m 54s		the patch passed
+1 💚	compile	21m 17s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	21m 17s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	3m 46s	/results-checkstyle-root.txt	root: The patch generated 1 new + 41 unchanged - 0 fixed = 42 total (was 41)
+1 💚	mvnsite	3m 17s		the patch passed
+1 💚	javadoc	2m 14s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	3m 19s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	6m 27s		the patch passed
+1 💚	shadedclient	26m 36s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	17m 35s		hadoop-common in the patch passed.
+1 💚	unit	325m 53s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 58s		The patch does not generate ASF License warnings.
		566m 31s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/3/artifact/out/Dockerfile
GITHUB PR	#4094
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux f93c1bd7146a 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `5d08361`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/3/testReport/
Max. process+thread count	2091 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-03-23T12:51:39Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 2s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 40s		Maven dependency ordering for branch
+1 💚	mvninstall	23m 5s		trunk passed
+1 💚	compile	22m 57s		trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	20m 1s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	3m 39s		trunk passed
+1 💚	mvnsite	3m 24s		trunk passed
+1 💚	javadoc	2m 29s		trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	3m 34s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 57s		trunk passed
+1 💚	shadedclient	23m 25s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 29s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 17s		the patch passed
+1 💚	compile	22m 6s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	22m 6s		the patch passed
+1 💚	compile	19m 52s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	19m 52s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	3m 38s		the patch passed
+1 💚	mvnsite	3m 22s		the patch passed
+1 💚	javadoc	2m 25s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	3m 33s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	6m 21s		the patch passed
+1 💚	shadedclient	23m 37s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	17m 52s		hadoop-common in the patch passed.
+1 💚	unit	379m 39s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 19s		The patch does not generate ASF License warnings.
		607m 56s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/4/artifact/out/Dockerfile
GITHUB PR	#4094
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 4a4f05c5f7f5 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `f04b423`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/4/testReport/
Max. process+thread count	2909 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4094/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

xinglin · 2022-03-23T16:08:25Z

Hi @omalley,

Could you take a look and commit it? I guess it should be a trivial cherry-pick to backport to 3.3 and 2.10. If not the case, please let me know. thanks,

…en from fallback FS (#4094) Signed-off-by: Owen O'Malley <oomalley@linkedin.com>

…en from fallback FS (apache#4094) HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from the fallback FS

HADOOP-18169. getDelegationTokens in ViewFs should also fetch the tok…

f5adca1

…en from the fallback FS

mccormickt12 reviewed Mar 22, 2022

View reviewed changes

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java Outdated Show resolved Hide resolved

mccormickt12 reviewed Mar 22, 2022

View reviewed changes

Xing Lin added 2 commits March 22, 2022 11:50

Fixed a variable name

4996eed

Renamed to rootFallBackFs

5d08361

Fixed a checkstyle warning

f04b423

omalley approved these changes Mar 31, 2022

View reviewed changes

omalley merged commit 15a5ea2 into apache:trunk Mar 31, 2022

omalley pushed a commit that referenced this pull request Mar 31, 2022

HADOOP-18169. getDelegationTokens in ViewFs should also fetch the tok…

20483f6

…en from fallback FS (#4094) Signed-off-by: Owen O'Malley <oomalley@linkedin.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from fallback FS #4094

HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from fallback FS #4094

xinglin commented Mar 22, 2022

Uh oh!

Uh oh!

mccormickt12 left a comment

Uh oh!

xinglin commented Mar 22, 2022

Uh oh!

hadoop-yetus commented Mar 23, 2022

Uh oh!

hadoop-yetus commented Mar 23, 2022

Uh oh!

hadoop-yetus commented Mar 23, 2022

Uh oh!

hadoop-yetus commented Mar 23, 2022

Uh oh!

xinglin commented Mar 23, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from fallback FS #4094

HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from fallback FS #4094

Conversation

xinglin commented Mar 22, 2022

Description of PR

How was this patch tested?

Uh oh!

Uh oh!

mccormickt12 left a comment

Choose a reason for hiding this comment

Uh oh!

xinglin commented Mar 22, 2022

Uh oh!

hadoop-yetus commented Mar 23, 2022

Uh oh!

hadoop-yetus commented Mar 23, 2022

Uh oh!

hadoop-yetus commented Mar 23, 2022

Uh oh!

hadoop-yetus commented Mar 23, 2022

Uh oh!

xinglin commented Mar 23, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants