HADOOP-19652. Fix dependency exclusion list of hadoop-client-runtime #7878

pan3793 · 2025-08-18T02:44:23Z

Description of PR

JIRA: HADOOP-19652. Fix dependency exclusion list of hadoop-client-runtime.

When trying to use the Hadoop trunk version client with Spark 4.1.0-SNAPSHOT (master branch), NoClassDefFoundError was raised.

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/shaded/javax/ws/rs/WebApplicationException
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1023)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
	at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:862)
	at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:760)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:681)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
	at org.apache.spark.deploy.yarn.YarnRMClient.getAmIpFilterParams(YarnRMClient.scala:109)
	at org.apache.spark.deploy.yarn.ApplicationMaster.addAmIpFilter(ApplicationMaster.scala:698)
	at org.apache.spark.deploy.yarn.ApplicationMaster.runExecutorLauncher(ApplicationMaster.scala:555)
	at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:265)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:942)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:941)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:714)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:525)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1959)
	at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:941)
	at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:973)
	at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.shaded.javax.ws.rs.WebApplicationException
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
	... 21 more

How was this patch tested?

Hadoop Client Runtime 3.4.2 RC2

Hadoop Client Runtime 3.5.0-SNAPSHOT trunk

Hadoop Client Runtime 3.5.0-SNAPSHOT HADOOP-19652

Tested by submitting a Spark application to a YARN cluster.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

pan3793 · 2025-08-18T02:57:40Z

cc @slfan1989 @cnauroth, this is caused by Jersey upgrading.

pan3793 · 2025-08-18T03:04:01Z

AFAIK, there is no solid integration test for the Hadoop Shaded client, I will continue to test it with Spark and fix issues I encounter.

hadoop-yetus · 2025-08-18T04:47:44Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 33s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	39m 10s		trunk passed
+1 💚	compile	0m 24s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 25s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	mvnsite	0m 29s		trunk passed
+1 💚	javadoc	0m 30s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 25s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	shadedclient	76m 30s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	5m 50s		the patch passed
+1 💚	compile	0m 16s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 16s		the patch passed
+1 💚	compile	0m 16s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	0m 16s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 17s		the patch passed
+1 💚	javadoc	0m 16s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 16s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
-1 ❌	shadedclient	35m 43s		patch has errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 20s		hadoop-client-runtime in the patch passed.
+1 💚	asflicense	0m 39s		The patch does not generate ASF License warnings.
		121m 57s

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7878/1/artifact/out/Dockerfile
GITHUB PR	#7878
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint
uname	Linux 84d5837e24b5 5.15.0-140-generic #150-Ubuntu SMP Sat Apr 12 06:00:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `16187ca`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7878/1/testReport/
Max. process+thread count	648 (vs. ulimit of 5500)
modules	C: hadoop-client-modules/hadoop-client-runtime U: hadoop-client-modules/hadoop-client-runtime
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7878/1/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

slfan1989 · 2025-08-18T04:52:56Z

hadoop-client-modules/hadoop-client-runtime/pom.xml

-                      <exclude>jakarta.servlet:*</exclude>
-                      <exclude>javax.annotation:*</exclude>
                      <exclude>org.hamcrest:*</exclude>
+                      <exclude>org.javassist:*</exclude>


Can this line remain unchanged?

this is the corrected version of

<exclude>javassist:*</exclude>

pan3793 · 2025-08-18T05:38:55Z

Not a real error, just caused by token expiration, push a new empty commit to retrigger CI.

12:47:42  ============================================================================
12:47:42  ============================================================================
12:47:42                           Adding comment to Github
12:47:42  ============================================================================
12:47:42  ============================================================================
12:47:42  
12:47:42  
12:47:46  ERROR: Failed to write github status. Token expired or missing repo:status write?
12:47:46  ERROR: Failed to write github status. Token expired or missing repo:status write?

https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7878/1/console

hadoop-yetus · 2025-08-18T07:40:08Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 35s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	38m 48s		trunk passed
+1 💚	compile	0m 25s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 24s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	mvnsite	0m 29s		trunk passed
+1 💚	javadoc	0m 31s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 25s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	shadedclient	76m 3s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	5m 47s		the patch passed
+1 💚	compile	0m 14s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 14s		the patch passed
+1 💚	compile	0m 15s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	0m 15s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 17s		the patch passed
+1 💚	javadoc	0m 15s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 16s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
-1 ❌	shadedclient	35m 29s		patch has errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 20s		hadoop-client-runtime in the patch passed.
+1 💚	asflicense	0m 49s		The patch does not generate ASF License warnings.
		121m 21s

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7878/2/artifact/out/Dockerfile
GITHUB PR	#7878
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint
uname	Linux f59cfeb748b8 5.15.0-140-generic #150-Ubuntu SMP Sat Apr 12 06:00:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `830cdae`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7878/2/testReport/
Max. process+thread count	560 (vs. ulimit of 5500)
modules	C: hadoop-client-modules/hadoop-client-runtime U: hadoop-client-modules/hadoop-client-runtime
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7878/2/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

slfan1989 · 2025-08-18T10:12:04Z

@pan3793 This PR still requires some adjustments.

https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7878/2/artifact/out/patch-shadedclient.txt

ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.5.0:enforce (enforce-banned-dependencies) on project hadoop-client-check-test-invariants: 
[ERROR] Rule 1: org.apache.maven.plugins.enforcer.BanDuplicateClasses failed with message:
[ERROR] Duplicate classes found:
[ERROR] 
[ERROR]   Found in:
[ERROR]     org.apache.hadoop:hadoop-client-minicluster:jar:3.5.0-SNAPSHOT:compile
[ERROR]     org.apache.hadoop:hadoop-client-runtime:jar:3.5.0-SNAPSHOT:compile
......
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :hadoop-client-check-test-invariants

pan3793 · 2025-08-18T10:14:45Z

@slfan1989 I see, will fix soon

hadoop-yetus · 2025-08-18T15:20:23Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 31s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	8m 54s		Maven dependency ordering for branch
+1 💚	mvninstall	32m 40s		trunk passed
+1 💚	compile	0m 27s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 27s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	mvnsite	0m 56s		trunk passed
+1 💚	javadoc	0m 53s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 50s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	shadedclient	79m 43s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 34s		Maven dependency ordering for patch
+1 💚	mvninstall	22m 9s		the patch passed
+1 💚	compile	0m 17s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 17s		the patch passed
+1 💚	compile	0m 17s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	0m 17s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 35s		the patch passed
+1 💚	javadoc	0m 32s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 32s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	shadedclient	36m 37s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 20s		hadoop-client-runtime in the patch passed.
+1 💚	unit	0m 21s		hadoop-client-minicluster in the patch passed.
+1 💚	asflicense	0m 38s		The patch does not generate ASF License warnings.
		143m 23s

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7878/3/artifact/out/Dockerfile
GITHUB PR	#7878
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint
uname	Linux 5bff763e4c87 5.15.0-140-generic #150-Ubuntu SMP Sat Apr 12 06:00:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `0e4ad11`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7878/3/testReport/
Max. process+thread count	546 (vs. ulimit of 5500)
modules	C: hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster U: hadoop-client-modules
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7878/3/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

slfan1989 · 2025-08-19T00:41:24Z

@slfan1989 I see, will fix soon

@pan3793 Thank you for your contribution! I don’t see any other issues. Let’s wait for one more day, and if there are no further comments, I’ll merge this PR.

slfan1989 · 2025-08-20T00:24:11Z

@pan3793 Thanks for the contribution! Merged into trunk.

…ient-runtime (apache#7878) Contributed by Cheng Pan. * HADOOP-19652. Fix dependency exlucsion list of hadoop-client-runtime Signed-off-by: Shilun Fan <[email protected]>

HADOOP-19652. Fix dependency exlucsion list of hadoop-client-runtime

16187ca

github-actions bot added build trunk labels Aug 18, 2025

slfan1989 reviewed Aug 18, 2025

View reviewed changes

slfan1989 approved these changes Aug 18, 2025

View reviewed changes

empty

830cdae

hadoop-client-minicluster

0e4ad11

steveloughran changed the title ~~HADOOP-19652. Fix dependency exlucsion list of hadoop-client-runtime~~ HADOOP-19652. Fix dependency exclusion list of hadoop-client-runtime Aug 19, 2025

slfan1989 merged commit 9e94953 into apache:trunk Aug 20, 2025
1 of 3 checks passed

HADOOP-19652. Fix dependency exclusion list of hadoop-client-runtime #7878

HADOOP-19652. Fix dependency exclusion list of hadoop-client-runtime #7878

Conversation

pan3793 commented Aug 18, 2025 • edited by slfan1989 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

Uh oh!

pan3793 commented Aug 18, 2025

Uh oh!

pan3793 commented Aug 18, 2025

Uh oh!

hadoop-yetus commented Aug 18, 2025

Uh oh!

slfan1989 Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

pan3793 Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

pan3793 commented Aug 18, 2025

Uh oh!

hadoop-yetus commented Aug 18, 2025

Uh oh!

slfan1989 commented Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pan3793 commented Aug 18, 2025

Uh oh!

hadoop-yetus commented Aug 18, 2025

Uh oh!

slfan1989 commented Aug 19, 2025

Uh oh!

Uh oh!

slfan1989 commented Aug 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pan3793 commented Aug 18, 2025 •

edited by slfan1989

Loading

slfan1989 commented Aug 18, 2025 •

edited

Loading