HADOOP-19674. [JDK 17] Add jaxb-runtime #7928
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
stoty
suggested changes
Sep 3, 2025
| <dependency> | ||
| <groupId>org.glassfish.jaxb</groupId> | ||
| <artifactId>jaxb-runtime</artifactId> | ||
| <version>${jaxb.version}</version> |
Contributor
There was a problem hiding this comment.
The version should be set in once in the hadoop-project pom dependencyManagement section
| <artifactId>jersey-media-jaxb</artifactId> | ||
| <scope>test</scope> | ||
| </dependency> | ||
| <dependency> |
Contributor
There was a problem hiding this comment.
How did you decide where to add jaxb-imp and with what scope ?
Contributor
Author
There was a problem hiding this comment.
I checked the usage of the JettisonJaxbContext in the projects where only used in the tests i added with test scope, otherwise i set no scope.
| <groupId>org.glassfish.jaxb</groupId> | ||
| <artifactId>jaxb-runtime</artifactId> | ||
| <version>${jaxb.version}</version> | ||
| <scope>test</scope> |
Contributor
There was a problem hiding this comment.
This is needed by Jetty. (at least)
This should be compile scope, as javax.xml.bind is directly used in the main code.
Contributor
Author
There was a problem hiding this comment.
I was not sure this needed in every case for the Jetty.
But in that case if i see well we will need this every where where we are using the HttpServer2.
So if i see well this will be needed in HDFS NN also.
In that case shall we add this to the hadoop-common project only and let others use it transitively?
Contributor
There was a problem hiding this comment.
I was not sure this needed in every case for the Jetty.
The stack strace seems to be coming from Jetty initialization.
But in that case if i see well we will need this every where where we are using the HttpServer2. So if i see well this will be needed in HDFS NN also.
Yes, that's likely.
In that case shall we add this to the
hadoop-commonproject only and let others use it transitively?
I think it's better to add it separately where needed.
Most modules don't use Jetty.
Contributor
There was a problem hiding this comment.
One other issue is the binary assembly
We also need to make sure that jaxb-impl is included there.
Contributor
Author
There was a problem hiding this comment.
I tried to upload a corrected version.
With my understanding every Hadoop based Jetty server is instance of the HttpServer2, what is in the hadoop-common, so i added the dependency there.
Some other modules what use hadoop-common, but not the Jetty server are already excludes the jetty-server.
I extended these cases with jaxb-runtime exculsion. This means: not 100% these modules dont need jaxb-runtime, but pretty sure not because the Jetty.
In the httpfs modul i dont know why but seems like the jetty-server transitive dependency is excluded from hadoop-common, but there is an explicit dependency for it also in the pom.
I run a clean install with -Pdist and i can see the jaxb-runtime jar in the hadoop-dist/target/hadoop-3.5.0-SNAPSHOT/share/hadoop/common/lib/jaxb-runtime-2.3.9.jar path
Contributor
|
Build issue observed on JDK 8 [ERROR] Rule 0: org.apache.maven.enforcer.rules.dependency.DependencyConvergence failed with message:
[ERROR] Failed while enforcing releasability.
[ERROR]
[ERROR] Dependency convergence error for org.glassfish.jaxb:jaxb-runtime:jar:2.3.1 paths to dependency are:
[ERROR] +-org.apache.hadoop:hadoop-mapreduce-client-app:jar:3.5.0-SNAPSHOT
[ERROR] +-org.apache.hadoop:hadoop-yarn-server-web-proxy:jar:3.5.0-SNAPSHOT:compile
[ERROR] +-org.apache.hadoop:hadoop-yarn-server-common:jar:3.5.0-SNAPSHOT:compile
[ERROR] +-org.ehcache:ehcache:jar:3.8.2:compile
[ERROR] +-org.glassfish.jaxb:jaxb-runtime:jar:2.3.1:compile
[ERROR] and
[ERROR] +-org.apache.hadoop:hadoop-mapreduce-client-app:jar:3.5.0-SNAPSHOT
[ERROR] +-org.glassfish.jaxb:jaxb-runtime:jar:2.3.9:compile |
Contributor
Author
Thanks @susheelgupta7 for the finding! |
K0K0V0K
force-pushed
the
HADOOP-19674
branch
from
41d266d to
6086bd5
Compare
- Since JDK11 javax.xml.bind modules is removed
source: https://docs.oracle.com/en/java/javase/24/migrate/removed-tools-and-components.html#GUID-11F78105-D735-430D-92DD-6C37958FCBC3
- Jetty needs a jaxb-impl to be able to start
- So we provide the jaxb-runtime artifact everywhere where Jetty Server is a dependency
- If some modules excludes jetty-server then most probably jaxb-runtime is not needed for them, so we exclude it.
- ehcache was using different version of jaxb-runtime, so it become excluded
K0K0V0K
force-pushed
the
HADOOP-19674
branch
from
6086bd5 to
450d122
Compare
stoty
reviewed
Sep 4, 2025
| <artifactId>jersey-media-json-jettison</artifactId> | ||
| <version>${jersey2.version}</version> | ||
| </dependency> | ||
| <dependency> |
Contributor
There was a problem hiding this comment.
Since we're globally dependency managing jaxb-runtime, I think that we don't need all the excludes in this patch.
They should all be dependency managed to 2.3.9 by maven, and the dependency convergence check should pass.
Contributor
There was a problem hiding this comment.
I think that the reported error was for the earlier patch without the global dependency management.
stoty
reviewed
Sep 4, 2025
Contributor
stoty
left a comment
stoty
left a comment
There was a problem hiding this comment.
Please check if the new exclusions are really needed.
|
💔 -1 overall
This message was automatically generated. |
K0K0V0K
force-pushed
the
HADOOP-19674
branch
from
a7da42e to
8359341
Compare
- Remove not necessary excludes
K0K0V0K
force-pushed
the
HADOOP-19674
branch
from
8359341 to
0b9139b
Compare
|
💔 -1 overall
This message was automatically generated. |
- Put back one exclude what seems to be necessary - Otherwise at and of install we can see the following error: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.5.0:enforce (enforce-banned-dependencies) on project hadoop-client-check-test-invariants: [ERROR] Rule 1: org.apache.maven.plugins.enforcer.BanDuplicateClasses failed with message: [ERROR] Duplicate classes found: [ERROR] [ERROR] Found in: [ERROR] org.apache.hadoop:hadoop-client-minicluster:jar:3.5.0-SNAPSHOT:compile [ERROR] org.apache.hadoop:hadoop-client-runtime:jar:3.5.0-SNAPSHOT:compile [ERROR] Duplicate classes: [ERROR] org/apache/hadoop/shaded/com/sun/activation/registries/MimeTypeEntry.class ... Cannon to right of them,
K0K0V0K
commented
Sep 5, 2025
| <exclusion> | ||
| <groupId>org.glassfish.jaxb</groupId> | ||
| <artifactId>jaxb-runtime</artifactId> | ||
| </exclusion> |
Contributor
Author
There was a problem hiding this comment.
I think this one is necessary otherwise at the end on install i am facing this error:
[INFO] Total time: 05:20 min (Wall Clock)
[INFO] Finished at: 2025-09-05T09:02:10+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.5.0:enforce (enforce-banned-dependencies) on project hadoop-client-check-test-invariants:
[ERROR] Rule 1: org.apache.maven.plugins.enforcer.BanDuplicateClasses failed with message:
[ERROR] Duplicate classes found:
[ERROR]
[ERROR] Found in:
[ERROR] org.apache.hadoop:hadoop-client-minicluster:jar:3.5.0-SNAPSHOT:compile
[ERROR] org.apache.hadoop:hadoop-client-runtime:jar:3.5.0-SNAPSHOT:compile
[ERROR] Duplicate classes:
[ERROR] org/apache/hadoop/shaded/com/sun/activation/registries/MimeTypeEntry.class
[ERROR] org/apache/hadoop/shaded/com/sun/istack/localization/LocalizableMessageFactory$ResourceBundleSupplier.class
[ERROR] org/apache/hadoop/shaded/com/sun/activation/registries/MailcapTokenizer.class
[ERROR] org/apache/hadoop/shaded/com/sun/istack/FinalArrayList.class
[ERROR] org/apache/hadoop/shaded/com/sun/istack/Pool$Impl.class
[ERROR] org/apache/hadoop/shaded/com/sun/activation/registries/MailcapParseException.class
[ERROR] org/apache/hadoop/shaded/com/sun/activation/registries/LogSupport.class
[ERROR] META-INF/versions/9/com/sun/istack/logging/StackHelper.class
[ERROR] org/apache/hadoop/shaded/com/sun/istack/NotNull.class
[ERROR] org/apache/hadoop/shaded/com/sun/istack/XMLStreamException2.class
[ERROR] org/apache/hadoop/shaded/com/sun/activation/viewers/TextEditor.class
[ERROR] org/apache/hadoop/shaded/com/sun/istack/XMLStreamReaderToContentHandler$1.class
[ERROR] org/apache/hadoop/shaded/com/sun/istack/localization/Localizable.class
[ERROR] org/apache/hadoop/shaded/com/sun/activation/viewers/ImageViewerCanvas.class
[ERROR] org/apache/hadoop/shaded/com/sun/activation/viewers/TextViewer.class
[ERROR] org/apache/hadoop/shaded/com/sun/istack/Interned.class
[ERROR] org/apache/hadoop/shaded/com/sun/istack/localization/Localizer.class
[ERROR] org/apache/hadoop/shaded/com/sun/istack/localization/LocalizableMessageFactory.class
[ERROR] org/apache/hadoop/shaded/com/sun/activation/registries/MailcapFile.class
[ERROR] org/apache/hadoop/shaded/com/sun/istack/logging/Logger.class
[ERROR] org/apache/hadoop/shaded/com/sun/istack/ByteArrayDataSource.class
[ERROR] org/apache/hadoop/shaded/com/sun/istack/Nullable.class
[ERROR] org/apache/hadoop/shaded/com/sun/istack/localization/NullLocalizable.class
[ERROR] org/apache/hadoop/shaded/com/sun/istack/SAXException2.class
[ERROR] org/apache/hadoop/shaded/com/sun/activation/registries/MimeTypeFile.class
[ERROR] org/apache/hadoop/shaded/com/sun/istack/logging/StackHelper.class
[ERROR] org/apache/hadoop/shaded/com/sun/activation/registries/LineTokenizer.class
[ERROR] org/apache/hadoop/shaded/com/sun/istack/Builder.class
[ERROR] org/apache/hadoop/shaded/com/sun/istack/FragmentContentHandler.class
[ERROR] org/apache/hadoop/shaded/com/sun/istack/Pool.class
[ERROR] org/apache/hadoop/shaded/com/sun/istack/localization/LocalizableMessage.class
[ERROR] org/apache/hadoop/shaded/com/sun/activation/viewers/ImageViewer.class
[ERROR] org/apache/hadoop/shaded/com/sun/istack/SAXParseException2.class
[ERROR] org/apache/hadoop/shaded/com/sun/istack/XMLStreamReaderToContentHandler.class
Contributor
There was a problem hiding this comment.
I have checked 3.4.2, and these classes are not present there.
Based on that, it seems that hadoop-client-runtime indeed does not include jaxb-impl traditionally.
brumi1024
approved these changes
Sep 5, 2025
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
- If we exclude the dependency from hadoop common, maybe we should do the same with the hdfs-client Cannon to left of them,
|
💔 -1 overall
This message was automatically generated. |
Contributor
|
@K0K0V0K Thank you for your contribution! LGTM. However, we need to trigger the compilation again as the current process seems to have been interrupted. I also noticed some warning messages. Could you please double-check? |
Contributor
Author
|
Hi @slfan1989! Thanks for the review. I checked the latest console, output and seems like the waring is not present anymore: |
- there were 2 empty line at end of file, so I removed one of them to retrigger the build. Cannon in front of them.
|
💔 -1 overall
This message was automatically generated. |
slfan1989
approved these changes
Sep 9, 2025
Contributor
Author
|
Thanks! @stoty @slfan1989 @brumi1024 @susheelgupta7 for the review and for the help here! |
p-szucs
pushed a commit
to p-szucs/hadoop
that referenced
this pull request
Oct 22, 2025
* HADOOP-19674. [JDK 17] Add jaxb-runtime
- Since JDK11 javax.xml.bind modules is removed
source: https://docs.oracle.com/en/java/javase/24/migrate/removed-tools-and-components.html#GUID-11F78105-D735-430D-92DD-6C37958FCBC3
- Jetty needs a jaxb-impl to be able to start
- So we provide the jaxb-runtime artifact everywhere where Jetty Server is a dependency
- If some modules excludes jetty-server then most probably jaxb-runtime is not needed for them, so we exclude it.
- ehcache was using different version of jaxb-runtime, so it become excluded
* HADOOP-19674. [JDK 17] Add jaxb-runtime
- Remove not necessary excludes
* HADOOP-19674. [JDK 17] Add jaxb-runtime
- Put back one exclude what seems to be necessary
- Otherwise at and of install we can see the following error:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.5.0:enforce (enforce-banned-dependencies) on project hadoop-client-check-test-invariants:
[ERROR] Rule 1: org.apache.maven.plugins.enforcer.BanDuplicateClasses failed with message:
[ERROR] Duplicate classes found:
[ERROR]
[ERROR] Found in:
[ERROR] org.apache.hadoop:hadoop-client-minicluster:jar:3.5.0-SNAPSHOT:compile
[ERROR] org.apache.hadoop:hadoop-client-runtime:jar:3.5.0-SNAPSHOT:compile
[ERROR] Duplicate classes:
[ERROR] org/apache/hadoop/shaded/com/sun/activation/registries/MimeTypeEntry.class
...
Cannon to right of them,
* HADOOP-19674. [JDK 17] Add jaxb-runtime
- If we exclude the dependency from hadoop common, maybe we should do the same with the hdfs-client
Cannon to left of them,
* HADOOP-19674. [JDK 17] Add jaxb-runtime
- there were 2 empty line at end of file, so I removed one of them to retrigger the build.
Cannon in front of them.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://issues.apache.org/jira/browse/HADOOP-19674
Description of PR
source: https://docs.oracle.com/en/java/javase/24/migrate/removed-tools-and-components.html#GUID-11F78105-D735-430D-92DD-6C37958FCBC3
How was this patch tested?
Unit tests
org.apache.hadoop.yarn.webapp.TestWebApptestorg.apache.hadoop.yarn.webapp.TestWebApptestFor code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?