Skip to content

HDDS-12462. Use exclude rules for defining shaded filesystem jar contents #8008

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
szetszwo merged 12 commits into from
Mar 21, 2025
Merged

HDDS-12462. Use exclude rules for defining shaded filesystem jar contents #8008

Changes from 5 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
0fde989
HDDS-12462. Shade third dependency for ozone client release package
xichen01 Mar 4, 2025
1eadebf
remove org.hadoop.ozone from org exclude
xichen01 Mar 5, 2025
696ecb3
make shade package include org.xerial.snappy
xichen01 Mar 5, 2025
520031e
Shade all the class which JVM or Hadoop can provide
xichen01 Mar 7, 2025
1e45f38
Fix test
xichen01 Mar 7, 2025
efe5471
Fix test
xichen01 Mar 7, 2025
892df45
Fix test
xichen01 Mar 8, 2025
90361b0
shade netty native libraries
xichen01 Mar 14, 2025
7111c6e
Merge branch 'master' into HDDS-12462
xichen01 Mar 14, 2025
b5d92f9
define plugin version in pluginManagement
xichen01 Mar 19, 2025
5c46400
Merge branch 'master' into HDDS-12462
xichen01 Mar 19, 2025
dc217c2
remove hadoop form shaded prefix
xichen01 Mar 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
129 changes: 101 additions & 28 deletions hadoop-ozone/ozonefs-shaded/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,10 @@
<groupId>org.apache.logging.log4j</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>org.junit</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>*</artifactId>
Expand Down Expand Up @@ -102,6 +106,34 @@
<phase>package</phase>
<configuration>
<skip>${maven.shade.skip}</skip>
Copy link
Contributor Author

@xichen01 xichen01 Mar 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adoroszlai @ChenSammi
I have an idea
https://github.com/apache/hadoop/blob/d552bb056c3daec8d540f166c41b463e8ef18645/hadoop-client-modules/hadoop-client-api/pom.xml#L131
For these classes that are not shaded in hadoop, should we choose to have Ozone not include these "hadoop client unshaded classes" because this may cause class conflicts? (exclude them in artifactSet and relocation)

We should have two options for Ozone dependencies

  1. Shade dependencies
  2. Do not shade dependencies, and do not include this dependency in the ozone release package

In this way, the ozone release package will only contain shaded dependencies, and for these unshaded dependencies, Ozone will assume that Hadoop env provides these dependencies.

How do you think ?

Copy link
Contributor

@adoroszlai adoroszlai Mar 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good to me. Hadoop is also not included in the fat jar, so it cannot work without Hadoop anyway.

Copy link
Contributor Author

@xichen01 xichen01 Mar 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a user directly depend on Ozone and use Ozone's Filesystem without introducing Hadoop's package, can the current Ozone fat-jar provide all necessary dependencies?

<artifactSet>
<excludes>
<exclude>com.sun.javadoc:*</exclude>
<exclude>com.sun.jndi:*</exclude>
<exclude>com.sun.management:*</exclude>
<exclude>com.sun.security:*</exclude>
<exclude>com.sun.tools:*</exclude>
<exclude>org.bouncycastle:*</exclude>
<exclude>org.ietf.jgss:*</exclude>
<exclude>org.omg:*</exclude>
<exclude>org.w3c.dom:*</exclude>
<exclude>org.xerial.snappy:*</exclude>
<exclude>org.xml.sax:*</exclude>
</excludes>
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<!-- Exclude android (dependency from io.grpc:grpc-core) to reduce unnecessary dependencies and avoid potential conflicts -->
<exclude>android/**</exclude>
<!-- Hadoop env provides these dependencies.
To prevent conflicts, remove them from Ozone release package.-->
<exclude>javax/annotation/**</exclude>
<exclude>mozilla/public-suffix-list.txt</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.DontIncludeResourceTransformer">
<resources>
Expand All @@ -120,38 +152,67 @@
<relocation>
<pattern>org</pattern>
<shadedPattern>${shaded.prefix}.org</shadedPattern>
<includes>
<include>org.yaml.**.*</include>
<include>org.sqlite.**.*</include>
<include>org.tukaani.**.*</include>
<include>org.bouncycastle.**.*</include>
<include>org.rocksdb.**.*</include>
<include>org.apache.commons.cli.**.*</include>
<include>org.apache.commons.compress.**.*</include>
<include>org.apache.commons.codec.**.*</include>
<include>org.apache.commons.beanutils.**.*</include>
<include>org.apache.commons.collections.**.*</include>
<include>org.apache.commons.digester.**.*</include>
<include>org.apache.commons.io.**.*</include>
<include>org.apache.commons.logging.**.*</include>
<include>org.apache.commons.validator.**.*</include>
<include>org.apache.commons.lang3.**.*</include>
<include>org.sqlite.**.*</include>
<include>org.apache.thrift.**.*</include>
</includes>
<excludes>
<exclude>org.apache.hadoop.**</exclude>
<exclude>org.apache.log4j.**</exclude>
<exclude>org.apache.ozone.**</exclude>
<exclude>org.apache.ratis.**</exclude>
<!-- These packages are the ones that are not shaded in the hadoop release package.
To prevent conflicts, we do not include these packages in the ozone client release package.
Ozone will use hadoop / JVM to provide these dependencies. -->
<exclude>org.bouncycastle.**</exclude>
<exclude>org.ietf.jgss.*</exclude>
<exclude>org.omg.**</exclude>
<exclude>org.slf4j.**</exclude>
<exclude>org.w3c.dom.**</exclude>
<exclude>org.xerial.snappy.**</exclude>
<exclude>org.xml.sax.**</exclude>
<exclude>org.wildfly.**</exclude>
</excludes>
</relocation>
<relocation>
<pattern>org.apache.ratis</pattern>
Copy link
Contributor Author

@xichen01 xichen01 Mar 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we shade ratis on the <pattern>org</pattern> , will cause exception for the ratis file in the META-INF

image

put ratis shade in a Independent unit can fix this issue

The normal ratis file
image

<shadedPattern>${shaded.prefix}.org.apache.ratis</shadedPattern>
</relocation>
<relocation>
<pattern>com</pattern>
<shadedPattern>${shaded.prefix}.com</shadedPattern>
<includes>
<include>com.google.common.**.*</include>
<include>com.google.gson.**.*</include>
<include>com.codahale.**.*</include>
<include>com.fasterxml.**.*</include>
<include>com.lmax.**.*</include>
<include>com.github.joshelser.**.*</include>
<include>com.twitter.**.*</include>
</includes>
<excludes>
<exclude>com.google.protobuf.**</exclude>
<!-- These packages are the ones that are not shaded in the hadoop release package.
To prevent conflicts, we do not include these packages in the ozone client release package.
Ozone will use hadoop / JVM to provide these dependencies. -->
<exclude>com.sun.javadoc.**</exclude>
<exclude>com.sun.jndi.**</exclude>
<exclude>com.sun.management.**</exclude>
<exclude>com.sun.security.**</exclude>
<exclude>com.sun.tools.**</exclude>
<exclude>com.ibm.security.**</exclude>
</excludes>
</relocation>
<relocation>
<pattern>google</pattern>
<shadedPattern>${shaded.prefix}.google</shadedPattern>
</relocation>
<relocation>
<pattern>net.jcip</pattern>
<shadedPattern>${shaded.prefix}.net.jcip</shadedPattern>
</relocation>
<relocation>
<pattern>javassist</pattern>
<shadedPattern>${shaded.prefix}.javassist</shadedPattern>
</relocation>
<relocation>
<pattern>javax.xml.bind</pattern>
<shadedPattern>${shaded.prefix}.javax.xml.bind</shadedPattern>
</relocation>
<relocation>
<pattern>javax.activation</pattern>
<shadedPattern>${shaded.prefix}.javax.activation</shadedPattern>
</relocation>
<relocation>
<pattern>jakarta.annotation</pattern>
<shadedPattern>${shaded.prefix}.jakarta.annotation</shadedPattern>
</relocation>
<relocation>
<pattern>kotlin</pattern>
Expand All @@ -170,6 +231,18 @@
<shadedPattern>${shaded.prefix}.io</shadedPattern>
<excludes>
<exclude>io!netty!*</exclude>
<!-- Exclude config keys for Hadoop that look like package names -->
<!-- These packages are the ones that are not shaded in the hadoop release package.
To prevent conflicts, we do not include these packages in the ozone client release package.
Ozone will use hadoop / JVM to provide these dependencies. -->
<exclude>io.compression.**</exclude>
<exclude>io.file.buffer.size</exclude>
<exclude>io.mapfile.**</exclude>
<exclude>io.map.index</exclude>
<exclude>io.seqfile.**</exclude>
<exclude>io.serializations</exclude>
<exclude>io.skip.checksum.errors</exclude>
<exclude>io.sort.**</exclude>
</excludes>
</relocation>

Expand Down
Loading