Skip to content

HDDS-12462. Use exclude rules for defining shaded filesystem jar contents #8008

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
szetszwo merged 12 commits into from
Mar 21, 2025
Merged

HDDS-12462. Use exclude rules for defining shaded filesystem jar contents #8008

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
0fde989
HDDS-12462. Shade third dependency for ozone client release package
xichen01 Mar 4, 2025
1eadebf
remove org.hadoop.ozone from org exclude
xichen01 Mar 5, 2025
696ecb3
make shade package include org.xerial.snappy
xichen01 Mar 5, 2025
520031e
Shade all the class which JVM or Hadoop can provide
xichen01 Mar 7, 2025
1e45f38
Fix test
xichen01 Mar 7, 2025
efe5471
Fix test
xichen01 Mar 7, 2025
892df45
Fix test
xichen01 Mar 8, 2025
90361b0
shade netty native libraries
xichen01 Mar 14, 2025
7111c6e
Merge branch 'master' into HDDS-12462
xichen01 Mar 14, 2025
b5d92f9
define plugin version in pluginManagement
xichen01 Mar 19, 2025
5c46400
Merge branch 'master' into HDDS-12462
xichen01 Mar 19, 2025
dc217c2
remove hadoop form shaded prefix
xichen01 Mar 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
254 changes: 226 additions & 28 deletions hadoop-ozone/ozonefs-shaded/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@
<properties>
<!-- no tests in this module so far -->
<maven.test.skip>true</maven.test.skip>
<ozone.shaded.native.prefix>org_apache_hadoop_ozone_shaded</ozone.shaded.native.prefix>
Copy link
Contributor

@szetszwo szetszwo Mar 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about removing hadoop, i.e. org_apache_ozone_shaded?

<!-- refer to ratis thirdparty ratis.thirdparty.shaded.native.prefix -->
<ratis.thirdparty.shaded.native.prefix>org_apache_ratis_thirdparty_</ratis.thirdparty.shaded.native.prefix>
<shaded.prefix>org.apache.hadoop.ozone.shaded</shaded.prefix>
Copy link
Contributor

@szetszwo szetszwo Mar 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's also remove hadoop here?

</properties>

Expand All @@ -35,6 +38,30 @@
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-resolver-dns-native-macos</artifactId>
<classifier>osx-x86_64</classifier>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative-boringssl-static</artifactId>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-transport-native-epoll</artifactId>
<classifier>linux-aarch_64</classifier>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-transport-native-epoll</artifactId>
<classifier>linux-x86_64</classifier>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-transport-native-kqueue</artifactId>
<classifier>osx-x86_64</classifier>
</dependency>
<dependency>
<groupId>org.apache.ozone</groupId>
<artifactId>ozone-filesystem-common</artifactId>
Expand Down Expand Up @@ -75,6 +102,10 @@
<groupId>org.apache.logging.log4j</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>org.junit</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>*</artifactId>
Expand Down Expand Up @@ -102,6 +133,52 @@
<phase>package</phase>
<configuration>
<skip>${maven.shade.skip}</skip>
Copy link
Contributor Author

@xichen01 xichen01 Mar 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adoroszlai @ChenSammi
I have an idea
https://github.com/apache/hadoop/blob/d552bb056c3daec8d540f166c41b463e8ef18645/hadoop-client-modules/hadoop-client-api/pom.xml#L131
For these classes that are not shaded in hadoop, should we choose to have Ozone not include these "hadoop client unshaded classes" because this may cause class conflicts? (exclude them in artifactSet and relocation)

We should have two options for Ozone dependencies

  1. Shade dependencies
  2. Do not shade dependencies, and do not include this dependency in the ozone release package

In this way, the ozone release package will only contain shaded dependencies, and for these unshaded dependencies, Ozone will assume that Hadoop env provides these dependencies.

How do you think ?

Copy link
Contributor

@adoroszlai adoroszlai Mar 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good to me. Hadoop is also not included in the fat jar, so it cannot work without Hadoop anyway.

Copy link
Contributor Author

@xichen01 xichen01 Mar 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a user directly depend on Ozone and use Ozone's Filesystem without introducing Hadoop's package, can the current Ozone fat-jar provide all necessary dependencies?

<artifactSet>
<excludes>
<exclude>com.sun.javadoc:*</exclude>
<exclude>com.sun.jndi:*</exclude>
<exclude>com.sun.management:*</exclude>
<exclude>com.sun.security:*</exclude>
<exclude>com.sun.tools:*</exclude>
<exclude>org.ietf.jgss:*</exclude>
<exclude>org.omg:*</exclude>
<exclude>org.w3c.dom:*</exclude>
<exclude>org.xerial.snappy:*</exclude>
<exclude>org.xml.sax:*</exclude>
</excludes>
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<!-- Exclude android (dependency from io.grpc:grpc-core) to reduce unnecessary dependencies and avoid potential conflicts -->
<exclude>android/**</exclude>
<!-- Hadoop env provides these dependencies.
To prevent conflicts, remove them from Ozone release package.-->
<exclude>javax/annotation/**</exclude>
<!-- Filter ratis thirdparty shaded netty_tcnative libs -->
<exclude>META-INF/native/liborg_apache_ratis_thirdparty_netty_resolver_dns_native_macos_x86_64.jnilib</exclude>
<exclude>META-INF/native/liborg_apache_ratis_thirdparty_netty_tcnative_linux_aarch_64.so</exclude>
<exclude>META-INF/native/liborg_apache_ratis_thirdparty_netty_tcnative_linux_x86_64.so</exclude>
<exclude>META-INF/native/liborg_apache_ratis_thirdparty_netty_tcnative_osx_aarch_64.jnilib</exclude>
<exclude>META-INF/native/liborg_apache_ratis_thirdparty_netty_tcnative_osx_x86_64.jnilib</exclude>
<exclude>META-INF/native/liborg_apache_ratis_thirdparty_netty_transport_native_epoll_aarch_64.so</exclude>
<exclude>META-INF/native/liborg_apache_ratis_thirdparty_netty_transport_native_epoll_x86_64.so</exclude>
<exclude>META-INF/native/liborg_apache_ratis_thirdparty_netty_transport_native_kqueue_x86_64.jnilib</exclude>
<exclude>META-INF/native/org_apache_ratis_thirdparty_netty_tcnative_windows_x86_64.dll</exclude>
<!-- Filter unshaded netty_tcnative libs -->
<exclude>META-INF/native/libnetty_resolver_dns_native_macos_x86_64.jnilib</exclude>
<exclude>META-INF/native/libnetty_tcnative_linux_aarch_64.so</exclude>
<exclude>META-INF/native/libnetty_tcnative_linux_x86_64.so</exclude>
<exclude>META-INF/native/libnetty_tcnative_osx_aarch_64.jnilib</exclude>
<exclude>META-INF/native/libnetty_tcnative_osx_x86_64.jnilib</exclude>
<exclude>META-INF/native/libnetty_transport_native_epoll_aarch_64.so</exclude>
<exclude>META-INF/native/libnetty_transport_native_epoll_x86_64.so</exclude>
<exclude>META-INF/native/libnetty_transport_native_kqueue_x86_64.jnilib</exclude>
<exclude>META-INF/native/netty_tcnative_windows_x86_64.dll</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.DontIncludeResourceTransformer">
<resources>
Expand All @@ -120,38 +197,66 @@
<relocation>
<pattern>org</pattern>
<shadedPattern>${shaded.prefix}.org</shadedPattern>
<includes>
<include>org.yaml.**.*</include>
<include>org.sqlite.**.*</include>
<include>org.tukaani.**.*</include>
<include>org.bouncycastle.**.*</include>
<include>org.rocksdb.**.*</include>
<include>org.apache.commons.cli.**.*</include>
<include>org.apache.commons.compress.**.*</include>
<include>org.apache.commons.codec.**.*</include>
<include>org.apache.commons.beanutils.**.*</include>
<include>org.apache.commons.collections.**.*</include>
<include>org.apache.commons.digester.**.*</include>
<include>org.apache.commons.io.**.*</include>
<include>org.apache.commons.logging.**.*</include>
<include>org.apache.commons.validator.**.*</include>
<include>org.apache.commons.lang3.**.*</include>
<include>org.sqlite.**.*</include>
<include>org.apache.thrift.**.*</include>
</includes>
<excludes>
<exclude>org.apache.hadoop.**</exclude>
<exclude>org.apache.log4j.**</exclude>
<exclude>org.apache.ozone.**</exclude>
<exclude>org.apache.ratis.**</exclude>
<!-- These packages are the ones that are not shaded in the hadoop release package.
To prevent conflicts, we do not include these packages in the ozone client release package.
Ozone will use hadoop / JVM to provide these dependencies. -->
<exclude>org.ietf.jgss.*</exclude>
<exclude>org.omg.**</exclude>
<exclude>org.slf4j.**</exclude>
<exclude>org.w3c.dom.**</exclude>
<exclude>org.xerial.snappy.**</exclude>
<exclude>org.xml.sax.**</exclude>
<exclude>org.wildfly.**</exclude>
</excludes>
</relocation>
<relocation>
<pattern>org.apache.ratis</pattern>
Copy link
Contributor Author

@xichen01 xichen01 Mar 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we shade ratis on the <pattern>org</pattern> , will cause exception for the ratis file in the META-INF

image

put ratis shade in a Independent unit can fix this issue

The normal ratis file
image

<shadedPattern>${shaded.prefix}.org.apache.ratis</shadedPattern>
</relocation>
<relocation>
<pattern>com</pattern>
<shadedPattern>${shaded.prefix}.com</shadedPattern>
<includes>
<include>com.google.common.**.*</include>
<include>com.google.gson.**.*</include>
<include>com.codahale.**.*</include>
<include>com.fasterxml.**.*</include>
<include>com.lmax.**.*</include>
<include>com.github.joshelser.**.*</include>
<include>com.twitter.**.*</include>
</includes>
<excludes>
<exclude>com.google.protobuf.**</exclude>
<!-- These packages are the ones that are not shaded in the hadoop release package.
To prevent conflicts, we do not include these packages in the ozone client release package.
Ozone will use hadoop / JVM to provide these dependencies. -->
<exclude>com.sun.javadoc.**</exclude>
<exclude>com.sun.jndi.**</exclude>
<exclude>com.sun.management.**</exclude>
<exclude>com.sun.security.**</exclude>
<exclude>com.sun.tools.**</exclude>
<exclude>com.ibm.security.**</exclude>
</excludes>
</relocation>
<relocation>
<pattern>google</pattern>
<shadedPattern>${shaded.prefix}.google</shadedPattern>
</relocation>
<relocation>
<pattern>net.jcip</pattern>
<shadedPattern>${shaded.prefix}.net.jcip</shadedPattern>
</relocation>
<relocation>
<pattern>javassist</pattern>
<shadedPattern>${shaded.prefix}.javassist</shadedPattern>
</relocation>
<relocation>
<pattern>javax.xml.bind</pattern>
<shadedPattern>${shaded.prefix}.javax.xml.bind</shadedPattern>
</relocation>
<relocation>
<pattern>javax.activation</pattern>
<shadedPattern>${shaded.prefix}.javax.activation</shadedPattern>
</relocation>
<relocation>
<pattern>jakarta.annotation</pattern>
<shadedPattern>${shaded.prefix}.jakarta.annotation</shadedPattern>
</relocation>
<relocation>
<pattern>kotlin</pattern>
Expand All @@ -170,6 +275,18 @@
<shadedPattern>${shaded.prefix}.io</shadedPattern>
<excludes>
<exclude>io!netty!*</exclude>
<!-- Exclude config keys for Hadoop that look like package names -->
<!-- These packages are the ones that are not shaded in the hadoop release package.
To prevent conflicts, we do not include these packages in the ozone client release package.
Ozone will use hadoop / JVM to provide these dependencies. -->
<exclude>io.compression.**</exclude>
<exclude>io.file.buffer.size</exclude>
<exclude>io.mapfile.**</exclude>
<exclude>io.map.index</exclude>
<exclude>io.seqfile.**</exclude>
<exclude>io.serializations</exclude>
<exclude>io.skip.checksum.errors</exclude>
<exclude>io.sort.**</exclude>
</excludes>
</relocation>

Expand All @@ -187,6 +304,87 @@
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>${maven-dependency-plugin.version}</version>
Copy link
Contributor

@adoroszlai adoroszlai Mar 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: version can be omitted, already defined in pluginManagement in root pom.xml.

Copy link
Contributor Author

@xichen01 xichen01 Mar 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

<executions>
<execution>
<id>unpack-dependencies</id>
<goals>
<goal>unpack-dependencies</goal>
</goals>
<phase>validate</phase>
<configuration>
<includeGroupIds>io.netty</includeGroupIds>
<includes>**/META-INF/native/*</includes>
<includeArtifactIds>netty-resolver-dns-native-macos,
netty-tcnative-boringssl-static,
netty-transport-native-epoll,
netty-transport-native-kqueue</includeArtifactIds>
<outputDirectory>${project.build.directory}/classes/</outputDirectory>
<overWriteReleases>true</overWriteReleases>
<overWriteSnapshots>true</overWriteSnapshots>
</configuration>
</execution>
</executions>
</plugin>
<!-- The native libraries need to be named in the same way we relocate the Netty classes via the shade-plugin -->
<!-- We shaded ratis thirdparty, so the dynamic library name here includes both ratis thirdparty shaded and ozone shaded-->
<plugin>
<groupId>com.coderplus.maven.plugins</groupId>
<artifactId>copy-rename-maven-plugin</artifactId>
<version>${copy-rename-maven-plugin.version}</version>
Copy link
Contributor

@adoroszlai adoroszlai Mar 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: please define version in pluginManagement in root pom.xml, similar to other plugins:

ozone/pom.xml

Lines 1331 to 1337 in 8489cc8

<pluginManagement>
<plugins>
<plugin>
<groupId>com.github.ekryd.sortpom</groupId>
<artifactId>sortpom-maven-plugin</artifactId>
<version>${sortpom-maven-plugin.version}</version>
</plugin>

Copy link
Contributor Author

@xichen01 xichen01 Mar 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

<executions>
<execution>
<id>rename-file</id>
<goals>
<goal>rename</goal>
</goals>
<phase>generate-sources</phase>
<configuration>
<fileSets>
<fileSet>
<sourceFile>${project.build.directory}/classes/META-INF/native/libnetty_resolver_dns_native_macos_x86_64.jnilib</sourceFile>
<destinationFile>${project.build.directory}/classes/META-INF/native/lib${ozone.shaded.native.prefix}_${ratis.thirdparty.shaded.native.prefix}netty_resolver_dns_native_macos_x86_64.jnilib</destinationFile>
</fileSet>
<fileSet>
<sourceFile>${project.build.directory}/classes/META-INF/native/libnetty_transport_native_epoll_aarch_64.so</sourceFile>
<destinationFile>${project.build.directory}/classes/META-INF/native/lib${ozone.shaded.native.prefix}_${ratis.thirdparty.shaded.native.prefix}netty_transport_native_epoll_aarch_64.so</destinationFile>
</fileSet>
<fileSet>
<sourceFile>${project.build.directory}/classes/META-INF/native/libnetty_transport_native_epoll_x86_64.so</sourceFile>
<destinationFile>${project.build.directory}/classes/META-INF/native/lib${ozone.shaded.native.prefix}_${ratis.thirdparty.shaded.native.prefix}netty_transport_native_epoll_x86_64.so</destinationFile>
</fileSet>
<fileSet>
<sourceFile>${project.build.directory}/classes/META-INF/native/libnetty_transport_native_kqueue_x86_64.jnilib</sourceFile>
<destinationFile>${project.build.directory}/classes/META-INF/native/lib${ozone.shaded.native.prefix}_${ratis.thirdparty.shaded.native.prefix}netty_transport_native_kqueue_x86_64.jnilib</destinationFile>
</fileSet>
<fileSet>
<sourceFile>${project.build.directory}/classes/META-INF/native/libnetty_tcnative_linux_aarch_64.so</sourceFile>
<destinationFile>${project.build.directory}/classes/META-INF/native/lib${ozone.shaded.native.prefix}_${ratis.thirdparty.shaded.native.prefix}netty_tcnative_linux_aarch_64.so</destinationFile>
</fileSet>
<fileSet>
<sourceFile>${project.build.directory}/classes/META-INF/native/libnetty_tcnative_linux_x86_64.so</sourceFile>
<destinationFile>${project.build.directory}/classes/META-INF/native/lib${ozone.shaded.native.prefix}_${ratis.thirdparty.shaded.native.prefix}netty_tcnative_linux_x86_64.so</destinationFile>
</fileSet>
<fileSet>
<sourceFile>${project.build.directory}/classes/META-INF/native/libnetty_tcnative_osx_aarch_64.jnilib</sourceFile>
<destinationFile>${project.build.directory}/classes/META-INF/native/lib${ozone.shaded.native.prefix}_${ratis.thirdparty.shaded.native.prefix}netty_tcnative_osx_aarch_64.jnilib</destinationFile>
</fileSet>
<fileSet>
<sourceFile>${project.build.directory}/classes/META-INF/native/libnetty_tcnative_osx_x86_64.jnilib</sourceFile>
<destinationFile>${project.build.directory}/classes/META-INF/native/lib${ozone.shaded.native.prefix}_${ratis.thirdparty.shaded.native.prefix}libnetty_tcnative_osx_x86_64.jnilib</destinationFile>
</fileSet>
<fileSet>
<sourceFile>${project.build.directory}/classes/META-INF/native/netty_tcnative_windows_x86_64.dll</sourceFile>
<destinationFile>${project.build.directory}/classes/META-INF/native/${ozone.shaded.native.prefix}_${ratis.thirdparty.shaded.native.prefix}netty_tcnative_windows_x86_64.dll</destinationFile>
</fileSet>
</fileSets>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
1 change: 1 addition & 0 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@
<commons-text.version>1.12.0</commons-text.version>
<commons-validator.version>1.6</commons-validator.version>
<compile-testing.version>0.21.0</compile-testing.version>
<copy-rename-maven-plugin.version>1.0</copy-rename-maven-plugin.version>
<curator.version>4.2.0</curator.version>
<cyclonedx.version>2.9.1</cyclonedx.version>
<!-- the version of Hadoop declared in the version resources; can be overridden
Expand Down