Skip to content

[HUDI-3983] ClassNotFoundException when using hudi-spark-bundle to write table with hbase index #6715

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
codope merged 3 commits into from
Sep 21, 2022
Merged

[HUDI-3983] ClassNotFoundException when using hudi-spark-bundle to write table with hbase index #6715

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1bbc7fd
[HUDI-3983] ClassNotFoundException when using hudi-spark-bundle to wr…
xicm Sep 19, 2022
888e4f0
Merge branch 'master' into HUDI-3983
xicm Sep 19, 2022
65ea048
remove commented conf
xicm Sep 21, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 0 additions & 7 deletions hudi-common/src/main/resources/hbase-site.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1699,13 +1699,6 @@ possible configurations would overwhelm and obscure the important.
Implementation of the status publication with a multicast message.
</description>
</property>
<property>
<name>hbase.status.listener.class</name>
<value>org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener</value>
Copy link
Contributor

@yihua yihua Sep 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xicm could you rename the class here with the shaded pattern, instead of removing the config?

Copy link
Contributor Author

@xicm xicm Sep 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yihua If we rename the class with the shaded name, there will be a ClassNotFoundException when referencing hudi-common.

Copy link
Contributor Author

@xicm xicm Sep 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean we use shaded name with bundle jar only. If the dependence we use is hudi-common and the listener class comes from original hbase-client, in this case we will get an exception.

<description>
Implementation of the status listener with a multicast message.
</description>
</property>
<property>
<name>hbase.status.multicast.address.ip</name>
<value>226.1.1.3</value>
Expand Down
3 changes: 3 additions & 0 deletions packaging/hudi-spark-bundle/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -300,6 +300,9 @@
<exclude>META-INF/services/javax.*</exclude>
<exclude>**/*.proto</exclude>
<exclude>hbase-webapps/**</exclude>
<!-- hbase-default.xml comes from hbase-common, hbase related classes used in hudi are in shaded
pattern, the default classes in hbase-default.xml can cause ClassNotFoundException. -->
<exclude>hbase-default.xml</exclude>
Copy link
Contributor

@wulei0302 wulei0302 Oct 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xicm Do we have the same problem on the Flink side?

Copy link
Contributor Author

@xicm xicm Oct 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wulei0302 I think flink has this problem as well.

</excludes>
</filter>
</filters>
Expand Down