Skip to content

[minor]The Kryo of Hudi conflicts with the Kryo of Flink #6522

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cuibo01 wants to merge 1 commit into from
Closed

[minor]The Kryo of Hudi conflicts with the Kryo of Flink #6522

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion hudi-common/src/main/java/org/apache/hudi/common/util/SerializationUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,9 @@ public Kryo newKryo() {
// ensure that kryo doesn't fail if classes are not registered with kryo.
kryo.setRegistrationRequired(false);
// This would be used for object initialization if nothing else works out.
kryo.setInstantiatorStrategy(new Kryo.DefaultInstantiatorStrategy(new StdInstantiatorStrategy()));
Kryo.DefaultInstantiatorStrategy instantiatorStrategy = new Kryo.DefaultInstantiatorStrategy();
instantiatorStrategy.setFallbackInstantiatorStrategy(new StdInstantiatorStrategy());
kryo.setInstantiatorStrategy(instantiatorStrategy);
// Handle cases where we may have an odd classloader setup like with libjars
Copy link
Contributor

@danny0405 danny0405 Aug 29, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already add some shade pattern for the kryo in hudi-flink-bundle jar in #4251, does that solve your conflicts here ?

Copy link
Contributor Author

@cuibo01 cuibo01 Aug 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes，now we use hudi-flink，and the hudi only needs to modify the code and does not need to use the bundle jar

Copy link
Member

@xushiyan xushiyan Nov 4, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes，now we use hudi-flink，and the hudi only needs to modify the code and does not need to use the bundle jar

@cuibo01 can you clarify the situation please? do you mean hudi flink bundle jar will have kryo lib conflicts? we have shaded kryo for every module

hudi/pom.xml

Line 450 in b2a84ee

<shadedPattern>org.apache.hudi.com.esotericsoftware.kryo.</shadedPattern>

can you point out in your environment which dependency brings in kryo that caused conflicts?

// for hadoop
kryo.setClassLoader(Thread.currentThread().getContextClassLoader());
Expand Down