[MINOR] Build failed using master #9726
Conversation
codope
changed the title
hudi-common/pom.xml
Outdated
| <version>${disruptor.version}</version> | ||
| </dependency> | ||
|
|
||
| <dependency> |
There was a problem hiding this comment.
Shouldn't this already be included? Perhaps the build is failing for some profile. Can you share the build command?
There was a problem hiding this comment.
mvn clean package install -DskipTests -Dspark3.3 -Dscala-2.12 -Dhadoop.version=3.3.0 -Dhive.version=3.1.2 -Phudi-platform-service -Dcheckstyle.skip=true -Drat.skip=true -Pm1-mac -Pthrift-gen-source -Pflink1.14
It's due to -Dhadoop.version=3.3.0?
There was a problem hiding this comment.
I just tried it, this problem will occur when using Hadoop 3.3.0 and Spark3.2+(3.2/3.3/3.4).
…endencies" This reverts commit 1804bbe.
…issing jackson dependencies when using hadoop 3.3 version.
pom.xml
Outdated
| <artifactId>jackson-core-asl</artifactId> | ||
| <version>1.9.13</version> | ||
| </dependency> | ||
| </dependencies> |
There was a problem hiding this comment.
Do we support hadoop 3.x compile?
There was a problem hiding this comment.
We have been using -Dhadoop.version=3.3.0 for a long time and there have been no problems.
There was a problem hiding this comment.
We have a long suspended PR #5786 before for Hadoop 3.x compatibility, not sure how it works for your env.
There was a problem hiding this comment.
PR #5786 is adapted to the hadoop3.x environment, but if parquet.version is configured with version 0.12.2, the problem of missing jackson package will still occur. The currently submitted PR #9726 is mainly to solve the problem of compilation failure caused by the lack of jackson dependency package in the Hadoop 3 environment. It is not adapted to the hadoop 3 environment.
As for adapting hadoop3, it may take longer to fully test and verify.
There was a problem hiding this comment.
Got it, jackson is a jar that is very easy to conflict.
There was a problem hiding this comment.
@Forus0322 Can you double check jackson-core-asl is included for hadoop3.3 profile only?
|
@danny0405 @codope are you good with the changes? |
codope
left a comment
codope
left a comment
There was a problem hiding this comment.
@Forus0322 Left one comment for clarification. Can you please rebase as well?
pom.xml
Outdated
| <artifactId>jackson-core-asl</artifactId> | ||
| <version>1.9.13</version> | ||
| </dependency> | ||
| </dependencies> |
There was a problem hiding this comment.
@Forus0322 Can you double check jackson-core-asl is included for hadoop3.3 profile only?
|
@codope Hi, I reanalyzed the problem. The essence is that parquet 1.10.1 contains the org.codehaus.jackson dependency package, and parquet 1.12.0 version does not contain the org.codehaus.jackson dependency package. Secondly, the JsonEncoder class under the hudi-common-base package uses the org.codehaus.jackson dependency package, and hudi-common-base references the parquet-avro dependency package, so when compiling with parquet.version 1.10.1, There is no problem; but when using parquet.version 1.12.0, there will be a missing JsonEncoder dependency problem. Therefore, this issue does not need to be included in the current PR, just specify parquet.version 1.10.1. But if you want to use parquet.version 1.12.0, you need to add an additional org.codehaus.jackson dependency. The current PR can modify the profile, not hadoop3.3, but add the org.codehaus.jackson dependency to any profile using version 1.12.0. Do you think it is necessary to modify it again?
|
|
If we are talking about the version coflicts between the explcity introduced jackson jar and the parquet jar, it is actually a bug, should have a fix. |
|
@danny0405 I have added jackson dependency to all profiles using parquet.version 1.12.x. How about modifying it like this? |
6 participants
Change Logs
Add jackson dependencies.
Fixes #9725
Impact
Fix master build
Risk level (write none, low medium or high below)
none
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change
ticket number here and follow the instruction to make
changes to the website.
Contributor's checklist