[HUDI-8179] Upgrade hudi flink connector to 1.20.0 #11966

guptashailesh92 · 2024-09-19T07:46:36Z

Change Logs

Upgrade hudi flink connector to 1.20.0

Impact

Added hudi flink connector for 1.20.0.

Risk level (write none, low medium or high below)

medium

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".

none

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

guptashailesh92 · 2024-09-25T14:10:56Z

@danny0405 please let me know if anything else is required here.

danny0405 · 2024-09-26T00:57:50Z

@guptashailesh92 Thanks for the contribution, have you already referenced #11779 for the changes? Especially we need to upload the docker images for bundle validation tests, only Hudi PMC got permission do do this now.

guptashailesh92 · 2024-09-26T07:27:26Z

Yes, i used the referenced cr: #11779 for raising this CR.

danny0405 · 2024-09-27T01:14:52Z

pom.xml

    <flink.avro.version>1.11.1</flink.avro.version>
    <flink.format.parquet.version>1.13.1</flink.format.parquet.version>
-    <flink.connector.kafka.version>3.2.0-1.19</flink.connector.kafka.version>
+    <!-- check kafka version -->


Is this right?

I see the discussion regarding flink connector kafka version to 1.20, but it has not been release yet.
apache/flink-connector-kafka#111

CTTY

Note:

hudi-flink1.20.x is a copy of existing hudi-flink1.19.x with only version changes, there is no changes made to adapt to Flink upgrade.
This change has passed EMR Hudi integtest.

Left some minor comments around avro version

CTTY · 2024-10-03T17:53:56Z

scripts/release/deploy_staging_jars.sh

 "-Dscala-2.12 -Dflink1.17 -Davro.version=1.11.1 -pl packaging/hudi-flink-bundle -am"
 "-Dscala-2.12 -Dflink1.18 -Davro.version=1.11.1 -pl packaging/hudi-flink-bundle -am"
 "-Dscala-2.12 -Dflink1.19 -Davro.version=1.11.1 -pl packaging/hudi-flink-bundle -am"
+"-Dscala-2.12 -Dflink1.20 -Davro.version=1.11.1 -pl packaging/hudi-flink-bundle -am"


avro version here should be 1.11.3

updated. thanks for the review.

CTTY · 2024-10-03T17:55:44Z

pom.xml

+    <hudi.flink.module>hudi-flink1.20.x</hudi.flink.module>
+    <flink.bundle.version>1.20</flink.bundle.version>
    <!-- This is fixed to match with version from flink-avro -->
    <flink.avro.version>1.11.1</flink.avro.version>


We should update this avro version as well since we are changing the default flink version to 1.20

yihua

I've built and uploaded the following bundle validation docker images to the Docker Hub, based on the scripts in this PR:

yihua · 2024-10-08T02:25:02Z

packaging/bundle-validation/ci_run.sh

  HIVE_VERSION=3.1.3
  DERBY_VERSION=10.14.1.0
-  FLINK_VERSION=1.19.0
+  FLINK_VERSION=1.20.0


The image tag IMAGE_TAG needs to be updated to properly test Hudi Flink bundle on the latest Flink version.

This reverts commit f5c8c45.

danny0405 · 2024-10-10T11:40:05Z

@guptashailesh92 Hi, the TestHoodieFlinkQuickstart.testHoodieFlinkQuickstart has been failing while swithing to 1.20 release, I have debug it a little, here is the info I can share:

local IDEA run can pass successfully;
use cmd mvn test -Punit-tests -D"scala-2.12" -D"flink1.20" -pl hudi-examples/hudi-examples-flink to trigger the test can reproduce the error;
I enabled the -Dmaven.surefire.debug for the mvn cmd and start a remote debug, and finds out that there are fatal errors arised by the RPC calls, the mini-cluster was indeed closed before the job was submitted.

But I still didn't find the root cause, maybe you can spend with your spare time to do some research.

…f reload4j

danny0405 · 2024-10-11T03:55:45Z

@guptashailesh92 I have addressed the issues of the test failure, it is because we have conflict dependency of reload4j in the classpath, it includes the legacy slf4j MDC adapter Reload4jMDCAdapter which triggers the NPE for invocation MDC.setContextMap, the NPE then causes the Flink resource manager to crush then the mini-cluster is shutdown.

The solution is to remove the reload4j related jars because we already have log4j-1.2-api jar dependency in module hudi-tests-common.

guptashailesh92 · 2024-10-11T06:05:20Z

thank you @danny0405, I am little occupied with things, haven't been able to spend much time.

hudi-bot · 2024-10-11T08:12:45Z

CI report:

c3b0b8f UNKNOWN
3fc2c86 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

addressed

github-actions bot added the size:XL PR with lines of changes > 1000 label Sep 19, 2024

danny0405 self-assigned this Sep 19, 2024

danny0405 reviewed Sep 27, 2024

View reviewed changes

danny0405 added the version-upgrade label Sep 27, 2024

CTTY reviewed Oct 3, 2024

View reviewed changes

yihua reviewed Oct 8, 2024

View reviewed changes

yihua previously requested changes Oct 8, 2024

View reviewed changes

Shailesh Gupta and others added 7 commits October 10, 2024 13:26

Upgrade hudi flink connector to 1.20.0

8326721

Update docker image for flink build

53acd5e

typo: Update flink verison

e6ce566

Address review comments

0deac41

Fix the Integration test of Flink to use 1.20

397bcb3

Add flink mini cluster resource definition for example test

f5c8c45

Fix the flink1.19 profile properties

c3b0b8f

danny0405 force-pushed the HUDI-8179 branch from 3cb36d6 to c3b0b8f Compare October 10, 2024 05:26

danny0405 added 2 commits October 10, 2024 13:30

Revert "Add flink mini cluster resource definition for example test"

bce19f6

This reverts commit f5c8c45.

Fix the job await of TestHoodieFlinkQuickstart

8f722d9

fix azure pipiline flink modules and remove the dependency conflict o…

5ff3cda

…f reload4j

add back slf4j-log4j12 dependency for spark profiles

b1b5932

cherry-pick HUDI-7930 to 1.19 and 1.20 releases

3fc2c86

danny0405 added the engine:flink Flink integration label Oct 11, 2024

danny0405 approved these changes Oct 12, 2024

View reviewed changes

danny0405 merged commit 7530e4f into apache:master Oct 12, 2024

danny0405 mentioned this pull request Oct 12, 2024

[HUDI-7930] Update Flink 1.19 to support Array and Map of Rows #12083

Closed

4 tasks

[HUDI-8179] Upgrade hudi flink connector to 1.20.0 #11966

[HUDI-8179] Upgrade hudi flink connector to 1.20.0 #11966

Uh oh!

Conversation

guptashailesh92 commented Sep 19, 2024

Change Logs

Impact

Risk level (write none, low medium or high below)

Documentation Update

Contributor's checklist

Uh oh!

guptashailesh92 commented Sep 25, 2024

Uh oh!

danny0405 commented Sep 26, 2024

Uh oh!

guptashailesh92 commented Sep 26, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CTTY left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yihua left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danny0405 commented Oct 10, 2024

Uh oh!

danny0405 commented Oct 11, 2024

Uh oh!

guptashailesh92 commented Oct 11, 2024

Uh oh!

hudi-bot commented Oct 11, 2024

CI report:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants