Skip to content

Upgrade Hudi version to 0.14.0#19810

Merged
electrum merged 3 commits intotrinodb:masterfrom
yihua:hudi-upgrade-0-14-0
Dec 13, 2023
Merged

Upgrade Hudi version to 0.14.0#19810
electrum merged 3 commits intotrinodb:masterfrom
yihua:hudi-upgrade-0-14-0

Conversation

@yihua
Copy link
Copy Markdown
Member

@yihua yihua commented Nov 18, 2023

Description

This PR upgrades Hudi version to 0.14.0 to make sure the Hudi connector can read tables with new Hudi table version created by Hudi 0.14.0 release.

Hudi metadata table (MDT) is disabled in the tests as the support of reading MDT is broken after the removal of Hudi dependencies from compile time (which happened some time back #17392).

Additional context and related issues

Hudi tables written by 0.14.0 with the new table version can be read.

Release notes

(x) This is not user-visible or is docs only, and no release notes are required.

@cla-bot
Copy link
Copy Markdown

cla-bot bot commented Nov 18, 2023

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

@github-actions github-actions bot added the hudi Hudi connector label Nov 18, 2023
Copy link
Copy Markdown
Member

@ebyhr ebyhr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please submit CLA and fix CI failure:

Error:  Errors: 
Error:    TestHudiConnectorTest>AbstractTestQueryFramework.init:111->createQueryRunner:35 » NoClassDefFound Could not initialize class org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex
Error:    TestHudiCopyOnWriteMinioConnectorSmokeTest>AbstractTestQueryFramework.init:111->createQueryRunner:38 » NoClassDefFound org/apache/hadoop/hbase/util/Bytes
Error:    TestHudiMergeOnReadMinioConnectorSmokeTest>AbstractTestQueryFramework.init:111->createQueryRunner:38 » NoClassDefFound Could not initialize class org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex

@yihua
Copy link
Copy Markdown
Member Author

yihua commented Nov 22, 2023

Please submit CLA and fix CI failure:

Error:  Errors: 
Error:    TestHudiConnectorTest>AbstractTestQueryFramework.init:111->createQueryRunner:35 » NoClassDefFound Could not initialize class org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex
Error:    TestHudiCopyOnWriteMinioConnectorSmokeTest>AbstractTestQueryFramework.init:111->createQueryRunner:38 » NoClassDefFound org/apache/hadoop/hbase/util/Bytes
Error:    TestHudiMergeOnReadMinioConnectorSmokeTest>AbstractTestQueryFramework.init:111->createQueryRunner:38 » NoClassDefFound Could not initialize class org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex

I've submitted my CLA 2 days ago. I'm still fixing the test failures due to the version upgrade.

@yihua yihua force-pushed the hudi-upgrade-0-14-0 branch from 7814db3 to 5d88149 Compare November 30, 2023 17:55
@cla-bot
Copy link
Copy Markdown

cla-bot bot commented Nov 30, 2023

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

@yihua yihua force-pushed the hudi-upgrade-0-14-0 branch from 5d88149 to 96cba3a Compare November 30, 2023 17:57
@cla-bot
Copy link
Copy Markdown

cla-bot bot commented Nov 30, 2023

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

@yihua
Copy link
Copy Markdown
Member Author

yihua commented Nov 30, 2023

@ebyhr I fixed the tests in Hudi connector. I have to disable Hudi metadata table (MDT) in the tests as the support of reading MDT is broken after the removal of Hudi dependencies from compile time (which happened some time back #17392).

I'll follow up with a separate PR to fix MDT support in Hudi.

@cla-bot
Copy link
Copy Markdown

cla-bot bot commented Nov 30, 2023

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

@yihua yihua force-pushed the hudi-upgrade-0-14-0 branch from c26f4d7 to 9ecbff1 Compare November 30, 2023 23:30
@cla-bot cla-bot bot added the cla-signed label Nov 30, 2023
@yihua yihua assigned yihua and unassigned yihua Dec 2, 2023
@yihua
Copy link
Copy Markdown
Member Author

yihua commented Dec 6, 2023

@ebyhr could you help review this PR again?

@ebyhr ebyhr self-requested a review December 7, 2023 00:07
@electrum electrum merged commit 79bc8fa into trinodb:master Dec 13, 2023
@github-actions github-actions bot added this to the 435 milestone Dec 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-signed hudi Hudi connector

Development

Successfully merging this pull request may close these issues.

3 participants