Support Hudi MOR snapshot

[willzgw]

Description

• Support Hudi MOR Tables #5132
• Support MOR snapshot querying for Hudi tables #14432

Non-technical explanation

• Support Hudi MOR Tables
• Support MOR snapshot querying for Hudi tables

Release notes

( ) Support Hudi MOR Tables
( ) Support MOR snapshot querying for Hudi tables

# Section
* Support Hudi MOR Tables. ({issue}`5132`)
* Support MOR snapshot querying for Hudi tables. ({issue}`14432`)

[codope]

@willzgw Thanks for adding the MOR snapshot query support. This would be very helpful for the community. Please edit the PR description accordingly and point to issue#14432.
Also, if you could add integ tests for MOR snapshot query that would be great. You can extend the existing setup. Note that you will have to add an upsert operation (folowwing after insert) in TpchHudiTablesInitializer#load.

[willzgw]

I am trying to add some unit test cases for this PR.

[codope]

@willzgw This is looking more and more in shape. It would be great if you could add a few tests as well.

[codope]

HudiSplit already carry configuration. See if we can pass from the split in page source provider.

[codope]

Just for my understanding, what's the intention of setting compression codecs here?

[ebyhr]

Could you rebase on upstream to resolve conflicts and add product tests?

[codope]

@willzgw I have a PR for async split processing #16034 that improves the Hudi connector performance. On top of it, I have MoR support in my fork - https://github.com/codope/trino/tree/mor-snapshot
Can you please review the code changes in both? Eventually, if we go with this PR, we would want it to be rebased with #16034

[willzgw]

@willzgw I have a PR for async split processing #16034 that improves the Hudi connector performance. On top of it, I have MoR support in my fork - https://github.com/codope/trino/tree/mor-snapshot Can you please review the code changes in both? Eventually, if we go with this PR, we would want it to be rebased with #16034

Hi @codope Let's continue with this PR, I will rebase this branch with #16034.
And, I recently did something similar to your PR on shopee's Trino to optimize the performance of hudi-connector, yours is much better, I just modified HudiReadOptimizedDirectoryLister and the batch-size of the load partition is a fixed value and not configurable.

[codope]

@codope It looks like even the latest version of HoodieMergeOnReadSnapshotReader still depends on Hadoop FileSystem. Do you have plans to address that, or will need a custom version in Trino?

@electrum The current version of HoodieMergeOnReadSnapshotReader only depends on Configuration of Hadoop. Client code need not create FileSystem. There shouldn't be any FileSystem imports in trino-hudi code. The next step is to create HudiFileSystem and HudiPath abstraction to remove Hadoop from hudi - https://issues.apache.org/jira/browse/HUDI-6497. This is planned for 1.0.0.

[zhuzhengjun01]

This pr is very helpful to me, but when we use 433 version and I tried to cherry-pick this pr found that there are many conflicts after this pr 【#18241 】，especially the RecordCursor is not used after this pr.

[mosabua]

👋 @willzgw - this PR has become inactive. We hope you are still interested in working on it. Please let us know, and we can try to get reviewers to help with that.

We're working on closing out old and inactive PRs, so if you're too busy or this has too many merge conflicts to be worth picking back up, we'll be making another pass to close it out in a few weeks.

Also cc @codope @brandyml

[willzgw]

👋 @willzgw - this PR has become inactive. We hope you are still interested in working on it. Please let us know, and we can try to get reviewers to help with that.

We're working on closing out old and inactive PRs, so if you're too busy or this has too many merge conflicts to be worth picking back up, we'll be making another pass to close it out in a few weeks.

Also cc @codope @brandyml

Hi @mosabua
I am still working on this. I found some bugs of the HudiTableFileSystemView and this PR.
I will raise an issue to follow these problems and fix them in this PR.

[mosabua]

Great .. feel free to reach out to us if you need help. @codope can help on the Hudi side..

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Guowei Zhang.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email email@example.com
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[mosabua]

Looks like github still thinks you have a lot of conflicts to resolve @willzgw

[willzgw]

Looks like github still thinks you have a lot of conflicts to resolve @willzgw

Yeah, still in progress.

[pravin1406]

@willzgw @codope Any timeline for this support ?

[mosabua]

I am not sure if it is worth refactoring this PR. Currently @yihua and @codope are working towards a new Hudi release that removes the Hadoop dependencies. Once that is out they will work on Hudi connector updates that will probably change the codebase quite a bit. So maybe its worth waiting to see if MOR snapshot support will come out of these PRs or just be easier to implement from scratch then.

[willzgw]

I am not sure if it is worth refactoring this PR. Currently @yihua and @codope are working towards a new Hudi release that removes the Hadoop dependencies. Once that is out they will work on Hudi connector updates that will probably change the codebase quite a bit. So maybe its worth waiting to see if MOR snapshot support will come out of these PRs or just be easier to implement from scratch then.

@mosabua
Thanks for the heads up. I will evaluate and follow up once HUDI is released.

[github-actions]

This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua

[github-actions]

Closing this pull request, as it has been stale for six weeks. Feel free to re-open at any time.

[ebyhr]

Currently @yihua and @codope are working towards a new Hudi release that removes the Hadoop dependencies

@yihua @codope Did Hudi 0.15.0 remove all Hadoop dependencies? I see https://issues.apache.org/jira/browse/HUDI-6497 contains 0.15.0 as a fix version.

[pravin1406]

@codope @willzgw Did we guys move forward on this ?

[codope]

Hey @pravin1406, I am not actively contributing. For MOR query support, I would recommend that you can check Onehouse's fork of Trino -- https://github.com/onehouseinc/trino. It does have MOR support and also integrates with some of the indexes.
cc @yihua

[yihua]

Hi @pravin1406 , as @codope mentioned, this Trino fork has the latest support on Hudi MOR table https://github.com/onehouseinc/trino . We're going to upstream the changes to Trino OSS. Stay tuned.

[pravin1406]

@yihua @codope Alright will have a go at onehousetrino then. Just wanna know one more thing, our production workloads are currently at hudi version 0.14.1. Will this trino fork for them ? or do i need to plan an upgrade before that? also what trino base version does onehousetrino use ?

[yihua]

Hi @pravin1406 ,

our production workloads are currently at hudi version 0.14.1. Will this trino fork for them ? or do i need to plan an upgrade before that?

The Trino fork (https://github.com/onehouseinc/trino) works on all table versions, including the Hudi tables created by Hudi 0.14.1.

also what trino base version does onehousetrino use ?

It is based on top of OSS Trino 472.