Improve Hudi connector performance

[codope]

Description

Previously, the query execution would wait for all the split generation to complete and splits were loaded in a single thread. With this PR, split generation and processing can happen asynchronously.

Additional context and related issues

Fixes apache/hudi#7643
Fixes #15564

Release notes

(x) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text:

# Section
* Fix some things. ({issue}`issuenumber`)

[findepi]

martint requested review from findepi and hashhar 2 weeks ago

@findinpath and @electrum should be the reviewers here.

[findinpath]

There are a lot of changes in this PR and (besides TestHudiConfig) no tests to accompanying them. Can you consider adding more test coverage for your changes?

[codope]

@findinpath @ksoullpwk Thanks for reviewing the PR. I will address your comments this week.

[codope]

There are a lot of changes in this PR and (besides TestHudiConfig) no tests to accompanying them. Can you consider adding more test coverage for your changes?

@findinpath Can you please review again?
Would appreciate if you can help with what kind of tests we can add here. There is no new functionality. This PR is a performance enhancement. Sure, there is some change in the behavior but existing tests will go through that path. So functional correctness is already tested.

[tooptoop4]

🦗

[codope]

@findinpath @electrum gentle ping to review the PR.

[vinothchandar]

cc @umehrot2 from Athena/EMR and @bvaradar from Robinhood, who were asking about this.

[vinothchandar]

@ksoullpwk Could you please review again or one of the trino maintainers chime in? We need to get this in, so Hudi users can migrate to the hudi connector away from hive connector.

[ksoullpwk]

As I see, some configs have changed to this PR. Don't forget to update the document https://github.com/trinodb/trino/blob/master/docs/src/main/sphinx/connector/hudi.rst.

[electrum]

Apologies for the delay on this. Can you rebase? We completed the removal of Hadoop from the Hudi connector and can now start merging these other changes.

[codope]

@electrum Thanks for removing Hadoop from Hudi connector. We are also working on exposing APIs in Hudi such that Hadoop is not required and then there won't be need to duplicate classes in hudi-trino module. Meanwhile, I have rebased and this PR is ready for another review.

[codope]

@electrum gentle ping

[codope]

@electrum I've rebased and addressed all comments. Please take a pass again.

[codope]

@trinodb/maintainers Please review this PR.

[electrum]

A few comments, otherwise looks good

[codope]

Thanks @electrum @ksoullpwk for quick feedback. I've updated the docs and addressed other comments. PR is ready to merge.

[yx-keith]

this pr use hive metastore's getPartition function when load parition info, it will take a long time if there are many partitions.

[yx-keith]

I optimized it in 406 version, brings 40x to 50x performance improment in my environment

743c414

[codope]

I optimized it in 406 version, brings 40x to 50x performance improment in my environment

743c414

@yx-keith This is great! Do you want to raise a PR so that we can port over your changes to the latest Trino version? I can help with the review. If you have some benchmarks on a large dataset (possibly thousands of partitions), feel free to share it as well. Once you have the PR, I can also help with benchmarking against TPC-DS dataset which has about ~2k partitions (good enough to bring out the perf diff).

[yx-keith]

I optimized it in 406 version, brings 40x to 50x performance improment in my environment
743c414

@yx-keith This is great! Do you want to raise a PR so that we can port over your changes to the latest Trino version? I can help with the review. If you have some benchmarks on a large dataset (possibly thousands of partitions), feel free to share it as well. Once you have the PR, I can also help with benchmarking against TPC-DS dataset which has about ~2k partitions (good enough to bring out the perf diff).

@codope
yes, thank you, Is it necessary to merge my code into master branch and then raise a PR?
I use table and dataset in this issue:
apache/hudi#7643
there are 744 partitions

[codope]

@yx-keith I would suggest to rebase your changes on top of latest master code and run your test with the dataset (744 partitions) that you have again. Ideally, metastore rpc happens for only matching partitions. It was fixed in

trino/plugin/trino-hudi/src/main/java/io/trino/plugin/hudi/partition/HudiPartitionInfoLoader.java

Lines 67 to 68 in 3661df8

	if (hudiPartitionInfo.doesMatchPredicates() || partitionName.equals("")) {
	List<HivePartitionKey> partitionKeys = hudiPartitionInfo.getHivePartitionKeys();

Also, check this comment where we discussed the additional metastore calls - #16034 (comment)

[yx-keith]

> @yx-keith I would suggest to rebase you changes on top of latest master code and run your test with the dataset (744 partitions) that you have again. Ideally, metastore rpc happens for only matching partitions. It was fixed in

trino/plugin/trino-hudi/src/main/java/io/trino/plugin/hudi/partition/HudiPartitionInfoLoader.java

Lines 67 to 68 in 3661df8

	if (hudiPartitionInfo.doesMatchPredicates() || partitionName.equals("")) {
	List<HivePartitionKey> partitionKeys = hudiPartitionInfo.getHivePartitionKeys();

Also, check this comment where we discussed the additional metastore calls - #16034 (comment)

ok, this is my pr
#18511
@codope