async and concurrently to process hudi table partitions in batches and create splits#18511
async and concurrently to process hudi table partitions in batches and create splits#18511yx-keith wants to merge 4 commits intotrinodb:masterfrom
Conversation
There was a problem hiding this comment.
@tooptoop4 thank you for pointing this out, I have fixed it
There was a problem hiding this comment.
All * are changed now, I think
|
@tooptoop4 @codope @martint |
|
@yx-keith apologies for the delay. Can you rebase this PR? |
|
Hi @yx-keith - When this will be merged. |
|
This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua |
|
This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua |
|
Closing this pull request, as it has been stale for six weeks. Feel free to re-open at any time. |
5 participants
Description
With this PR, generate partition will happen in batches and concurrently, at the same time, generate splits asynchronous and concurrently.
can improve performance more than 5 times in trino 406 version in tpcds query with 1800 partitions
Additional context and related issues
fixs apache/hudi#7643
fixs #15564
Release notes
( ) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text: