[HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path #10836
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nsivabalan
changed the title
nsivabalan
reviewed
Mar 7, 2024
Contributor
nsivabalan
left a comment
nsivabalan
left a comment
There was a problem hiding this comment.
Can you add reference to when this was added (offending commit) in the PR desc.
jonvex
commented
Mar 8, 2024
...udi-client-common/src/main/java/org/apache/hudi/index/bucket/ConsistentBucketIndexUtils.java
Show resolved
Hide resolved
danny0405
reviewed
Mar 8, 2024
| } | ||
|
|
||
| writer.getWriteStatuses.asScala.iterator | ||
| }), SQLConf.get).collect() |
Contributor
Author
|
@hudi-bot run azure |
Contributor
Author
|
|
Collaborator
danny0405
approved these changes
Mar 12, 2024
xushiyan
pushed a commit
that referenced
this pull request
Apr 18, 2024
… path (#10836) * get rid of collect in row writer clustering * fix race condition * add logging --------- Co-authored-by: Jonathan Vexler <=>
xushiyan
pushed a commit
that referenced
this pull request
Apr 18, 2024
… path (#10836) * get rid of collect in row writer clustering * fix race condition * add logging --------- Co-authored-by: Jonathan Vexler <=>
yihua
pushed a commit
that referenced
this pull request
May 14, 2024
… path (#10836) * get rid of collect in row writer clustering * fix race condition * add logging --------- Co-authored-by: Jonathan Vexler <=>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change Logs
Collect of all the write statuses will be a ton of data on the driver. We can avoid this because the data is parallelized again immediately after the collect.
This is in the original feature pr: #6046
Impact
Prevent OOM and speed up row writer clustering
Risk level (write none, low medium or high below)
low
Documentation Update
none
Contributor's checklist