[HUDI-1606]align BaseJavaCommitActionExecuto#execute method with BaseSparkCommitActionExecutor #2560
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #2560 +/- ##
=========================================
Coverage 51.15% 51.15%
- Complexity 3212 3213 +1
=========================================
Files 436 436
Lines 19987 19987
Branches 2057 2057
=========================================
+ Hits 10224 10225 +1
Misses 8922 8922
+ Partials 841 840 -1
Flags with carried forward coverage won't be shown. Click here to find out more.
|
leesf
reviewed
Feb 10, 2021
| recordsSoFar.stream().map(r -> new HoodieRecord<HoodieAvroPayload>(r)).collect(Collectors.toList()); | ||
| client.upsert(writeRecords, newCommitTime); | ||
| List<WriteStatus> insertStatus = client.upsert(writeRecords, newCommitTime); | ||
| client.commit(newCommitTime,insertStatus); |
Contributor
There was a problem hiding this comment.
would we just change the BaseJavaCommitActionExecutor#execute method to add updateIndexAndCommitIfNeeded method and align with BaseSparkCommitActionExecutor?
Author
There was a problem hiding this comment.
@leesf many thanks for your review. You suggestion make more sense and i have change the code accordingly.
…SparkCommitActionExecutor
caidezhi
changed the title
leesf
reviewed
Feb 21, 2021
| result.setWriteStatuses(statuses); | ||
| } | ||
|
|
||
| protected void updateIndexAndCommitIfNeeded(List<WriteStatus> writeStatuses, HoodieWriteMetadata<List<WriteStatus>> result) { |
Contributor
Author
There was a problem hiding this comment.
@leesf if it is already covered by other PR, please feel free to close this one. Thanks for your review.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Tips
What is the purpose of the pull request
run HoodieJavaWriteClientExample locally ,the code fail with exception. The root cause is that BaseJavaCommitActionExecutor#execute method does not commit the index when it is needed.
the PR is to fix the issue.
Brief change log
org.apache.hudi.table.action.commit.BaseJavaCommitActionExecutor
Verify this pull request
This pull request is already covered by existing tests: HoodieJavaWriteClientExample.main()
I manually verified the change by running a job locally.
Committer checklist
Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.