[HUDI-910]Introduce HoodieWriteInput for hudi write client

[wangxianghu]

Tips

• Thank you very much for contributing to Apache Hudi.
• Please review https://hudi.apache.org/contributing.html before opening a pull request.

What is the purpose of the pull request

This pull request introduce HoodieWriteInput for hudi write client as the unified input format.

Brief change log

• add a new module hudi-writer-commom
• introduce HoodieWriteInput for hudi write client as the unified input format.

Verify this pull request

This pull request is a trivial rework / code cleanup without any test coverage.

Committer checklist

• Has a corresponding JIRA in PR title & commit

• Commit message is descriptive of the change

• CI is green

• Necessary doc changes done or have another open PR

• For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

[wangxianghu]

Hi @yanghua , could you please take a look when free, thanks!

[codecov-commenter]

Codecov Report

Merging #1665 into master will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master    #1665   +/-   ##
=========================================
  Coverage     18.21%   18.21%           
  Complexity      856      856           
=========================================
  Files           348      348           
  Lines         15332    15332           
  Branches       1523     1523           
=========================================
  Hits           2792     2792           
  Misses        12183    12183           
  Partials        357      357           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bde7a70...9f0121f. Read the comment docs.

[vinothchandar]

Is there an umbrella task to understand how all the follow up work will be.. on this..For e.g I am wondering what the eventual methods on HoodieWriteInput will be and how it will abstract away the RDD construct

[wangxianghu]

Is there an umbrella task to understand how all the follow up work will be.. on this..For e.g I am wondering what the eventual methods on HoodieWriteInput will be and how it will abstract away the RDD construct

Is there an umbrella task to understand how all the follow up work will be.. on this..For e.g I am wondering what the eventual methods on HoodieWriteInput will be and how it will abstract away the RDD construct

yes, here it is : https://issues.apache.org/jira/browse/HUDI-909
currently only four subtasks are filed, which is the very foundation of the entire abstraction:

1. Introduce HoodieWriteInput for hudi write client: https://issues.apache.org/jira/browse/HUDI-910
2. Introduce HoodieWriteOutput for hudi write client: https://issues.apache.org/jira/browse/HUDI-911
3. Introduce HoodieWriteKey for hudi write client: https://issues.apache.org/jira/browse/HUDI-912
4. Introduce HoodieEngineContext for hudi write client: https://issues.apache.org/jira/browse/HUDI-913

For Spark these could be :

JavaRDD<HoodieRecord<T>> records = ... ; // read from souce
HoodieWriteInput<JavaRDD<HoodieRecord<T>>> inputRecords = new HoodieWriteInput(records);
JavaRDD<HoodieRecord<T>> inputRdds = inputRecords.getInputs();

JavaSparkContext jsc = ...;
SerializableConfiguration hadoopConf = ...;`
HoodieEngineContext  hec = new HoodieSparkEngineContext(hadoopConf, jsc); //HoodieSparkEngineContext extends HoodieEngineContext`
JavaSparkContext jsc = ((HoodieSparkEngineContext)hec).getContext();

The HoodieWriteKey and HoodieWriteOutput are the same as HoodieWriteInput.

upsert api could be like this:

public HoodieWriteOutput<JavaRDD<WriteStatus>> upsert(HoodieWriteInput<JavaRDD<HoodieRecord<T>>> records, final String instantTime) {...}

The content of the method is almost the same as before.

For Java and Flink, just replace the JavaRDD with List.

[leesf]

I am ok with the abstraction and reviewed before in other branch.

[leesf]

@vinothchandar just a reminder on this PR.

[vinothchandar]

@wangxianghu @leesf sorry for that delays. I was trying to understand the relation to this work
https://issues.apache.org/jira/browse/HUDI-538 cc @yanghua
This is not still clear to me where we are moving towards. While I agree that abstracting RDD into a WriteInput, like to understand how the existing code is going to evolve further. May I ask that we do a draft or first that replaces the entire code in hudi-client with these abstractions ( no need to even have the code compile. But want to understand the final shape we are looking at). I am also happy to do that myself and discuss.. Please let me know if that seems like a reasonable ask

[vinothchandar]

This pr itself is fine. But given we are adding a new module and this is a critical thing to get right, trying to understand more upfront

[wangxianghu]

@wangxianghu @leesf sorry for that delays. I was trying to understand the relation to this work
https://issues.apache.org/jira/browse/HUDI-538 cc @yanghua
This is not still clear to me where we are moving towards. While I agree that abstracting RDD into a WriteInput, like to understand how the existing code is going to evolve further. May I ask that we do a draft or first that replaces the entire code in hudi-client with these abstractions ( no need to even have the code compile. But want to understand the final shape we are looking at). I am also happy to do that myself and discuss.. Please let me know if that seems like a reasonable ask

Hi @vinothchandar thanks for your suggestion. Replacing the entire code in hudi-client with new abstractions is a good choose, I'll do it :)

[vinothchandar]

@wangxianghu Thanks for being a sport! That will give us good confidence that we can ultimately pull this off.. Happy to help along as needed..

[wangxianghu]

@wangxianghu Thanks for being a sport! That will give us good confidence that we can ultimately pull this off.. Happy to help along as needed..

[wangxianghu]

Hi @vinothchandar , This is the abstration of hudi-client module（not finished yet）:
https://github.com/wangxianghu/hudi/tree/HUDI-xxx
please take a look when free.
cc @yanghua @leesf

[leesf]

Hi @vinothchandar , This is the abstration of hudi-client module（not finished yet）:
https://github.com/wangxianghu/hudi/tree/HUDI-xxx
please take a look when free.
cc @yanghua @leesf

Ack, look at the branch, the structure looks good to me, @vinothchandar would you please also take a look when you are free, we need land this feature ASAP.

[vinothchandar]

Sg.. Will jump on #1727 . Closing this one