[HUDI-875] Introduce a new pom module named hudi-common-sync

[lw309637554]

Tips

• Thank you very much for contributing to Apache Hudi.
• Please review https://hudi.apache.org/contributing.html before opening a pull request.

What is the purpose of the pull request

Introduce a new pom module named hudi-common-sync, before start abstract common meta sync module support multiple meta service. Design is in https://cwiki.apache.org/confluence/display/HUDI/RFC+-+17+Abstract+common+meta+sync+module+support+multiple+meta+service

Brief change log

Introduce a new pom module named hudi-common-sync, before start abstract common meta sync module support multiple meta service.

Verify this pull request

(Please pick either of the following options)

This pull request is a trivial rework / code cleanup without any test coverage.

Committer checklist

• Has a corresponding JIRA in PR title & commit

• Commit message is descriptive of the change

• CI is green

• Necessary doc changes done or have another open PR

• For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

[codecov-commenter]

Codecov Report

Merging #1716 into master will decrease coverage by 54.10%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##             master    #1716       +/-   ##
=============================================
- Coverage     72.25%   18.14%   -54.11%     
- Complexity      294      859      +565     
=============================================
  Files           374      352       -22     
  Lines         16371    15411      -960     
  Branches       1654     1525      -129     
=============================================
- Hits          11829     2797     -9032     
- Misses         3806    12256     +8450     
+ Partials        736      358      -378     

Impacted Files	Coverage Δ	Complexity Δ
.../java/org/apache/hudi/client/HoodieReadClient.java	0.00% <0.00%> (-100.00%)	0.00% <0.00%> (ø%)
.../java/org/apache/hudi/metrics/MetricsReporter.java	0.00% <0.00%> (-100.00%)	0.00% <0.00%> (ø%)
.../java/org/apache/hudi/common/model/ActionType.java	0.00% <0.00%> (-100.00%)	0.00% <0.00%> (ø%)
...java/org/apache/hudi/io/HoodieRangeInfoHandle.java	0.00% <0.00%> (-100.00%)	0.00% <0.00%> (ø%)
.../java/org/apache/hudi/hadoop/InputPathHandler.java	0.00% <0.00%> (-100.00%)	0.00% <0.00%> (ø%)
...a/org/apache/hudi/exception/HoodieIOException.java	0.00% <0.00%> (-100.00%)	0.00% <0.00%> (ø%)
...org/apache/hudi/table/action/commit/SmallFile.java	0.00% <0.00%> (-100.00%)	0.00% <0.00%> (ø%)
...rg/apache/hudi/index/bloom/KeyRangeLookupTree.java	0.00% <0.00%> (-100.00%)	0.00% <0.00%> (ø%)
...g/apache/hudi/exception/HoodieInsertException.java	0.00% <0.00%> (-100.00%)	0.00% <0.00%> (ø%)
...g/apache/hudi/exception/HoodieUpsertException.java	0.00% <0.00%> (-100.00%)	0.00% <0.00%> (ø%)
... and 340 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update acb1ada...82e999f. Read the comment docs.

[leesf]

@vinothchandar What do you think about the PR, introducing a new module.

[vinothchandar]

Good with the module.. Few thoughts/suggestions

• Can we nest this under hudi-sync as hudi-sync-common, hudi-sync-hive and you can then introduce the aliyun metastore as hudi-sync-aliyun?
• We need to just ensure there are no upgrades/backwards incompatible changes for existing users .

[vinothchandar]

is this pom same as the current hudi-hive-sync pom?

[lw309637554]

Good with the module.. Few thoughts/suggestions

• Can we nest this under hudi-sync as hudi-sync-common, hudi-sync-hive and you can then introduce the aliyun metastore as hudi-sync-aliyun?
• We need to just ensure there are no upgrades/backwards incompatible changes for existing users .

thanks ,

• i think "we nest this under hudi-sync as hudi-sync-common" is a good idea,i will modify like this
• "there are no upgrades/backwards incompatible changes for existing users ." i will add more tests to guarantee

[lw309637554]

is this pom same as the current hudi-hive-sync pom?
not exactly the same,i will minimize it

[lw309637554]

@vinothchandar What do you think about the updated PR

[vinothchandar]

is the plan to eventually also move hudi-hive-sync under hudi-sync?

[vinothchandar]

@umehrot2 @zhedoubushishi please weigh in with your thoughts as well

[lw309637554]

is the plan to eventually also move hudi-hive-sync under hudi-sync?

yes，when /hudi-sync/hudi-sync-common is ready, will move hudi-hive-sync under hudi-sync

[vinothchandar]

Sorry for the delay. Was spending time on another large review.

Can we make those changes in this pr itself? Would love to see end-end functionality working. IIRC your ultimate goal is to plug in alliyun metastore?

[lw309637554]

Sorry for the delay. Was spending time on another large review.

Can we make those changes in this pr itself? Would love to see end-end functionality working. IIRC your ultimate goal is to plug in alliyun metastore?

ok , then i add the changes in this pr. aliyun metastore can plug is valuable for users.Now users already use kafka -> spark -> hudi + aliyun metastore -> presto + spark program.

[vinothchandar]

Now users already use kafka -> spark -> hudi + aliyun metastore -> presto + spark program.

Thanks for letting me know.. Thats great! I am a fan of doing the refactor with a precise goal in mind.. so please feel free to reopen if you feel we need to restructure for something else.. AWS Glue also works with the current abstractions..