[HUDI-3336][HUDI-FLINK] Support custom hadoop config options for flink #4699

cuibo01 · 2022-01-27T10:12:40Z

…t take effect

Tips

Thank you very much for contributing to Apache Hudi.
Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.

What is the purpose of the pull request

(For example: This pull request adds quick-start document.)

Brief change log

(for example:)

Modify AnnotationLocation checkstyle rule in checkstyle.xml

Verify this pull request

(Please pick either of the following options)

This pull request is a trivial rework / code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

Added integration tests for end-to-end.
Added HoodieClientWriteTest to verify the change.
Manually verified the change by running a job locally.

Committer checklist

Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

xiarixiaoyao · 2022-01-27T12:41:48Z

@hudi-bot run azure

xiarixiaoyao · 2022-01-29T01:38:02Z

@hudi-bot run azure

cuibo01 · 2022-01-29T01:39:37Z

@danny0405 pls review thx

danny0405 · 2022-01-29T04:48:00Z

Thanks, can you explain a little why we need theses hadoop. prefixed config options ?

cuibo01 · 2022-01-29T06:44:19Z

Thanks, can you explain a little why we need theses hadoop. prefixed config options ?

In the same application, the same core/hdfs-site is used
but some configurations may be different for different jobs, for example, the compact memory size, the FileSystem...

cuibo01 · 2022-01-29T06:44:54Z

@hudi-bot run azure

danny0405 · 2022-01-29T11:53:16Z

You may need to change the commit title: such as: Support custom hadoop config options for flink

cuibo01 · 2022-01-29T12:03:19Z

You may need to change the commit title: such as: Support custom hadoop config options for flink

ok Thanks for the review.

cuibo01 · 2022-01-29T12:04:23Z

@hudi-bot run azure

cuibo01 · 2022-01-29T12:31:54Z

@hudi-bot run azure

cuibo01 · 2022-01-30T01:27:13Z

@danny0405 can u approval workflow and merge the pr, thx

cuibo01 · 2022-02-06T00:51:47Z

@hudi-bot run azure

cuibo01 · 2022-02-10T07:28:01Z

@danny0405 can u merge the PR？ thx

danny0405 · 2022-02-10T10:59:53Z

hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java

   */
-  public static Map<String, String> getHoodiePropertiesWithPrefix(Map<String, String> options, String prefix) {
+  public static Map<String, String> getPropertiesWithPrefix(Map<String, String> options, String subprefix) {
    final Map<String, String> hoodieProperties = new HashMap<>();


Can we use the prefix directly ? There is no need to prefix the option with properties., prefix the option with hadoop. directly is okey.

And can we also add a too method named getHadoopOptions(Configuration conf) here ?

Parqurt also uses the getHoodiePropertiesWithPrefix, but the getHoodiePropertiesWithPrefix only use properties., not append parqurt.. To ensure consistency, hadoop and parquet add prefix(properties.) .
properties.parqurt.* and parqurt.*, which one you think is better

hi @danny0405 pls review :)
replace getPropertiesWithPrefix with DelegatingConfiguration#toMap of Flink
and Added validation to HoodieTableFactory

hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java

hudi-flink/src/main/java/org/apache/hudi/util/ViewStorageProperties.java

hudi-flink/src/test/java/org/apache/hudi/utils/TestStreamerUtil.java

hudi-flink/src/test/java/org/apache/hudi/utils/TestViewStorageProperties.java

danny0405 · 2022-02-11T02:56:40Z

The DelegatingConfiguration would prefix all the options with specified prefix which is not what we want
Remove the validation of HoodieTableFactory because we want to support all the hoodie builtin options
can we move all the prefix into FlinkOptions and in there, add a util to add options with specified prefix into the given hadoop configuration

cuibo01 · 2022-02-11T07:00:05Z

sorry ,i dont understand
DelegatingConfiguration#toMap returns all the specified prefixe options, the logic is the same as that of getHoodiePropertiesWithPrefix.
https://github.com/apache/flink/blob/0caed30f9aaa0ce9b1488000f843f9932c19ce5a/flink-core/src/main/java/org/apache/flink/configuration/DelegatingConfiguration.java#L314

danny0405 · 2022-02-11T07:57:16Z

sorry ,i dont understand DelegatingConfiguration#toMap returns all the specified prefixe options, the logic is the same as that of getHoodiePropertiesWithPrefix. https://github.com/apache/flink/blob/0caed30f9aaa0ce9b1488000f843f9932c19ce5a/flink-core/src/main/java/org/apache/flink/configuration/DelegatingConfiguration.java#L314

Yes, the toMap has similar functionality, but the getXXX and setXXX methods always prefix the option with prefix, which is not what we want, i still think there is no need to introduce this too class.

cuibo01 · 2022-02-11T08:57:57Z

Yes, the toMap has similar functionality, but the getXXX and setXXX methods always prefix the option with prefix, which is not what we want, i still think there is no need to introduce this too class.

got it, thx @danny0405 , i will update the PR :)

cuibo01 · 2022-02-16T04:02:18Z

@danny0405 hi pls review :)

cuibo01 · 2022-02-19T07:25:02Z

@danny0405 can u merge the PR?

danny0405 · 2022-02-19T13:28:42Z

I think a little and we should hold for this patch, people usually do not pass hadoop config options through SQL options, can you describe again your use cases again, what kind of options do you want to pass around in per-job level ?

cuibo01 · 2022-02-21T07:43:45Z

I think a little and we should hold for this patch, people usually do not pass hadoop config options through SQL options, can you describe again your use cases again, what kind of options do you want to pass around in per-job level ?

1, Multiple SQL Job in the same process have different hadoop config in our production environment, we can configure different jobs by with
2, fix getParquetConf logic. :)

XuQianJin-Stars · 2022-05-07T09:32:32Z

hi @cuibo01 can rebase this pr？

…gtoHdfsConfig2 Conflicts: hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableFactory.java hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSource.java

hudi-bot · 2022-05-07T12:38:21Z

CI report:

ec269f1 UNKNOWN
5085945 UNKNOWN
ccfcae2 UNKNOWN
643062f Azure: FAILURE
655551e Azure: PENDING

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

cuibo01 · 2022-05-07T12:59:43Z

#5528

cuibo01 force-pushed the flinkConfigtoHdfsConfig branch 2 times, most recently from 8b1a75a to 8da29b1 Compare January 27, 2022 13:57

cuibo01 changed the title ~~[HUDI][HUDI-FLINK] Configurations transferred through Flink SQL canno…~~ [HUDI-3336][HUDI-FLINK] Configurations transferred through Flink SQL canno… Jan 28, 2022

cuibo01 force-pushed the flinkConfigtoHdfsConfig branch from 8da29b1 to 6e9036e Compare January 28, 2022 12:24

cuibo01 force-pushed the flinkConfigtoHdfsConfig branch from 4cc376c to ec269f1 Compare January 29, 2022 11:51

cuibo01 changed the title ~~[HUDI-3336][HUDI-FLINK] Configurations transferred through Flink SQL canno…~~ [HUDI-3336][HUDI-FLINK] Support custom hadoop config options for flink Jan 29, 2022

cuibo01 force-pushed the flinkConfigtoHdfsConfig branch 2 times, most recently from 5085945 to 33f419e Compare January 29, 2022 12:22

[HUDI-3336][HUDI-FLINK] Support custom hadoop config options for flink

099a340

cuibo01 force-pushed the flinkConfigtoHdfsConfig branch from 33f419e to 099a340 Compare February 5, 2022 08:26

danny0405 reviewed Feb 10, 2022

View reviewed changes

cuibo01 force-pushed the flinkConfigtoHdfsConfig branch from ccfcae2 to 1cb3039 Compare February 11, 2022 02:17

[HUDI-3336][HUDI-FLINK] Support custom hadoop config options for flink

0c496e6

cuibo01 force-pushed the flinkConfigtoHdfsConfig branch from 1cb3039 to 0c496e6 Compare February 11, 2022 02:45

[HUDI-3336][HUDI-FLINK] Support custom hadoop config options for flink

eb0b3c9

cuibo01 force-pushed the flinkConfigtoHdfsConfig branch from d83f5bd to eb0b3c9 Compare February 12, 2022 07:07

nsivabalan assigned danny0405 Feb 16, 2022

cuibo01 added 4 commits March 3, 2022 17:36

Merge branch 'master' into flinkConfigtoHdfsConfig

15d3c2e

Merge branch 'apache:master' into flinkConfigtoHdfsConfig

f700543

[HUDI-3336][HUDI-FLINK] Support custom hadoop config options for flink

3d3d2ac

[HUDI-3336][HUDI-FLINK] Support custom hadoop config options for flink

643062f

cuibo01 added 3 commits May 7, 2022 19:43

Merge branch 'master' of https://gitee.com/cui01/hudi into flinkConfi…

c23d81c

…gtoHdfsConfig2 Conflicts: hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableFactory.java hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSource.java

merge

42a480f

Merge branch 'flinkConfigtoHdfsConfig2' into flinkConfigtoHdfsConfig

655551e

cuibo01 closed this May 7, 2022

[HUDI-3336][HUDI-FLINK] Support custom hadoop config options for flink #4699

[HUDI-3336][HUDI-FLINK] Support custom hadoop config options for flink #4699

Uh oh!

Conversation

cuibo01 commented Jan 27, 2022

Tips

What is the purpose of the pull request

Brief change log

Verify this pull request

Committer checklist

Uh oh!

xiarixiaoyao commented Jan 27, 2022

Uh oh!

xiarixiaoyao commented Jan 29, 2022

Uh oh!

cuibo01 commented Jan 29, 2022

Uh oh!

danny0405 commented Jan 29, 2022

Uh oh!

cuibo01 commented Jan 29, 2022

Uh oh!

cuibo01 commented Jan 29, 2022

Uh oh!

danny0405 commented Jan 29, 2022

Uh oh!

cuibo01 commented Jan 29, 2022

Uh oh!

cuibo01 commented Jan 29, 2022

Uh oh!

cuibo01 commented Jan 29, 2022

Uh oh!

cuibo01 commented Jan 30, 2022

Uh oh!

cuibo01 commented Feb 6, 2022

Uh oh!

cuibo01 commented Feb 10, 2022

Uh oh!

danny0405 Feb 10, 2022

Choose a reason for hiding this comment

Uh oh!

cuibo01 Feb 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cuibo01 Feb 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danny0405 commented Feb 11, 2022

Uh oh!

cuibo01 commented Feb 11, 2022

Uh oh!

danny0405 commented Feb 11, 2022

Uh oh!

cuibo01 commented Feb 11, 2022

Uh oh!

cuibo01 commented Feb 16, 2022

Uh oh!

cuibo01 commented Feb 19, 2022

Uh oh!

danny0405 commented Feb 19, 2022

Uh oh!

cuibo01 commented Feb 21, 2022 • edited by danny0405 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

XuQianJin-Stars commented May 7, 2022

Uh oh!

hudi-bot commented May 7, 2022

CI report:

Uh oh!

cuibo01 commented May 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

cuibo01 Feb 10, 2022 •

edited

Loading

cuibo01 Feb 11, 2022 •

edited

Loading

cuibo01 commented Feb 21, 2022 •

edited by danny0405

Loading

cuibo01 commented May 7, 2022 •

edited

Loading