[HUDI-2069] Refactored String constants #3172

sbernauer · 2021-06-28T11:07:47Z

What is the purpose of the pull request

Follow-Up of #3111 adressing the feedback
Revert a change from #3092 that broke the feature again by not passing the needed properties to kafka-client

Brief change log

Refactored String constants
Pass the hoodie.deltastreamer.source.kafka.value.deserializer.* configs to the kafka clienr

Verify this pull request

This pull request is a trivial rework / code cleanup without any test coverage.

Committer checklist

Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

… String constants

hudi-bot · 2021-06-28T11:09:34Z

CI report:

f1adb61 UNKNOWN
ccb58db Azure: SUCCESS Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run travis re-run the last Travis build
@hudi-bot run azure re-run the last Azure build

sbernauer · 2021-06-28T12:12:21Z

@hudi-bot run azure

hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala

hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/AvroKafkaSource.java

…d String constants

codecov-commenter · 2021-06-29T22:23:33Z

Codecov Report

Merging #3172 (ccb58db) into master (7173d13) will decrease coverage by 0.08%.
The diff coverage is 15.38%.

@@             Coverage Diff              @@
##             master    #3172      +/-   ##
============================================
- Coverage     47.51%   47.42%   -0.09%     
+ Complexity     5428     5424       -4     
============================================
  Files           922      923       +1     
  Lines         40966    41178     +212     
  Branches       4100     4152      +52     
============================================
+ Hits          19464    19530      +66     
- Misses        19784    19914     +130     
- Partials       1718     1734      +16

Flag	Coverage Δ
hudicli	`39.97% <ø> (-0.04%)`	⬇️
hudiclient	`34.57% <ø> (-0.01%)`	⬇️
hudicommon	`48.38% <ø> (-0.02%)`	⬇️
hudiflink	`59.80% <ø> (-0.11%)`	⬇️
hudihadoopmr	`51.29% <ø> (ø)`
hudisparkdatasource	`67.22% <ø> (-0.02%)`	⬇️
hudisync	`54.48% <ø> (+0.43%)`	⬆️
huditimelineservice	`64.07% <ø> (ø)`
hudiutilities	`56.41% <15.38%> (-1.64%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...apache/hudi/utilities/sources/AvroKafkaSource.java	`0.00% <0.00%> (ø)`
...hudi/utilities/sources/helpers/KafkaOffsetGen.java	`87.68% <50.00%> (-0.65%)`	⬇️
...i/utilities/deser/KafkaAvroSchemaDeserializer.java	`81.25% <100.00%> (ø)`
.../apache/hudi/keygen/constant/KeyGeneratorType.java	`63.63% <0.00%> (-36.37%)`	⬇️
...apache/hudi/utilities/deltastreamer/DeltaSync.java	`52.88% <0.00%> (-18.50%)`	⬇️
.../keygen/factory/HoodieAvroKeyGeneratorFactory.java	`71.42% <0.00%> (-10.93%)`	⬇️
...keygen/factory/HoodieSparkKeyGeneratorFactory.java	`71.42% <0.00%> (-10.93%)`	⬇️
...in/scala/org/apache/hudi/IncrementalRelation.scala	`76.13% <0.00%> (-5.69%)`	⬇️
...ache/hudi/sink/StreamWriteOperatorCoordinator.java	`68.96% <0.00%> (-2.64%)`	⬇️
...e/hudi/common/table/log/HoodieLogFormatWriter.java	`78.12% <0.00%> (-1.57%)`	⬇️
... and 19 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7173d13...ccb58db. Read the comment docs.

nsivabalan

Looks like you need some rebasing. Can you please check it out.
@vinothchandar : Do we have a guidelines or link for adding new configs. I think we need to do something extra so that configs are auto generated right.

nsivabalan · 2021-07-02T21:10:49Z

hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala

-  val KAFKA_AVRO_VALUE_DESERIALIZER = "hoodie.deltastreamer.source.kafka.value.deserializer.class"
-  // Schema to be used in custom kakfa deserializer
-  val KAFKA_AVRO_VALUE_DESERIALIZER_SCHEMA = "hoodie.deltastreamer.source.kafka.value.deserializer.schema"
+  val KAFKA_AVRO_VALUE_DESERIALIZER_OPT_KEY = "hoodie.deltastreamer.source.kafka.value.deserializer.class"


nit: can we reuse the property prefix variable here as well? KAFKA_AVRO_VALUE_DESERIALIZER_PROPERTY_PREFIX

I also thought of that but I was unable to import the hudi-utilities Class in the hudi-spark module I think. It would be great if you could give a hint how to achieve this.

Just tried it out again. Adding hudi-utilities as dependency to the pom of hudi-spark introduces a circular dependency :/

hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/AvroKafkaSource.java

sbernauer · 2021-07-03T08:27:30Z

Rebase done

sbernauer · 2021-07-06T06:23:46Z

I should have addressed all feedback. We have tested this patch again and have rolled it out to production.
Can you please have a look if everything is fine?

vinothchandar

LGTM.

reg, auto generation of configs. HUDI-1241 is tracking it.

vinothchandar · 2021-07-06T06:53:52Z

@hudi-bot run azure

sbernauer · 2021-07-06T06:56:49Z

Thanks @vinothchandar!
As the patch uses the ConfigProperty class the generation of the docs should work fine as my understanding

sbernauer · 2021-07-07T18:23:15Z

Thanks a lot for merging!

change the insret overwrte return type [HUDI-1860] Test wrapper for insert_overwrite and insert_overwrite_table [HUDI-2084] Resend the uncommitted write metadata when start up (apache#3168) Co-authored-by: 喻兆靖 <[email protected]> [HUDI-2081] Move schema util tests out from TestHiveSyncTool (apache#3166) [HUDI-2094] Supports hive style partitioning for flink writer (apache#3178) [HUDI-2097] Fix Flink unable to read commit metadata error (apache#3180) [HUDI-2085] Support specify compaction paralleism and compaction target io for flink batch compaction (apache#3169) [HUDI-2092] Fix NPE caused by FlinkStreamerConfig#writePartitionUrlEncode null value (apache#3176) [HUDI-2006] Adding more yaml templates to test suite (apache#3073) [HUDI-2103] Add rebalance before index bootstrap (apache#3185) Co-authored-by: 喻兆靖 <[email protected]> [HUDI-1944] Support Hudi to read from committed offset (apache#3175) * [HUDI-1944] Support Hudi to read from committed offset * [HUDI-1944] Adding group option to KafkaResetOffsetStrategies * [HUDI-1944] Update Exception msg [HUDI-2052] Support load logFile in BootstrapFunction (apache#3134) Co-authored-by: 喻兆靖 <[email protected]> [HUDI-89] Add configOption & refactor all configs based on that (apache#2833) Co-authored-by: Wenning Ding <[email protected]> [MINOR] Update .asf.yaml to codify notification settings, turn on jira comments, gh discussions (apache#3164) - Turn on comment for jira, so we can track PR activity better - Create a notification settings that match https://gitbox.apache.org/schemes.cgi?hudi - Try and turn on "discussions" on Github, to experiment [MINOR] Fix broken build due to FlinkOptions (apache#3198) [HUDI-2088] Missing Partition Fields And PreCombineField In Hoodie Properties For Table Written By Flink (apache#3171) [MINOR] Add Documentation to KEYGENERATOR_TYPE_PROP (apache#3196) [HUDI-2105] Compaction Failed For MergeInto MOR Table (apache#3190) [HUDI-2051] Enable Hive Sync When Spark Enable Hive Meta For Spark Sql (apache#3126) [HUDI-2112] Support reading pure logs file group for flink batch reader after compaction (apache#3202) [HUDI-2114] Spark Query MOR Table Written By Flink Return Incorrect Timestamp Value (apache#3208) [HUDI-2121] Add operator uid for flink stateful operators (apache#3212) [HUDI-2123] Exception When Merge With Null-Value Field (apache#3214) [HUDI-2124] A Grafana dashboard for HUDI. (apache#3216) [HUDI-2057] CTAS Generate An External Table When Create Managed Table (apache#3146) [HUDI-1930] Bootstrap support configure KeyGenerator by type (apache#3170) * [HUDI-1930] Bootstrap support configure KeyGenerator by type [HUDI-2116] Support batch synchronization of partition datas to hive metastore to avoid oom problem (apache#3209) [HUDI-2126] The coordinator send events to write function when there are no data for the checkpoint (apache#3219) [HUDI-2127] Initialize the maxMemorySizeInBytes in log scanner (apache#3220) [HUDI-2058]support incremental query for insert_overwrite_table/insert_overwrite operation on cow table (apache#3139) [HUDI-2129] StreamerUtil.medianInstantTime should return a valid date time string (apache#3221) [HUDI-2131] Exception Throw Out When MergeInto With Decimal Type Field (apache#3224) [HUDI-2122] Improvement in packaging insert into smallfiles (apache#3213) [HUDI-2132] Make coordinator events as POJO for efficient serialization (apache#3223) [HUDI-2106] Fix flink batch compaction bug while user don't set compaction tasks (apache#3192) [HUDI-2133] Support hive1 metadata sync for flink writer (apache#3225) [HUDI-2089]fix the bug that metatable cannot support non_partition table (apache#3182) [HUDI-2028] Implement RockDbBasedMap as an alternate to DiskBasedMap in ExternalSpillableMap (apache#3194) Co-authored-by: Rajesh Mahindra <[email protected]> [HUDI-2135] Add compaction schedule option for flink (apache#3226) [HUDI-2055] Added deltastreamer metric for time of lastSync (apache#3129) [HUDI-2046] Loaded too many classes like sun/reflect/GeneratedSerializationConstructorAccessor in JVM metaspace (apache#3121) Loaded too many classes when use kryo of spark to hudi Co-authored-by: weiwei.duan <[email protected]> [HUDI-1996] Adding functionality to allow the providing of basic auth creds for confluent cloud schema registry (apache#3097) * adding support for basic auth with confluent cloud schema registry [HUDI-2093] Fix empty avro schema path caused by duplicate parameters (apache#3177) * [HUDI-2093] Fix empty avro schema path caused by duplicate parameters * rename shcmea option key * fix doc * rename var name [HUDI-2113] Fix integration testing failure caused by sql results out of order (apache#3204) [HUDI-2016] Fixed bootstrap of Metadata Table when some actions are in progress. (apache#3083) Metadata Table cannot be bootstrapped when any action is in progress. This is detected by the presence of inflight or requested instants. The bootstrapping is initiated in preWrite and postWrite of each commit. So bootstrapping will be retried again until it succeeds. Also added metrics for when the bootstrapping fails or a table is re-bootstrapped. This will help detect tables which are not getting bootstrapped. [HUDI-2140] Fixed the unit test TestHoodieBackedMetadata.testOnlyValidPartitionsAdded. (apache#3234) [HUDI-2115] FileSlices in the filegroup is not descending by timestamp (apache#3206) [HUDI-1104] Adding support for UserDefinedPartitioners and SortModes to BulkInsert with Rows (apache#3149) [HUDI-2069] Refactored String constants (apache#3172) [HUDI-1105] Adding dedup support for Bulk Insert w/ Rows (apache#2206) [HUDI-2134]Add generics to avoif forced conversion in BaseSparkCommitActionExecutor#partition (apache#3232) [HUDI-2009] Fixing extra commit metadata in row writer path (apache#3075) [HUDI-2099]hive lock which state is WATING should be released, otherwise this hive lock will be locked forever (apache#3186) [MINOR] Fix build broken from apache#3186 (apache#3245) [HUDI-2136] Fix conflict when flink-sql-connector-hive and hudi-flink-bundle are both in flink lib (apache#3227) [HUDI-2087] Support Append only in Flink stream (apache#3174) Co-authored-by: 喻兆靖 <[email protected]> UnitTest for deltaSync Removing cosmetic changes and reuse function for insert_overwrite_table unit test intial unit test for the insert_overwrite and insert_over_write_table Adding failed test code for insert_overwrite Revert "[HUDI-2087] Support Append only in Flink stream (apache#3174)" (apache#3251) This reverts commit 3715267. [HUDI-2147] Remove unused class AvroConvertor in hudi-flink (apache#3243) [MINOR] Fix some wrong assert reasons (apache#3248) [HUDI-2087] Support Append only in Flink stream (apache#3252) Co-authored-by: 喻兆靖 <[email protected]> [HUDI-2143] Tweak the default compaction target IO to 500GB when flink async compaction is off (apache#3238) [HUDI-2142] Support setting bucket assign parallelism for flink write task (apache#3239) [HUDI-1483] Support async clustering for deltastreamer and Spark streaming (apache#3142) - Integrate async clustering service with HoodieDeltaStreamer and HoodieStreamingSink - Added methods in HoodieAsyncService to reuse code [HUDI-2107] Support Read Log Only MOR Table For Spark (apache#3193) [HUDI-2144]Bug-Fix:Offline clustering(HoodieClusteringJob) will cause insert action losing data (apache#3240) * fixed * add testUpsertPartitionerWithSmallFileHandlingAndClusteringPlan ut * fix CheckStyle Co-authored-by: yuezhang <[email protected]> [MINOR] Fix EXTERNAL_RECORD_AND_SCHEMA_TRANSFORMATION config (apache#3250) [HUDI-2168] Fix for AccessControlException for anonymous user (apache#3264) [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And DeltaStreamer test with insert-overwrite and insert-overwrite-table removing hardcoded action to pass the unit test [HUDI-1969] Support reading logs for MOR Hive rt table (apache#3033) [HUDI-2171] Add parallelism conf for bootstrap operator using delta-commit for insert_overwrite

[HUDI-2069] Enable custom avro deserializer by default and refactored…

499b74b

… String constants

sbernauer mentioned this pull request Jun 28, 2021

[HUDI-2069] Fix KafkaAvroSchemaDeserializer to not rely on reflection #3111

Merged

5 tasks

wangxianghu added the engine:spark Spark integration label Jun 29, 2021

sbernauer added 2 commits June 29, 2021 16:31

Merge branch 'master' into HUDI-2069

22455a5

Use static import for string constants

72a2bad

sbernauer mentioned this pull request Jun 29, 2021

[HUDI-1910] Commit Offset to Kafka after successful Hudi commit #3092

Merged

5 tasks

nsivabalan requested changes Jun 29, 2021

View reviewed changes

[HUDI-2069] Disable custom avro deserializer by default and refactore…

3adabda

…d String constants

sbernauer changed the title ~~[HUDI-2069] Enable custom avro deserializer by default and refactored String constants~~ [HUDI-2069] Refactored String constants Jun 30, 2021

nsivabalan reviewed Jul 2, 2021

View reviewed changes

sbernauer added 3 commits July 3, 2021 10:02

Merge branch 'master' into HUDI-2069

3a0d5be

Refactored KAFKA_AVRO_VALUE_DESERIALIZER_CLASS config

f1adb61

Optimized imports

7ac4dc2

sbernauer added 2 commits July 3, 2021 11:59

Added version info to KAFKA_AVRO_VALUE_DESERIALIZER_CLASS

33e4f39

Fixed getting Name of Class

ccb58db

vinothchandar approved these changes Jul 6, 2021

View reviewed changes

nsivabalan approved these changes Jul 7, 2021

View reviewed changes

nsivabalan merged commit 8f7ad8b into apache:master Jul 7, 2021

ghost pushed a commit to shivagowda/hudi that referenced this pull request Jul 15, 2021

[HUDI-2069] Refactored String constants (apache#3172)

f7627c3

ghost pushed a commit to shivagowda/hudi that referenced this pull request Aug 1, 2021

[HUDI-2069] Refactored String constants (apache#3172)

efb660e

[HUDI-2069] Refactored String constants #3172

[HUDI-2069] Refactored String constants #3172

Uh oh!

Conversation

sbernauer commented Jun 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is the purpose of the pull request

Brief change log

Verify this pull request

Committer checklist

Uh oh!

hudi-bot commented Jun 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

sbernauer commented Jun 28, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Jun 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

nsivabalan left a comment

Choose a reason for hiding this comment

Uh oh!

nsivabalan Jul 2, 2021

Choose a reason for hiding this comment

Uh oh!

sbernauer Jul 3, 2021

Choose a reason for hiding this comment

Uh oh!

sbernauer Jul 3, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sbernauer commented Jul 3, 2021

Uh oh!

sbernauer commented Jul 6, 2021

Uh oh!

vinothchandar left a comment

Choose a reason for hiding this comment

Uh oh!

vinothchandar commented Jul 6, 2021

Uh oh!

sbernauer commented Jul 6, 2021

Uh oh!

sbernauer commented Jul 7, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

sbernauer commented Jun 28, 2021 •

edited

Loading

hudi-bot commented Jun 28, 2021 •

edited

Loading

codecov-commenter commented Jun 29, 2021 •

edited

Loading