[HUDI-2052] Support load logFile in BootstrapFunction #3134

yuzhaojing · 2021-06-22T10:57:03Z

Support load logfile in BootstrapFunction.

What is the purpose of the pull request

(For example: This pull request adds quick-start document.)

Brief change log

(for example:)

Modify AnnotationLocation checkstyle rule in checkstyle.xml

Verify this pull request

(Please pick either of the following options)

This pull request is a trivial rework / code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

Added integration tests for end-to-end.
Added HoodieClientWriteTest to verify the change.
Manually verified the change by running a job locally.

Committer checklist

Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

yuzhaojing · 2021-06-22T10:58:01Z

@danny0405 please review this pr, thanks.

hudi-bot · 2021-06-22T11:02:17Z

CI report:

26c3164 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run travis re-run the last Travis build
@hudi-bot run azure re-run the last Azure build

codecov-commenter · 2021-06-22T15:53:19Z

Codecov Report

Merging #3134 (26c3164) into master (0749cc8) will increase coverage by 2.03%.
The diff coverage is 78.00%.

@@             Coverage Diff              @@
##             master    #3134      +/-   ##
============================================
+ Coverage     44.25%   46.28%   +2.03%     
- Complexity     4595     5409     +814     
============================================
  Files           825      921      +96     
  Lines         36577    40122    +3545     
  Branches       3942     4306     +364     
============================================
+ Hits          16187    18571    +2384     
- Misses        18642    19665    +1023     
- Partials       1748     1886     +138

Flag	Coverage Δ
hudicli	`39.95% <ø> (ø)`
hudiclient	`30.45% <ø> (+14.00%)`	⬆️
hudicommon	`47.58% <ø> (ø)`
hudiflink	`60.05% <78.00%> (-0.02%)`	⬇️
hudihadoopmr	`51.29% <ø> (ø)`
hudisparkdatasource	`67.06% <ø> (ø)`
hudisync	`54.05% <ø> (ø)`
huditimelineservice	`64.36% <ø> (ø)`
hudiutilities	`59.13% <ø> (+0.72%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
.../apache/hudi/sink/bootstrap/BootstrapFunction.java	`75.67% <78.00%> (+0.99%)`	⬆️
...va/org/apache/hudi/configuration/FlinkOptions.java	`96.37% <0.00%> (ø)`
.../org/apache/hudi/streamer/FlinkStreamerConfig.java	`0.00% <0.00%> (ø)`
...apache/hudi/sink/compact/HoodieFlinkCompactor.java	`0.00% <0.00%> (ø)`
...pache/hudi/sink/compact/FlinkCompactionConfig.java	`0.00% <0.00%> (ø)`
...i/table/action/clean/SparkCleanActionExecutor.java	`100.00% <0.00%> (ø)`
...metadata/SparkHoodieBackedTableMetadataWriter.java	`72.36% <0.00%> (ø)`
...udi/table/action/commit/SparkBulkInsertHelper.java	`100.00% <0.00%> (ø)`
...e/hudi/client/common/HoodieSparkEngineContext.java	`54.83% <0.00%> (ø)`
...on/commit/SparkBulkInsertCommitActionExecutor.java	`63.63% <0.00%> (ø)`
... and 94 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0749cc8...26c3164. Read the comment docs.

danny0405 · 2021-06-24T02:32:04Z

hudi-flink/src/main/java/org/apache/hudi/sink/bootstrap/BootstrapFunction.java

To simplify the logic, fetch the fileslice directly and send the parquet and log record keys, the file group id is solid, which is recorded by the fileslice.

I think so, I will modify this logic.

danny0405 · 2021-06-24T03:37:54Z

hudi-flink/src/main/java/org/apache/hudi/sink/bootstrap/BootstrapFunction.java

You can read the logs all together with the scanner.

ok, I already modify this.

danny0405 · 2021-06-24T06:20:40Z

hudi-flink/src/test/java/org/apache/hudi/sink/TestWriteMergeOnRead.java

This seems the only different with the parent method, can we abstract it out to override.

TestWriteMergeOnRead already override this function.

We override before to skip the tests, now you open it again, we should reuse the code as much as possible.

Yes, I will abstract this method.

danny0405 · 2021-06-24T11:36:07Z

hudi-flink/src/test/java/org/apache/hudi/sink/TestWriteMergeOnRead.java

No need to override anymore

danny0405 · 2021-06-24T11:37:18Z

hudi-flink/src/test/java/org/apache/hudi/sink/TestWriteMergeOnRead.java

Revert the import change.

danny0405 · 2021-06-24T11:37:30Z

hudi-flink/src/test/java/org/apache/hudi/sink/TestWriteCopyOnWrite.java

yuzhaojing · 2021-06-24T13:50:51Z

Already modify.

nsivabalan · 2021-06-24T16:32:38Z

There are some checkstyle issues reported by CI. Can you please look into fixing

yuzhaojing · 2021-06-25T01:07:55Z

There are some checkstyle issues reported by CI. Can you please look into fixing

I will fix that.

yanghua · 2021-06-25T06:23:08Z

hudi-flink/src/main/java/org/apache/hudi/sink/bootstrap/BootstrapFunction.java

Should it be return or continue?

Thanks for review, I will fix that.

danny0405 · 2021-06-25T09:10:38Z

hudi-flink/src/main/java/org/apache/hudi/sink/bootstrap/BootstrapFunction.java

Can we send the keys directly and do not buffer the records ? The cache takes too much memory footprint.

yes, this cache use memory unnecessary.

danny0405

+1, LGTM.

yanghua

@yuzhaojing Thanks for your contribution. Left two minor comments, otherwise LGTM.

yanghua · 2021-06-30T06:08:33Z

hudi-flink/src/main/java/org/apache/hudi/sink/bootstrap/BootstrapFunction.java

shouldLoadFile seems better?

yanghua · 2021-06-30T06:11:16Z

hudi-flink/src/main/java/org/apache/hudi/sink/bootstrap/BootstrapFunction.java

we follow hudi class first rule, please split it.

yuzhaojing · 2021-06-30T12:11:34Z

@yuzhaojing Thanks for your contribution. Left two minor comments, otherwise LGTM.

Thanks for review, already modify it.

yanghua

LGTM, @yuzhaojing thanks for addressing my concerns.

change the insret overwrte return type [HUDI-1860] Test wrapper for insert_overwrite and insert_overwrite_table [HUDI-2084] Resend the uncommitted write metadata when start up (apache#3168) Co-authored-by: 喻兆靖 <[email protected]> [HUDI-2081] Move schema util tests out from TestHiveSyncTool (apache#3166) [HUDI-2094] Supports hive style partitioning for flink writer (apache#3178) [HUDI-2097] Fix Flink unable to read commit metadata error (apache#3180) [HUDI-2085] Support specify compaction paralleism and compaction target io for flink batch compaction (apache#3169) [HUDI-2092] Fix NPE caused by FlinkStreamerConfig#writePartitionUrlEncode null value (apache#3176) [HUDI-2006] Adding more yaml templates to test suite (apache#3073) [HUDI-2103] Add rebalance before index bootstrap (apache#3185) Co-authored-by: 喻兆靖 <[email protected]> [HUDI-1944] Support Hudi to read from committed offset (apache#3175) * [HUDI-1944] Support Hudi to read from committed offset * [HUDI-1944] Adding group option to KafkaResetOffsetStrategies * [HUDI-1944] Update Exception msg [HUDI-2052] Support load logFile in BootstrapFunction (apache#3134) Co-authored-by: 喻兆靖 <[email protected]> [HUDI-89] Add configOption & refactor all configs based on that (apache#2833) Co-authored-by: Wenning Ding <[email protected]> [MINOR] Update .asf.yaml to codify notification settings, turn on jira comments, gh discussions (apache#3164) - Turn on comment for jira, so we can track PR activity better - Create a notification settings that match https://gitbox.apache.org/schemes.cgi?hudi - Try and turn on "discussions" on Github, to experiment [MINOR] Fix broken build due to FlinkOptions (apache#3198) [HUDI-2088] Missing Partition Fields And PreCombineField In Hoodie Properties For Table Written By Flink (apache#3171) [MINOR] Add Documentation to KEYGENERATOR_TYPE_PROP (apache#3196) [HUDI-2105] Compaction Failed For MergeInto MOR Table (apache#3190) [HUDI-2051] Enable Hive Sync When Spark Enable Hive Meta For Spark Sql (apache#3126) [HUDI-2112] Support reading pure logs file group for flink batch reader after compaction (apache#3202) [HUDI-2114] Spark Query MOR Table Written By Flink Return Incorrect Timestamp Value (apache#3208) [HUDI-2121] Add operator uid for flink stateful operators (apache#3212) [HUDI-2123] Exception When Merge With Null-Value Field (apache#3214) [HUDI-2124] A Grafana dashboard for HUDI. (apache#3216) [HUDI-2057] CTAS Generate An External Table When Create Managed Table (apache#3146) [HUDI-1930] Bootstrap support configure KeyGenerator by type (apache#3170) * [HUDI-1930] Bootstrap support configure KeyGenerator by type [HUDI-2116] Support batch synchronization of partition datas to hive metastore to avoid oom problem (apache#3209) [HUDI-2126] The coordinator send events to write function when there are no data for the checkpoint (apache#3219) [HUDI-2127] Initialize the maxMemorySizeInBytes in log scanner (apache#3220) [HUDI-2058]support incremental query for insert_overwrite_table/insert_overwrite operation on cow table (apache#3139) [HUDI-2129] StreamerUtil.medianInstantTime should return a valid date time string (apache#3221) [HUDI-2131] Exception Throw Out When MergeInto With Decimal Type Field (apache#3224) [HUDI-2122] Improvement in packaging insert into smallfiles (apache#3213) [HUDI-2132] Make coordinator events as POJO for efficient serialization (apache#3223) [HUDI-2106] Fix flink batch compaction bug while user don't set compaction tasks (apache#3192) [HUDI-2133] Support hive1 metadata sync for flink writer (apache#3225) [HUDI-2089]fix the bug that metatable cannot support non_partition table (apache#3182) [HUDI-2028] Implement RockDbBasedMap as an alternate to DiskBasedMap in ExternalSpillableMap (apache#3194) Co-authored-by: Rajesh Mahindra <[email protected]> [HUDI-2135] Add compaction schedule option for flink (apache#3226) [HUDI-2055] Added deltastreamer metric for time of lastSync (apache#3129) [HUDI-2046] Loaded too many classes like sun/reflect/GeneratedSerializationConstructorAccessor in JVM metaspace (apache#3121) Loaded too many classes when use kryo of spark to hudi Co-authored-by: weiwei.duan <[email protected]> [HUDI-1996] Adding functionality to allow the providing of basic auth creds for confluent cloud schema registry (apache#3097) * adding support for basic auth with confluent cloud schema registry [HUDI-2093] Fix empty avro schema path caused by duplicate parameters (apache#3177) * [HUDI-2093] Fix empty avro schema path caused by duplicate parameters * rename shcmea option key * fix doc * rename var name [HUDI-2113] Fix integration testing failure caused by sql results out of order (apache#3204) [HUDI-2016] Fixed bootstrap of Metadata Table when some actions are in progress. (apache#3083) Metadata Table cannot be bootstrapped when any action is in progress. This is detected by the presence of inflight or requested instants. The bootstrapping is initiated in preWrite and postWrite of each commit. So bootstrapping will be retried again until it succeeds. Also added metrics for when the bootstrapping fails or a table is re-bootstrapped. This will help detect tables which are not getting bootstrapped. [HUDI-2140] Fixed the unit test TestHoodieBackedMetadata.testOnlyValidPartitionsAdded. (apache#3234) [HUDI-2115] FileSlices in the filegroup is not descending by timestamp (apache#3206) [HUDI-1104] Adding support for UserDefinedPartitioners and SortModes to BulkInsert with Rows (apache#3149) [HUDI-2069] Refactored String constants (apache#3172) [HUDI-1105] Adding dedup support for Bulk Insert w/ Rows (apache#2206) [HUDI-2134]Add generics to avoif forced conversion in BaseSparkCommitActionExecutor#partition (apache#3232) [HUDI-2009] Fixing extra commit metadata in row writer path (apache#3075) [HUDI-2099]hive lock which state is WATING should be released, otherwise this hive lock will be locked forever (apache#3186) [MINOR] Fix build broken from apache#3186 (apache#3245) [HUDI-2136] Fix conflict when flink-sql-connector-hive and hudi-flink-bundle are both in flink lib (apache#3227) [HUDI-2087] Support Append only in Flink stream (apache#3174) Co-authored-by: 喻兆靖 <[email protected]> UnitTest for deltaSync Removing cosmetic changes and reuse function for insert_overwrite_table unit test intial unit test for the insert_overwrite and insert_over_write_table Adding failed test code for insert_overwrite Revert "[HUDI-2087] Support Append only in Flink stream (apache#3174)" (apache#3251) This reverts commit 3715267. [HUDI-2147] Remove unused class AvroConvertor in hudi-flink (apache#3243) [MINOR] Fix some wrong assert reasons (apache#3248) [HUDI-2087] Support Append only in Flink stream (apache#3252) Co-authored-by: 喻兆靖 <[email protected]> [HUDI-2143] Tweak the default compaction target IO to 500GB when flink async compaction is off (apache#3238) [HUDI-2142] Support setting bucket assign parallelism for flink write task (apache#3239) [HUDI-1483] Support async clustering for deltastreamer and Spark streaming (apache#3142) - Integrate async clustering service with HoodieDeltaStreamer and HoodieStreamingSink - Added methods in HoodieAsyncService to reuse code [HUDI-2107] Support Read Log Only MOR Table For Spark (apache#3193) [HUDI-2144]Bug-Fix:Offline clustering(HoodieClusteringJob) will cause insert action losing data (apache#3240) * fixed * add testUpsertPartitionerWithSmallFileHandlingAndClusteringPlan ut * fix CheckStyle Co-authored-by: yuezhang <[email protected]> [MINOR] Fix EXTERNAL_RECORD_AND_SCHEMA_TRANSFORMATION config (apache#3250) [HUDI-2168] Fix for AccessControlException for anonymous user (apache#3264) [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And DeltaStreamer test with insert-overwrite and insert-overwrite-table removing hardcoded action to pass the unit test [HUDI-1969] Support reading logs for MOR Hive rt table (apache#3033) [HUDI-2171] Add parallelism conf for bootstrap operator using delta-commit for insert_overwrite

Co-authored-by: 喻兆靖 <[email protected]>

yuzhaojing force-pushed the HUDI-2052 branch from 31e04ec to 886976b Compare June 23, 2021 03:27

danny0405 reviewed Jun 24, 2021

View reviewed changes

yuzhaojing force-pushed the HUDI-2052 branch from 886976b to 6658f31 Compare June 24, 2021 03:27

danny0405 reviewed Jun 24, 2021

View reviewed changes

yuzhaojing force-pushed the HUDI-2052 branch 2 times, most recently from a71e70e to 51d09c6 Compare June 24, 2021 05:38

danny0405 reviewed Jun 24, 2021

View reviewed changes

yuzhaojing force-pushed the HUDI-2052 branch from 51d09c6 to 56a9c15 Compare June 24, 2021 09:31

danny0405 reviewed Jun 24, 2021

View reviewed changes

yuzhaojing force-pushed the HUDI-2052 branch from 56a9c15 to b428ad5 Compare June 24, 2021 13:38

yanghua self-assigned this Jun 25, 2021

yuzhaojing force-pushed the HUDI-2052 branch from b428ad5 to 92bbb23 Compare June 25, 2021 01:10

yanghua reviewed Jun 25, 2021

View reviewed changes

yuzhaojing force-pushed the HUDI-2052 branch from 92bbb23 to cc2e757 Compare June 25, 2021 09:01

danny0405 reviewed Jun 25, 2021

View reviewed changes

yuzhaojing force-pushed the HUDI-2052 branch 2 times, most recently from b91eda3 to 31d3303 Compare June 26, 2021 05:14

danny0405 approved these changes Jun 28, 2021

View reviewed changes

yuzhaojing force-pushed the HUDI-2052 branch from b8a1be7 to c4494bc Compare June 29, 2021 07:26

yanghua reviewed Jun 30, 2021

View reviewed changes

yuzhaojing force-pushed the HUDI-2052 branch from c4494bc to 7b8bf59 Compare June 30, 2021 06:23

[HUDI-2052] Support load logFile in BootstrapFunction

26c3164

yuzhaojing force-pushed the HUDI-2052 branch from 7b8bf59 to 26c3164 Compare June 30, 2021 06:28

yuzhaojing closed this Jun 30, 2021

yuzhaojing reopened this Jun 30, 2021

yanghua approved these changes Jun 30, 2021

View reviewed changes

yanghua merged commit 07e93de into apache:master Jun 30, 2021

ghost pushed a commit to shivagowda/hudi that referenced this pull request Jul 29, 2021

[HUDI-2052] Support load logFile in BootstrapFunction (apache#3134)

5a148e6

Co-authored-by: 喻兆靖 <[email protected]>

[HUDI-2052] Support load logFile in BootstrapFunction #3134

[HUDI-2052] Support load logFile in BootstrapFunction #3134

Uh oh!

Conversation

yuzhaojing commented Jun 22, 2021

What is the purpose of the pull request

Brief change log

Verify this pull request

Committer checklist

Uh oh!

yuzhaojing commented Jun 22, 2021

Uh oh!

hudi-bot commented Jun 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

codecov-commenter commented Jun 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danny0405 Jun 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yuzhaojing commented Jun 24, 2021

Uh oh!

nsivabalan commented Jun 24, 2021

Uh oh!

yuzhaojing commented Jun 25, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danny0405 left a comment

Choose a reason for hiding this comment

Uh oh!

yanghua left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yuzhaojing commented Jun 30, 2021

Uh oh!

yanghua left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

hudi-bot commented Jun 22, 2021 •

edited

Loading

codecov-commenter commented Jun 22, 2021 •

edited

Loading

danny0405 Jun 24, 2021 •

edited

Loading