[HUDI-1104] Adding support for UserDefinedPartitioners and SortModes to BulkInsert with Rows #3149

nsivabalan · 2021-06-24T12:41:45Z

What is the purpose of the pull request

Adding bulk_insert sort modes for row writer path

(for example:)

Adding bulk_insert sort modes for row writer path
Caching RowCreateHandles if applicable

Verify this pull request

This pull request is already covered by existing tests, such as TestBulkInsertInternalPartitionerForRows

This change added tests and can be verified as follows:

TestHoodieBulkInsertDataInternalWriter

Committer checklist

Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

hudi-bot · 2021-06-24T12:48:00Z

CI report:

bbf3285 Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run travis re-run the last Travis build
@hudi-bot run azure re-run the last Azure build

codecov-commenter · 2021-06-24T14:25:17Z

Codecov Report

Merging #3149 (bbf3285) into master (6e24434) will decrease coverage by 0.00%.
The diff coverage is 56.00%.

@@             Coverage Diff              @@
##             master    #3149      +/-   ##
============================================
- Coverage     47.61%   47.61%   -0.01%     
- Complexity     5487     5496       +9     
============================================
  Files           924      929       +5     
  Lines         41206    41242      +36     
  Branches       4134     4135       +1     
============================================
+ Hits          19621    19637      +16     
- Misses        19843    19860      +17     
- Partials       1742     1745       +3

Flag	Coverage Δ
hudicli	`39.97% <ø> (ø)`
hudiclient	`34.60% <52.63%> (+0.02%)`	⬆️
hudicommon	`48.56% <ø> (-0.02%)`	⬇️
hudiflink	`59.58% <ø> (ø)`
hudihadoopmr	`51.29% <ø> (ø)`
hudisparkdatasource	`67.23% <58.06%> (-0.10%)`	⬇️
hudisync	`54.48% <ø> (ø)`
huditimelineservice	`64.07% <ø> (ø)`
hudiutilities	`58.59% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...a/org/apache/hudi/config/HoodieInternalConfig.java	`0.00% <0.00%> (ø)`
...java/org/apache/hudi/config/HoodieWriteConfig.java	`42.88% <ø> (ø)`
...n/java/org/apache/hudi/internal/DefaultSource.java	`0.00% <0.00%> (ø)`
...org/apache/hudi/spark3/internal/DefaultSource.java	`0.00% <0.00%> (ø)`
...nal/HoodieDataSourceInternalBatchWriteBuilder.java	`0.00% <0.00%> (ø)`
...spark3/internal/HoodieDataSourceInternalTable.java	`0.00% <0.00%> (ø)`
.../BulkInsertInternalPartitionerWithRowsFactory.java	`50.00% <50.00%> (ø)`
...ecution/bulkinsert/NonSortPartitionerWithRows.java	`66.66% <66.66%> (ø)`
...n/bulkinsert/PartitionSortPartitionerWithRows.java	`66.66% <66.66%> (ø)`
...tion/bulkinsert/GlobalSortPartitionerWithRows.java	`75.00% <75.00%> (ø)`
... and 16 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6e24434...bbf3285. Read the comment docs.

vinothchandar

Left some comments.

vinothchandar · 2021-07-06T07:17:05Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java

this name needs some work.

also seems like you are deriving this value in HoodieSparkSqlWriter? I would like to avoid this new config if possible. can you clarify why we need this. it was not very clear to me

When switching over to a new RowCreateHandle, this config will decide whether to cache and reuse the write handles or close it right away when switching to a diff partition. I have left a comment at the exact place where its used as well.

I named the variable based on the method we have in the interface.

boolean arePartitionRecordsSorted();

Thot it will be consistent.
I can may be name this variable as "arePartitionerRecordsSortedInBulkInsert"

also, guess this is not a public config as such. Do we know where these configs should go? I see we have defined BULKINSERT_INPUT_DATA_SCHEMA_DDL in HoodieWriteConfig which is mainly used for internal purposes.

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java

.../java/org/apache/hudi/execution/bulkinsert/BulkInsertInternalPartitionerWithRowsFactory.java

vinothchandar · 2021-07-06T07:23:20Z

.../java/org/apache/hudi/execution/bulkinsert/BulkInsertInternalPartitionerWithRowsFactory.java

drop Internal from the name?

Existing factory class for write client path is called BulkInsertInternalPartitionerFactory. hence named it this way. Reason is that, we have an interface called BulkInsertPartitioner. we have few out of the box partitioners and we could have user defined as well. hence the naming for these factories as internal. I can fix the name for both the factories if you prefer.

.../src/main/java/org/apache/hudi/execution/bulkinsert/RDDPartitionSortPartitionerWithRows.java

hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala

...-spark-common/src/main/java/org/apache/hudi/internal/BulkInsertDataInternalWriterHelper.java

…r Class (apache#1927)

…r to bulk insert of Rows

change the insret overwrte return type [HUDI-1860] Test wrapper for insert_overwrite and insert_overwrite_table [HUDI-2084] Resend the uncommitted write metadata when start up (apache#3168) Co-authored-by: 喻兆靖 <[email protected]> [HUDI-2081] Move schema util tests out from TestHiveSyncTool (apache#3166) [HUDI-2094] Supports hive style partitioning for flink writer (apache#3178) [HUDI-2097] Fix Flink unable to read commit metadata error (apache#3180) [HUDI-2085] Support specify compaction paralleism and compaction target io for flink batch compaction (apache#3169) [HUDI-2092] Fix NPE caused by FlinkStreamerConfig#writePartitionUrlEncode null value (apache#3176) [HUDI-2006] Adding more yaml templates to test suite (apache#3073) [HUDI-2103] Add rebalance before index bootstrap (apache#3185) Co-authored-by: 喻兆靖 <[email protected]> [HUDI-1944] Support Hudi to read from committed offset (apache#3175) * [HUDI-1944] Support Hudi to read from committed offset * [HUDI-1944] Adding group option to KafkaResetOffsetStrategies * [HUDI-1944] Update Exception msg [HUDI-2052] Support load logFile in BootstrapFunction (apache#3134) Co-authored-by: 喻兆靖 <[email protected]> [HUDI-89] Add configOption & refactor all configs based on that (apache#2833) Co-authored-by: Wenning Ding <[email protected]> [MINOR] Update .asf.yaml to codify notification settings, turn on jira comments, gh discussions (apache#3164) - Turn on comment for jira, so we can track PR activity better - Create a notification settings that match https://gitbox.apache.org/schemes.cgi?hudi - Try and turn on "discussions" on Github, to experiment [MINOR] Fix broken build due to FlinkOptions (apache#3198) [HUDI-2088] Missing Partition Fields And PreCombineField In Hoodie Properties For Table Written By Flink (apache#3171) [MINOR] Add Documentation to KEYGENERATOR_TYPE_PROP (apache#3196) [HUDI-2105] Compaction Failed For MergeInto MOR Table (apache#3190) [HUDI-2051] Enable Hive Sync When Spark Enable Hive Meta For Spark Sql (apache#3126) [HUDI-2112] Support reading pure logs file group for flink batch reader after compaction (apache#3202) [HUDI-2114] Spark Query MOR Table Written By Flink Return Incorrect Timestamp Value (apache#3208) [HUDI-2121] Add operator uid for flink stateful operators (apache#3212) [HUDI-2123] Exception When Merge With Null-Value Field (apache#3214) [HUDI-2124] A Grafana dashboard for HUDI. (apache#3216) [HUDI-2057] CTAS Generate An External Table When Create Managed Table (apache#3146) [HUDI-1930] Bootstrap support configure KeyGenerator by type (apache#3170) * [HUDI-1930] Bootstrap support configure KeyGenerator by type [HUDI-2116] Support batch synchronization of partition datas to hive metastore to avoid oom problem (apache#3209) [HUDI-2126] The coordinator send events to write function when there are no data for the checkpoint (apache#3219) [HUDI-2127] Initialize the maxMemorySizeInBytes in log scanner (apache#3220) [HUDI-2058]support incremental query for insert_overwrite_table/insert_overwrite operation on cow table (apache#3139) [HUDI-2129] StreamerUtil.medianInstantTime should return a valid date time string (apache#3221) [HUDI-2131] Exception Throw Out When MergeInto With Decimal Type Field (apache#3224) [HUDI-2122] Improvement in packaging insert into smallfiles (apache#3213) [HUDI-2132] Make coordinator events as POJO for efficient serialization (apache#3223) [HUDI-2106] Fix flink batch compaction bug while user don't set compaction tasks (apache#3192) [HUDI-2133] Support hive1 metadata sync for flink writer (apache#3225) [HUDI-2089]fix the bug that metatable cannot support non_partition table (apache#3182) [HUDI-2028] Implement RockDbBasedMap as an alternate to DiskBasedMap in ExternalSpillableMap (apache#3194) Co-authored-by: Rajesh Mahindra <[email protected]> [HUDI-2135] Add compaction schedule option for flink (apache#3226) [HUDI-2055] Added deltastreamer metric for time of lastSync (apache#3129) [HUDI-2046] Loaded too many classes like sun/reflect/GeneratedSerializationConstructorAccessor in JVM metaspace (apache#3121) Loaded too many classes when use kryo of spark to hudi Co-authored-by: weiwei.duan <[email protected]> [HUDI-1996] Adding functionality to allow the providing of basic auth creds for confluent cloud schema registry (apache#3097) * adding support for basic auth with confluent cloud schema registry [HUDI-2093] Fix empty avro schema path caused by duplicate parameters (apache#3177) * [HUDI-2093] Fix empty avro schema path caused by duplicate parameters * rename shcmea option key * fix doc * rename var name [HUDI-2113] Fix integration testing failure caused by sql results out of order (apache#3204) [HUDI-2016] Fixed bootstrap of Metadata Table when some actions are in progress. (apache#3083) Metadata Table cannot be bootstrapped when any action is in progress. This is detected by the presence of inflight or requested instants. The bootstrapping is initiated in preWrite and postWrite of each commit. So bootstrapping will be retried again until it succeeds. Also added metrics for when the bootstrapping fails or a table is re-bootstrapped. This will help detect tables which are not getting bootstrapped. [HUDI-2140] Fixed the unit test TestHoodieBackedMetadata.testOnlyValidPartitionsAdded. (apache#3234) [HUDI-2115] FileSlices in the filegroup is not descending by timestamp (apache#3206) [HUDI-1104] Adding support for UserDefinedPartitioners and SortModes to BulkInsert with Rows (apache#3149) [HUDI-2069] Refactored String constants (apache#3172) [HUDI-1105] Adding dedup support for Bulk Insert w/ Rows (apache#2206) [HUDI-2134]Add generics to avoif forced conversion in BaseSparkCommitActionExecutor#partition (apache#3232) [HUDI-2009] Fixing extra commit metadata in row writer path (apache#3075) [HUDI-2099]hive lock which state is WATING should be released, otherwise this hive lock will be locked forever (apache#3186) [MINOR] Fix build broken from apache#3186 (apache#3245) [HUDI-2136] Fix conflict when flink-sql-connector-hive and hudi-flink-bundle are both in flink lib (apache#3227) [HUDI-2087] Support Append only in Flink stream (apache#3174) Co-authored-by: 喻兆靖 <[email protected]> UnitTest for deltaSync Removing cosmetic changes and reuse function for insert_overwrite_table unit test intial unit test for the insert_overwrite and insert_over_write_table Adding failed test code for insert_overwrite Revert "[HUDI-2087] Support Append only in Flink stream (apache#3174)" (apache#3251) This reverts commit 3715267. [HUDI-2147] Remove unused class AvroConvertor in hudi-flink (apache#3243) [MINOR] Fix some wrong assert reasons (apache#3248) [HUDI-2087] Support Append only in Flink stream (apache#3252) Co-authored-by: 喻兆靖 <[email protected]> [HUDI-2143] Tweak the default compaction target IO to 500GB when flink async compaction is off (apache#3238) [HUDI-2142] Support setting bucket assign parallelism for flink write task (apache#3239) [HUDI-1483] Support async clustering for deltastreamer and Spark streaming (apache#3142) - Integrate async clustering service with HoodieDeltaStreamer and HoodieStreamingSink - Added methods in HoodieAsyncService to reuse code [HUDI-2107] Support Read Log Only MOR Table For Spark (apache#3193) [HUDI-2144]Bug-Fix:Offline clustering(HoodieClusteringJob) will cause insert action losing data (apache#3240) * fixed * add testUpsertPartitionerWithSmallFileHandlingAndClusteringPlan ut * fix CheckStyle Co-authored-by: yuezhang <[email protected]> [MINOR] Fix EXTERNAL_RECORD_AND_SCHEMA_TRANSFORMATION config (apache#3250) [HUDI-2168] Fix for AccessControlException for anonymous user (apache#3264) [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And DeltaStreamer test with insert-overwrite and insert-overwrite-table removing hardcoded action to pass the unit test [HUDI-1969] Support reading logs for MOR Hive rt table (apache#3033) [HUDI-2171] Add parallelism conf for bootstrap operator using delta-commit for insert_overwrite

…to BulkInsert with Rows (apache#3149)

nsivabalan mentioned this pull request Jun 24, 2021

[HUDI-1104] Adding support for UserDefinedPartitioners and SortModes to BulkInsert with Rows #2049

Closed

4 tasks

nsivabalan force-pushed the bulk_insert_simplified_prep_internal_custom_ds branch from b12f427 to bd1ddae Compare June 25, 2021 17:10

nsivabalan added the priority:critical Production degraded; pipelines stalled label Jun 25, 2021

nsivabalan force-pushed the bulk_insert_simplified_prep_internal_custom_ds branch from bd1ddae to 86705be Compare June 28, 2021 15:39

vinothchandar added the priority:blocker Production down; release blocker label Jul 5, 2021

vinothchandar requested changes Jul 6, 2021

View reviewed changes

nsivabalan commented Jul 6, 2021

View reviewed changes

...-spark-common/src/main/java/org/apache/hudi/internal/BulkInsertDataInternalWriterHelper.java Outdated Show resolved Hide resolved

lamberken and others added 7 commits July 6, 2021 10:37

trigger rebuild

212523a

[HUDI-1156] Remove unused dependencies from HoodieDeltaStreamerWrappe…

8f46bff

…r Class (apache#1927)

Adding bulk insert sort modes and user defined bulk insert partitione…

3fda2ee

…r to bulk insert of Rows

Rebasing and adding tests for BulkInsertPartitioners with Rows

1752f5e

Fixing build issues

a304570

fetching and rebasing with master

82f373c

Caching row create handles

af3e97a

nsivabalan force-pushed the bulk_insert_simplified_prep_internal_custom_ds branch from 86705be to f5ad980 Compare July 6, 2021 15:05

Addressing feedback

bbf3285

nsivabalan force-pushed the bulk_insert_simplified_prep_internal_custom_ds branch from f5ad980 to bbf3285 Compare July 6, 2021 22:45

nsivabalan merged commit ea9e5d0 into apache:master Jul 7, 2021

ghost pushed a commit to shivagowda/hudi that referenced this pull request Jul 15, 2021

[HUDI-1104] Adding support for UserDefinedPartitioners and SortModes …

d1385c9

…to BulkInsert with Rows (apache#3149)

ghost pushed a commit to shivagowda/hudi that referenced this pull request Aug 1, 2021

[HUDI-1104] Adding support for UserDefinedPartitioners and SortModes …

e41b03d

…to BulkInsert with Rows (apache#3149)

[HUDI-1104] Adding support for UserDefinedPartitioners and SortModes to BulkInsert with Rows #3149

[HUDI-1104] Adding support for UserDefinedPartitioners and SortModes to BulkInsert with Rows #3149

Uh oh!

Conversation

nsivabalan commented Jun 24, 2021

What is the purpose of the pull request

Verify this pull request

Committer checklist

Uh oh!

hudi-bot commented Jun 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

codecov-commenter commented Jun 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

vinothchandar left a comment

Choose a reason for hiding this comment

Uh oh!

vinothchandar Jul 6, 2021

Choose a reason for hiding this comment

Uh oh!

vinothchandar Jul 6, 2021

Choose a reason for hiding this comment

Uh oh!

nsivabalan Jul 6, 2021

Choose a reason for hiding this comment

Uh oh!

nsivabalan Jul 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nsivabalan Jul 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vinothchandar Jul 6, 2021

Choose a reason for hiding this comment

Uh oh!

nsivabalan Jul 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

hudi-bot commented Jun 24, 2021 •

edited

Loading

codecov-commenter commented Jun 24, 2021 •

edited

Loading

nsivabalan Jul 6, 2021 •

edited

Loading

nsivabalan Jul 6, 2021 •

edited

Loading

nsivabalan Jul 6, 2021 •

edited

Loading