[HUDI-2815] add partial overwrite payload to support partial overwrit…#4724

Closed

stayrascal wants to merge 137 commits intoapache:masterfrom

stayrascal:HUDI-2815

Commits on Jan 30, 2022

[HUDI-2815] add partial overwrite payload to support partial overwrite case
stayrascal
committed

Commits on Feb 6, 2022

Commits on Mar 22, 2022

Commits on Mar 25, 2022

Merge branch 'master' into HUDI-2815
stayrascal
committed

Commits on Apr 2, 2022

[HUDI-3521] Fixing kakfa key and value serializer value type from class to string (#4919)

nsivabalan
authored and
stayrascal
committed
[HUDI-3018] Adding validation to dataframe scheme to ensure reserved field does not have diff data type (#4852)

nsivabalan
authored and
stayrascal
committed
[MINOR] Change MINI_BATCH_SIZE to 2048 (#4862)

cuibo01
authored and
stayrascal
committed
[HUDI-2917] rollback insert data appended to log file when using Hbase Index (#4840)

authored and
stayrascal
committed
[HUDI-3528] Fix String convert issue and overwrite putAll method in TypedProperties.java (#4920)
stayrascal
committed
[HUDI-3341] Fix log file reader for S3 with hadoop-aws 2.7.x (#4897)

yihua
authored and
stayrascal
committed
[HUDI-3450] Avoid passing empty string spark master to hudi cli (#4844)

authored and
stayrascal
committed
[HUDI-3418] Save timeout option for remote RemoteFileSystemView (#4809)

yuzhaojing
authored and
stayrascal
committed
[HUDI-3465] Add validation of column stats and bloom filters in HoodieMetadataTableValidator (#4878)

yihua
authored and
stayrascal
committed
[HUDI-3497] Adding Datatable validator tool (#4902)

nsivabalan
authored and
stayrascal
committed
[HUDI-3441] Add support for "marker delete" in hudi-cli (#4922)

XuQianJin-Stars
authored and
stayrascal
committed
[HUDI-3516] Implement record iterator for HoodieDataBlock (#4909)

cuibo01
authored and
stayrascal
committed
[HUDI-2631] In CompactFunction, set up the write schema each time with the latest schema (#4000)

yuzhaojing
authored and
stayrascal
committed
[HUDI-3469] Refactor `HoodieTestDataGenerator` to provide for reproducible Builds (#4866)

Alexey Kudinkin
authored and
stayrascal
committed
[HUDI-3315] RFC-35 Part-1 Support bucket index in Flink writer (#4679)

garyli1019
authored and
stayrascal
committed
[minor] Cosmetic changes following HUDI-3315 (#4934)

danny0405
authored and
stayrascal
committed
[MINOR] Adding more test props to integ tests (#4935)

nsivabalan
authored and
stayrascal
committed
[MINOR] RFC-38 markdown content error (#4933)

liujinhui1994
authored and
stayrascal
committed
[HUDI-3264]: made schema registry urls configurable with MTDS (#4779)

pratyakshsharma
authored and
stayrascal
committed
[HUDI-2973] RFC-27: Data skipping index to improve query performance (#4728)

manojpec
authored and
stayrascal
committed
[HUDI-3544] Fixing "populate meta fields" update to metadata table (#4941)

authored and
stayrascal
committed
[HUDI-3552] Strength the NetworkUtils#getHostname by checking network interfaces first (#4942)

danny0405
authored and
stayrascal
committed
[HUDI-3548] Fix if user specify key "hoodie.datasource.clustering.async.enable" directly, async clustering not work (#4905)

authored and
stayrascal
committed
[HUDI-3445] Support Clustering Command Based on Call Procedure Command for Spark SQL (#4901)

authored and
stayrascal
committed
[HUDI-3161][RFC-47] Add Call Produce Command for Spark SQL (#4607)

XuQianJin-Stars
authored and
stayrascal
committed
[MINOR] fix UTC timezone config (#4950)

YuweiXiao
authored and
stayrascal
committed
[HUDI-3348] Add UT to verify HoodieRealtimeFileSplit serde (#4951)

xushiyan
authored and
stayrascal
committed
[HUDI-3460] Add reader merge memory option for flink (#4911)

cuibo01
authored and
stayrascal
committed
[HUDI-2761] Fixing timeline server for repeated refreshes (#4812)

nsivabalan
authored and
stayrascal
committed
[HUDI-3130] Fixing Hive getSchema for RT tables addressing different partitions having different schemas (#4468)

authored and
stayrascal
committed
[HUDI-3520] Introduce DeleteSupportSchemaPostProcessor to support adding _hoodie_is_deleted column to schema (#4921)

wangxianghu
authored and
stayrascal
committed
[HUDI-3525] Introduce JsonkafkaSourceProcessor to support data preprocess before it is transformed to DataSet (#4930)

wangxianghu
authored and
stayrascal
committed
[HUDI-3069] Improve HoodieMergedLogRecordScanner avoid putting unnecessary hoodie records (#4932)

authored and
stayrascal
committed
[HUDI-3213] Making commit preserve metadata to true for compaction (#4811)

nsivabalan
authored and
stayrascal
committed
[HUDI-3561] Avoid including whole `MultipleSparkJobExecutionStrategy` object into the closure for Spark to serialize (#4954)

Alexey Kudinkin
authored and
stayrascal
committed
[HUDI-3365] Make sure Metadata Table records are updated appropriately on HDFS (#4739)

Alexey Kudinkin
authored and
stayrascal
committed
[HUDI-2747] support set --sparkMaster for MDT cli (#4964)

authored and
stayrascal
committed
[HUDI-3576] Configuring timeline refreshes based on latest commit (#4973)

nsivabalan
authored and
stayrascal
committed
[HUDI-3573] flink cleanFuntion execute clean on initialization (#4936)

todd5167
authored and
stayrascal
committed
[MINOR][HUDI-3460]Fix HoodieDataSourceITCase

cuibo01
authored and
stayrascal
committed
[HUDI-2677] Add DFS based message queue for flink writer[part3] (#4961)

danny0405
authored and
stayrascal
committed
[HUDI-3574] Improve maven module configs for different spark profiles (#4970)

XuQianJin-Stars
authored and
stayrascal
committed
[HUDI-3584] Skip integ test modules by default (#4986)

xushiyan
authored and
stayrascal
committed
[HUDI-3356][HUDI-3203] HoodieData for metadata index records; BloomFilter construction from index based on the type param (#4848)

authored and
stayrascal
committed
[HUDI-3221] Support querying a table as of a savepoint (#4720)

XuQianJin-Stars
authored and
stayrascal
committed
[HUDI-3587] Making SupportsUpgradeDowngrade serializable (#4991)

nsivabalan
authored and
stayrascal
committed
[HUDI-3568] Introduce ChainedSchemaPostProcessor to support setting multi processors at once (#4968)

wangxianghu
authored and
stayrascal
committed
[HUDI-3383] Sync column comments while syncing a hive table (#4960)

MrSleeping123
authored and
stayrascal
committed
[MINOR] Add IT CI Test timeout option (#5003)

XuQianJin-Stars
authored and
stayrascal
committed
[HUDI-3396] Make sure `BaseFileOnlyViewRelation` only reads projected columns (#4818)

Alexey Kudinkin
authored and
stayrascal
committed
[HUDI-3581] Reorganize some clazz for hudi flink (#4983)

danny0405
authored and
stayrascal
committed
[HUDI-3602][DOCS] Update docker README to build multi-arch images using buildx (#5011)

codope
authored and
stayrascal
committed
[HUDI-3586] Add Trino Queries in integration tests (#4988)

yihua
authored and
stayrascal
committed
[HUDI-3595] Fixing NULL schema provider for empty batch (#5002)

nsivabalan
authored and
stayrascal
committed
[HUDI-3522] Introduce DropColumnSchemaPostProcessor to support drop columns from schema (#4972)

wangxianghu
authored and
stayrascal
committed
[HUDI-2999] [RFC-42] RFC for consistent hashing index (#4326)

authored and
stayrascal
committed
[HUDI-3566] Add thread factory in BoundedInMemoryExecutor (#4926)

authored and
stayrascal
committed
[HUDI-3575] Use HoodieTestDataGenerator#TRIP_SCHEMA as example schema in TestSchemaPostProcessor (#5019)

wangxianghu
authored and
stayrascal
committed
[HUDI-3567] Refactor HoodieCommonUtils to make code more reasonable (#4982)

huberylee
authored and
stayrascal
committed
[HUDI-3513] Make sure Column Stats does not fail in case it fails to load previous Index Table state (#5015)

Alexey Kudinkin
authored and
stayrascal
committed
[HUDI-3592] Fix NPE of DefaultHoodieRecordPayload if Property is empty (#4999)

authored and
stayrascal
committed
[HUDI-3569] Introduce ChainedJsonKafkaSourePostProcessor to support setting multi processors at once (#4969)

wangxianghu
authored and
stayrascal
committed
[HUDI-3556] Re-use rollback instant for rolling back of clustering and compaction if rollback failed mid-way (#4971)

nsivabalan
authored and
stayrascal
committed
[HUDI-3593] Restore TypedProperties and flush checksum in table config (#5013)

codope
authored and
stayrascal
committed
[HUDI-3583] Fix MarkerBasedRollbackStrategy NoSuchElementException (#4984)

authored and
stayrascal
committed
[HUDI-3501] Support savepoints command based on Call Produce Command (#5025)

XuQianJin-Stars
authored and
stayrascal
committed
[HUDI-3613] Adding/fixing yamls for metadata (#5029)

nsivabalan
authored and
stayrascal
committed
[HUDI-3600] Tweak the default cleaning strategy to be more streaming friendly for flink (#5010)

danny0405
authored and
stayrascal
committed
fix NPE when run schdule using spark-sql if the commits time < hoodie.compact.inline.max.delta.commits (#4976)

peanut-chenzhong
authored and
stayrascal
committed
[MINODR] Remove repeated kafka-clients dependencies (#5034)

wangxianghu
authored and
stayrascal
committed
[HUDI-3621] Fixing NullPointerException in DeltaStreamer (#5039)

nsivabalan
authored and
stayrascal
committed
[HUDI-3623] Removing hive sync node from non hive yamls (#5040)

nsivabalan
authored and
stayrascal
committed
[HUDI-3620] Adding spark3.2.0 profile (#5038)

nsivabalan
authored and
stayrascal
committed
[HUDI-3547] Introduce MaxwellSourcePostProcessor to extract data from Maxwell json string (#4987)

wangxianghu
authored and
stayrascal
committed
[HUDI-3606] Add `org.objenesis:objenesis` to hudi-timeline-server-bundle pom (#5017)

cdmikechen
authored and
stayrascal
committed
[HUDI-3619] Fix HoodieOperation fromValue using wrong constant value (#5033)

authored and
stayrascal
committed
[HUDI-3514] Rebase Data Skipping flow to rely on MT Column Stats index (#4948)

Alexey Kudinkin
authored and
stayrascal
committed
[HUDI-3633] Allow non-string values to be set in TypedProperties (#5045)

codope
authored and
stayrascal
committed
[HUDI-3589] flink sync hive metadata supports table properties and serde properties (#4995)

todd5167
authored and
stayrascal
committed
[HUDI-3588] Remove hudi-common and hudi-hadoop-mr jars in Presto Docker image (#4997)

yihua
authored and
stayrascal
committed
[HUDI-3607] Support backend switch in HoodieFlinkStreamer (#5032)

liufangqi
authored and
stayrascal
committed
[Hudi-3376] Add an option to skip under deletion files for HoodieMetadataTableValidator (#4994)

authored and
stayrascal
committed
[HUDI-3404] Automatically adjust write configs based on metadata table and write concurrency mode (#4975)

yihua
authored and
stayrascal
committed
[HUDI-3494] Consider triggering condition of MOR compaction during archival (#4974)

yihua
authored and
stayrascal
committed
[HUDI-3645] Fix NPE caused by multiple threads accessing non-thread-safe HashMap (#5028)

fengjian428
authored and
stayrascal
committed
[HUDI-2439] Replace RDD with HoodieData in HoodieSparkTable and commit executors (#4856)

xushiyan
authored and
stayrascal
committed
[MINOR] HoodieFileScanRDD could print null path (#5056)

authored and
stayrascal
committed
[HUDI-3598] Row Data to Hoodie Record Operator parallelism needs to always be consistent with input operator (#5049)

authored and
stayrascal
committed
[HUDI-3656] Adding medium sized dataset for clustering and minor fixes to integ tests (#5063)

nsivabalan
authored and
stayrascal
committed
[HUDI-3659] Reducing the validation frequency with integ tests (#5067)

nsivabalan
authored and
stayrascal
committed
[HUDI-3457] Refactored Spark DataSource Relations to avoid code duplication (#4877)

Alexey Kudinkin
authored and
stayrascal
committed
[HUDI-3663] Fixing Column Stats index to properly handle first Data Table commit (#5070)

Alexey Kudinkin
authored and
stayrascal
committed
[MINOR] Remove flaky assert in TestInLineFileSystem (#5069)

yihua
authored and
stayrascal
committed
[HUDI-3665] Support flink multiple versions (#5072)

danny0405
authored and
stayrascal
committed
[MINOR] Fixing sparkUpdateNode for record generation (#5079)

nsivabalan
authored and
stayrascal
committed
[HUDI-3559] Flink bucket index with COW table throws NoSuchElementException

wxplovecc
authored and
stayrascal
committed
[HUDI-1436]: Provide an option to trigger clean every nth commit (#4385)

authored and
stayrascal
committed
[HUDI-3640] Set SimpleKeyGenerator as default in 2to3 table upgrade for Spark engine (#5075)

yihua
authored and
stayrascal
committed
[HUDI-2883] Refactor hive sync tool / config to use reflection and standardize configs (#4175)

authored and
stayrascal
committed
[HUDI-3642] Handle NPE due to empty requested replacecommit metadata (#5090)

codope
authored and
stayrascal
committed
Fixing non partitioned all files record in MDT (#5108)

nsivabalan
authored and
stayrascal
committed
[minor] Checks the data block type for archived timeline (#5106)

danny0405
authored and
stayrascal
committed
[HUDI-3689] Fix glob path and hive sync in deltastreamer tests (#5117)

authored and
stayrascal
committed
[HUDI-3684] Fixing NPE in `ParquetUtils` (#5102)

Alexey Kudinkin
authored and
stayrascal
committed
[HUDI-3689] Remove Azure CI cache (#5121)

xushiyan
authored and
stayrascal
committed
[HUDI-3689] Fix UT failures in TestHoodieDeltaStreamer (#5120)

xushiyan
authored and
stayrascal
committed
[HUDI-3706] Downgrade maven surefire and failsafe version (#5123)

yihua
authored and
stayrascal
committed
[HUDI-3689] Fix delta streamer tests (#5124)

xushiyan
authored and
stayrascal
committed
[HUDI-3689] Disable flaky tests in TestHoodieDeltaStreamer (#5127)

yihua
authored and
stayrascal
committed
[HUDI-3624] Check all instants before starting a commit in metadata table (#5098)

yihua
authored and
stayrascal
committed
[HUDI-3638] Make ZookeeperBasedLockProvider serializable (#5112)

yihua
authored and
stayrascal
committed
[HUDI-3701] Flink bulk_insert support bucket hash index (#5118)

danny0405
authored and
stayrascal
committed
[HUDI-1180] Upgrade HBase to 2.4.9 (#5004)

authored and
stayrascal
committed
[HUDI-3703] Reset taskID in restoreWriteMetadata (#5122)

yuzhaojing
authored and
stayrascal
committed
[HUDI-3580] Claim RFC number 48 for LogCompaction action RFC (#5128)

suryaprasanna
authored and
stayrascal
committed
[HUDI-3678] Fix record rewrite of create handle when 'preserveMetadata' is true (#5088)

danny0405
authored and
stayrascal
committed
[HUDI-3594] Supporting Composite Expressions over Data Table Columns in Data Skipping flow (#4996)

Alexey Kudinkin
authored and
stayrascal
committed
[HUDI-3711] Fix typo in MaxwellJsonKafkaSourcePostProcessor.Config#PRECOMBINE_FIELD_TYPE_PROP (#5096)

wangxianghu
authored and
stayrascal
committed
[HUDI-3563] Make quickstart examples covered by CI tests (#5082)

XuQianJin-Stars
authored and
stayrascal
committed

Commits

Commits on Jan 30, 2022

Commits on Feb 6, 2022

Commits on Feb 7, 2022

Commits on Feb 21, 2022

Commits on Feb 25, 2022

Commits on Feb 26, 2022

Commits on Feb 27, 2022

Commits on Mar 8, 2022

Commits on Mar 16, 2022

Commits on Mar 22, 2022

Commits on Mar 25, 2022

Commits on Apr 2, 2022