[HUDI-6423] Incremental cleaning should consider inflight compaction instant #9038

zhuanshenbsj1 · 2023-06-22T15:33:13Z

Change Logs

Illustration of issue before fix

Incremental has a sliding window that bounds it's cleaning partition as shown below. If there is an async table service action as shown in the illustration below, partitions might be skipped if it falls behind the cleaning window.

Note: The illustration might not be the entirely correct in the determination of earliest commit to retain, but it does give a general illustration of the sliding window.

Fix

In #7568, clean will be blocked by any pending action. As such, by factoring COMPACTION actions into the active timeline, the sliding window of incremental range will be bounded correctly to not ahead of any pending writes made via compaction.

Impact

Describe any public API or user-facing feature change or any performance impact.

Risk level (write none, low medium or high below)

none

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

zhuanshenbsj1 · 2023-06-22T15:40:16Z

...client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanner.java

  public Option<HoodieInstant> getEarliestCommitToRetain() {
    return CleanerUtils.getEarliestCommitToRetain(
-        hoodieTable.getMetaClient().getActiveTimeline().getCommitsTimeline(),
+        hoodieTable.getMetaClient().getActiveTimeline().getCommitsAndMergesTimeline(),


Previously, there was no consideration of inflight compaction instant. When partitioning by time, if the inflight compaction instant is in the previous partition and is skipped for incremental cleaning, it will cause the log file not be cleaned up.

Nice catch ~

danny0405 · 2023-06-26T10:21:21Z

hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieDefaultTimeline.java

+   */
+  public HoodieTimeline getCommitsAndMergesTimeline() {
+    return getTimelineOfActions(CollectionUtils.createSet(COMMIT_ACTION, DELTA_COMMIT_ACTION, REPLACE_COMMIT_ACTION, COMPACTION_ACTION));
+  }


getCommitsAndMergesTimeline -> getCommitsAndCompactionTimeline

Can we also add a test case for this incremental cleaning scenario, where partition path got switched and the old partition files could not be cleaned.

zhuanshenbsj1 · 2023-06-29T08:31:07Z

@hudi-bot run azure

danny0405 · 2023-07-01T01:22:53Z

@hudi-bot run azure

danny0405 · 2023-07-03T01:43:44Z

There are many test failures:

https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=18249&view=logs&j=600e7de6-e133-5e69-e615-50ee129b3c08&t=bbbd7bcc-ae73-56b8-887a-cd2d6deaafc7&l=16819

https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=18249&view=logs&j=b1544eb9-7ff1-5db9-0187-3e05abf459bc&t=e0ae894b-41c9-5f4b-7ed2-bdf5243b02e7&l=569172

zhuanshenbsj1 · 2023-07-04T01:31:27Z

There are many test failures:

https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=18249&view=logs&j=600e7de6-e133-5e69-e615-50ee129b3c08&t=bbbd7bcc-ae73-56b8-887a-cd2d6deaafc7&l=16819

https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=18249&view=logs&j=b1544eb9-7ff1-5db9-0187-3e05abf459bc&t=e0ae894b-41c9-5f4b-7ed2-bdf5243b02e7&l=569172

TestCleaner.testKeepLatestCommitsWithPendingCompactions has been fixed, seems that TestGlobalIndexEnableUpdatePartitions is not my problem.

danny0405

+1，cc @yihua for visibility ~

zhuanshenbsj1 · 2023-07-04T13:48:25Z

hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/TestCleaner.java

    HoodieTableMetadataWriter metadataWriter = SparkHoodieBackedTableMetadataWriter.create(hadoopConf, config, context);

    final String partition = "2016/03/15";
+    String timePrefix = "00000000000";


Clean instant is after instant-000 before adjustment, this is unreasonable, it should be at the end

before adjustment：
localTimeline=[[000__commit__COMPLETED__20230704200732288], [000000001__clean__COMPLETED__20230704200742843], [001__commit__COMPLETED__20230704200733129], [003__commit__COMPLETED__20230704200734117], [==>004__compaction__REQUESTED__20230704200734125], [005__commit__COMPLETED__20230704200734948], [0055__commit__COMPLETED__20230704200735880], [==>006__compaction__REQUESTED__20230704200735885], [007__commit__COMPLETED__20230704200736807], [0075__commit__COMPLETED__20230704200737690], [==>008__compaction__REQUESTED__20230704200737694], [009__commit__COMPLETED__20230704200738629], [0095__commit__COMPLETED__20230704200739576],[==>010__compaction__REQUESTED__20230704200739580], [011__commit__COMPLETED__20230704200740426], [013__commit__COMPLETED__20230704200741363]

after adjustment：
localTimeline=[[00000000000000__commit__COMPLETED__20230704200400940], [00000000000001__commit__COMPLETED__20230704200401790], [00000000000003__commit__COMPLETED__20230704200402888], [==>00000000000004__compaction__REQUESTED__20230704200402896], [00000000000005__commit__COMPLETED__20230704200403841], [000000000000055__commit__COMPLETED__20230704200404879], [==>00000000000006__compaction__REQUESTED__20230704200404883], [00000000000007__commit__COMPLETED__20230704200405861], [000000000000075__commit__COMPLETED__20230704200406790], [==>00000000000008__compaction__REQUESTED__20230704200406797], [00000000000009__commit__COMPLETED__20230704200407808], [000000000000095__commit__COMPLETED__20230704200408834], [==>00000000000010__compaction__REQUESTED__20230704200408839], [00000000000011__commit__COMPLETED__20230704200410653], [00000000000013__commit__COMPLETED__20230704200411934], [00000000000014__clean__COMPLETED__20230704200413695]]

zhuanshenbsj1 · 2023-07-04T14:03:58Z

@hudi-bot run azure

hudi-bot · 2023-07-04T22:31:46Z

CI report:

a65a29c UNKNOWN
43c37c8 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

voonhous · 2023-07-20T08:51:11Z

Description and context please...

Illustration of issue before fix

Incremental has a sliding window that bounds it's cleaning partition as shown below. If there is an async table service action as shown in the illustration below, partitions might be skipped if it falls behind the cleaning window.

Note: The illustration might not be the entirely correct in the determination of earliest commit to retain, but it does give a general illustration of the sliding window.

Fix

In #7568, clean will be blocked by any pending action. As such, by factoring COMPACTION actions into the active timeline, the sliding window of incremental range will be bounded correctly to not ahead of any pending writes made via compaction.

zhuanshenbsj1 · 2023-07-20T09:14:23Z

Thanks for the description of the scene~~ Added to current pr change-logs. @voonhous

zhuanshenbsj1 commented Jun 22, 2023

View reviewed changes

danny0405 assigned yihua Jun 26, 2023

danny0405 added the area:table-service Table services label Jun 26, 2023

danny0405 reviewed Jun 26, 2023

View reviewed changes

zhuanshenbsj1 force-pushed the HUDI-6423 branch 2 times, most recently from a65a29c to 34f8823 Compare June 29, 2023 06:19

zhuanshenbsj1 force-pushed the HUDI-6423 branch from 34f8823 to b58c9f0 Compare July 1, 2023 12:39

zhuanshenbsj1 force-pushed the HUDI-6423 branch from f7bbb54 to 33e1774 Compare July 3, 2023 14:50

zhuanshenbsj1 force-pushed the HUDI-6423 branch from 33e1774 to 0801d38 Compare July 4, 2023 06:10

danny0405 added release-0.14.0 priority:blocker Production down; release blocker labels Jul 4, 2023

danny0405 approved these changes Jul 4, 2023

View reviewed changes

zhuanshenbsj1 commented Jul 4, 2023

View reviewed changes

zhuanshenbsj1 added 3 commits July 4, 2023 22:08

addInflightCompaction

ebef627

add ut

42bfdfc

fxt testKeepLatestCommitsWithPendingCompaction error

6880871

zhuanshenbsj1 force-pushed the HUDI-6423 branch 2 times, most recently from 5b354dd to 68d67e5 Compare July 4, 2023 15:12

adjust instant index for testPendingCompactions

43c37c8

zhuanshenbsj1 force-pushed the HUDI-6423 branch from 68d67e5 to 43c37c8 Compare July 4, 2023 15:17

danny0405 approved these changes Jul 5, 2023

View reviewed changes

danny0405 merged commit 0716440 into apache:master Jul 5, 2023

danny0405 mentioned this pull request Jul 18, 2023

[SUPPORT] Why some delta commit logs files are not converted to parquet ？ #8267

Open

voonhous mentioned this pull request Jul 21, 2023

[HUDI-6574] Fix the problem that incremental clean cannot be executed when the earliest ActiveTimeline is a pending commit. #9243

Closed

4 tasks

lokeshj1703 mentioned this pull request Jul 25, 2023

[HUDI-6548] Two log compaction instants can be scheduled at the same time #9246

Closed

4 tasks

zhuanshenbsj1 deleted the HUDI-6423 branch April 11, 2024 08:00

[HUDI-6423] Incremental cleaning should consider inflight compaction instant #9038

[HUDI-6423] Incremental cleaning should consider inflight compaction instant #9038

Uh oh!

Conversation

zhuanshenbsj1 commented Jun 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Logs

Illustration of issue before fix

Fix

Impact

Risk level (write none, low medium or high below)

Documentation Update

Contributor's checklist

Uh oh!

zhuanshenbsj1 Jun 22, 2023

Choose a reason for hiding this comment

Uh oh!

danny0405 Jun 23, 2023

Choose a reason for hiding this comment

Uh oh!

danny0405 Jun 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhuanshenbsj1 Jun 28, 2023

Choose a reason for hiding this comment

Uh oh!

zhuanshenbsj1 commented Jun 29, 2023

Uh oh!

danny0405 commented Jul 1, 2023

Uh oh!

danny0405 commented Jul 3, 2023

Uh oh!

zhuanshenbsj1 commented Jul 4, 2023

Uh oh!

danny0405 left a comment

Choose a reason for hiding this comment

Uh oh!

zhuanshenbsj1 Jul 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhuanshenbsj1 commented Jul 4, 2023

Uh oh!

hudi-bot commented Jul 4, 2023

CI report:

Uh oh!

voonhous commented Jul 20, 2023

Illustration of issue before fix

Fix

Uh oh!

zhuanshenbsj1 commented Jul 20, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

zhuanshenbsj1 commented Jun 22, 2023 •

edited

Loading

danny0405 Jun 26, 2023 •

edited

Loading

zhuanshenbsj1 Jul 4, 2023 •

edited

Loading