[HUDI-3494] Consider triggering condition of MOR compaction during archival

[yihua]

What is the purpose of the pull request

For Hudi MOR table, the scheduling of compaction is triggered under certain conditions, configured by hoodie.compact.inline.trigger.strategy. The default triggering condition is the number of delta commits, with the config of hoodie.compact.inline.max.delta.commits. If this setting is larger than the archival config of hoodie.keep.max.commits, there is not enough delta commits in the active timeline and the compaction will never happen.

To guard around such configs, for MOR table with triggering strategy of NUM_COMMITS (trigger compaction when reach N delta commits) and NUM_AND_TIME (trigger compaction when both NUM_COMMITS and TIME_ELAPSED are satisfied), the archival always makes sure that there are enough delta commits in the active timeline to trigger compaction scheduling, besides other conditions.

Brief change log

• Add new logic in HoodieTimelineArchiver to make sure that there are enough delta commits in the active timeline to trigger compaction scheduling, when the trigger strategy of compaction is NUM_COMMITS or NUM_AND_TIME.
• Add util methods in CompactionUtils and refactor ScheduleCompactionActionExecutor to use the same method for checking latest complete compaction and delta commits as HoodieTimelineArchiver
• Fix an issue of checking delta commits in active timeline when the timeline does not have any delta commit
• Add new tests for the new logic.

Verify this pull request

This PR adds new tests in TesthoodieTimelineArchiver and TestCompactionUtils for the new logic.

Committer checklist

• Has a corresponding JIRA in PR title & commit

• Commit message is descriptive of the change

• CI is green

• Necessary doc changes done or have another open PR

• For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

[hudi-bot]

CI report:

• 681207d Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[Zouxxyy]

The default triggering condition is the number of delta commits, with the config of hoodie.compact.inline.max.delta.commits. If this setting is larger than the archival config of hoodie.keep.max.commits, there is not enough delta commits in the active timeline and the compaction will never happen.

why not just throw exception when hoodie.compact.inline.max.delta.commits > hoodie.keep.max.commits

[yihua]

The default triggering condition is the number of delta commits, with the config of hoodie.compact.inline.max.delta.commits. If this setting is larger than the archival config of hoodie.keep.max.commits, there is not enough delta commits in the active timeline and the compaction will never happen.

why not just throw exception when hoodie.compact.inline.max.delta.commits > hoodie.keep.max.commits

I think auto adjustment in archival process is better than failing the job in this case. Also, in Hudi 1.x we plan to deprecate archival as the new LSM timeline design can provide fast lookup on much larger number of commits, and the problem in this PR will not be relevant in Hudi 1.x.