[HUDI-6068] Improve logic of getOldestInstantToRetainForClustering wh…

[hbgstc123]

Change Logs

Original ClusteringUtils::getOldestInstantToRetainForClustering is based on inflight clean instant, but there maybe a moment when the last clean is complete and the next clean plan not generated, if timeline archive execute at this moment, no replace commit will be retained.
This pr propose to decide OldestInstantToRetainForClustering based on latest completed clean instant, return the first replace commit after the earliestInstantToRetain of last complete clean or first replace commit after last clean instant if earliestInstantToRetain is empty, and return the first replace commit in active timeline if there is no clean instant.

Impact

no

Risk level (write none, low medium or high below)

low

Documentation Update

no

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[danny0405]

Nice catch, using completed clean instants is more reliable.

[danny0405]

Generally looks good, cc @bvaradar for another round of review~

[SteNicholas]

@hbgstc123, which border situation could cause that there maybe a moment when the last clean is complete and the next clean plan not generated, if timeline archive execute at this moment, no replace commit will be retained？

[hbgstc123]

@hbgstc123, which border situation could cause that there maybe a moment when the last clean is complete and the next clean plan not generated, if timeline archive execute at this moment, no replace commit will be retained？

For example in flink pipeline, clean is scheduled and executed in class Cleanfunction when function snapshotState() is invoked at the beginning of a checkpoint, so after a clean is complete and before the next checkpoint is triggered, there is no inflight clean instant in timeline.
And even if last clean operation complete the moment before the next checkpoint begin, the generation of clean plan will take time, the bigger the table the longer it may takes, during this clean planning time, there is no inflight clean instant in timeline neither.

[hudi-bot]

CI report:

• 0dff5a6 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[hbgstc123]

@bvaradar pls cc when convenient, thanks

[bvaradar]

Good catch. Thanks for the fix.

[danny0405]

The test is failing continously after the fix: https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=16792&view=logs&j=dcedfe73-9485-5cc5-817a-73b61fc5dcb0&t=746585d8-b50a-55c3-26c5-517d93af9934&l=33925,

can you take a look ~