[HUDI-5289] Fix writeStatus RDD recalculated in cluster

[Zouxxyy]

Change Logs

fix https://issues.apache.org/jira/projects/HUDI/issues/HUDI-5289

There is a patch similar to this problem, but after this patch is added, the problem still exists.

Impact

Could improve the clustering performance.

Risk level (write none, low medium or high below)

low

Documentation Update

None

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[nsivabalan]

when I put up the fix, I did verify that clustering was not triggered multiple times. Can you confirm that you are seeing clustering being triggered twice?
If you keep inspecting the data directory, you will find additional data files just during the validaiton.
Or other way to test this is, marker directory reconciliation will delete some additional files if dag was triggered twice. If not, you should not see any marker file deletions.

[Zouxxyy]

@nsivabalan yes, I confirm, and @boneanxs find it too, So I think we should persist it

[hudi-bot]

CI report:

• 5f13f4c Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[boneanxs]

@nsivabalan yes, I confirm, and @boneanxs find it too, So I think we should persist it

I find there was a issue when I first implemented clustering as row, but didn't investigate too much at that time.

But I'm thinking whether we can do collectAsList and then parallelize this list to build HoodieData[WriteStatus] in performClusteringWithRecordsRDD, just like performClusteringWithRecordsAsRow has already done (dereference the RDD to a list) to permanently fix this?

[nsivabalan]

we can't rely on cache since cache could be invalidated and we have had experiences where actual files written did not match w/ whats in writeStatus. Hence if something fails, we let dag to get retriggered here. we added guard rails at BasecommitActionExecutor by means of cloning.
CC @alexeykudinkin

[nsivabalan]

Are you guys saying that, even in a successful path, the dag is getting triggered twice. Or it happens only during exception path.

[Zouxxyy]

@nsivabalan
yes, even in a successful path, the dag is getting triggered twice. But you must turn off hoodie.datasource.write.row.writer.enable, because when it is turned on, it will avoid recalculation becauce of the collect at HoodieDatasetBulkInsertHelper. At present, it seems that we do not recommend using the persist, and I will close these two PRs.