[HUDI-5238] Fixing HoodieMergeHandle shutdown sequence

[alexeykudinkin]

Change Logs

This PR addresses the #7234 related to HoodieMergeHandle shutdown sequence:
in introduced at #4264 we changed the ordering in which we shut down the handle relative to the executor:

Before it was

1. Handle
2. Executor

After

1. Executor
2. Handle

The reason it was switched was to handle the case when during exception thrown executor might still be writing out records, and closing of the handle (before the executor) was leaving some of the produced Parquet files corrupted.

This PR, addresses this issue by making sure that in the successful path we close the Handle immediately as soon as writing has finished (before we shutdown the executor), which would make sure this will not result in any PipeBroken exceptions in GCS

Impact

No impact

Risk level (write none, low medium or high below)

Low

Documentation Update

N/A

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[hudi-bot]

CI report:

• 10b74fc Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[prithiviiiiiiii]

@xushiyan / @alexeykudinkin / @nsivabalan i am facing this exact same issue with 12.0 #7234.

One question: why this happens selectively? In my case, many tables' upserts are running fine with 12.0 except one table. Why?
i just want to understand on what's different in this one table of mine such that it is not upserting on 12.0.

Also how should i fix this. Do i need to migrate to a later version? or is there a better way to fix this