[HUDI-2917] rollback insert data appended to log file when using Hbase Index

[nsivabalan]

What is the purpose of the pull request

Redo of #4446

We may find some data which should be rollbacked in hudi table.

Root cause:

Let's first recall how rollback plan generated about log blocks for deltaCommit. Hudi takes two cases into consideration.

For some log file with no base file, they are comprised by records which are all 'insert record'. Delete them directly. Here we assume all inserted record should be covered by this way.
For those fileID which are updated according to inflight commit meta of instant we want to rollback, we append command block to these log file to rollback. Here all updated record are handled.
However, the first condition is not always true. For indexes which can index log file, they could insert record to some existing log file. In current process, inflight hoodieCommitMeta was generated before they are assigned to specific filegroup.

Brief change log

1. make upsert partitioner generate an execution workload stats which including all fileGroup will be written into comparing with workload generated by input data. This will cover the case that insert data is written into some log files which is recognized as small file when using Hbase Index.
2. In such case, we cannot guarantee that all log files which contains only insert data could be deleted in rollback. They may be rollback using command block. So handle this case in compactor.

Verify this pull request

• Added integration tests for end-to-end: spark client

Committer checklist

• Has a corresponding JIRA in PR title & commit

• Commit message is descriptive of the change

• CI is green

• Necessary doc changes done or have another open PR

• For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

[nsivabalan]

@codope : can you review this patch.

[nsivabalan]

@guanziyue : feel free to review the patch.

[nsivabalan]

@codope : good to review again. fixed unnecessary updates/populating output workload stats if not required.

[codope]

LGTM. If the else block is not necessary then it's better to remove it. You can land it after that.

[nsivabalan]

@hudi-bot run azure

[hudi-bot]

CI report:

• 260c29f Azure: FAILURE Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build