[HUDI-1822][RFC-27][WIP] range index support with metadata table

[satishkotha]

What is the purpose of the pull request

Add range index support with metadata table

Brief change log

• This is a WIP PR primarily used for data collection.
• Lot of fundamental metadata table bottlenecks have to be resolved. For example, bootstrapping rangeIndex for 30 partitions of large table (4000 columns) runs into memory issues and driver result too large errors. We also need to implement async metadata table design in V2
• Tried to implement primitive bucketing by column name to improve index locality
• Only implemented commit integration. Clean and other action integration needs to be added.

Verify this pull request

• Verified end-to-end by bootstrapping range index on few partitions of large table (4000 columns)
• Unit Tests TBD

Committer checklist

• Has a corresponding JIRA in PR title & commit

• Commit message is descriptive of the change

• CI is green

• Necessary doc changes done or have another open PR

• For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

[nsivabalan]

@satishkotha : may I know why we need this extra condition. base file is always going to be HFile for metadata table right? Or do we have plans for any other partition where base file will not be Hfile in metadata table?

[satishkotha]

@nsivabalan We were initially considering parquet file format as an option of range index. So I added it as a config to support sorting on parquet files in metadata table. Now that we have finalized HFile format (at least for short term), we can probably remove config. Sorry for delay, I missed your mention. Feel free to ping me on slack if you have any followups

[hudi-bot]

CI report:

• 75b5bc5 Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[nsivabalan]

closing this as have another PR #4352