[HUDI-53] Record Level Index #7429
Conversation
同步 hudi master
|
hey, thanks for the patch. we do already have a patch on this end #5581 just that it was based of of 0.10.1. And it has some enhancements to foundational metadata table before we can bring in record level index. Give me 2 days. I am half way through the other patch to understand how we can split it into multiple patches and for actual record level index, we can incorporate from this patch may be. I will keep you posted. |
| "doc": "FileId of file group" | ||
| }, | ||
| { | ||
| "name": "rowGroupIndex", |
There was a problem hiding this comment.
may I know whats the purpose of row group index ?
There was a problem hiding this comment.
Thanks for review.
This is a parameter reserved for parquet partial updates
#6612
| return HoodieMetadataPayload.createRecordLevelIndexRecord(next.getRecordKey(), next.getPartitionPath(), fileId, rowGroupIndex, isDeleted, fileCommitTime, HoodieOperation.INSERT); | ||
| } | ||
|
|
||
| // int findRowGroupIndex() { |
There was a problem hiding this comment.
guess its not yet used as of this patch. but w/ hfile, may I know how does this row group index help us?
There was a problem hiding this comment.
RowGroupIndex is a reserved attribute, which indicates the serial number of the rowGroup in the parquet file, and it has not been collected in this pr.
|
@nsivabalan Too disappointed, after half a year, the community has not made any progress |
|
Closing this as the Record Level Index is landed in #8758 and included in Hudi 0.14.0 release. |
5 participants
Change Logs
This is the implementation of RFC-08, HUDI-53
Impact
Record Level Index is a new HoodieIndexType.
The mapping relationship between the primary key and the fileId is stored in the hudi meta table.
Risk level (write none, low medium or high below)
Documentation Update
Contributor's checklist