[HUDI-3356][HUDI-3142] Metadata index initialization for bloom filters and column stats partitions #4746
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…lter construction from index based on the type param - Write stats are converted to metadata index records during the commit. Making them use the HoodieData<HoodieRecord> type so that the record generation scales up with needs. - When building the BloomFilter from the index records, using the type param stored in the payload instead of hardcoded type.
…n stats partitions - When metadata table is created and when it decides to the initialization from the filesystem for the user dataset, all the enabled partitions need to be initialized along with FILES partition. This fix adds init support for bloom filter and column stats partitions.
5 tasks
Collaborator
codope
reviewed
Feb 7, 2022
|
|
||
| return appendedFileMap.entrySet().stream().flatMap(appendedFileNameLengthPair -> { | ||
| if (!FSUtils.isBaseFile(new Path(appendedFileNameLengthPair.getKey())) | ||
| || !appendedFileNameLengthPair.getKey().endsWith(HoodieFileFormat.PARQUET.getFileExtension())) { |
Member
There was a problem hiding this comment.
I think we should try to generalize this if-condition to a method. I know right now we just get stats from base parquet file but it might change tomorrow.
5 tasks
codope
reviewed
Feb 10, 2022
| List<String> partitions = partitionInfoList.stream().map(p -> | ||
| p.getRelativePath().isEmpty() ? NON_PARTITIONED_NAME : p.getRelativePath()).collect(Collectors.toList()); | ||
| final int totalFiles = partitionInfoList.stream().mapToInt(p -> p.getTotalFiles()).sum(); | ||
| private void initialCommit(String createInstantTime) { |
Member
There was a problem hiding this comment.
+1 for extracting the actual commit part out of bootstrap (initialize filegroup). This will help me for async indexing.
How about adding partitionType as another parameter and making initialCommit part of the HoodieTableMetadataWriter interface? That way we can call from write client or any action executor if needed. Not necessary to do this in PR but just a suggestion to consider.
Member
|
Covered in #4848 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the purpose of the pull request
When metadata table is created and when it decides to the initialization
from the filesystem for the user dataset, all the enabled partitions need
to be initialized along with FILES partition. This fix adds init support
for bloom filter and column stats partitions.
Brief change log
Since the initialization can be for a huge set of files, the records prepared
are in HoodieData format so that they can be passed on
to engine specific commit without materialization at the driver and avoid OOME.
PS: This is stacked on top of #4740
Commit to review: 9c4b90b
(for example:)
Verify this pull request
(Please pick either of the following options)
This pull request is a trivial rework / code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)
Committer checklist
Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.