[HUDI-3782] Fixing table config when any of the index is disabled #5222
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nsivabalan
requested changes
Apr 4, 2022
Contributor
nsivabalan
left a comment
nsivabalan
left a comment
There was a problem hiding this comment.
mostly minor comments
...di-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
Show resolved
Hide resolved
| } | ||
|
|
||
| if (partitionTypes.contains(MetadataPartitionType.FILES)) { | ||
| // Record which saves the list of all partitions |
Contributor
There was a problem hiding this comment.
may I know why do we need this if condition. can you help clarify.
Member
Author
There was a problem hiding this comment.
This just saves some duplicate effort. There was no correctness issue in absence of this if condition. For e.g., when colstats is re-enabled, we are reusing this method and then we don't really need to redo the files partition, hence this if condition. In absence of this condition, HoodieMetadataPayload.createPartitionListRecord(partitions) would have been called everytime.
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java
Outdated
Show resolved
Hide resolved
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java
Outdated
Show resolved
Hide resolved
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java
Outdated
Show resolved
Hide resolved
...di-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
Show resolved
Hide resolved
Collaborator
nsivabalan
approved these changes
Apr 6, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the purpose of the pull request
Fix HUDI-3782
Now that we have table configs for inflight and completed metadata partitions, and given that both reader and writer are going to rely on this config, we need to ensure that this config is updated properly even when any one of the metadata indexes is enabled/disabled.
For instance, let's say user started with metadata enabled and colstats enabed. Colstats partition was fully built and the table config got updated. Now, they disable it and we miss to update table config. So the writers won't update the colstats partition and it would be out of sync. Readers think that the table config has it so it's already in sync but that's incorrect. This patch fixes this issue.
Brief change log
Verify this pull request
Added a unti test which first writes with colstats enabled, then disabled (btu metadata enabled), and then re-enabled. Validated table config and metadata.
Committer checklist
Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.