[DO_NOT_MERGE][WIP][HUDI-2285]Sync metadata debug #3716
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…of commits in data timelime. - Added support to do synchronous updates to metadata table. A write will first get commited to metadata table followed by data table. - Reader will ignore any commits found only in metadata table while serving files via metadata table. - Removed option for runtime validation of metadata table against file listing. Validation does not work in several cases especially with multi-writer. So its best to remove it. - Fixed compactions on the Metadata Table. We cannot perform compaction if there are previous inflight operations on the dataset. This is because a compacted metadata base file at time Tx should represent all the actions on the dataset till time Tx. - Added support for buckets in metadata table partitions. 1. There will be fixed number of buckets for each Metadata Table partition. 2. Buckets are implemented using filenames of format bucket-ABCD where ABCD is the bucket number. This allows easy identification of the files and their order while still keeping the names unique. 3. Buckets are pre-allocated during the time of bootstrap. 4. Currently only "files" partition has 1 bucket. But this building block is required for record-level-index and other indices and so implemented here. - Do not archive instants on the dataset if they are newer than latest compaction on metadata table. LogBlocks written to the log file of Metadata Table need to be validated - they are used only if they correspond to a completed action on the dataset. - Handle Metadata Table upgrade/downgrade by deleting the table and re-bootstrapping. The two versions differ in schema and its complicated to check whether the table is in sync. So its simpler to re-bootstrap as its only the file listing which needs to be re-bootstrapped. - Multi-writers in data table is also supported with metadata table. Since each operation on metadata table writes to the same files (file-listing partition has a single FileSlice), we can only allow single-writer access to the metadata table. To ensure this, any commit that happens in data table is guarded within the data table lock. Prior to this patch, table services like compaction and clustering was not taking any locks while committing to data table. But with this patch, added the lock. - Enabling metadata table by default.
nsivabalan
changed the title
Collaborator
nsivabalan
force-pushed
the
sync_metadata_debug
branch
3 times, most recently
from
34d27aa to
0597e01
Compare
Contributor
Author
|
@hudi-bot azure run |
1 similar comment
Contributor
Author
|
@hudi-bot azure run |
nsivabalan
force-pushed
the
sync_metadata_debug
branch
4 times, most recently
from
edf9f3a to
6b38fee
Compare
nsivabalan
force-pushed
the
sync_metadata_debug
branch
from
6b38fee to
e98b2fe
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the purpose of the pull request
Same as #3590. Added log statements to debug CI failures. Locally not reproducible.
Brief change log
(for example:)
Verify this pull request
(Please pick either of the following options)
This pull request is a trivial rework / code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)
Committer checklist
Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.