Merge ManifestEntry and DataFile classes #952
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This updates DataFile to contain metadata from ManifestEntry because the separation no longer makes sense. V1 metadata files still use ManifestEntry, but v2 will not.
This reverts commit 76bded6.
rdblue
commented
Apr 23, 2020
|
|
||
| static Schema wrapFileSchema(StructType fileType) { | ||
| return new Schema(STATUS, SNAPSHOT_ID, SEQUENCE_NUMBER, required(DATA_FILE_ID, "data_file", fileType)); | ||
| // remove ManifestEntry fields from the file type when wrapping to avoid duplication |
Contributor
Author
There was a problem hiding this comment.
I should note that this PR doesn't actually add the manifest entry fields to DataFile, it just sets up that change.
Contributor
Author
|
I'm closing this because we plan to keep the separate structs. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This updates
DataFileto contain metadata fromManifestEntrybecause the separation no longer makes sense. V1 metadata files still useManifestEntry, but v2 will not.GenericManifestEntryis split into two parts. First, a wrapper classManifestEntryWrapperthat is used byManifestWriterto replace fields like status when writing aDataFilebecause theDataFilestatus is not set by the caller. Second, a class that makes aDataFileappear like aManifestEntry, calledGenericDataFile.AsManifestEntry. TheGenericManifestEntryclass now extendsAsManifestEntryso that reads that used it don't need to be updated.