Add content types to DataFile and ManifestFile #1030
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This also updates AllEntriesTable to return the correct sequence number.
aokolnychyi
approved these changes
May 23, 2020
Contributor
aokolnychyi
left a comment
aokolnychyi
left a comment
There was a problem hiding this comment.
LGTM. The v1 metadata seems to stay unchanged.
| /** | ||
| * @return the content stored in the file; one of DATA, POSITION_DELETES, or EQUALITY_DELETES | ||
| */ | ||
| default FileContent content() { |
Contributor
There was a problem hiding this comment.
Will DeleteFile extend this or will it be totally separate?
|
|
||
| return CloseableIterable.transform(manifests, manifest -> new BaseFileScanTask( | ||
| DataFiles.fromManifest(manifest), schemaString, specString, ResidualEvaluator.unpartitioned(rowFilter))); | ||
| return CloseableIterable.transform(manifests, manifest -> new ManifestEntriesTable.ManifestReadTask( |
Contributor
There was a problem hiding this comment.
Ugh, this one wouldn't probably work correctly before if snapshot id inheritance was enabled.
| return wrapped.snapshotId(); | ||
| case 2: | ||
| DataFile file = wrapped.file(); | ||
| if (file == null || file instanceof GenericDataFile) { |
Contributor
There was a problem hiding this comment.
Is this change needed because we added FileContent in GenericDataFile?
This was referenced May 25, 2020
waterlx
added a commit
to waterlx/incubator-iceberg
that referenced
this pull request
May 27, 2020
…pdate GenericFlinkManifestFile due to PR apache#1030
waterlx
added a commit
to waterlx/incubator-iceberg
that referenced
this pull request
May 28, 2020
…pdate GenericFlinkManifestFile due to PR apache#1030
waterlx
added a commit
to waterlx/incubator-iceberg
that referenced
this pull request
Jun 3, 2020
…pdate GenericFlinkManifestFile due to PR apache#1030
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds
FileContentandManifestContentto encode the content type in aDataFileorManifestFile. Readers and writers are updated to handle the new metadata fields and values from v1 metadata default toDATA.DataFilealways usesFileContent.DATA. Although the schema is the same in a manifest,DataFilewill be used in the API only for data files, andDeleteFilewill be added to handle delete deltas.This also adds documentation comments to fields, and introduces an
IndexedDataFilefor writing v2 metadata that no longer writes block size and makes content required.