Add PartitionKey to the public API #1195
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
chenjunjiedada
approved these changes
Jul 11, 2020
Collaborator
chenjunjiedada
left a comment
chenjunjiedada
left a comment
There was a problem hiding this comment.
+1, The Flink row wrapper should be very simple since it uses java native types.
openinx
approved these changes
Jul 12, 2020
Member
openinx
left a comment
openinx
left a comment
There was a problem hiding this comment.
The patch looks good to me, it don't need the public BasePartitionKey any more, and simplified the logics about Accessor. , Please merge it first, then I cloud provide the flink partitionKey.
Thanks.
|
Been following this from afar. +1 |
Contributor
|
+1 LGTM |
rdblue
added a commit
to rdblue/iceberg
that referenced
this pull request
Jul 29, 2020
* Add PartitionKey to the public API. * Handle null values.
aokolnychyi
pushed a commit
to aokolnychyi/iceberg
that referenced
this pull request
Aug 18, 2020
* Add PartitionKey to the public API. * Handle null values.
cmathiesen
pushed a commit
to ExpediaGroup/iceberg
that referenced
this pull request
Aug 19, 2020
* Add PartitionKey to the public API. * Handle null values.
rdblue
added a commit
to rdblue/iceberg
that referenced
this pull request
Aug 19, 2020
* Add PartitionKey to the public API. * Handle null values.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a refactor similar to #1175 that is intended to make the
PartitionKeyclass used by Spark public. This PR uses a different approach that doesn't require creating sub-classes ofPartitionKeyand customAccessorclasses. Instead, the publicPartitionKeyclass can partition anyStructLike, and a reusable wrapper is added to adapt Spark'sInternalRowtoStructLike.This approach separates concerns. Conversion to the internal representations used by Iceberg classes is handled by the row wrapper, which allows the same
Accessorclass to be used everywhere. That wrapper class could be used to pass rows to an expression evaluator or toPartitionKey, as in #983.