Parquet: Support constant map for partition values #909
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rdblue
commented
Apr 9, 2020
|
|
||
| protected abstract void set(S struct, int pos, Object value); | ||
|
|
||
| protected Object prepareConstant(Type type, Object value) { |
Contributor
Author
There was a problem hiding this comment.
I'm moving this out of Avro and adding a callback to convert the constants to PartitionUtil.constantsMap. That way, Spark can supply a conversion function and use it in both places, instead of duplicating the conversion in Avro and Parquet readers.
rdblue
force-pushed
the
parquet-support-constant-map
branch
from
6965f40 to
6c5db2d
Compare
rdblue
commented
Apr 9, 2020
| } | ||
|
|
||
| @Override | ||
| protected Object prepareConstant(Type type, Object value) { |
rdblue
force-pushed
the
parquet-support-constant-map
branch
from
3e102e6 to
db7a4c7
Compare
rdsr
reviewed
Apr 11, 2020
Contributor
rdsr
left a comment
rdsr
left a comment
There was a problem hiding this comment.
Minor comment. Otherwise LGTM!
| appender.addAll(expected); | ||
| } | ||
|
|
||
| // add the Avro data file to the source table |
Contributor
There was a problem hiding this comment.
Why not write the data for the parameterized format for which the test is running?
Contributor
Author
There was a problem hiding this comment.
This is just source data for the write from Spark with the target format. Since it isn't part of the test, we don't want it to change at all in ways that might affect the test.
Contributor
Author
|
Thanks for reviewing @rdsr! |
Fokko
pushed a commit
to Fokko/iceberg
that referenced
this pull request
Apr 21, 2020
This is a follow-up to apache#896, which added the same constant map support for Avro. Fixes #575 for Parquet and replaces #585. Andrei did a lot of the work for this in #585. Co-authored-by: Andrei Ionescu <[email protected]>
sunchao
pushed a commit
to sunchao/iceberg
that referenced
this pull request
May 10, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a follow-up to #896, which added the same constant map support for Avro.
Fixes #575 for Parquet and replaces #585. Andrei did a lot of the work for this in #585.
Co-authored-by: Andrei Ionescu [email protected]