-
Notifications
You must be signed in to change notification settings - Fork 4k
ARROW-6187: [C++] Fallback to storage type when writing ExtensionType to Parquet #5436
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ARROW-6187: [C++] Fallback to storage type when writing ExtensionType to Parquet #5436
Conversation
|
It would be better to store the extension name and serialization in the Parquet metadata, no? Then you can recreate a true extension array when reading? |
|
Yes, that was the idea to look into next. But I can probably directly try that in this PR. |
|
I think that would be nice, since that would validate the whole approach. The metadata keys are currently private in Perhaps you could instead use Parquet-specific metadata keys if Parquet has a well-known convention for them, though. |
jorisvandenbossche
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To a potential reviewer already looking at it, this is currently a very ugly implementation(just wanted to have something working now to pass the tests), will try to clean-up tomorrow.
|
The implementation looks good on the principle, feel free to ping when this is ready for review. |
7b54d86 to
61d245e
Compare
6143946 to
e56164b
Compare
|
This should be ready for review now. |
pitrou
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks!
https://issues.apache.org/jira/browse/ARROW-6187