-
Notifications
You must be signed in to change notification settings - Fork 4k
ARROW-8425: [Rust] [Parquet] Correct temporal IO #8926
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Codecov Report
@@ Coverage Diff @@
## master #8926 +/- ##
==========================================
- Coverage 83.22% 83.18% -0.04%
==========================================
Files 196 196
Lines 48232 48341 +109
==========================================
+ Hits 40142 40214 +72
- Misses 8090 8127 +37
Continue to review full report at Codecov.
|
jorgecarleitao
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I did not see any changes in the tests, which I would expect for a change in semantics. Aren't we not testing this yet, or how can I verify that it is now correct?
Contains the following: - Fixes the date64 writer by writing the array as a timestamp milli - Adds support for reading and writing interval types > Apache Spark doesn't read intervals (as they're fixed len binary) > It looks like the CPP impl at v2.0.0 just reads back the binary data. The binary data read is however correct.
There were existing tests, but they were ignored. So I removed the I was writing date64 as timestampmilli, but this wasn't per the spec, so I fixed that now. I also added an interval test to increase test coverage. |
jorgecarleitao
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
…nt types This PR adds capabilities to read decimal columns in parquet files that store them as i32 or i64. I tried to follow the approach in #8926 by using casts. Closes #9047 from sweb/ARROW-11072/support-int-types Authored-by: Florian Müller <[email protected]> Signed-off-by: Andrew Lamb <[email protected]>
Contains the following:
The result is that we can now correctly read and write the temporal types.
The Duration type remains unsupported.