Handle NaN in Parquet statistics#4467
Conversation
30c9e29 to
97e9af8
Compare
findepi
left a comment
There was a problem hiding this comment.
LGTM
do we have storage formats smoke tests to tests full write-read path?
this should be run for ORC as well, even though there isn't a problem to fix there
(like io.prestosql.plugin.hive.TestHiveIntegrationSmokeTest#testNaNPartition)
presto-parquet/src/main/java/io/prestosql/parquet/predicate/TupleDomainParquetPredicate.java
Outdated
Show resolved
Hide resolved
presto-parquet/src/main/java/io/prestosql/parquet/predicate/TupleDomainParquetPredicate.java
Outdated
Show resolved
Hide resolved
presto-parquet/src/main/java/io/prestosql/parquet/predicate/TupleDomainParquetPredicate.java
Outdated
Show resolved
Hide resolved
presto-parquet/src/main/java/io/prestosql/parquet/predicate/TupleDomainParquetPredicate.java
Outdated
Show resolved
Hide resolved
aalbu
left a comment
There was a problem hiding this comment.
do we have storage formats smoke tests to tests full write-read path?
this should be run for ORC as well, even though there isn't a problem to fix there
I did write a round-trip test, but I wasn't able to create stats with NaN, so it didn't fail without the fix 🤨 Inserting data through Presto didn't do it for me and I couldn't figure out how to insert NaN through Hive.
presto-parquet/src/main/java/io/prestosql/parquet/predicate/TupleDomainParquetPredicate.java
Outdated
Show resolved
Hide resolved
97e9af8 to
1bcbbee
Compare
Since we use hive's writer, we depend in hive/parquet version we bundle. |
presto-parquet/src/main/java/io/prestosql/parquet/predicate/TupleDomainParquetPredicate.java
Outdated
Show resolved
Hide resolved
presto-parquet/src/main/java/io/prestosql/parquet/predicate/TupleDomainParquetPredicate.java
Outdated
Show resolved
Hide resolved
presto-parquet/src/main/java/io/prestosql/parquet/predicate/TupleDomainParquetPredicate.java
Outdated
Show resolved
Hide resolved
I have generated Parquet files with Athena and Spark, but I have not had any luck creating |
1bcbbee to
2235b45
Compare
|
Merged, thanks! |
Fixes #4267