Skip to content

Commit 9341c95

Browse files
henryrgatorsmile
authored andcommitted
[SPARK-23852][SQL] Add test that fails if PARQUET-1217 is not fixed
## What changes were proposed in this pull request? Add a new test that triggers if PARQUET-1217 - a predicate pushdown bug - is not fixed in Spark's Parquet dependency. ## How was this patch tested? New unit test passes. Author: Henry Robinson <[email protected]> Closes #21284 from henryr/spark-23852.
1 parent 9e3bb31 commit 9341c95

File tree

2 files changed

+10
-0
lines changed

2 files changed

+10
-0
lines changed
321 Bytes
Binary file not shown.

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -648,6 +648,16 @@ class ParquetFilterSuite extends QueryTest with ParquetTest with SharedSQLContex
648648
}
649649
}
650650
}
651+
652+
test("SPARK-23852: Broken Parquet push-down for partially-written stats") {
653+
// parquet-1217.parquet contains a single column with values -1, 0, 1, 2 and null.
654+
// The row-group statistics include null counts, but not min and max values, which
655+
// triggers PARQUET-1217.
656+
val df = readResourceParquetFile("test-data/parquet-1217.parquet")
657+
658+
// Will return 0 rows if PARQUET-1217 is not fixed.
659+
assert(df.where("col > 0").count() === 2)
660+
}
651661
}
652662

653663
class NumRowGroupsAcc extends AccumulatorV2[Integer, Integer] {

0 commit comments

Comments
 (0)