Prune columns are all null in ParquetExec by row_counts , handle IS NOT NULL #9989

Ted-Jiang · 2024-04-08T02:53:59Z

…atistics

Which issue does this PR close?

Part #9961 .

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

…atistics

Ted-Jiang · 2024-04-08T02:56:38Z

datafusion/core/src/datasource/physical_plan/parquet/row_groups.rs

@@ -1026,15 +1028,17 @@ mod tests {
        column_statistics: Vec<ParquetStatistics>,
    ) -> RowGroupMetaData {
        let mut columns = vec![];
+        let number_row = 1000;


Before all unit test set each col with default 0 row, will trigger num_rows == num_nulls

I was wondering if this could cause problems in real files (for example, if the row counts were not included in the statistics that were written into the file).

However, I double checked the code and it seems like ColumnChunkMetaData::num_values is non nullable so I think we are good.

Ted-Jiang · 2024-04-08T03:50:52Z

@alamb @waynexia PTAL

waynexia

Looks good to me 👍 Thanks @Ted-Jiang

datafusion/core/tests/parquet/row_group_pruning.rs

waynexia · 2024-04-08T08:00:02Z

datafusion/core/src/datasource/physical_plan/parquet/row_groups.rs

-        None
+    fn row_counts(&self, column: &Column) -> Option<ArrayRef> {
+        let (c, _) = self.column(&column.name)?;
+        let scalar = ScalarValue::UInt64(Some(c.num_values() as u64));


Another implementor returns an Int64 array. I'm not sure if the type matters?

https://github.com/apache/arrow-datafusion/blob/215f30f74a12e91fd7dca0d30e37014c8c3493ed/datafusion/core/src/physical_optimizer/pruning.rs#L1699-L1709

Try to keep with https://github.com/apache/arrow-datafusion/blob/18c9f4d7e68b3eb470b2b2ef3f2297e018dd4298/datafusion/core/src/datasource/physical_plan/parquet/row_groups.rs#L340-L342 avoid cast during = , i think Int64Array only use in test

UInt64 also matches the documentation https://docs.rs/datafusion/latest/datafusion/physical_optimizer/pruning/trait.PruningStatistics.html#tymethod.row_counts 👍

I made a PR to fix the tests: #10004

Co-authored-by: Ruihang Xia <[email protected]>

alamb · 2024-04-08T19:11:30Z

Thanks @Ted-Jiang -- this looks awesome -- I look forward to reviewing it later today

alamb

Thanks @Ted-Jiang -- and @waynexia . I think the code looks good, though I think a few more tests would improve the coverage we could add them as a follow on PR.

Do you plan to add support in page_filter.rs as well (maybe that is why the PR is marked "Part #9961 ")?

alamb · 2024-04-08T20:40:32Z

datafusion/core/src/datasource/physical_plan/parquet/row_groups.rs

@@ -1026,15 +1028,17 @@ mod tests {
        column_statistics: Vec<ParquetStatistics>,
    ) -> RowGroupMetaData {
        let mut columns = vec![];
+        let number_row = 1000;


I was wondering if this could cause problems in real files (for example, if the row counts were not included in the statistics that were written into the file).

However, I double checked the code and it seems like ColumnChunkMetaData::num_values is non nullable so I think we are good.

datafusion/core/tests/parquet/mod.rs

alamb · 2024-04-08T20:47:10Z

datafusion/core/tests/parquet/row_group_pruning.rs

+    // After pruning, only row group 1,3 should be selected
+    RowGroupPruningTest::new()
+        .with_scenario(Scenario::AllNullValues)
+        .with_query("SELECT * FROM t WHERE \"i8\" is Null")


Could you also add a tests:

i16 IS NOT NULL (to cover the opposite)

i32 > 7 (prune via nulls and some via counts)

@alamb thanks! Add test in 11567d9, and support the isNotNull

Do you plan to add support in page_filter.rs as well (maybe that is why the PR is marked "Part
#9961 ")?

As the page level prune i prefer in next pr to keep this pr short and clean.

alamb · 2024-04-08T21:00:22Z

datafusion/core/src/datasource/physical_plan/parquet/row_groups.rs

-        None
+    fn row_counts(&self, column: &Column) -> Option<ArrayRef> {
+        let (c, _) = self.column(&column.name)?;
+        let scalar = ScalarValue::UInt64(Some(c.num_values() as u64));


UInt64 also matches the documentation https://docs.rs/datafusion/latest/datafusion/physical_optimizer/pruning/trait.PruningStatistics.html#tymethod.row_counts 👍

I made a PR to fix the tests: #10004

alamb

Thanks @Ted-Jiang -- the support for IS NOT NULL is also quite neat.

Ted-Jiang · 2024-04-12T05:07:56Z

Thanks @Ted-Jiang -- the support for IS NOT NULL is also quite neat.

@alamb sorry there is a bug here i will fix this it should

/// If set `with_not` to true: which means is not null
/// datafusion use false flag to prune unit (row group, page ..)
/// Given an expression reference to `expr`, if `expr` is a column expression,
/// returns a pruning expression in terms of IsNotNull that will evaluate to true
/// if the column may contain any nonnull values, and false if definitely does not contain
/// nonnull values which any null values.

Prune columns are all null in ParquetExec by row_counts in pruning st…

de93d2a

…atistics

github-actions bot added the core Core DataFusion crate label Apr 8, 2024

Ted-Jiang commented Apr 8, 2024

View reviewed changes

fix clippy

18c9f4d

Ted-Jiang requested review from alamb and waynexia April 8, 2024 03:50

waynexia approved these changes Apr 8, 2024

View reviewed changes

Update datafusion/core/tests/parquet/row_group_pruning.rs

4c908f0

Co-authored-by: Ruihang Xia <[email protected]>

This was referenced Apr 8, 2024

DataFusion weekly project plan (Andrew Lamb) - April 8, 2024 #10002

Closed

Minor: Clarify documentation on PruningStatistics::row_counts and PruningStatistics::null_counts and make test match #10004

Merged

alamb approved these changes Apr 8, 2024

View reviewed changes

Ted-Jiang added 4 commits April 9, 2024 14:04

fix comment and support isNotNUll

11567d9

add test

93f9bea

fix conflict

5ecdc5c

Merge branch 'main' into issue-9961

0d91757

alamb changed the title ~~Prune columns are all null in ParquetExec by row_counts in pruning st…~~ Prune columns are all null in ParquetExec by row_counts , handle IS NOT NULL Apr 10, 2024

alamb approved these changes Apr 10, 2024

View reviewed changes

alamb merged commit ed37467 into apache:main Apr 10, 2024
25 checks passed

Ted-Jiang mentioned this pull request Apr 12, 2024

Parquet pruneing incorrectly handles IS NOT NULL predicate #10049

Closed

alamb mentioned this pull request Apr 22, 2024

Support IS NOT NULL predicates in PruningPredicate #9231

Closed

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prune columns are all null in ParquetExec by row_counts , handle IS NOT NULL #9989

Prune columns are all null in ParquetExec by row_counts , handle IS NOT NULL #9989

Ted-Jiang commented Apr 8, 2024

Ted-Jiang Apr 8, 2024

alamb Apr 8, 2024

Ted-Jiang commented Apr 8, 2024

waynexia left a comment

waynexia Apr 8, 2024

Ted-Jiang Apr 8, 2024 •

edited

Loading

alamb Apr 8, 2024

alamb commented Apr 8, 2024

alamb left a comment

alamb Apr 8, 2024

alamb Apr 8, 2024

Ted-Jiang Apr 9, 2024 •

edited

Loading

alamb Apr 8, 2024

alamb left a comment

Ted-Jiang commented Apr 12, 2024 •

edited

Loading

Prune columns are all null in ParquetExec by row_counts , handle IS NOT NULL #9989

Prune columns are all null in ParquetExec by row_counts , handle IS NOT NULL #9989

Conversation

Ted-Jiang commented Apr 8, 2024

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Ted-Jiang Apr 8, 2024

Choose a reason for hiding this comment

alamb Apr 8, 2024

Choose a reason for hiding this comment

Ted-Jiang commented Apr 8, 2024

waynexia left a comment

Choose a reason for hiding this comment

waynexia Apr 8, 2024

Choose a reason for hiding this comment

Ted-Jiang Apr 8, 2024 • edited Loading

Choose a reason for hiding this comment

alamb Apr 8, 2024

Choose a reason for hiding this comment

alamb commented Apr 8, 2024

alamb left a comment

Choose a reason for hiding this comment

alamb Apr 8, 2024

Choose a reason for hiding this comment

alamb Apr 8, 2024

Choose a reason for hiding this comment

Ted-Jiang Apr 9, 2024 • edited Loading

Choose a reason for hiding this comment

alamb Apr 8, 2024

Choose a reason for hiding this comment

alamb left a comment

Choose a reason for hiding this comment

Ted-Jiang commented Apr 12, 2024 • edited Loading

Ted-Jiang Apr 8, 2024 •

edited

Loading

Ted-Jiang Apr 9, 2024 •

edited

Loading

Ted-Jiang commented Apr 12, 2024 •

edited

Loading