parquet: Add tests for pruning on Int8/Int16/Int64 columns #9778
Conversation
progval
changed the title
There was a problem hiding this comment.
Thank you @progval
[Do not merge this PR, it highlights a bug in Int8 and Int16 columns, through correct_bloom_filters: false. See https://github.com//issues/9779 for a discussion]
What would you think about merging this PR (but keeping #9779 open)?
Then the fix for #9779 could just update the tests to set correct_bloom_filters?
Agreed. For others following along, the issue is upstream in parquet-rs: #9779 (comment) I think what we should do with this PR is to update the comments explaining that the tests now demonstrate there is a bug in DataFusion and link to the upstream issue. Once the upstream issue is fixed, we can then update the tests / close the false nagatives bug #9779 |
|
As a follow up here, I plan to add the comments explaining what is going on here (and that there are tests that show incorrect results, that are tracked by a ticket) and then merge it in. |
|
I merged up from main to make sure we have a clean CI run and then I think we can merge this one in |
|
Thanks again @progval |
* parquet: Add tests for Bloom filters on Int8/Int16/Int64 columns * Document int_tests macro --------- Co-authored-by: Andrew Lamb <[email protected]>
[Do not merge this PR, it highlights a bug in Int8 and Int16 columns, throughcorrect_bloom_filters: false. See #9779 for a discussion]Which issue does this PR close?
Closes #9777.
Rationale for this change
What changes are included in this PR?
Generalizes the Int32 tests, using a macro.
Are these changes tested?
Are there any user-facing changes?