Fix Parquet predicate pushdown for smallint, tinyint#131
Merged
findepi merged 6 commits intotrinodb:masterfrom Feb 5, 2019
Merged
Fix Parquet predicate pushdown for smallint, tinyint#131findepi merged 6 commits intotrinodb:masterfrom
findepi merged 6 commits intotrinodb:masterfrom
Conversation
This is refactoring commit. It is supposed not to change code behavior regardless of anything, unless introduced short-circuiting prevents an exception from being thrown.
This commit changes code behavior in the unlikely edge case where: 1. Domain constructed from file stats is empty (`isNone()`) 2. we did not skip the file/stripe based on `numberOfRows == 0` condition. 3. there is no effective predicate for a column Previously, the code would construct realize the `Domain` constructed from file stats `isNone()`. Now, the `Domain` construction will not be attempted, as the code short-circuits when there no effective predicate for a column. This change is based on assumption that the `Domain` constructed from file stats cannot `isNone()` in any practical situation.
When creating `Domain` from file stats, we need to know actual Presto Type of the column, as Types must match for `Domain#intersect`. Instead of reconstructing the type based on the type in the file, which is not always possible, as Parquet does not distinguish between INTEGER, SMALLINT etc, use the Type passed with effective predicate.
Member
Author
Member
Author
|
@nishantrayan I saw yours prestodb/presto#11118 , which fixes |
This was referenced Mar 4, 2019
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #123