[native] Convert input field names to lower case in plan conversion#21265
Merged
amitkdutta merged 1 commit intoprestodb:masterfrom Nov 14, 2023
Merged
Conversation
kevinwilfong
force-pushed
the
native_fields_case_sensitive
branch
3 times, most recently
from
1c45dab to
5116aa8
Compare
presto-native-execution/presto_cpp/main/types/PrestoToVeloxQueryPlan.cpp
Outdated
Show resolved
Hide resolved
majetideepak
approved these changes
Oct 30, 2023
xiaoxmeng
approved these changes
Oct 30, 2023
Contributor
xiaoxmeng
left a comment
xiaoxmeng
left a comment
There was a problem hiding this comment.
@kevinwilfong LGTM. Thanks!
kevinwilfong
force-pushed
the
native_fields_case_sensitive
branch
2 times, most recently
from
ee1a432 to
0b05eb5
Compare
Presto is case insensitive, and so always converts column names and field names to lower case in its plan. When Presto constructs the Velox query plan it passes in the types for Hive tables as they were read from the metastore, with capital letters if they were there in the metastore. Velox uses these types to construct the output type of the TableScan. This leads to errors as the output of the TableScan will have capital letters in field names, while field accesses in the rest of the plan will use the field names Presto used in its original plan, which are all lower case. Converting the field names from the types in the input Hive tables to lower case fixes this inconsistency.
kevinwilfong
force-pushed
the
native_fields_case_sensitive
branch
from
0b05eb5 to
b69c7c6
Compare
26 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Presto is case insensitive, and so always converts column names and field names to lower case in its plan.
When Presto constructs the Velox query plan it passes in the types for Hive tables as they were read from the metastore, with capital letters if they were there in the metastore. Velox uses these types to construct the output type of the TableScan.
This leads to errors as the output of the TableScan will have capital letters in field names, while field accesses in the rest of the plan will use the field names Presto used in its original plan, which are all lower case.
Converting the field names from the types in the input Hive tables to lower case fixes this inconsistency.