-
Notifications
You must be signed in to change notification settings - Fork 3.6k
Optimize effectively literal values #10663
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
ca7acde
c9cbf80
7d4d5c0
3de0c9b
e562eb6
4af8552
2b5f035
2936ce0
9c05db6
71045b5
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -14,33 +14,53 @@ | |
| package io.trino.sql; | ||
|
|
||
| import com.google.common.collect.ImmutableList; | ||
| import com.google.common.collect.ImmutableMap; | ||
| import com.google.common.collect.Iterables; | ||
| import io.trino.Session; | ||
| import io.trino.execution.warnings.WarningCollector; | ||
| import io.trino.metadata.Metadata; | ||
| import io.trino.metadata.ResolvedFunction; | ||
| import io.trino.security.AllowAllAccessControl; | ||
| import io.trino.spi.type.Type; | ||
| import io.trino.sql.analyzer.ExpressionAnalyzer; | ||
| import io.trino.sql.analyzer.Scope; | ||
| import io.trino.sql.planner.DeterminismEvaluator; | ||
| import io.trino.sql.planner.ExpressionInterpreter; | ||
| import io.trino.sql.planner.LiteralEncoder; | ||
| import io.trino.sql.planner.NoOpSymbolResolver; | ||
| import io.trino.sql.planner.Symbol; | ||
| import io.trino.sql.planner.SymbolsExtractor; | ||
| import io.trino.sql.planner.TypeProvider; | ||
| import io.trino.sql.tree.Cast; | ||
| import io.trino.sql.tree.Expression; | ||
| import io.trino.sql.tree.ExpressionRewriter; | ||
| import io.trino.sql.tree.ExpressionTreeRewriter; | ||
| import io.trino.sql.tree.FunctionCall; | ||
| import io.trino.sql.tree.GenericDataType; | ||
| import io.trino.sql.tree.Identifier; | ||
| import io.trino.sql.tree.IsNullPredicate; | ||
| import io.trino.sql.tree.LambdaExpression; | ||
| import io.trino.sql.tree.Literal; | ||
| import io.trino.sql.tree.LogicalExpression; | ||
| import io.trino.sql.tree.LogicalExpression.Operator; | ||
| import io.trino.sql.tree.NodeRef; | ||
| import io.trino.sql.tree.QualifiedName; | ||
| import io.trino.sql.tree.RowDataType; | ||
| import io.trino.sql.tree.SymbolReference; | ||
|
|
||
| import java.util.Arrays; | ||
| import java.util.Collection; | ||
| import java.util.HashSet; | ||
| import java.util.List; | ||
| import java.util.Map; | ||
| import java.util.Set; | ||
| import java.util.function.Function; | ||
| import java.util.function.Predicate; | ||
|
|
||
| import static com.google.common.base.Predicates.not; | ||
| import static com.google.common.collect.ImmutableList.toImmutableList; | ||
| import static io.trino.metadata.LiteralFunction.LITERAL_FUNCTION_NAME; | ||
| import static io.trino.metadata.ResolvedFunction.isResolved; | ||
| import static io.trino.sql.tree.BooleanLiteral.FALSE_LITERAL; | ||
| import static io.trino.sql.tree.BooleanLiteral.TRUE_LITERAL; | ||
| import static java.util.Objects.requireNonNull; | ||
|
|
@@ -255,6 +275,59 @@ public static Function<Expression, Expression> expressionOrNullSymbols(Predicate | |
| }; | ||
| } | ||
|
|
||
| /** | ||
| * Returns whether expression is effectively literal. An effectitvely literal expression is a simple constant value, or null, | ||
| * in either {@link Literal} form, or other form returned by {@link LiteralEncoder}. In particular, other constant expressions | ||
| * like a deterministic function call with constant arguments are not considered effectitvely literal. | ||
| */ | ||
| public static boolean isEffectivelyLiteral(PlannerContext plannerContext, Session session, Expression expression) | ||
| { | ||
| if (expression instanceof Literal) { | ||
| return true; | ||
| } | ||
| if (expression instanceof Cast) { | ||
| return ((Cast) expression).getExpression() instanceof Literal | ||
| // a Cast(Literal(...)) can fail, so this requires verification | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why is it important that the cast doesn't fail? I guess, we don't want the query to fail in the stats calculator, or in However, in Also, I suggest a different approach on handling failures: we could skip the validation here, and if the "effectively literal" value is to be evaluated during the optimization phase, then we could use the safe
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
the Note that we don't need to be 'smart' about failing cast case. In a typical case,
True. However, no optimizer / optimizer rule should do this. Literals coming from parser should be validated in ExpressionAnalyzer.
That would work. Note however that my intention was to capture literals, and other things produced by LiteralEncoder. Hence the method name & javadoc and that's also why I choose not to fail.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
My question was: why do we try to filter-out failing casts instead of just reporting them as "effectively literals". I wasn't trying to suggest that
Yes, but we shouldn't depend on the rule order, especially that
If we drop the failure check in If we drop the failure check in On the other hand, if
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Considering
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As discussed offline, the intended contract of this method is to filter simple constant expressions which do not fail, so that the caller does not need to deal with potential failure. I consider this a fair decision, as the majority of failing expressions should never reach this point (supposed that they go through the However, for the contract to hold, we need to validate Literals also, not only the casts. Maybe add the validation for Literals for now, and a TODO that user-provided Literals should ba validated in the
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Done in #10720, so let me skip adding it here. |
||
| && constantExpressionEvaluatesSuccessfully(plannerContext, session, expression); | ||
| } | ||
| if (expression instanceof FunctionCall) { | ||
| QualifiedName functionName = ((FunctionCall) expression).getName(); | ||
| if (isResolved(functionName)) { | ||
| ResolvedFunction resolvedFunction = plannerContext.getMetadata().decodeFunction(functionName); | ||
| return LITERAL_FUNCTION_NAME.equals(resolvedFunction.getSignature().getName()); | ||
| } | ||
| } | ||
|
|
||
| return false; | ||
| } | ||
|
|
||
| private static boolean constantExpressionEvaluatesSuccessfully(PlannerContext plannerContext, Session session, Expression constantExpression) | ||
| { | ||
| Map<NodeRef<Expression>, Type> types = getExpressionTypes(plannerContext, session, constantExpression, TypeProvider.empty()); | ||
| ExpressionInterpreter interpreter = new ExpressionInterpreter(constantExpression, plannerContext, session, types); | ||
| Object literalValue = interpreter.optimize(NoOpSymbolResolver.INSTANCE); | ||
| return !(literalValue instanceof Expression); | ||
| } | ||
|
|
||
| /** | ||
| * @deprecated Use {@link io.trino.sql.planner.TypeAnalyzer#getTypes(Session, TypeProvider, Expression)}. | ||
| */ | ||
| @Deprecated | ||
| public static Map<NodeRef<Expression>, Type> getExpressionTypes(PlannerContext plannerContext, Session session, Expression expression, TypeProvider types) | ||
|
findepi marked this conversation as resolved.
Outdated
|
||
| { | ||
| ExpressionAnalyzer expressionAnalyzer = ExpressionAnalyzer.createWithoutSubqueries( | ||
| plannerContext, | ||
| new AllowAllAccessControl(), | ||
| session, | ||
| types, | ||
| ImmutableMap.of(), | ||
| node -> new IllegalStateException("Unexpected node: " + node), | ||
| WarningCollector.NOOP, | ||
| false); | ||
| expressionAnalyzer.analyze(expression, Scope.create()); | ||
| return expressionAnalyzer.getExpressionTypes(); | ||
| } | ||
|
|
||
| /** | ||
| * Removes duplicate deterministic expressions. Preserves the relative order | ||
| * of the expressions in the list. | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Where is this case handled? Is it SimplifyExpressions?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in
FilterStatsCalculator,trino/core/trino-main/src/main/java/io/trino/cost/FilterStatsCalculator.java
Lines 102 to 120 in 8e40a44