Skip to content

Conversation

@tdcmeehan
Copy link
Contributor

@tdcmeehan tdcmeehan commented Nov 22, 2024

Description

To support constant folding and consistent semantics between the Presto coordinator (Java) and the Presto C++ worker, it is necessary to use consistent expression evaluation. To support this, a native expression evaluation endpoint, v1/expressions, has been added to the Presto sidecar process, and a plugin has been created which can utilize Velox expression evaluation behind a standard ExpressionOptimizer.

Adds ExpressionOptimizer in Velox to optimize and constant fold TypedExprs via API optimizeExpression(); depends on Velox changes. Optimizes Presto RowExpressions in the Presto native sidecar with following conversion:

RowExpression == (PrestoToVeloxExpr) ==> TypedExpr == (Velox API optimizeExpression()) ==> optimized `TypedExpr` == (RowExpressionConverter) ==> optimized RowExpression

Adds helper class RowExpressionConverter to convert the optimized/constant folded Velox core::TypedExpr to a Presto protocol::RowExpression.

These changes are being reviewed in PR: #22927.

Motivation and Context

Consistency between C++ and Java semantics. Support for using C++ functions during constant folding of expressions in the planner.

Impact

Fix some lingering bugs with C++ clusters.

Test Plan

Tests have been added by extending the TestRowExpressionInterpreter class to also test native expression evaluation in TestNativeExpressionOptimizer.java. The Velox expression fuzzer is extended to ensure the expression optimizations are tested exhaustively. However, this feature is still in Beta, and to support production workloads with complete certainty a fuzzer will be created to surface any remaining bugs with the integration at a later time.

Release Notes

== RELEASE NOTES ==

General Changes
* ... :pr:`12345`
* ... :pr:`12345`

Hive Connector Changes
* ... :pr:`12345`
* ... :pr:`12345`

@tdcmeehan tdcmeehan requested a review from presto-oss November 22, 2024 20:14
@prestodb-ci prestodb-ci added the from:IBM PR from IBM label Nov 22, 2024
@prestodb-ci prestodb-ci requested review from a team, aaneja and zuyu and removed request for a team November 22, 2024 20:14
@tdcmeehan tdcmeehan marked this pull request as draft November 22, 2024 20:27
@pramodsatya pramodsatya force-pushed the expr-spi branch 2 times, most recently from 21db147 to 7056fae Compare March 12, 2025 19:47
@pramodsatya pramodsatya force-pushed the expr-spi branch 2 times, most recently from 3c0dad2 to 203dc63 Compare April 19, 2025 02:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

from:IBM PR from IBM

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants