feat(native): Add endpoint for expression optimization in sidecar by pramodsatya · Pull Request #26475 · prestodb/presto

pramodsatya · 2025-10-30T03:52:59Z

Description

To support constant folding and consistent semantics between the Presto Java coordinator and the Presto C++ worker, it is necessary to use consistent expression evaluation. To support this, a native expression evaluation endpoint, v1/expressions, has been added to the Presto C++ sidecar. This endpoint leverages the ExprOptimizer in Velox to optimize Presto expressions.

The optimized Velox core::TypedExpr returned by Velox's ExprOptimizer is converted to a Presto protocol::RowExpression in the Presto native sidecar with helper class VeloxToPrestoExprConverter. The end to end flow between the coordinator and sidecar is as follows:

RowExpression == (PrestoToVeloxExpr) ==> TypedExpr == (Velox ExprOptimizer optimize() API) ==> optimized TypedExpr == (VeloxToPrestoExprConverter) ==> optimized RowExpression

When OptimizerLevel is EVALUATED and an error is encountered during expression evaluation, it is converted to NativeSidecarFailureInfo with presto protocol. The list of optimized RowExpressions and NativeSidecarFailureInfos is returned along with their corresponding indices in the input expressions list. With the fuzzer testing (see test plan), we expect this endpoint to be ready for production workloads.

Motivation and Context

The native-sidecar-plugin will implement the ExpressionOptimizer interface from Presto SPI to utilize the v1/expressions endpoint on the sidecar for optimizing Presto expressions using the native expression evaluation engine.

The primary goal is to achieve consistency between C++ and Java semantics for expression optimization. With this change, C++ functions including UDFs can be used for constant folding of expressions in the Presto planner.
Please refer to RFC-0006.

Test Plan

Tests have been added by abstracting testcases from TestRowExpressionInterpreter to an interface AbstractTestExpressionInterpreter. This test interface is implemented in TestNativeExpressionInterpreter to test the v1/expressions endpoint on the sidecar end to end.

This feature is still in Beta, and to support production workloads with complete certainty, the Velox expression fuzzer will be extended to test this endpoint with fuzzer generated expressions in a follow-up PR. This should surface any remaining bugs, such as in VeloxToPrestoExpr.
Note: The Velox ExprOptimizer and logical rewrites are already tested with fuzzer.

Release Notes

Prestissimo (Native Execution) Changes
* Add http endpoint `v1/expressions` in sidecar for expression optimization. See :doc:`/presto_cpp/sidecar`.

sourcery-ai

Sorry @pramodsatya, your pull request is larger than the review limit of 150000 diff characters

sourcery-ai

Sorry @pramodsatya, your pull request is larger than the review limit of 150000 diff characters

pramodsatya · 2025-10-30T23:01:01Z

@aditi-pandit, @tdcmeehan, could you please take a look?

presto-native-execution/presto_cpp/main/types/VeloxToPrestoExpr.cpp

presto-native-execution/presto_cpp/main/types/VeloxToPrestoExpr.h

presto-native-execution/presto_cpp/main/types/VeloxToPrestoExpr.cpp

aditi-pandit

Thanks @pramodsatya. Have bunch of comments.

presto-native-execution/presto_cpp/main/PrestoServer.cpp

presto-native-execution/presto_cpp/main/types/VeloxToPrestoExpr.cpp

presto-native-execution/presto_cpp/main/types/tests/VeloxToPrestoExprTest.cpp

pramodsatya · 2025-11-11T00:07:48Z

Thanks @aditi-pandit, made the changes as suggested. Could you PTAL?

presto-native-execution/presto_cpp/main/types/ExpressionOptimizer.cpp

pramodsatya · 2025-12-03T07:48:31Z

Thanks @aditi-pandit, addressed the comments, could you please take another look?

aditi-pandit

Thanks @pramodsatya. Code looks good to me.

@czentgr : Do you want to give it a read as well ? If it looks good, then we can get Tim to approve on the Java part as well.

steveburnett

LGTM! (docs)
Pull branch, local doc build. Thanks!

presto-native-execution/presto_cpp/main/types/ExpressionOptimizer.cpp

aditi-pandit

Thanks @pramodsatya

…d expression and failure

Co-authored-by: Pramod Satya <pramod.satya@ibm.com>

aditi-pandit

Thanks @pramodsatya for all the iterations.

) ## Description Add a `ExpressionOptimizer` which delegates to the native sidecar process to evaluate expressions with Velox. ## Motivation and Context #26475 added support for an endpoint in the sidecar for constant folding expressions. This follows up on that by adding an expression interpreter to call that endpoint. For more context: [RFC-0006](https://github.com/prestodb/rfcs/blob/main/RFC-0006-expression-eval.md). ## Impact No impact by default as the old in-memory evaluation is the default. ## Test Plan Tests have been added. ## Contributor checklist - [x] Please make sure your submission complies with our [contributing guide](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md), in particular [code style](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#code-style) and [commit standards](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#commit-standards). - [x] PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced. - [x] Documented new properties (with its default value), SQL syntax, functions, or other functionality. - [x] If release notes are required, they follow the [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines). - [x] Adequate tests were added if applicable. - [x] CI passed. ## Release Notes Please follow [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines) and fill in the release notes below. ``` == RELEASE NOTES == Prestissimo (Native Execution) Changes * Add a native expression optimizer for optimizing expressions in the sidecar. ``` --------- Co-authored-by: Timothy Meehan <tdm@fb.com>

prestodb-ci added the from:IBM PR from IBM label Oct 30, 2025

sourcery-ai bot reviewed Oct 30, 2025

View reviewed changes

pramodsatya requested review from aditi-pandit and tdcmeehan October 30, 2025 03:53

pramodsatya force-pushed the expr_opt branch 2 times, most recently from d3f5a74 to 4224f36 Compare October 30, 2025 23:00

pramodsatya marked this pull request as ready for review October 30, 2025 23:00

pramodsatya requested review from a team, feilong-liu, jaystarshot and vivek-bharathan as code owners October 30, 2025 23:00

prestodb-ci requested review from a team and pratyakshsharma and removed request for a team October 30, 2025 23:00

sourcery-ai bot reviewed Oct 30, 2025

View reviewed changes

pramodsatya force-pushed the expr_opt branch from 4224f36 to 742c4ad Compare October 31, 2025 02:48

tdcmeehan reviewed Nov 3, 2025

View reviewed changes

presto-native-execution/presto_cpp/main/types/VeloxToPrestoExpr.cpp Outdated Show resolved Hide resolved

aditi-pandit reviewed Nov 4, 2025

View reviewed changes

pramodsatya force-pushed the expr_opt branch 2 times, most recently from 8034891 to 69c7c37 Compare November 5, 2025 18:37

pramodsatya requested review from aditi-pandit and tdcmeehan November 5, 2025 18:38

aditi-pandit reviewed Nov 8, 2025

View reviewed changes

pramodsatya force-pushed the expr_opt branch from 69c7c37 to 388eb89 Compare November 11, 2025 00:06

pramodsatya requested a review from aditi-pandit November 11, 2025 00:08

pramodsatya force-pushed the expr_opt branch from 388eb89 to dfb9a2b Compare November 11, 2025 07:25

aditi-pandit reviewed Nov 11, 2025

View reviewed changes

presto-native-execution/presto_cpp/main/types/ExpressionOptimizer.cpp Outdated Show resolved Hide resolved

pramodsatya force-pushed the expr_opt branch 2 times, most recently from 02fa18e to b809033 Compare December 3, 2025 07:47

pramodsatya requested review from aditi-pandit and czentgr December 4, 2025 02:59

aditi-pandit reviewed Dec 4, 2025

View reviewed changes

steveburnett previously approved these changes Dec 5, 2025

View reviewed changes

czentgr previously approved these changes Dec 5, 2025

View reviewed changes

presto-native-execution/presto_cpp/main/types/ExpressionOptimizer.cpp Show resolved Hide resolved

pramodsatya requested a review from aditi-pandit December 5, 2025 22:37

aditi-pandit previously approved these changes Dec 8, 2025

View reviewed changes

pramodsatya added 3 commits December 7, 2025 19:19

feat(sidecar): Represent expression optimization result with optimize…

3de8d32

…d expression and failure

chore(native): Update presto protocol

bef95a4

feat(native): Add Velox to Presto expression converter

c364169

tdcmeehan previously approved these changes Dec 8, 2025

View reviewed changes

pramodsatya and others added 3 commits December 7, 2025 19:43

feat(native): Add expression optimization endpoint in sidecar

bfc52a9

test(native): Add e2e tests for expression optimization endpoint

f53088f

docs(native): Document v1/expressions endpoint

d4c428f

Co-authored-by: Pramod Satya <pramod.satya@ibm.com>

pramodsatya dismissed stale reviews from tdcmeehan, aditi-pandit, czentgr, and steveburnett via d4c428f December 8, 2025 03:54

pramodsatya force-pushed the expr_opt branch from b809033 to d4c428f Compare December 8, 2025 03:54

aditi-pandit approved these changes Dec 8, 2025

View reviewed changes

pramodsatya requested a review from tdcmeehan December 8, 2025 07:17

tdcmeehan approved these changes Dec 8, 2025

View reviewed changes

aditi-pandit merged commit 18323c3 into prestodb:master Dec 8, 2025
143 of 202 checks passed

pdabre12 mentioned this pull request Dec 8, 2025

feat(plugin-native-sidecar): Add native row expression optimizer #24602

Merged

6 tasks

pramodsatya deleted the expr_opt branch December 8, 2025 21:23

pramodsatya mentioned this pull request Feb 5, 2026

Expand Presto C++ function test coverage #27088

Open

Conversation

pramodsatya commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

Test Plan

Release Notes

Uh oh!

sourcery-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

sourcery-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

pramodsatya commented Oct 30, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aditi-pandit left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pramodsatya commented Nov 11, 2025

Uh oh!

Uh oh!

pramodsatya commented Dec 3, 2025

Uh oh!

aditi-pandit left a comment

Choose a reason for hiding this comment

Uh oh!

steveburnett left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aditi-pandit left a comment

Choose a reason for hiding this comment

Uh oh!

aditi-pandit left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

pramodsatya commented Oct 30, 2025 •

edited

Loading