ESQL: Physical Planning on the Lookup Node

[julian-elastic]

Physical Planning on the Lookup Node

Summary

Logical Planning on the Lookup Node will be added with a separate PR.

The change applies only to Streaming Lookup Join which is behind snapshot build flag, so no production changes due to the physical planning are expected.

Moves the physical plan construction and optimization for LOOKUP JOIN to the lookup node itself, enabling proper optimizer rules (like filter pushdown to Lucene) to run against the lookup-side plan.

Previously, the lookup node received raw request fields (match fields, extract fields, join conditions, pre-join filter plan) and manually constructed a physical plan tree (ParameterizedQueryExec -> FieldExtractExec -> ProjectExec -> OutputExec). This bypassed the standard optimizer pipeline, meaning right-side-only filters could only be pushed to Lucene via a special code path in ExpressionQueryList.buildPreJoinFilter.

Now, the data node builds a logical plan (rooted at ParameterizedQuery, with an optional Filter node and a Project on top) and sends it to the lookup node inside a FragmentExec. The lookup node then runs LocalMapper -> LookupPhysicalPlanOptimizer to produce the physical plan, reusing existing optimizer rules like PushFiltersToSource and InsertFieldExtraction.

Key changes

• Refactored doLookupStreaming : Remove usages of the use of Request after logical plan is created. The idea is that all the information needed should be already in the plan, and we should not read anything from the request as some of the rules could have modified it.
• New ParameterizedQuery logical plan node: Replaces EsRelation as the leaf of the lookup-side logical plan, carrying match field metadata and join-on conditions.
• New LookupPhysicalPlanOptimizer: Runs PushFiltersToSource, InsertFieldExtraction and ParameterizedQueryExec rules on the lookup physical plan.
• PushFiltersToSource extended: Now handles FilterExec -> ParameterizedQueryExec and FilterExec -> EvalExec -> ParameterizedQueryExec paths, pushing eligible filters into ParameterizedQueryExec.query(). Duplicated filter classification logic refactored into shared classifyFilters helper.
• ExpressionQueryList enhanced: Accepts pre-separated Expression rightOnlyFilter and QueryBuilder pushedQuery instead of walking a PhysicalPlan tree to extract filters. Old code is still there for bwc reasons with doLookup still getting called from old nodes.
• LookupExecutionPlanner refactored: Removed EnrichQuerySourceOperatorFactory (dead code after streaming migration). Introduced QueryListFromPlanFactory to build query lists from plan data rather than transport request fields. Simplified LookupDriverContext by removing request and queryListFactory fields.
• BWC path: extractOrBuildLogicalPlan detects whether the incoming rightPreJoinPlan already contains a ParameterizedQuery (new path) or falls back to building it locally (rolling upgrade compatibility with older data nodes).
• New transport version (esql_lookup_planning): Serializes Configuration in the lookup transport request so the lookup node has access to fold context and query pragmas. In the future, Configuration will also be needed for other functionality such as timezone sensitive functions.
• Bug fix: LookupFromIndexOperator.Status.emittedRows() was returning emittedPages instead of emittedRows.

Test plan

• New LookupPhysicalPlanOptimizerTests covering: simple lookup, pushable right-only filter, non-pushable filter, mixed filters, ON expression filters, filter-through-eval pushdown, and two consecutive LOOKUP JOINs.
• New ParameterizedQuerySerializationTests for serialization round-trip.
• Updated LookupExecutionPlannerTests, LookupFromIndexOperatorTests, StreamingLookupFromIndexOperatorTests to use new APIs.
• Updated EsqlNodeSubclassTests and ApproximationSupportTests to account for new ParameterizedQuery and MatchConfig types.

Assisted by Cursor

[coderabbitai]

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one) (2)

• Team:Delivery
• Team:Search - Inference

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: f8e72262-eb15-4ed4-93fe-bdd90077aa6a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[alex-spies]

I'm not sure if re-using the LocalPhysicalOptimizerContext is appropriate. It has stuff in it that's not really relevant for non-data node plans.

[julian-elastic]

PushFiltersToSource, InsertFieldExtraction rules are parameterized on LocalPhysicalOptimizerContext. I think it should be ok to create LocalPhysicalOptimizerContext here, so we can avoid duplicating the rules or having a complex rule hierarchy.
public class PushFiltersToSource extends PhysicalOptimizerRules.ParameterizedOptimizerRule<FilterExec, LocalPhysicalOptimizerContext> {