ES|QL: allow executing pre-built parsed statements by felixbarny · Pull Request #144415 · elastic/elasticsearch

felixbarny · 2026-03-17T16:19:03Z

The context of this change is that I'm working on a Prometheus query_range endpoint that internally executes an ES|QL PROMQL command and then translates the response to the expected JSON format.

This change allows the query range rest action to provide a LogicalPlan instead of having to construct a temporary query string. We're also planning to add more Prometheus endpoints that would benefit from this. For example, the different Prometheus metadata APIs that will rely on the TS_INFO command under the hood.

Overview of the changes:

add EsqlQueryRequest.syncEsqlQueryRequestWithPlan(EsqlStatement) so internal callers can pass a pre-built logical plan
add transient parsedStatement support to EsqlQueryRequest and execute it directly in EsqlSession instead of reparsing query text
add unit and integration coverage (EsqlQueryRequestTests, EsqlActionIT#testRowWithParsedStatement) to verify parsed plans are actually executed

Stack context

Overview: Prometheus: Add query_range REST endpoint backed by ES|QL PROMQL command #144294
Independent PR in the stack; can be reviewed/merged standalone.
Consumed by
- Prometheus: execute query_range via parsed EsqlStatement plan #144416
- Add PrometheusSeriesPlanBuilder for series endpoint #144493

Allow internal callers to supply a pre-built EsqlStatement that bypasses ES|QL string construction and parsing. EsqlSession now checks for a parsed statement before invoking the parser.

elasticsearchmachine · 2026-03-17T16:30:32Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

EsqlAsyncActionIT.run() was re-creating the request from only the query string, discarding the parsedStatement. For a pre-built plan, the query string is "[pre-built plan]" which fails ES|QL parsing with a token recognition error on '['. Add asyncEsqlQueryRequestWithPlan() factory method and propagate the parsedStatement when wrapping a sync request into an async one.

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlQueryRequest.java

When EsqlQueryRequest carries a pre-built EsqlStatement (bypassing the parser), TelemetryAware command labels and function names were not recorded because telemetry collection normally happens inside the parser (LogicalPlanBuilder.telemetryAccounting and ExpressionBuilder.visitFunctionName/castToType). Add EsqlSession.gatherPlanTelemetry, which does a single forEachDown pass over the plan tree to fill in PlanTelemetry post-hoc. UnresolvedFunction nodes (named calls as produced by the parser) are recorded by name; concrete Function instances (inline casts from the parser, or functions instantiated directly in programmatic plan builders such as the Prometheus plugin's PromqlQueryPlanBuilder) are resolved via the function registry.

When a pre-built parsedStatement is provided in EsqlQueryRequest, the normal parsing path is bypassed, which also skipped QuerySettings.validate(). This meant snapshot-only settings (e.g. SET approximation=...) could pass through on non-snapshot builds. Apply the same validation as the regular parse path.

…rsed-statement

felixbarny · 2026-03-20T11:41:33Z

The PrometheusSeriesPlanBuilder in #144493 demonstrates why building a LogicalPlan is needed and how I'm planning to use the functionality introduced in this PR.

The flow looks like this:

A request to GET /_prometheus/api/v1/series comes in with a matcher[] query parameter such as ?match[]=http_requests_total{job="elasticsearch"} (note that there can be multiple matchers).
The existing PromqlParser is used to parse the match expressions into an InstantSelector LogicalPlan.
The InstantSelector is translated to a Filter expression using the TranslatePromqlToEsqlPlan.translateLabelMatcher method we also use in context of the PROMQL command.
The filters are combined with a time range filter and added as a child of the TsInfo command.
The resulting LogicalPlan is executed using the EsqlQueryRequest.syncEsqlQueryRequestWithPlan method added in this PR.

Creating a dynamic ES|QL string to achieve that doesn't seem feasible and wouldn't allow us to re-use the existing parsing logic, making this very brittle.

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlQueryRequest.java

...esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/EsqlAsyncActionIT.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlQueryRequest.java

…st-hoc pass Previously, telemetry (commands, functions, settings) was collected inline during parsing via context.telemetry() hooks in LogicalPlanBuilder and ExpressionBuilder. This moves all collection to a single post-hoc gatherPlanTelemetry pass over the resolved plan tree. - Remove PlanTelemetry from ParsingContext and all context.telemetry() call sites in ExpressionBuilder and LogicalPlanBuilder; delete telemetryAccounting - Remove the PlanTelemetry parameter from EsqlParser.createStatement/parse/etc. - Call gatherPlanTelemetry on viewResolution.plan() in analyseAndExecute() so view-expanded nodes are also included - Guard gatherPlanTelemetry against TelemetryAware nodes with a null label (e.g. LOOKUP JOIN relation refs) and Function subclasses not registered in the function registry (binary operators, predicates, etc.) - Add EsqlFunctionRegistry.functionExists(Class) and expose PlanTelemetry.functionRegistry() to support those guards - Remove ViewService's dead PlanTelemetry field - Delete EsqlSessionTelemetryTests, which existed solely to verify equivalence between the old parser-inline and post-hoc paths

Adds a PreparedEsqlQueryRequest subclass that carries a pre-built EsqlStatement, bypassing the parser. Includes all supporting changes: - Override writeTo to signal local-only execution - Move QuerySettings validation into EsqlQueryRequest#parse so it applies to both parsed and pre-built paths - Add copy constructor to EsqlQueryRequest; use it in maybeWrapAsPrepared to eliminate the redundant garbage query-string overwrite - Separate sourceText (full original query) from display string in EsqlQueryRequest; fix sourceTextForConfiguration to return the full original query and fix a Source position validation crash on multi-node clusters - Smoke-test PreparedEsqlQueryRequest via random promotion in ITs

… parser signatures

alex-spies

Thanks @felixbarny , I think this came out rather nicely.

This mostly LGTM, except that I think this is currently breaking the plan telemetry tests. Before merging, it'd be great if @luigidellaquila or @idegtiarenko could have another look at the plan telemetry refactoring, specifically.

Except for that, I have mostly minor comments. Please resolve these at your own discretion. (Maybe don't skip those that aren't marked "nit"/"thought", though.)

.../plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/PreparedEsqlQueryRequest.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/view/ViewService.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/LogicalPlanBuilder.java

.../internalClusterTest/java/org/elasticsearch/xpack/esql/action/AbstractEsqlIntegTestCase.java

.../plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/PreparedEsqlQueryRequest.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java

…ing statement.plan()

Guard against regressions where Sources with real line/column positions are deserialized against a Configuration whose query is null/empty: text must degrade gracefully to empty without throwing.

When wrapping a request as a PreparedEsqlQueryRequest, the parser was called with Settings.EMPTY, baking in default InferenceSettings (e.g. rerank/completion row limits and enabled flags) into the pre-built plan. Since PreparedEsqlQueryRequest.parse() returns the plan directly without re-parsing, any cluster-settings-dependent parse behaviour (row limits, enabled/disabled checks) was silently bypassed on the prepared path. Fix by reading the current effective cluster settings from the cluster state metadata so the pre-built plan reflects the actual configured values.

When INLINE STATS is used, the plan tree contains InlineStats wrapping an Aggregate. The generic forEachDown pass in gatherPlanTelemetry visited both nodes and recorded both "INLINE STATS" and "STATS", causing TelemetryIT to fail with an unexpected STATS entry. The old parser-based telemetry (removed in 0c63022) only recorded the top-level command node per pipeline step, so the inner Aggregate was never counted. Fix the post-hoc pass to mirror that behaviour: collect inner Aggregate nodes of all InlineStats commands first, then skip them when recording command telemetry.

When maybeWrapAsPrepared wraps a request as a PreparedEsqlQueryRequest, configuration.query() is null so Source text cannot be reconstructed during plan deserialization on remote nodes. The profile and EXPLAIN requests are wrapped independently, so one plan may carry expression text (e.g. "value > 50") while the other has an empty source — causing the structural comparison to fail spuriously. Fix by normalising source text in determinizePlanString alongside the existing position normalisation: expression text before "@_:_" in "source" fields is stripped, so the comparison is immune to whether either plan was built from a prepared request.

The old parser-based telemetry (telemetryAccounting) was called only on top-level pipeline step nodes, so Subquery nodes — created internally inside visitFromCommand while building UnionAll — were never reached. The test expectation reflected this limitation rather than intent. The new post-hoc gatherPlanTelemetry pass traverses the full plan tree and correctly picks up Subquery nodes, which were already TelemetryAware. Update the test to expect SUBQUERY to appear alongside UNIONALL.

…rsed-statement

…atement' into promql-query-range-parsed-statement

- Pass query description to PreparedEsqlQueryRequest.sync (new required arg added in elastic#144415) - Pass index path param instead of hardcoded "*" to PromqlQueryPlanBuilder - Add INDEX_PARAM constant; replace unused EsqlQueryRequest import with PreparedEsqlQueryRequest

Allow internal callers to supply a pre-built EsqlStatement that bypasses ES|QL string construction and parsing. EsqlSession now checks for a parsed statement before invoking the parser.

Add parsed statement support to EsqlQueryRequest

487eed3

Allow internal callers to supply a pre-built EsqlStatement that bypasses ES|QL string construction and parsing. EsqlSession now checks for a parsed statement before invoking the parser.

elasticsearchmachine added v9.4.0 needs:triage Requires assignment of a team area label external-contributor Pull request authored by a developer outside the Elasticsearch team labels Mar 17, 2026

felixbarny added >non-issue :Analytics/ES|QL AKA ESQL labels Mar 17, 2026

elasticsearchmachine added Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) and removed needs:triage Requires assignment of a team area label labels Mar 17, 2026

idegtiarenko reviewed Mar 18, 2026

View reviewed changes

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java Outdated Show resolved Hide resolved

alex-spies reviewed Mar 18, 2026

View reviewed changes

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlQueryRequest.java Outdated Show resolved Hide resolved

This was referenced Mar 18, 2026

Add PrometheusSeriesPlanBuilder for series endpoint #144493

Merged

Add PrometheusSeriesRestAction for /_prometheus/api/v1/series endpoint #144494

Merged

felixbarny added 3 commits March 20, 2026 11:47

Merge remote-tracking branch 'origin/main' into promql-query-range-pa…

c01ea41

…rsed-statement

idegtiarenko reviewed Mar 20, 2026

View reviewed changes

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java Outdated Show resolved Hide resolved

alex-spies reviewed Mar 20, 2026

View reviewed changes

felixbarny added 2 commits March 23, 2026 07:35

felixbarny force-pushed the promql-query-range-parsed-statement branch from d220dbb to d6d25cc Compare March 23, 2026 06:37

Merge branch 'main' into promql-query-range-parsed-statement

bf9db00

felixbarny and others added 2 commits March 23, 2026 09:08

Fix benchmark compilation errors after PlanTelemetry was removed from…

43e9e7d

… parser signatures

[CI] Auto commit changes from spotless

3abafe7

felixbarny requested a review from alex-spies March 23, 2026 08:29

alex-spies approved these changes Mar 23, 2026

View reviewed changes

alex-spies requested review from alex-spies, idegtiarenko and luigidellaquila and removed request for alex-spies March 23, 2026 13:52

idegtiarenko reviewed Mar 23, 2026

View reviewed changes

.../plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/PreparedEsqlQueryRequest.java Show resolved Hide resolved

idegtiarenko reviewed Mar 23, 2026

View reviewed changes

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java Show resolved Hide resolved

felixbarny and others added 6 commits March 23, 2026 17:26

Address review comments

647c995

Require callers to provide a query description rather than materialis…

7178423

…ing statement.plan()

Test Source deserialization with a null-query Configuration

99cd9bb

Guard against regressions where Sources with real line/column positions are deserialized against a Configuration whose query is null/empty: text must degrade gracefully to empty without throwing.

Fix telemetry collection for settings (fixes TelemetryIT)

0c5cfeb

Merge branch 'main' into promql-query-range-parsed-statement

6cbadd3

idegtiarenko approved these changes Mar 24, 2026

View reviewed changes

felixbarny self-assigned this Mar 24, 2026

felixbarny added 2 commits March 24, 2026 16:40

felixbarny enabled auto-merge (squash) March 24, 2026 17:06

felixbarny and others added 4 commits March 24, 2026 22:14

Merge branch 'main' into promql-query-range-parsed-statement

352ed86

Merge remote-tracking branch 'origin/main' into promql-query-range-pa…

899b63b

…rsed-statement

Merge remote-tracking branch 'felixbarny/promql-query-range-parsed-st…

c5981fa

…atement' into promql-query-range-parsed-statement

felixbarny merged commit 8a7327e into elastic:main Mar 25, 2026
36 checks passed

felixbarny deleted the promql-query-range-parsed-statement branch March 25, 2026 10:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ES|QL: allow executing pre-built parsed statements#144415

ES|QL: allow executing pre-built parsed statements#144415
felixbarny merged 22 commits intoelastic:mainfrom
felixbarny:promql-query-range-parsed-statement

felixbarny commented Mar 17, 2026 •

edited

Loading

Uh oh!

elasticsearchmachine commented Mar 17, 2026

Uh oh!

Uh oh!

Uh oh!

felixbarny commented Mar 20, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alex-spies left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

felixbarny commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Stack context

Uh oh!

elasticsearchmachine commented Mar 17, 2026

Uh oh!

Uh oh!

Uh oh!

felixbarny commented Mar 20, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

felixbarny commented Mar 17, 2026 •

edited

Loading