[ES|QL] Non-Correlated Subquery in FROM command

[fang-xing-esql]

This feature is behind snapshot, and not formally released yet.

This PR enables support for non-correlated subqueries within the FROM command. Related to https://github.com/elastic/esql-planning/issues/89

A non-correlated subquery in this context is one that is fully self-contained and does not reference attributes from the outer query. Enabling support for these subqueries in the FROM command provides an additional way to define a data source, beyond directly specifying index patterns in an ES|QL query.

Example

FROM index1, (FROM index2
              | WHERE a > 10
              | EVAL b = a * 2
              | STATS cnt = COUNT(*) BY c
              | SORT cnt desc
              | LIMIT 10)
, index3, (FROM index4 
           | stats count(*))
| WHERE d > 10
| STATS max = max(*) BY e
| SORT max desc

This feature is built on top of Fork. Subqueries are processed in a manner similar to how Fork operates today, with modifications made to the following components to support this functionality:

• Grammar: FROM_MODE is updated to support subquery syntax.
• Parser: LogicalPlanBuilder creates a UnionAll logical plan on top of multiple data sources. Each data source can be either index patterns or subqueries. UnionAll extends Fork, but unlike Fork, each UnionAll leg may fetch data from different indices—this is one of the key differences between UnionAll and Fork.
• PreAnalyzer: Extracts index patterns from subqueries and issues fieldcaps calls to build an IndexResolution for each subquery.
• Analyzer: Resolves indices referenced by subqueries and handles union-typed fields referenced within them. Since subquery index patterns and main query index patterns are accessed separately behind each UnionAll leg, InvalidMappedField are not created across them. If conversion functions are required for common fields between the main index and subquery indices, those conversion functions must be pushed down into each UnionAll leg.
• LogicalPlanOptimizer: Pushes down eligible filters/predicates from the main query into subqueries. This is another key distinction between UnionAll and Fork, as predicate pushdown applies only to UnionAll, while Fork remains unchanged.

Restrictions and follow ups to be addressed in the next PRs:

• CCS: Grammar and parser don't block remote clusters, however subqueries with CCS is not fully tested yet. This is tracked in [ES|QL] Support remote cluster in subqueries in from command #136035.
• Nested subqueries: Most of the nested subqueries are not allowed at the moment, verifier of LogicalPlanOptimizer will error out, if the subquery has commands besides FROM command. This is tracked in [ES|QL] Allow nested non-correlated subqueries in from command #136034.
• Improve FieldNameUtils.resolveFieldNames to identify subquery field names for field caps call, instead of using all fields *. [ES|QL] Improve FieldNameUtils.resolveFieldNames to identify subquery field names for field caps call, instead of using all fields * #137283
• Improve the error messages when a full text function cannot be applied to a subquery.
• When a predicate from the main query eliminates a subquery, the LocalRelation created by PruneFilters includes all of the output of the subquery, and the output is the superset of the outputs from each subquery, which looks confusing, the outputs that are not directly from the current subquery can be excluded from the LocalRelation.
• Push down Filters/Predicates on mixed data type UnionAll output with explicit casting. [ES|QL] Push down Filters/Predicates on mixed data type UnionAll output with explicit casting. #137284
• Implicit casting for mixed data type UnionAll output
• Documentation related to subqueries.
• Add new performance tests for subqueries. [E|QL] Add performance tests to subqueries #137285
• Remove implicit LIMIT appended to each subquery(UnionAll branch). [ES|QL] Remove the implicit LIMIT command added to each subquery (UnionAll branch) #138106

[elasticsearchmachine]

Hi @fang-xing-esql, I've created a changelog YAML for you.

[elasticsearchmachine]

Hi @fang-xing-esql, I've created a changelog YAML for you.

[Copilot]

Pull Request Overview

This PR introduces support for non-correlated subqueries within the FROM command in ES|QL, allowing queries to reference multiple data sources including both index patterns and subqueries. The implementation enables subqueries to be processed similarly to Fork operations, with key distinctions in index resolution and predicate pushdown capabilities.

• Adds grammar and parser support for subquery syntax in FROM commands
• Implements UnionAll logical plan to handle mixed index patterns and subqueries
• Enables predicate pushdown optimization specifically for UnionAll operations

Reviewed Changes

Copilot reviewed 36 out of 39 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
EsqlBaseParser.g4	Updates grammar to support subquery syntax in FROM_MODE
LogicalPlanBuilder.java	Creates UnionAll plans and handles subquery/index pattern combinations
UnionAll.java	New logical plan extending Fork with union-typed field support
Subquery.java	New logical plan node representing subquery placeholders
Analyzer.java	Resolves subquery indices and handles union-typed fields
PushDownAndCombineFilters.java	Adds predicate pushdown optimization for UnionAll
EsqlSession.java	Implements subquery index resolution during pre-analysis
Various test files	Adds comprehensive test coverage for subquery functionality

Comments suppressed due to low confidence (1)

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/parser/SubqueryTests.java:1

• There's a typo in "nested fork/subquery is not supported, it passes Analyzer" - should be "nested fork/subquery is not supported; it passes Analyzer" (semicolon instead of comma for better grammar).

/*

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

The error message "FROM is required in a subquery" is misleading since the grammar already enforces this requirement. Consider a more descriptive message like "Invalid subquery structure" or remove the comment and exception if this code path is truly unreachable.

Suggested change

	throw new ParsingException("FROM is required in a subquery");
	throw new ParsingException("Invalid subquery structure");

[Copilot]

The TODO comment indicates incomplete functionality. Consider implementing the general push down logic or at least provide a more specific plan for when this will be addressed, as returning null could lead to silent failures in optimization.

Suggested change

	default -> null; // TODO add a general push down for unexpected pattern
	default -> {
	// Fallback: unknown child type, do not push down filter for this child.
	// Consider implementing general push down logic here in the future.
	yield child;
	}

[fang-xing-esql]

Merging this subqueryIndices into the mainIndices is another option, it will require changes to EsqlCCSUtils.initCrossClusterState and EsqlCCSUtils.createIndexExpressionFromAvailableClusters, as they associate the ExecutionInfo with only one index pattern today.

[fang-xing-esql]

When there is subquery exists in the query convertToRemoteIndices doesn't generate a correct remote index pattern yet, the query becomes invalid. Subqueries are not fully tested in CCS yet, working on it as a follow up.

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[elasticsearchmachine]

Pinging @elastic/kibana-esql (ES|QL-ui)

[fang-xing-esql]

Thank you so much for reviewing this PR in such depth, I really appreciate the time and thought you put into it, @astefan !

I'll create a follow up issue for improving FieldNameUtils.resolveFieldNames, we can do a better job here to recognize field names referenced in subqueries, instead of sending * to the field caps call., thank you for reminding me about it!

I made a couples of changes and added additional tests related to subqueries combined with fork and full-text functions, both in the main query and within subqueries. Initially, I planned to address these in a separate PR since this one is already quite large, but I think these scenarios are important enough to include here.

First, FieldNameUtils.resolveFieldNames currently asserts when encountering nested forks. For cases involving a subquery and a fork, this check has been deferred to LogicalVerifier, which now throws a VerificationException instead of an assertion error. as allowing nested subqueries and subquery + fork is planned as the next step.

Second, the validation of commands before a full-text function, as well as the verification of the field referenced by that function, is now deferred to LogicalVerifier when a UnionAll plan exists in its child plans. Since the output of a UnionAll plan is always a ReferenceAttribute, and certain full-text functions apply only to fields, deferring this validation allows the PushDownFilterAndLimitIntoUnionAll rule to push the full-text functions down into individual subqueries. This enables those validations to occur within the subquery context, increasing the chances that full-text functions can be properly evaluated.

[fang-xing-esql]

Thanks @fang-xing-esql, that looks pretty good. I left another round of comments, most of them are very minor changes, but one needs to be fixed before moving forward

Thank you so much for your thoughtful reviews @luigidellaquila! They really help this PR to become a better PR. I think I addressed all of them, please just let me know if there is anything that I missed.

[luigidellaquila]

Thanks @fang-xing-esql
There are still a couple of things I'd like to check in detail (the pushdown logic in particular), but for what I could see it looks good, so I'm approving to unblock the merge.

[luigidellaquila]

nit: I'd expect this logic to be in EsqlDataTypeConverter.commonType() (that you are using below btw).
The only logical difference is noCounter().

Maybe commonTypes() could just do

                for (List<Attribute> out : outputs) {
                    type = EsqlDataTypeConverter.commonType(type, out.get(i).dataType()).noCounter();
                }

[bpintea]

Started my journey on this PR. Minor/optional notes only.

[bpintea]

It'd be great to have some javadoc for this class.
(I would seem it's mostly useful in PushDownFilterAndLimitIntoUnionAll, where we first push "below" UnionAll and then once there, below Subquery?
Seems like the LogicalPlanBuilder#visitRelation always places the index patern first, followed by the subqueries.
I'm wondering thus if this marker node is really needed. Wouldn't these push downs be doable on each branch by existing rules, if Subquery wasn't there?
But still to look into it and didn't grasp it all yet. Javadoc would be great in any case, though. :) )

[fang-xing-esql]

There are a couple of reasons that Subquery node exists in the plan.

First, originally I added this node because in the future we may want to support qualifiers with subqueries(like FROM idx1, (FROM idx2, idx3) as idx23), and this is a place that we can store the name(qualifier) of the subqueries, that can be referenced by the parent query.

Second, when I started working on pushing down filters into subqueries, I realize PushDownAndCombineFilters does not push down filters below limit intentionally, refer to here, however UnionAll/Fork adds a implicit limit for each branch, so this rule doesn't help the predicate pushdown for subqueries. The Subquery node is also used as a pattern for predicate pushdown. I'll add more comments in the code.

[bpintea]

One issue I see that we could align is the treatment of the "union"/conflicting types: in case these emerge out of an index pattern, we return the type as unsupported, along with the original_types and a suggested_cast. User is in the know, all good.

With UnionAll, we return it of type keyword, even if no index in the union has the conflicting field of that type and null values, but with no other indication whatsoever that there's a conflict. The user won't know of the conflict, which I think is problematic, as this could occur frequently.

Don't know if we have a decision here or track this somewhere?

(Some more comments to follow, sorry :) )

[bpintea]

Can this now ever happen? Shouldn't it be guaranteed by the grammar that it's never null?

[fang-xing-esql]

The piece of code that processes subquery index patterns in EsqlSession will be in a separate PR from @craigtaverner, and it will be shared by subqueries and views, stay tuned :).

[bpintea]

This isn't currently supported by the grammar, no?

[bpintea]

Suggested change

	indexResolution = context.subqueryResolution().getOrDefault(indexPattern, indexResolution);
	indexResolution = context.subqueryResolution().get(indexPattern);
	if (indexResolution == null) {...

Wouldn't it be a bug if the simple get() would return a null? And wouldn't we introduce a new one, if an "unkown" pattern resolves then to the "main" index?

[bpintea]

I think there are some fixes needed on conflicting/union types from different branches. I've left a few notes, out of which one is, I think, a bug.

Potentially related to that: when running the query in that comment - with index1 only having "x": 1 and index2 having "x": "2" (i.e. both text and keyword fields) - I get a CCE:

[2025-10-17T12:49:31,742][ERROR][o.e.r.ChunkedRestResponseBodyPart] [runTask-0] failure encoding chunk java.lang.ClassCastException: class org.elasticsearch.compute.data.IntVectorBlock cannot be cast to class org.elasti
csearch.compute.data.LongBlock (org.elasticsearch.compute.data.IntVectorBlock and org.elasticsearch.compute.data.LongBlock are in unnamed module of loader java.net.FactoryURLClassLoader @11d2dd2d)
        at org.elasticsearch.xpack.esql.action.PositionToXContent$1.valueToXContent(PositionToXContent.java:72)
        at org.elasticsearch.xpack.esql.action.PositionToXContent.positionToXContent(PositionToXContent.java:53)
        at org.elasticsearch.xpack.esql.action.ResponseXContentUtils.lambda$rowValues$7(ResponseXContentUtils.java:114)
        at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.rest.ChunkedRestResponseBodyPart$1.encodeChunk(ChunkedRestResponseBodyPart.java:161)
        at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.rest.RestController$EncodedLengthTrackingChunkedRestResponseBodyPart.encodeChunk(RestController.java:1012)
        at org.elasticsearch.transport.netty4@9.3.0-SNAPSHOT/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.writeChunk(Netty4HttpPipeliningHandler.java:436)
        at org.elasticsearch.transport.netty4@9.3.0-SNAPSHOT/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.doWriteChunkedResponse(Netty4HttpPipeliningHandler.java:263)
        at org.elasticsearch.transport.netty4@9.3.0-SNAPSHOT/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.doWrite(Netty4HttpPipeliningHandler.java:231)
        at org.elasticsearch.transport.netty4@9.3.0-SNAPSHOT/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.write(Netty4HttpPipeliningHandler.java:184)
        at io.netty.transport@4.1.126.Final/io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:891)
        at io.netty.transport@4.1.126.Final/io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:956)
        at io.netty.transport@4.1.126.Final/io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1263)
        at io.netty.common@4.1.126.Final/io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
        at io.netty.common@4.1.126.Final/io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
        at io.netty.common@4.1.126.Final/io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
        at io.netty.transport@4.1.126.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
        at io.netty.common@4.1.126.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
        at io.netty.common@4.1.126.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.base/java.lang.Thread.run(Thread.java:1575)

Also, something like:

FROM index1, (FROM index2) | EVAL l = x::LONG | EVAL i = x::INTEGER | KEEP x, l, i

fails with a verification exception:

"org.elasticsearch.xpack.esql.VerificationException: Found 1 problem\nline 130:47: Output has changed from [[x{r}#204, l{r}#191, i{r}#194]] to [[x{r}#204, l{r}#191, i{r}#194]].
	at org.elasticsearch.xpack.esql.optimizer.LogicalPlanOptimizer.optimize(LogicalPlanOptimizer.java:121)
	at org.elasticsearch.xpack.esql.session.EsqlSession.optimizedPlan(EsqlSession.java:947)
	at org.elasticsearch.xpack.esql.session.EsqlSession$1.lambda$onResponse$1(EsqlSession.java:228)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:261)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:406)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:326)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:222)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.lambda$andThen$1(SubscribableListener.java:534)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:465)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.newForked(SubscribableListener.java:138)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.andThen(SubscribableListener.java:534)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.andThen(SubscribableListener.java:489)
	at org.elasticsearch.xpack.esql.session.EsqlSession$1.onResponse(EsqlSession.java:228)
	at org.elasticsearch.xpack.esql.session.EsqlSession$1.onResponse(EsqlSession.java:219)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:406)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:326)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:355)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:262)
	at org.elasticsearch.xpack.esql.session.EsqlSession.analyzeWithRetry(EsqlSession.java:885)
	at org.elasticsearch.xpack.esql.session.EsqlSession.lambda$resolveIndices$14(EsqlSession.java:538)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:261)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:406)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:326)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:355)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:262)
	at org.elasticsearch.xpack.esql.session.EsqlSession.preAnalyzeSubqueryIndices(EsqlSession.java:554)
	at org.elasticsearch.xpack.esql.session.EsqlSession.lambda$preAnalyzeSubqueryIndices$15(EsqlSession.java:551)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:261)
	at org.elasticsearch.xpack.esql.session.EsqlSession.lambda$preAnalyzeSubqueryIndex$16(EsqlSession.java:589)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:233)
	at org.elasticsearch.xpack.esql.session.IndexResolver.lambda$resolveAsMergedMapping$0(IndexResolver.java:100)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:261)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:413)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:228)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:222)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:350)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:413)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:33)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:111)
	at org.elasticsearch.xpack.esql.action.EsqlResolveFieldsAction.finishHim(EsqlResolveFieldsAction.java:338)
	at org.elasticsearch.xpack.esql.action.EsqlResolveFieldsAction.lambda$doExecuteForked$7(EsqlResolveFieldsAction.java:232)
	at org.elasticsearch.base@9.3.0-SNAPSHOT/org.elasticsearch.core.AbstractRefCounted$1.closeInternal(AbstractRefCounted.java:125)
	at org.elasticsearch.base@9.3.0-SNAPSHOT/org.elasticsearch.core.AbstractRefCounted.decRef(AbstractRefCounted.java:77)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.support.RefCountingRunnable.close(RefCountingRunnable.java:113)
	at org.elasticsearch.base@9.3.0-SNAPSHOT/org.elasticsearch.core.Releasables$4.close(Releasables.java:178)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.RunOnce.run(RunOnce.java:41)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.fieldcaps.RequestDispatcher.innerExecute(RequestDispatcher.java:177)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.fieldcaps.RequestDispatcher$1.doRun(RequestDispatcher.java:146)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction$1.onResponse(TransportFieldCapabilitiesAction.java:328)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction$1.onResponse(TransportFieldCapabilitiesAction.java:324)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractThrottledTaskRunner$1.doRun(AbstractThrottledTaskRunner.java:136)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1067)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1575)\n"

[bpintea]

I think this is problematic: we convert an emerging union output attribute to the first[*] type showing up in a conversion function, irrespective of this conversion function applying to the union output attribute or not.
That is: if x is a field in two indices (of different types), part of the output of something like:

FROM index1, (FROM index2) | EVAL some_field = x::LONG | EVAL some_other_field = x::INTEGER

x's type will be either LONG or INTEGER, depending on the order of those EVALs. Which I think is wrong.

[*] first in order of walking the tree.

I guess we probably want to restrict the map to only those attributes that apply a conversion function to another one with the same name? In which case, I guess we'll want to simply push down the conversion function to the UnionAll branches -- so not just copy it, but push it away downwards: right now the conversions stay, even if done on the same name attribute.

[fang-xing-esql]

It is a good idea, I'll see how to deal with it. I'd like to keep the explicit casting push down in this PR. And I'm going to remove the implicit casting among subqueries and main index patterns from this PR, and do it as a follow up, as suggested by @alex-spies in the design review, if you haven't review the implicit casting part, you can skip it for now.

[fang-xing-esql]

Thank you so much for catching this! I think we can handle multiple conversion functions on the same UnionAll output better. The collectConvertFunctions method has been updated to collect all explicit conversion functions from the main query that reference the outputs of UnionAll. These explicit conversion functions are now pushed down into each UnionAll branch and returned as new output attributes. And the subsequent commands after UnionAll can reference the outputs without confusion.

[bpintea]

Why do we need to check for the EsqlProject presence / pattern?

[fang-xing-esql]

Up to here, the branches of UnionAll follow a common pattern like below:

UnionAll/Fork
  Project
    Eval (optional)
      Subquery (main index patterns do not have this node)
        ...

In both ResolveUnionTypesInUnionAll and PushDownFilterAndLimitIntoUnionAll, the pattern checks are intentionally strict — these rules only apply their transformations when a branch of UnionAll exactly matches the expected patterns. The main goal is to have a tight/better control over their behavior and ensure they perform only the intended transformations. If the structure of subqueries changes in the future and no longer fits the expected pattern, these rules will simply stop applying, allowing us to detect such changes early in the process.

[bpintea]

I think this condition can only be true if convertFunctions.containsKey(oldAttr.name()). Maybe we can simplify a bit the code here for better legibility.

[diaryfolio]

hi @fang-xing-esql , just checking the feature is still available in snapshot build and NOT still into GA?