[ES|QL] Remove implicit limit appended to each subquery branch

[fang-xing-esql]

Resolves: #138106

The implicit limit appended to each subquery adds limitations for subquery, especially when working with inline stats. It may also cause non-deterministic results from subqueries when there is no sort in the subquery to ensure the order of the intermediate results.

This PR attempts to remove the implicit limit append to each subquery branch, and enables more capabilities of subquery. There is potential risk that the intermediate results returned by a subquery is huge, and they need to be processed in batches or CB if it is too big to process.

The major changes are listed here, there is NO change to Fork. More follow ups on heap attack tests are needed, this PR focus on functionality, making sure correct results are returned without the implicit limit appended to each subquery. The subquery feature is still behind snapshot.

• HeapAttackSubqueryIT is added to catch potential OOM or CBE caused by large intermediate results from subqueries. Some tests with 8 subqueries hit OOM, instead of CBE, there are PR and new issues created to address those OOMs as follow ups. Currently most of the subquery heap attack tests run with 2 subqueries, root causes of the OOMs and TODOs and commented in each test.
• PushDownFilterAndLimitIntoUnionAll is updated accordingly after Analyzer does not add the implicit limit to each subquery branch.
  • There is a special case related to the knn function. When a subquery contains a knn, it still requires a limit to appear after the knn, because knn has an implicitK that is not serialized or sent to the remote nodes. This was not an issue previously, as an implicit limit was appended to each branch. The PushDownFilterAndLimitIntoUnionAll rule has been updated to handle this knn case. Without a limit following the knn, the query either fails LogicalVerifier or produces incorrect results.
  • The sort operator has a similar situation, as unbounded sort is not supported. If a subquery contains a sort without a limit, removing the implicit limit previously appended to each branch causes LogicalVerifier to fail. In this PR, we do not append an implicit limit for this case, instead, it lets LogicalVerifier fail and inform users of the unbounded sort. If support for this query pattern is required in the future, an implicit limit could be appended, using an approach similar to the one applied for knn.

Some details on the OOM happened to the new subquery heap attack tests
There are some OOM discovered by the new subquery heap attack tests, they exposed some untracked memory used during reading(unsorted or sorted) data from lucene, the list of issues and PRs are below. These will be fixed as follow ups. This PR focus on functionality, instead of memory intensive queries.

• Convert PackedValuesBlockHash.bytes to BreakingBytesRefBuilder for better memory tracking.
• Push sorts(with many large fields) to lucene may take lot of untracked memory
• Read many large keyword/text fields may take a lot of untracked memory

[elasticsearchmachine]

Hi @fang-xing-esql, I've created a changelog YAML for you.

[fang-xing-esql]

There are some unrelated failures in release tests, remove the test-release label for now.

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[astefan]

LGTM.
Left only some minor comments.

[nik9000]

• HeapAttackSubqueryIT is added to catch potential OOM or CBE caused by large intermediate results from subqueries. Some tests with 8 subqueries hit OOM, instead of CBE, there are PR and new issues created to address those OOMs as follow ups. Currently most of the subquery heap attack tests run with 2 subqueries, root causes of the OOMs and TODOs and commented in each test.

Talked with @fang-xing-esql, all of the changes this PR powers are currently only available in SNAPSHOT builds. So it's quite safe to merge without fixing the OOMs. So long as we lock the removal of the SNAPSHOT behind fixing the OOMs, we're safe.

I'll look at the PR soon with that in mind.

[nik9000]

Found an out of date comment, otherwise looks good to me.

[fang-xing-esql]

Thanks for the review @astefan @nik9000 ! All the comments have been address.