[ESQL] Adds planner implementation for LIMIT BY

[ncordon]

This is part of the new LIMIT BY ESQL command. LIMIT N BY expr1, expr2,... retains at most N documents per group, where groups are defined by one or more grouping key expressions.

It introduces a new LimitBy logical plan node and LimitByExec physical plan node. It doesn't modify existing Limit and LimitExec nodes.

This PR knits the parser changes and the planner changes for the compute side, GroupedLimitOperator that was added in #143458.

This means it enables queries like this:

FROM test
| LIMIT 5 BY languages
| KEEP first_name, last_name, salary, languages

but we have restricted using a SORT before the LIMIT BY until #143476 and #144279 have been merged. We have also done this for cleanliness because otherwise this PR would be even bigger. This means queries like this one would fail:

FROM test
| SORT first_name
| LIMIT 5 BY languages
| KEEP first_name, last_name, salary, languages

Part of #112918, https://github.com/elastic/esql-planning/issues/238

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[Copilot]

Pull request overview

This PR introduces the initial end-to-end plumbing for the new ESQL command LIMIT N BY expr1, expr2, ..., adding parsing support, logical/physical plan nodes, optimizer rules, and compute-engine planning to execute grouped limiting (without supporting SORT directly before LIMIT BY yet).

Changes:

• Add new logical (LimitBy) and physical (LimitByExec) plan nodes, including (de)serialization and planner wiring.
• Extend the parser/grammar to recognize LIMIT ... BY ... (dev-version gated) and add verification to reject SORT directly before LIMIT BY.
• Add logical/physical optimizer rules and tests to handle expression groupings, pruning foldable group keys, and combining/pushing down LIMIT BY.

Reviewed changes

Copilot reviewed 40 out of 41 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/EsqlBaseParser.g4	Grammar: adds optional BY ... group key to LIMIT command (dev build gated).
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/LogicalPlanBuilder.java	Parser: builds LimitBy logical node when BY is present.
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/LimitBy.java	New logical plan node representing grouped limit and its execution-location hints.
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/physical/LimitByExec.java	New physical plan node for grouped limit, including row-size estimation and serialization.
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/mapper/Mapper.java	Planner mapping: wires LimitBy -> LimitByExec and injects local limits into fragments under LimitBy.
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/mapper/LocalMapper.java	Local planner mapping for LimitBy -> LimitByExec.
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/PlannerUtils.java	Reduction planning: skips “local” pipeline breakers when selecting reduction boundaries.
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/LocalExecutionPlanner.java	Compute planning: adds GroupedLimitOperator planning for LimitByExec; refactors attribute-channel lookup.
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/LogicalPlanOptimizer.java	Adds new LIMIT BY optimizer rules to the optimization pipeline.
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/logical/ReplaceLimitByExpressionWithEval.java	New rule: extracts non-attribute groupings into synthetic Eval + wraps with Project.
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/logical/PruneLiteralsInLimitBy.java	New rule: drops foldable grouping keys and degenerates to plain LIMIT when all keys are foldable.
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/logical/PushDownAndCombineLimitBy.java	New rule: combines adjacent LimitBy, pushes/duplicates across certain unary/binary nodes.
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Verifier.java	Adds verification: blocks SORT directly before LIMIT BY (temporary restriction).
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/core/expression/Expressions.java	Adds listSemanticEquals utility for comparing grouping lists.
x-pack/plugin/esql/qa/testFixtures/src/main/resources/limit.csv-spec	QA coverage: adds LIMIT BY spec tests behind limit_by capability.
x-pack/plugin/esql/src/test/java/...	Adds extensive unit tests for parser, verifier, optimizer rules, and physical planning/serialization.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[idegtiarenko]

Do we need this limit here and in other cases? I believe we have much fewer records in employees, especially above condition.

[idegtiarenko]

Is it to avoid sorts over the entire result set?

[ivancea]

Technically no, as the default "limit 1000" automatically added. We can remove it

[idegtiarenko]

Maybe it is to avoid warning about default limit?
I do not think it is suppressed by new limit by, is it?

[ivancea]

It's not, so yes, it could be for the warning 👀

[ncordon]

We want stable results so we have to use SORT, but we cannot use SORT + LIMIT BY yet because we need a different logical and physical plan for that, so it's disallowed. The intermediate LIMIT is just temporary. I've added removing it as a task to do in the future so we don't forget: https://github.com/elastic/esql-planning/issues/262

[ivancea]

It's not, so yes, it could be for the warning 👀

[ncordon]

@ncordon It looks like the PR you referred to is already merged. #143476 . Should we remove the SORT restriction or you want to do that in a separate PR?

We cannot remove the check in this PR yet. The planner to enable SORT before is the pr we should review after #144279. I've updated the description.