[ESQL] Adds wriring for compute side of LIMIT BY command by ncordon · Pull Request #143458 · elastic/elasticsearch

ncordon · 2026-03-03T11:52:13Z

Adds a compute engine operator GroupedLimitOperator, for a new LIMIT BY ESQL command. LIMIT N BY expr1, expr2,... retains at most N documents per group, where groups are defined by one or more grouping key expressions.

FROM test
| SORT salary DESC
| LIMIT 5 BY languages
| KEEP first_name, last_name, salary, languages

This should be a no-op because it's not used anywhere in the language yet.

Part of #112918, https://github.com/elastic/esql-planning/issues/238

elasticsearchmachine · 2026-03-03T12:12:40Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

Copilot

Pull request overview

Adds compute-engine support for the ESQL LIMIT BY plan shape by threading grouping expressions through logical/physical Limit nodes and introducing a new compute operator to enforce per-group row limits, including transport-version gated serialization.

Changes:

Extend Limit (logical) and LimitExec (physical) with groupings and add mixed-version serialization behavior for LIMIT BY.
Wire planning/mapping to pass groupings through and execute GroupedLimitOperator when groupings are present.
Introduce GroupedLimitOperator + PositionKeyEncoder with accompanying unit tests and register the new operator status writeable.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/plan/physical/LimitExecSerializationTests.java	Updates physical plan serialization tests for new `groupings` field + mixed-version behavior.
x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/plan/logical/LimitSerializationTests.java	Updates logical plan serialization tests for new `groupings` field + mixed-version behavior.
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/EsqlPlugin.java	Registers `GroupedLimitOperator.Status` in named writeables.
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/mapper/Mapper.java	Passes `Limit.groupings()` into `LimitExec`.
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/mapper/LocalMapper.java	Passes `Limit.groupings()` into `LimitExec` for local planning.
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/LocalExecutionPlanner.java	Plans `LimitExec` into `LimitOperator` vs `GroupedLimitOperator` based on groupings.
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/physical/LimitExec.java	Adds `groupings` to physical node and gates serialization by transport version.
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/Limit.java	Adds `groupings` to logical node and gates serialization by transport version.
x-pack/plugin/esql/compute/src/test/java/org/elasticsearch/compute/operator/PositionKeyEncoderTests.java	Adds coverage for composite key encoding semantics.
x-pack/plugin/esql/compute/src/test/java/org/elasticsearch/compute/operator/GroupedLimitOperatorTests.java	Adds behavioral tests for grouped limiting and status fields.
x-pack/plugin/esql/compute/src/test/java/org/elasticsearch/compute/operator/GroupedLimitOperatorStatusTests.java	Adds wire/xcontent tests for `GroupedLimitOperator.Status`.
x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/PositionKeyEncoder.java	Implements grouping-key encoding across channels/values.
x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/GroupedLimitOperator.java	Implements per-group limiting operator + status.
server/src/main/resources/transport/upper_bounds/9.4.csv	Updates the 9.4 transport upper bound to include `esql_limit_by`.
server/src/main/resources/transport/definitions/referable/esql_limit_by.csv	Adds referable transport version definition for `esql_limit_by`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...ugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/GroupedLimitOperator.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/physical/LimitExec.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/Limit.java

...ugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/GroupedLimitOperator.java

nik9000 · 2026-03-03T16:56:40Z

...ugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/GroupedLimitOperator.java

+    private final int limitPerGroup;
+    private final PositionKeyEncoder keyEncoder;
+    private final BigArrays bigArrays;
+    private final BytesRefHashTable seenKeys;


It's worth a comment saying that you want this and not the a BlockHash implementation because those expand multivalues and this doesn't. I could see a world where one day we have other implementations of BlockHash that don't expand keys, but today is not that day. This is a good way to do it.

Could you make the comment javadoc? It'll read nicer if you mouseover the item.

...ugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/GroupedLimitOperator.java

ivancea · 2026-03-04T12:40:45Z

...ck/plugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/GroupKeyEncoder.java

+     * below 128 are encoded in a single byte, making this compact for the
+     * small numbers typical of value counts and byte-array lengths.
+     */
+    private void writeVInt(int value) {


It would be nice to be able to share logic with DefaultUnsortableTopNEncoder in these methods, as they do basically the same

We could have a TopNEncoder.DEFAULT_UNSORTABLE inside this class? I've done that here: d2bd9f3. Not sure if I like we are calling inside. Any thoughts?

...ugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/GroupedLimitOperator.java

...esql/compute/src/test/java/org/elasticsearch/compute/operator/GroupedLimitOperatorTests.java

...ugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/GroupedLimitOperator.java

cimequinox · 2026-03-04T14:27:07Z

...ugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/GroupedLimitOperator.java

+                BytesRef key = keyEncoder.encode(page, pos);
+                long hashOrd = seenKeys.add(key);
+                int count;
+                long ord;


No need to change anything but TIL BytesRefHashTable.add returns >= 0 when a new key was added and (-1-id) which is < 0 when it was already present.

I found that quite clever when I read about it 😄

nik9000 · 2026-03-04T14:04:06Z

...ugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/GroupedLimitOperator.java

+    private final int limitPerGroup;
+    private final PositionKeyEncoder keyEncoder;
+    private final BigArrays bigArrays;
+    private final BytesRefHashTable seenKeys;


Could you make the comment javadoc? It'll read nicer if you mouseover the item.

...ugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/GroupedLimitOperator.java

nik9000 · 2026-03-04T14:07:33Z

...ugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/GroupedLimitOperator.java

+        this.counts = bigArrays.newIntArray(16, false);
+    }
+
+    public static final class Factory implements Operator.OperatorFactory {


Could you move the Factory up to the top? I get confused when the factory is below the ctor. Mostly when I'm scanning the files quickly.

nik9000 · 2026-03-04T14:09:45Z

...ugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/GroupedLimitOperator.java

+            }
+        }
+        return result;
+    }


Could you move this into Page?

I see we do it like 40 times in other places. It'd be lovely have it in one. But it'd be nice to dedupe in a follow-up change.

Yep, Ivan pointed to me we already have a filter method in Page

nik9000 · 2026-03-04T14:30:31Z

...ugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/GroupedLimitOperator.java

+    @Override
+    public void finish() {
+        finished = true;
+    }


For later: there are cases where we know the hash can never grow any larger - like, we know the number of unique values we're grouping on. In that case we could early terminate - just like LimitOperator does. Say you are doing:

FROM foo | SORT @timestamp DESC | LIMIT 10 BY hostname

It's quite possible to know the number of unique hostnames, especially if it's less than a million. The old aggs framework uses a thing called "global ordinals" for this. We don't want those because they are very expensive at unexpected times. But their existence proves that such counts are possible/easy/good.

That is for later though.

👍 I've added it to the issue we have tracking things to polish LIMIT BY: https://github.com/elastic/esql-planning/issues/262

ncordon · 2026-03-04T17:36:10Z

I've addressed all of the feedback, just missing an opinion on this #143458 (comment) from @ivancea

ncordon · 2026-03-04T17:44:40Z

...ck/plugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/GroupKeyEncoder.java

-public class GroupKeyEncoder {
+public class GroupKeyEncoder implements Releasable {
+
+    private static final DefaultUnsortableTopNEncoder encoder = TopNEncoder.DEFAULT_UNSORTABLE;


Not sure whether I like this

Looks good to me, and we reuse some code. Plus, having the concrete class declared there should help with inlining.
Is there something you don't like specifically?

The naming perhaps (we are calling something from TopN inside limit), but it's fine, I can live with it

Uhm yep, the namespacing seems odd, but I would live with it. Ideally, if we decide to change to a BlockHash or similar, that would be "fixed" (Fun, because BlockHash is in the aggregations package, so not perfect either! 😆)

...esql/compute/src/test/java/org/elasticsearch/compute/operator/GroupedLimitOperatorTests.java

ivancea · 2026-03-04T18:05:50Z

...ck/plugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/GroupKeyEncoder.java

-public class GroupKeyEncoder {
+public class GroupKeyEncoder implements Releasable {
+
+    private static final DefaultUnsortableTopNEncoder encoder = TopNEncoder.DEFAULT_UNSORTABLE;


Looks good to me, and we reuse some code. Plus, having the concrete class declared there should help with inlining.
Is there something you don't like specifically?

...ugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/GroupedLimitOperator.java

ivancea

LGTM!

...ugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/GroupedLimitOperator.java

) --------- Co-authored-by: Iván Cea Fontenla <ivan.cea@elastic.co>

[ESQL] Adds wriring for compute side of LIMIT BY command

d893856

elasticsearchmachine added needs:triage Requires assignment of a team area label v9.4.0 labels Mar 3, 2026

Adds serialization tests for LIMIT with groupings

03d0830

ncordon added >feature Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) :Analytics/ES|QL AKA ESQL and removed needs:triage Requires assignment of a team area label labels Mar 3, 2026

ncordon requested a review from Copilot March 3, 2026 12:14

Copilot started reviewing on behalf of ncordon March 3, 2026 12:15 View session

ncordon review requested due to automatic review settings March 3, 2026 12:22

ncordon and others added 4 commits March 3, 2026 13:45

Adds more tests and tweaks

57a6798

[CI] Auto commit changes from spotless

d7ed4e1

Adds transport version tests

b232ec0

Merge remote-tracking branch 'upstream/main' into esql-limit-by

1bef15c

ivancea requested review from Copilot and ivancea March 3, 2026 14:42

Copilot started reviewing on behalf of ivancea March 3, 2026 14:43 View session

Copilot AI reviewed Mar 3, 2026

View reviewed changes

Undoes any logical or physical plan part

6a24032

ncordon force-pushed the esql-limit-by branch from f3532ee to 6a24032 Compare March 3, 2026 14:52

Rename

e0d2c30

nik9000 reviewed Mar 3, 2026

View reviewed changes

ncordon added 2 commits March 3, 2026 19:27

Adds code for the edge case limitPerGroup = 0

10b97f4

Adds some explanatory comments

3fbeb25

ncordon force-pushed the esql-limit-by branch from 7253a98 to 3fbeb25 Compare March 3, 2026 19:08

Merge branch 'main' into esql-limit-by

e53190b

ncordon mentioned this pull request Mar 4, 2026

ESQL: Added GroupedTopNOperator for LIMIT BY, compute only #143476

Merged

ivancea reviewed Mar 4, 2026

View reviewed changes

cimequinox reviewed Mar 4, 2026

View reviewed changes

nik9000 approved these changes Mar 4, 2026

View reviewed changes

Uses javadoc instead

cb5bf9a

Addresses pr review feedback

d2bd9f3

ncordon force-pushed the esql-limit-by branch from 83a5552 to d2bd9f3 Compare March 4, 2026 17:41

ncordon commented Mar 4, 2026

View reviewed changes

ncordon added 2 commits March 4, 2026 18:51

Adds more multivalue tests

f3f05ab

Last nit

a45aef8

ncordon force-pushed the esql-limit-by branch from ebb54f0 to a45aef8 Compare March 4, 2026 18:08

ivancea reviewed Mar 4, 2026

View reviewed changes

Addresses last bits of feedback, hopefully

88664e7

ncordon force-pushed the esql-limit-by branch from a16e089 to 88664e7 Compare March 4, 2026 19:24

Merge branch 'main' into esql-limit-by

e28884e

ivancea reviewed Mar 5, 2026

View reviewed changes

...ugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/GroupedLimitOperator.java Show resolved Hide resolved

ncordon merged commit 3b090df into elastic:main Mar 5, 2026
35 checks passed

ncordon deleted the esql-limit-by branch March 5, 2026 15:17

jfreden pushed a commit to jfreden/elasticsearch that referenced this pull request Mar 5, 2026

[ESQL] Adds wriring for compute side of LIMIT BY command (elastic#143458

634b84b

) --------- Co-authored-by: Iván Cea Fontenla <ivan.cea@elastic.co>

spinscale pushed a commit to spinscale/elasticsearch that referenced this pull request Mar 6, 2026

[ESQL] Adds wriring for compute side of LIMIT BY command (elastic#143458

1a21533

) --------- Co-authored-by: Iván Cea Fontenla <ivan.cea@elastic.co>

This was referenced Mar 6, 2026

[ML] Wait for cluster state in test #143767

Merged

[Transform] Disable PIT for CPS #143876

Closed

This was referenced Mar 9, 2026

[ESQL] Adds parsing and planner wiring for LIMIT BY #143762

Closed

[ESQL] Adds planner implementation for LIMIT BY #144069

Merged

Conversation

ncordon commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Mar 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ncordon Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cimequinox Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ncordon Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ncordon commented Mar 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ivancea left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

ncordon commented Mar 3, 2026 •

edited

Loading

ncordon Mar 4, 2026 •

edited

Loading

cimequinox Mar 4, 2026 •

edited

Loading

ncordon Mar 4, 2026 •

edited

Loading