[ESQL] Add a BY clause to CHANGE_POINT command by darius-vil · Pull Request #145210 · elastic/elasticsearch

darius-vil · 2026-03-30T15:24:19Z

This introduces a BY clause to CHANGE_POINT: https://www.elastic.co/docs/reference/query-languages/esql/commands/change-point

The most complicated change is in ChangePointOperator, which now has the added complexity of keeping track of group changes in the sorted input.

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/ChangePoint.java

darius-vil · 2026-03-31T10:01:10Z

.../esql/compute/src/test/java/org/elasticsearch/compute/operator/ChangePointOperatorTests.java

+        // Group A: [0×15, 1×15] -> step at row 15 of page0
+        // Group B: [1×15, 0×15] -> step at row 15 of page1
+        // Group C: [0×15, 1×15] -> step at row 15 of page2
+        List<Long> valuesColumn = Stream.of(


These are somewhat hard to read, but I really wanted some tests around groups spanning across pages...

darius-vil · 2026-03-31T10:04:28Z

x-pack/plugin/esql/src/main/antlr/EsqlBaseParser.g4

 changePointConfiguration
    : ON key=qualifiedName
    | AS targetType=qualifiedName COMMA targetPvalue=qualifiedName
+    | {this.isDevVersion()}? BY grouping=qualifiedName


I assume {this.isDevVersion()}? is needed since LimitBy has it and we depend on it.

Update: looks like LimitBy is about to remove it: #145225
Should we follow suit?

darius-vil · 2026-03-31T10:04:58Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java


        /**
         * Enables the feature LIMIT n BY expr1, expr2 for retaining at most n docs per group.
-         * The feature will not work if we had SORT | LIMIT n BY


No longer relevant:#144279

elasticsearchmachine · 2026-03-31T11:19:26Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2026-03-31T11:19:28Z

Hi @darius-vil, I've created a changelog YAML for you.

leemthompo · 2026-03-31T11:32:29Z

docs/reference/query-languages/esql/_snippets/commands/layout/change_point.md


 ```esql
-CHANGE_POINT value [ON key] [AS type_name, pvalue_name]
+CHANGE_POINT value [ON key] [BY group] [AS type_name, pvalue_name]


@darius-vil we must keep version changes relevant for all users on 9.x

this change is only relevant to 9.4+ so we need to tag it appropriately with applies_to tags

See this comment for pointers on that : #144300 (comment)

you could keep the old one in a 9.whatever-9.foo tab and add a new one in a 9.4+ versioned tab (AKA applies switch)

Thanks for pointing this out!

I meant to ask how do these work, but it completely slipped my mind in the end... I'll fix this in a sec

For the time being, I marked all the new switches/sections/inline changes with stack: preview 9.4 and serverless: preview - I'm not sure what the correct values should be.

Since CHANGE_POINT BY hinges on recently introduced LIMIT BY, I guess our release lifecycle should follow theirs?

I'll keep this comment open until I find the answer

cool you'll definitely need stack: ga|preview 9.4 but if it's ga you won't need any serverless tags :)

docs/reference/query-languages/esql/_snippets/commands/layout/change_point.md

jan-elastic · 2026-04-02T14:52:05Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/ChangePoint.java

+        Attribute key,
+        Attribute targetType,
+        Attribute targetPvalue,
+        Attribute grouping


What about multiple groupings (CHANGE_POINT ... BY host, day, etc)) or general expressions (BY n+1, LENGTH(host))

I think this should be a List<Expression> comparable to LimitBy.

Never considered multiple groups, my original proposal has always been CHANGE_POINT v [ON t] [AS k, l] [BY g] instead of CHANGE_POINT v [ON t] [AS k, l] [BY g1 [, g2 [, ... [, gN]]]].

I should have probably guessed though 😄 Thanks for pointing this out!

Luckily, this doesn't complicate things that much. This includes two major changes:

change in grammar and the boilerplate around it

tracking when changepoint needs to be invoked is the complicated part, but thanks to reusing

elasticsearch/x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/GroupKeyEncoder.java

Line 34 in 81dd6ce

public class GroupKeyEncoder implements Accountable, Releasable {

this becomes trivial

Regarding supporting Expressions, I got them working by following the same path as LimitBy - they have a custom optimizer that replaces expressions with attributes:

elasticsearch/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/logical/ReplaceLimitByExpressionWithEval.java

Line 33 in d690e76

public final class ReplaceLimitByExpressionWithEval extends OptimizerRules.OptimizerRule<LimitBy> {

I copied this approach, it works, but I am unsure whether it's the only and correct way of achieving this yet.

jan-elastic · 2026-04-02T14:56:17Z

x-pack/plugin/esql/src/main/antlr/EsqlBaseParser.g4

 changePointConfiguration
    : ON key=qualifiedName
    | AS targetType=qualifiedName COMMA targetPvalue=qualifiedName
+    | {this.isDevVersion()}? BY grouping=qualifiedName


I think this should be (BY grouping=fields)? (see also StatsCommand)

Hmm, not sure. Why do you think ()? is necessary here?

Notice that the actual changePointCommand is a few lines above

changePointCommand : CHANGE_POINT value=qualifiedName (changePointConfiguration)* ;

that's where changePointConfiguration is used and it's wrapped in ()*, meaning any rules in changePointConfiguration can appear zero or more times. It's later verified in LogicalPlanBuilder that any changePointConfiguration rule appears at most 1 time.

May be helpful here: https://github.com/antlr/antlr4/blob/dev/doc/parser-rules.md#subrules

jan-elastic · 2026-04-02T14:57:09Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/change_point.csv-spec

 ;
+
+
+detect step change by group not nulled


Please add test for:

multiple grouping (by a,b,c)

expression grouping (by a+1, "hi")

jan-elastic · 2026-04-02T14:57:55Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/ChangePoint.java

+        // Grouping must be sortable
+        if (grouping != null) {
+            type = grouping.dataType();
+            if (DataType.isSortable(type) == false) {


I don't see why this must be sortable.

Is that the case with LimitBy as well?

My thinking at the time of writing was:

ChangePoint invokes LimitBy with the expectation that it sorts our input by grouping(s), which means grouping(s) has to be sortable and this is the place to verify whether grouping(s) is sortable?

I think if we don't do this here, it would simply fail in OrderBy with a similar cause:

elasticsearch/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/OrderBy.java

Line 118 in 478fac8

order.forEach(order -> {

, but maybe it's worth keeping this check in ChangePoint anyway for the more precise fail message? CHANGE_POINT grouping only supports sortable values, found expression

LimitBy doesn't necessarily sorts the input by grouping(s) (as far as I know). It just groups by it. You can do that without sorting, e.g. by using a hash table.

jan-elastic · 2026-04-02T15:09:55Z

...lugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/ChangePointOperator.java

 * data that is passed to it, runs the change point detector on the data (which
 * is a compute-heavy process), and then outputs all data with the change points.
 */
 public class ChangePointOperator extends CompleteInputCollectorOperator {


This is not correct anymore:

ChangePointOperator used to be a CompleteInputCollectorOperator, because it would collect 1001 rows at max.

The new changepoint by, on the other hand, can get unlimited amounts of data, and needs to be streaming.

jan-elastic · 2026-04-02T15:13:07Z

...lugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/ChangePointOperator.java

+    private void createOutputPages() {
+        List<Double> values = new ArrayList<>();
+        List<Integer> bucketIndexes = new ArrayList<>();
+        ArrayDeque<DetectedChangePoint> detectedChangePoints = new ArrayDeque<>();


I don't see why we need this.

The way I imagine this working:

process addInput(Page page) calls until you hit a row with group key != previous group key

different group key triggers detecting the change point in the group

create the output for the group

jan-elastic

I gave this a first pass, see comments. I think there's some functionality missing.

darius-vil added 17 commits March 26, 2026 11:43

Add BY to grammar

0307f52

Ad csv tests

081a557

Implement CHANGE_POINT BY

769fc78

Add column sortable check

7379341

Fix factory describe

16db485

Apply spotless

b2f0161

Introduce nullable groupingChannel

a70e5b9

Refactor methods

e14cccb

Fix edge case

294dec4

Collect changepoints in a queue

057d781

Add tests around changepoints split across pages

254a3f6

Fix bug and add more unit tests

59a30a3

Merge branch 'main' into changepoint-limit-by

bed2254

Add BY parser tests after merge

f75395c

Fix empty input and nulls in the middle group

8233ba2

Refactor unit test

decb5a9

Remove TODOs

6dbd9cd

elasticsearchmachine added the v9.4.0 label Mar 30, 2026

darius-vil commented Mar 30, 2026

View reviewed changes

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/ChangePoint.java Show resolved Hide resolved

elasticsearchmachine and others added 7 commits March 30, 2026 15:39

[CI] Auto commit changes from spotless

8113031

Count value limit per group

a1d9975

Update docs

d4cdc6b

Minor fixes

1ac27e9

Fix test checkstyle

d66805c

[CI] Auto commit changes from spotless

862fbae

Fix typo in test

5d2337a

darius-vil changed the title ~~Changepoint limit by~~ [ESQL] Add a BY clause to CHANGE_POINT command Mar 31, 2026

darius-vil commented Mar 31, 2026

View reviewed changes

darius-vil added >enhancement Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) labels Mar 31, 2026

darius-vil added Team:ML Meta label for the ML team >feature :Analytics/ES|QL AKA ESQL and removed >enhancement labels Mar 31, 2026

darius-vil marked this pull request as ready for review March 31, 2026 11:18

darius-vil requested a review from a team as a code owner March 31, 2026 11:18

elasticsearchmachine removed the Team:ML Meta label for the ML team label Mar 31, 2026

Update docs/changelog/145210.yaml

7a45883

leemthompo reviewed Mar 31, 2026

View reviewed changes

docs/reference/query-languages/esql/_snippets/commands/layout/change_point.md Outdated Show resolved Hide resolved

leemthompo reviewed Mar 31, 2026

View reviewed changes

docs/reference/query-languages/esql/_snippets/commands/layout/change_point.md Outdated Show resolved Hide resolved

leemthompo reviewed Mar 31, 2026

View reviewed changes

docs/reference/query-languages/esql/_snippets/commands/layout/change_point.md Outdated Show resolved Hide resolved

Add versioning to docs

583c302

darius-vil mentioned this pull request Mar 31, 2026

ESQL: Added LIMIT BY docs (Tech preview) #144323

Closed

darius-vil and others added 4 commits April 1, 2026 14:41

Merge branch 'main' into changepoint-limit-by

f1a0a57

Add CHANGE_POINT BY generative test

f8f57b6

Merge branch 'main' into changepoint-limit-by

2a15f9a

[CI] Auto commit changes from spotless

fead1ff

jan-elastic reviewed Apr 2, 2026

View reviewed changes

jan-elastic requested changes Apr 2, 2026

View reviewed changes

brianseeders added v9.5.0 and removed v9.4.0 labels Apr 10, 2026

Conversation

darius-vil commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Mar 31, 2026

Uh oh!

elasticsearchmachine commented Mar 31, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leemthompo Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jan-elastic Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jan-elastic Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jan-elastic left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

darius-vil commented Mar 30, 2026 •

edited

Loading

leemthompo Mar 31, 2026 •

edited

Loading

jan-elastic Apr 2, 2026 •

edited

Loading

jan-elastic Apr 2, 2026 •

edited

Loading