ES|QL: Improve aggregation over constants handling by astefan · Pull Request #112392 · elastic/elasticsearch

astefan · 2024-08-30T11:48:31Z

This change consists of:

Add separate rule for dealing with nulls in aggregations
Duplicate SubstituteSurrogate in "Operator Optimization" batch
Many more tests
Add test for median_absolute_deviation function
Add mv handling to top function
Allows PropagateEvalFoldables rule to also deal with aggregate functions

Addresses part of #100634. Missing bits:

AwaitsFixes from LogicalPlanOptimizerTests cannot be removed yet. This likely has also to do with substitutions() batch -> NormalizeAggregate() rule (that needs to be added?) in LogicalPlanOptimizer
median_absolute_deviation is still pending on having its own mv_ function
st_centroid agg mv_* function still needs addressing
recent update: mv_values sister function is pending addition

Fixes #110257
Fixes #104430
Fixes #100170

Needs more tests for the new rule and the existent ones in LogicalPlanOptimizerTests.

Duplicate SubstituteSurrogate in "Operator Optimization" batch Many more tests Add tests for mad Add mv handling to top function

astefan · 2024-08-30T11:49:33Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/spatial.csv-spec

+null             |null                     |null
+;
+
+########### failing :-( with InvalidArgumentException: Does not support yet aggregations over constants


This will be removed once we have mv_ function for st_centroid_agg.

astefan · 2024-08-30T11:52:04Z

...lugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/aggregate/Top.java

@@ -197,11 +202,18 @@ public AggregatorFunctionSupplier supplier(List<Integer> inputChannels) {
    public Expression surrogate() {


The surrogate method, as it stands now, is more a "surrogate-expression-for-foldable-scenario" kind of method. This implies that the behavior that existed below before this change is not possible anymore.

Right - we should probably look into introducing a different interface altogether: surrogate was initially used for expressions that knew they'd be transformed.
But it evolved into a mechanism for "folding" however not to a value, but another expression (which itself might be foldable or not).

I'll leave this one for a follow up I think.

...k/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/LogicalPlanOptimizer.java

astefan · 2024-08-30T12:00:44Z

...ql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/ReplaceAggregatesWithNull.java

+ * All aggregate functions that are also nullable (COUNT_DISTINCT and COUNT are exceptions), will get a NULL
+ * field replacement by the FoldNull rule, COUNT_DISTINCT will benefit from PropagateEvalFoldables.
+ */
+public final class ReplaceAggregatesWithNull extends OptimizerRules.OptimizerRule<Aggregate> {


This rule is a simplified variant of SubstituteSurrogates.

astefan · 2024-08-30T12:01:25Z

...in/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/SubstituteSurrogates.java

+        Map<AggregateFunction, Attribute> aggFuncToAttr = new HashMap<>(); // existing aggregate and their respective attributes
+        List<Alias> transientEval = new ArrayList<>(); // surrogate functions eval
        boolean changed = false;
+        boolean hasSurrogates = false;


I've done this to shortcircuit the execution earlier in the execution.

astefan · 2024-08-30T12:02:10Z

...gin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LogicalPlanOptimizerTests.java

        assertEquals(countd, rule.rule(countd));
        countd = new CountDistinct(EMPTY, NULL, NULL);
-        assertEquals(new Literal(EMPTY, null, LONG), rule.rule(countd));
+        assertEquals(countd, rule.rule(countd));


This is the consequence of CountDistinct not being nullable anymore.

costin

LGTM - great tests and comments!

costin · 2024-09-02T16:02:48Z

.../src/main/java/org/elasticsearch/xpack/esql/expression/function/aggregate/CountDistinct.java


+    @Override
+    public Nullability nullable() {
+        return Nullability.FALSE;


costin · 2024-09-02T16:05:37Z

...lugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/aggregate/Top.java

@@ -197,11 +202,18 @@ public AggregatorFunctionSupplier supplier(List<Integer> inputChannels) {
    public Expression surrogate() {


Right - we should probably look into introducing a different interface altogether: surrogate was initially used for expressions that knew they'd be transformed.
But it evolved into a mechanism for "folding" however not to a value, but another expression (which itself might be foldable or not).

ivancea · 2024-09-02T16:44:25Z

...in/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/aggregate/Values.java

+
+    @Override
+    public Expression surrogate() {
+        return field().foldable() ? field() : null;


Values not only merges values, but also removes duplicates (If no test was triggered because of this, we should add some!)

ROW x = [1, 1, 2] | STATS a = VALUES(x) -> [1, 2]

Good catch, BUT I think we have a problem with the documentation. It's not mentioning this aspect. There were other misses in our functions docs (which are fixed in this PR), I think we need to review our documentation on functions and double check its correctness and completeness. I will create an issue.

@ivancea thank you for pro-actively checking this PR 🙏, that was very helpful.
I've created two issues:

improving our documentation: ES|QL: review, double check and add missing bits to functions documentation #112437

this PR also now depends on pending mv_values addition: ES|QL: add mv_values function #112445

alex-spies

This is great @astefan ! I think this change is sound and added mostly minor remarks.

My only major remark is: I think we need LogicalPlanOptimizerTests cases that prove that the foldable propagation actually takes place. The csv tests are great, but they do not prove that foldable propagation actually takes place, only that the result is correct.

But you already mentioned more optimizer tests as one of the tasks to un-draft :)

x-pack/plugin/esql/qa/testFixtures/src/main/resources/meta.csv-spec

alex-spies · 2024-09-03T12:27:08Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/row.csv-spec

Out of scope: stats.csv-spec has a bunch of ...OfConst tests that overlap a lot with the tests here, except that they normally start with from employees. Because these test stats more than row, maybe we should move test cases like row ... | stats ... from here to stats.csv-spec in a follow-up PR.

x-pack/plugin/esql/qa/testFixtures/src/main/resources/stats.csv-spec

alex-spies · 2024-09-03T14:53:50Z

.../esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/PropagateEvalFoldables.java

+            } else if (p instanceof Aggregate agg) {
+                List<NamedExpression> newAggs = new ArrayList<>(agg.aggregates().size());
+                agg.aggregates().forEach(e -> {
+                    if (Alias.unwrap(e) instanceof AggregateFunction) {


This looks like it cannot propagate into the groups, as in

... | eval x = [1,2,3] | stats sum(field) by x

right? Maybe it's worth adding a comment.

That's another thing we could optimize though if needed, as I think STATS ... BY const is the same as STATS ... | eval x = mv_values(const) | mv_expand x. Not sure that's worth maintaining an optimization rule for, though.

Yeah, it's not covered. Unintentionally, just I didn't think about this use case.
Will leave it for a follow up, though. There are many things going on in this PR.

...ql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/ReplaceAggregatesWithNull.java

alex-spies · 2024-09-03T15:28:47Z

...ql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/ReplaceAggregatesWithNull.java

+            } else {
+                // All aggs actually have been optimized away
+                // \_Aggregate[[],[AVG([NULL][NULL]) AS s]]
+                // Replace by a local relation with one row, followed by an eval, e.g.
+                // \_Eval[[MVAVG([NULL][NULL]) AS s]]
+                // \_LocalRelation[[{e}#21],[ConstantNullBlock[positions=1]]]
+                plan = new LocalRelation(
+                    source,
+                    List.of(new EmptyAttribute(source)),
+                    LocalSupplier.of(new Block[] { BlockUtils.constantBlock(PlannerUtils.NON_BREAKING_BLOCK_FACTORY, null, 1) })
+                );
+            }


The code in lines 86-106 also happens in SubstituteSurrogates, and kinda-sorta also in ReplaceStatsAggExpressionWithEval. I opened #110345 but maybe, instead of reducing the number of opt. rules, we should just refactor the code path that moves expressions out of aggregates and into evals. We could start here and make sure the code is the same as in SubstituteSurrogates.

…aggregations_over_constants

argument

…aggregations_over_constants

Introduce isConstantFoldable() for aggregate functions

…aggregations_over_constants

error message

alex-spies

Heya, just wanted to pick this up again and summarize what I think we need to do:

Per the discussion with Costin, percentile(field, null) is still not well defined (null vs invalid query) - okay to hash this out in a follow-up but maybe invalidating for now is safer w.r.t. bwc.
ST_CENTROID_AGG(null) should return null instead of Point(NaN NaN).
Some additional test cases won't hurt.
Fixing this edge case where all agg functions are optimized away

Consider this unblocked from my side as my main reason for requesting changes was the discussion about percentile(field, null) and similar cases. We could solve some problems in follow-up PRs as well, as I think the general approach here works :)

alex-spies · 2024-11-18T15:12:47Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/stats_percentile.csv-spec

We don't have to hash this out now, but maybe it'd be safer to start with a validation exception now - we can still go back and return null later, while the other way around could be considered a breaking change, albeit in a very edge case scenario.

…aggregations_over_constants

felixbarny · 2026-01-16T07:06:49Z

Is this PR superseded by #139797 or is it something we're planning to do additionally?

astefan · 2026-01-16T07:20:17Z

Is this PR superseded by #139797 or is it something we're planning to do additionally?

I haven't checked the other PR, only speaking about this one I created and explored some time ago.
If that other PR uses the same tests that this PR has in it (which I believe is the better part of the PR) and they all pass, I think it would ok to close this PR and consider it superseded by #139797.

ivancea · 2026-01-16T10:55:13Z

I'll take a look in case the other PR missed something we should move 👀

Add separate rule for dealing with nulls in aggregations

5159109

Duplicate SubstituteSurrogate in "Operator Optimization" batch Many more tests Add tests for mad Add mv handling to top function

astefan requested review from alex-spies and costin August 30, 2024 11:48

elasticsearchmachine added the v8.16.0 label Aug 30, 2024

astefan commented Aug 30, 2024

View reviewed changes

...k/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/LogicalPlanOptimizer.java Show resolved Hide resolved

astefan commented Aug 30, 2024

View reviewed changes

One more test from one of the reported bugs

74129a0

costin approved these changes Sep 2, 2024

View reviewed changes

ivancea reviewed Sep 2, 2024

View reviewed changes

This was referenced Sep 2, 2024

ES|QL: review, double check and add missing bits to functions documentation #112437

Open

ES|QL: add mv_values function #112445

Closed

alex-spies reviewed Sep 3, 2024

View reviewed changes

mark-vieira added v9.0.0 and removed v8.16.0 labels Sep 11, 2024

astefan added 7 commits September 18, 2024 19:18

Addressing reviews

ef3960e

Make count_distinct deal with Validateable interface

c2db11c

Merge branch 'main' of https://github.com/elastic/elasticsearch into …

d66004b

…aggregations_over_constants

More count_distinct fixes, more tests for percentile's foldable second

aedad55

argument

Merge branch 'main' of https://github.com/elastic/elasticsearch into …

47db145

…aggregations_over_constants

Add more tests

4c03ce2

Introduce isConstantFoldable() for aggregate functions

Merge branch 'main' of https://github.com/elastic/elasticsearch into …

0e8d484

…aggregations_over_constants

astefan marked this pull request as ready for review October 3, 2024 09:53

elasticsearchmachine added the needs:triage Requires assignment of a team area label label Oct 3, 2024

astefan added >bug auto-backport-and-merge and removed needs:triage Requires assignment of a team area label labels Oct 3, 2024

brianseeders added v8.17.0 and removed v8.16.0 labels Oct 16, 2024

Bug fix related to the source() used when creating the Validations

db2259b

error message

alex-spies requested review from alex-spies and removed request for alex-spies November 8, 2024 14:13

alex-spies reviewed Nov 18, 2024

View reviewed changes

elasticsearchmachine added v8.18.0 and removed v8.17.0 labels Nov 20, 2024

astefan added 3 commits November 27, 2024 18:23

Merge branch 'main' of https://github.com/elastic/elasticsearch into …

db006d6

…aggregations_over_constants

Fix some things after pull from main

1ef3b87

Merge branch 'main' of https://github.com/elastic/elasticsearch into …

968dcfb

…aggregations_over_constants

This was referenced Dec 6, 2024

ESQL: Rework isNull #118101

Merged

Add nullable to Categorize #118176

Closed

Mechanism for foldable aggregations in ES|QL #118292

Open

elasticsearchmachine added v8.19.0 v9.1.0 and removed v8.18.0 v9.0.0 labels Jan 30, 2025

elasticsearchmachine added v9.2.0 and removed v9.1.0 labels Jun 26, 2025

elasticsearchmachine added v9.3.0 and removed v9.2.0 labels Oct 2, 2025

astefan mentioned this pull request Nov 7, 2025

Fixes esql class cast bug in STATS at planning level #137511

Merged

elasticsearchmachine added v9.4.0 and removed v9.3.0 labels Dec 17, 2025

astefan mentioned this pull request Feb 4, 2026

ESQL: Double check type resolution of functions when the NULL type is used #140577

Closed

		@@ -197,11 +202,18 @@ public AggregatorFunctionSupplier supplier(List<Integer> inputChannels) {
		public Expression surrogate() {

Conversation

astefan commented Aug 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

costin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

felixbarny commented Jan 16, 2026

Uh oh!

astefan commented Jan 16, 2026

Uh oh!

ivancea commented Jan 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

astefan commented Aug 30, 2024 •

edited

Loading