ESQL: Nested expressions inside stats command

[costin]

Allow nested expressions to be used both for grouping or inside
aggregate functions inside the stats command.
This allows the following stats command:

stats c = count( a / 2 ) by x + 1

Since the grouping now can be an expression, the grammar has
been tweaked to allow the stats group to have optional aliasing
similar to aggregate functions

stats c = count( a / 2 ) by z = b + 1

Fix #99828

[elasticsearchmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[elasticsearchmachine]

Hi @costin, I've created a changelog YAML for you.

[astefan]

There's a small overlook while parsing the groupings, the fix is simple, but the results of the queries is highly impacted.

[luigidellaquila]

Looks good in general.

There is only a small regression in the validation process:

row a = 1, b = 2 | stats max(max(a)) by b

in main branch returns

{
    "error": {
        "root_cause": [
            {
                "type": "verification_exception",
                "reason": "Found 1 problem\nline 1:26: aggregate function's field must be an attribute or literal; found [max(a)] of type [Max]"
            }
        ],
        "type": "verification_exception",
        "reason": "Found 1 problem\nline 1:26: aggregate function's field must be an attribute or literal; found [max(a)] of type [Max]"
    },
    "status": 400
}

while with this PR returns

{
    "error": {
        "root_cause": [
            {
                "type": "ql_illegal_argument_exception",
                "reason": "Unsupported expression [max(a)]"
            }
        ],
        "type": "ql_illegal_argument_exception",
        "reason": "Unsupported expression [max(a)]"
    },
    "status": 500
}

It's a different error message, but most importantly, it's a different error code (500 vs 400).

Even though the root problem is not strictly related to this PR (it depends on the fact that EVAL command does not properly validate the usage of aggregate functions), IMHO it's worth fixing now, both because of the regression and because it's a 500 error.

Notice that the same problem happens with the following queries:

row a = 1, b = 2 | stats max(a) by max(b)
row a = 1, b = 2 | eval x = max(a)

[luigidellaquila]

Unrelated to this PR: this was never implemented, we should remove it probably.

[bpintea]

@luigi, trying to understand your note:

The error is not so clear (aggregate function's field must be an attribute or literal: which field? And why is that expression unsupported?),

Do you mean we should make it clearer that the outer max is the one the failure is about? As in, the existing explanation found [max(a)] of type [Max] isn't enough?

the root problem is not strictly related with this PR (it depends on the fact that EVAL command does not properly validate the usage of aggregate functions)

Wondering too why we didn't verify aggs in EVAL. But is this issue the same as how we verify the arguments of aggs themselves?

[luigidellaquila]

Do you mean we should make it clearer that the outer max is the one the failure is about? As in, the existing explanation found [max(a)] of type [Max] isn't enough?

@bpintea yeah, I removed that part of the comment already, I realized the message is actually good enough in the context of aggs validation.

Wondering too why we didn't verify aggs in EVAL. But is this issue the same as how we verify the arguments of aggs themselves?

until now they were two different problems, but they could become the same if we rely on EVAL checks after expression extraction from STATS.
IMHO the best solution is to do the aggs validation before the extraction, and in case fail fast.
EVAL validation can be fixed separately probably.

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[costin]

I've added the validation for eval and stats; the previous validation was checking for something else and caught the case of aggs inside aggs.

[astefan]

LGTM

[costin]

/cc @abdonpijpelink please update the stats docs with the expanded grammar - thanks!