[SPARK-41985][SQL] Centralize more column resolution rules

[cloud-fan]

What changes were proposed in this pull request?

This is a followup of #38888 .

When I search for all the matching of UnresolvedAttribute, I found that there are still a few rules doing column resolution:

1. ResolveAggAliasInGroupBy
2. ResolveGroupByAll
3. ResolveOrderByAll
4. ResolveDefaultColumns

This PR merges the first 3 into ResolvedReferences. The last one will be done with a separate PR, as it's more complicated.

To avoid making the rule ResolvedReferences bigger and bigger, this PR pulls out the resolution code for Aggregate to a separated virtual rule (only be used by ResolvedReferences). The same to Sort. We can refactor and add more virtual rules later.

Why are the changes needed?

It's problematic to not centralize all the column resolution logic, as the execution order of the rules is not reliable. It actually leads to regression after #38888 : select a from t where exists (select 1 as a group by a). The group by a should be resolved as 1 as a, but now it's resolved as outer reference a. This is because ResolveReferences runs before ResolveAggAliasInGroupBy, and resolves outer references too early.

Does this PR introduce any user-facing change?

Fixes a bug, but the bug is not released yet.

How was this patch tested?

new tests

[cloud-fan]

these functions are pulled out to ColumnResolutionHelper without actual changes.

[cloud-fan]

This is the actual change. Now we resolve group by alias and group by all before resolving outer references.

[cloud-fan]

cc @viirya

[viirya]

There are some conflicts.

[cloud-fan]

No new code, all code are moved from Analyzer.

[anchovYu]

I commented one issue with a failed case. Tricky, we need to note that once an expression is resolved to OuterReference there is no way back.

[anchovYu]

Add tests for this resolution order: 1) resolve group by alias 2) resolve group by all.
E.g. select .. as all .. group by all

[anchovYu]

As we discussed, select a + 1 as b, b + 1 as c .. group by c will fail as LCA now doesn't handle grouping expr. Need a followup if we want to support above query (which I think we should).

[anchovYu]

It can be problematic, try with this query:

      sql(s"select * from values(1) as t(something) " +
        s"where exists (select salary * 1.5 as something from $testTable group by something)")

where testTable has a salary field. (I just wrote in LCA suite.)

[cloud-fan]

Good catch! It's important to remember that rules will be executed repeatedly and we need to make sure the rule won't conflict with its previous runs.

[dtenedor]

Thanks @cloud-fan for doing this as well! Very useful for the health of the analyzer.

[gengliangwang]

It seems that the agg exprs are not changed from here.
Shall we abstract the code as

val resolvedGroupExprs = if (resolvedAggExprsWithOuter.forall(_.resolved)) {
  if (resolvedAggExprsWithOuter.exists(_.containsPattern(LATERAL_COLUMN_ALIAS_REFERENCE))) {
    ...
  } else {
    ...
  }
} else {
...
}
a.copy(...)

[anchovYu]

+1, even use single-layer if, because all conditions are on one variable resolvedAggExprsWithOuter

[gengliangwang]

Shall we use "aggregation expressions" instead of Aggregate.aggregateExpressions? The comment is using "grouping expressions" below

[gengliangwang]

nit: column => columns

[gengliangwang]

+1 for putting the outer ref at the last order. outer ref can only be used in where/having so it makes sense to resolve as the other references first.

[viirya]

If it is not group by alias but group by outer reference, it works? From ResolveReferencesInAggregate seems so, just want to confirm.

[anchovYu]

Actually reading the code I think group by alias is triggered, because the whole resolvedAggExprsWithOuter are resolved (to outer reference)..

[viirya]

GROUP BY alias takes precedence than outer reference?

[cloud-fan]

yes

[anchovYu]

Also add test for group by all:

SELECT 1 AS x, x + 1 AS k FROM testData GROUP BY all;

[anchovYu]

+1, even use single-layer if, because all conditions are on one variable resolvedAggExprsWithOuter

[anchovYu]

Can we add extra handling to report user-friendly error message, showing that queries of group by all and group by alias are not supported when there is LCA in the select clause? For example, check if the grouping expression is all or alias, throw exception immediately.

Currently I don't have a clear solution how to support the above case with the LCA design. So this unsupported situation may last for some time, and we better provide clean error messages.

[anchovYu]

Actually reading the code I think group by alias is triggered, because the whole resolvedAggExprsWithOuter are resolved (to outer reference)..

[anchovYu]

Curious, do we require the child of Sort to be resolved? Or is it implied somewhere? Any risk of not doing so (the outer reference thing)? Though i can't think of any negative case.

[cloud-fan]

Yes we do, ResolveRefererences only invokes these sub-rules if all children are resolved

[anchovYu]

haha I forgot ..

[anchovYu]

Any tests covering this order: resolveColWithAgg > resolveOrderByAll ?

[cloud-fan]

@anchovYu can you help review the new tests? I think I've covered all the cases.

[cloud-fan]

ditto

[anchovYu]

Missing reference won't be able to cover this case, it should only add grouping expressions to the Aggregate. v1.b is not a grouping expression, so if order by v1.b the query should throw exception.

[anchovYu]

Test is good. just comment needs correction.

[anchovYu]

haha I forgot ..

[cloud-fan]

this extends the fix of https://issues.apache.org/jira/browse/SPARK-31670.

Alias in grouping expressions is harmful. Semantically, alias in grouping expressions should be ignored. We trim aliases in grouping expressions at the end of analysis by the rule CleanupAliases, so that CheckAnalysis can correctly match grouping expressions with aggregate expressions. However, there are rules doing the same matching during analysis, and CleanupAliases is too late for them.

[anchovYu]

For details in how it affects LCA:
Without the trim,

=== Applying Rule org.apache.spark.sql.catalyst.analysis.ResolveLateralColumnAliasReference ===
!Aggregate [salary#19 AS lca#15], [salary#19 AS lca#15, (lateralAliasReference(lca#15, lca, lca#15) + 1) AS col#16]   Project [lca#15, (lateralAliasReference(lca#15, lca, lca#15) + 1) AS col#16]
!+- SubqueryAlias spark_catalog.default.employee                                                                      +- Aggregate [salary#19 AS lca#15], [salary#19 AS lca#15]
!   +- Relation spark_catalog.default.employee[dept#17,name#18,salary#19,bonus#20,properties#21] orc                     +- SubqueryAlias spark_catalog.default.employee
!                                                                                                                           +- Relation spark_catalog.default.employee[dept#17,name#18,salary#19,bonus#20,properties#21] orc

ResolveLateralColumnAliasReference matches the whole salary#19 AS lca#15 same as the grouping expression and pushes it down to the Aggregate (this rule finds matches in grouping expression and aggregate functions to push down). It causes problem for later application of this rule on Project: there is no alias in the Project any more so the lateralAliasReference can't find an alias to match and be removed.

[cloud-fan]

The pyspark failure is unrelated, I'm merging it to master/3.4 (as it fixes a regression), thanks for the review!