[SPARK-32344][SQL][2.4] Unevaluable expr is set to FIRST/LAST ignoreNullsExpr in distinct aggregates

[maropu]

What changes were proposed in this pull request?

This PR intends to fix a bug of distinct FIRST/LAST aggregates in v2.4.6;

scala> sql("SELECT FIRST(DISTINCT v) FROM VALUES 1, 2, 3 t(v)").show()
...
Caused by: java.lang.UnsupportedOperationException: Cannot evaluate expression: false#37
  at org.apache.spark.sql.catalyst.expressions.Unevaluable$class.eval(Expression.scala:258)
  at org.apache.spark.sql.catalyst.expressions.AttributeReference.eval(namedExpressions.scala:226)
  at org.apache.spark.sql.catalyst.expressions.aggregate.First.ignoreNulls(First.scala:68)
  at org.apache.spark.sql.catalyst.expressions.aggregate.First.updateExpressions$lzycompute(First.scala:82)
  at org.apache.spark.sql.catalyst.expressions.aggregate.First.updateExpressions(First.scala:81)
  at org.apache.spark.sql.execution.aggregate.HashAggregateExec$$anonfun$15.apply(HashAggregateExec.scala:268)

A root cause of this bug is that the Aggregation strategy replaces a foldable boolean ignoreNullsExpr expr with a Unevaluable expr (AttributeReference) for distinct FIRST/LAST aggregate functions. But, this operation cannot be allowed because the Analyzer has checked that it must be foldabe;

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/First.scala

Lines 74 to 76 in ffdbbae

	} else if (!ignoreNullsExpr.foldable) {
	TypeCheckFailure(
	s"The second argument of First must be a boolean literal, but got: ${ignoreNullsExpr.sql}")

So, this PR proposes to change a vriable for IGNORE NULLS from Expression to Boolean to avoid the case.

This is the backport of #29143.

Why are the changes needed?

Bugfix.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Added a test in DataFrameAggregateSuite.

[dongjoon-hyun]

+1, LGTM. (Pending Jenkins.)

[maropu]

Thanks, @dongjoon-hyun, for the update in the jira.

[dongjoon-hyun]

:)

[dongjoon-hyun]

This PR removed UnsupportedOperationException and all relevant test cases are passed in Scala/Java. Also, Python UTs passed. Currently, it's running on R mllib testing. So, I merged this to branch-2.4. Thanks!

[SparkQA]

Test build #126141 has finished for PR 29157 at commit c190886.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• case class First(child: Expression, ignoreNulls: Boolean)
• case class Last(child: Expression, ignoreNulls: Boolean)