[SPARK-10437][SQL] Support aggregation expressions in Order By

[viirya]

JIRA: https://issues.apache.org/jira/browse/SPARK-10437

If an expression in SortOrder is a resolved one, such as count(1), the corresponding rule in Analyzer to make it work in order by will not be applied.

[SparkQA]

Test build #41996 has finished for PR 8599 at commit 452cfb5.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

We need to set up a stop condition for this rule, or something like SELECT a, SUM(b) FROM t GROUP BY a ORDER BY a will go through this rule again and again until reach the fixed point. How about changing the end of this rule to:

if (evaluatedOrderings == sortOrder) {
  sort
} else {
  Project(aggregate.output,
    Sort(evaluatedOrderings, global,
      aggregate.copy(aggregateExpressions = originalAggExprs ++ needsPushDown)))
}

[viirya]

Thanks. I've updated it.

[SparkQA]

Test build #42003 has finished for PR 8599 at commit e65f4db.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

LGTM except a minor comment for test.
cc @marmbrus

[SparkQA]

Test build #42036 has finished for PR 8599 at commit 8f73c40.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

retest this please.

[SparkQA]

Test build #42038 has finished for PR 8599 at commit 8f73c40.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #42040 has finished for PR 8599 at commit 6023c8b.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

retest this please.

[viirya]

retest this please.

[SparkQA]

Test build #42057 has finished for PR 8599 at commit 6023c8b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class BlockFetchException(messages: String, throwable: Throwable)

[viirya]

ping @liancheng

[viirya]

ping @liancheng @marmbrus

[chenghao-intel]

Nit: And also add more typical unit tests like:
"""
|SELECT a
|FROM orderByData
|GROUP BY a
|ORDER BY a, count(*), sum(b)
""".stripMargin),

[viirya]

Thanks. I've added it.

[SparkQA]

Test build #42186 has finished for PR 8599 at commit 046a8cb.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Instance(w: Double, a: Vector, b: Double)
  • class WeibullGenerator(

[liancheng]

[SQ] in the PR description should be [SQL].

[viirya]

@liancheng Updated. Thanks.

[viirya]

@liancheng Any more comment?

[marmbrus]

Thanks, merged to master.