Skip to content

[SPARK-17590][SQL] Analyze CTE definitions at once and allow CTE subquery to define CTE #15146

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
viirya wants to merge 3 commits into from
Closed

[SPARK-17590][SQL] Analyze CTE definitions at once and allow CTE subquery to define CTE #15146

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
394495d
Analyze CTE definition at once.
viirya Sep 19, 2016
2f8e79f
Allow subquery to define CTE.
viirya Sep 19, 2016
baf239b
Adjust style.
viirya Sep 20, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -262,7 +262,7 @@ ctes
;

namedQuery
: name=identifier AS? '(' queryNoWith ')'
: name=identifier AS? '(' query ')'
Copy link
Contributor

@cloud-fan cloud-fan Sep 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you explain more about this grammar change?

Copy link
Member Author

@viirya viirya Sep 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently we only allow a query without With used in the CTE's named query. This change loose this limit to allow a query with With to be a named query. So we can define CTE in CTE subquery.

Copy link
Member Author

@viirya viirya Sep 19, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, query is defined as "ctes? queryNoWith".

;

tableProvider
Expand Down
5 changes: 2 additions & 3 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,15 +116,14 @@ class Analyzer(
)

/**
* Substitute child plan with cte definitions
* Analyze cte definitions and substitute child plan with analyzed cte definitions.
*/
object CTESubstitution extends Rule[LogicalPlan] {
// TODO allow subquery to define CTE
def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
case With(child, relations) =>
substituteCTE(child, relations.foldLeft(Seq.empty[(String, LogicalPlan)]) {
case (resolved, (name, relation)) =>
resolved :+ name -> ResolveRelations(substituteCTE(relation, resolved))
resolved :+ name -> execute(substituteCTE(relation, resolved))
})
case other => other
}
Expand Down
2 changes: 1 addition & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with Logging {
* This is only used for Common Table Expressions.
*/
override def visitNamedQuery(ctx: NamedQueryContext): SubqueryAlias = withOrigin(ctx) {
SubqueryAlias(ctx.name.getText, plan(ctx.queryNoWith), None)
SubqueryAlias(ctx.name.getText, plan(ctx.query), None)
}

/**
Expand Down
25 changes: 25 additions & 0 deletions sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,31 @@ class SubquerySuite extends QueryTest with SharedSQLContext {
)
}

test("define CTE in CTE subquery") {
checkAnswer(
sql(
"""
| with t2 as (with t1 as (select 1 as b, 2 as c) select b, c from t1)
| select a from (select 1 as a union all select 2 as a) t
| where a = (select max(b) from t2)
""".stripMargin),
Array(Row(1))
)
checkAnswer(
sql(
"""
| with t2 as (with t1 as (select 1 as b, 2 as c) select b, c from t1),
| t3 as (
| with t4 as (select 1 as d, 3 as e)
| select * from t4 cross join t2 where t2.b = t4.d
| )
| select a from (select 1 as a union all select 2 as a)
| where a = (select max(d) from t3)
""".stripMargin),
Array(Row(1))
)
}

test("uncorrelated scalar subquery in CTE") {
checkAnswer(
sql("with t2 as (select 1 as b, 2 as c) " +
Expand Down