Skip to content

[SPARK-19799] Support WITH clause (CTE) in subqueries #22936

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
gbloisi wants to merge 3 commits into from
Closed

[SPARK-19799] Support WITH clause (CTE) in subqueries #22936

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
66cd537
Because of SPARK-17590 support of WITH clause in subqueries requires …
gbloisi Nov 3, 2018
5377087
Fix scala check issue
gbloisi Nov 7, 2018
3080ed4
Resolve scalastyle check error
gbloisi Nov 7, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -372,7 +372,7 @@ queryPrimary
: querySpecification #queryPrimaryDefault
| TABLE tableIdentifier #table
| inlineTable #inlineTableDefault1
| '(' queryNoWith ')' #subquery
| '(' query ')' #subquery
;

sortItem
Expand Down Expand Up @@ -510,7 +510,7 @@ identifierComment

relationPrimary
: tableIdentifier sample? tableAlias #tableName
| '(' queryNoWith ')' sample? tableAlias #aliasedQuery
| '(' query ')' sample? tableAlias #aliasedQuery
| '(' relation ')' sample? tableAlias #aliasedRelation
| inlineTable #inlineTableDefault2
| functionTable #tableValuedFunction
Expand Down
4 changes: 2 additions & 2 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -795,7 +795,7 @@ class AstBuilder(conf: SQLConf) extends SqlBaseBaseVisitor[AnyRef] with Logging
* Create a logical plan for a sub-query.
*/
override def visitSubquery(ctx: SubqueryContext): LogicalPlan = withOrigin(ctx) {
plan(ctx.queryNoWith)
plan(ctx.query)
}

/**
Expand Down Expand Up @@ -883,7 +883,7 @@ class AstBuilder(conf: SQLConf) extends SqlBaseBaseVisitor[AnyRef] with Logging
* }}}
*/
override def visitAliasedQuery(ctx: AliasedQueryContext): LogicalPlan = withOrigin(ctx) {
val relation = plan(ctx.queryNoWith).optionalMap(ctx.sample)(withSample)
val relation = plan(ctx.query).optionalMap(ctx.sample)(withSample)
if (ctx.tableAlias.strictIdentifier == null) {
// For un-aliased subqueries, use a default alias name that is not likely to conflict with
// normal subquery names, so that parent operators can only access the columns in subquery by
Expand Down
12 changes: 12 additions & 0 deletions sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -535,7 +535,19 @@ class SQLQuerySuite extends QueryTest with SharedSQLContext {
|q2 as (select * from testData where key = '4')
|select * from q1 union all select * from q2""".stripMargin),
Row(5, "5") :: Row(4, "4") :: Nil)
}

test("SPARK-19799: Support WITH clause in subqueries") {
Copy link
Member

@gatorsmile gatorsmile Jan 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add more test cases? For example, can a with clause in a subquery shadow a with clause in an enclosing query with the same name? Another example, use with clauses in a subquery expression?

Copy link
Contributor Author

@gbloisi gbloisi Feb 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to find some spare time for that, but I'll surely do sooner or later. I assumed that most of the correctness comes from #15146 but surely there may be interactions that goes beyond that.

checkAnswer(
sql("""
|select avg(b) from (
| with maxearnings as (
| select course, year, max(earnings) as earnings FROM courseSales
| GROUP BY course, year
| )
| select course, sum(earnings) as b FROM maxearnings GROUP BY course
| )""".stripMargin),
Row(54000) :: Nil)
}

test("Allow only a single WITH clause per query") {
Expand Down