[SPARK-17590][SQL] Analyze CTE definitions at once and allow CTE subquery to define CTE #15146
Conversation
viirya
changed the title
|
Test build #65586 has finished for PR 15146 at commit
|
|
Test build #65588 has finished for PR 15146 at commit
|
|
|
||
| namedQuery | ||
| : name=identifier AS? '(' queryNoWith ')' | ||
| : name=identifier AS? '(' query ')' |
There was a problem hiding this comment.
can you explain more about this grammar change?
There was a problem hiding this comment.
Currently we only allow a query without With used in the CTE's named query. This change loose this limit to allow a query with With to be a named query. So we can define CTE in CTE subquery.
There was a problem hiding this comment.
BTW, query is defined as "ctes? queryNoWith".
|
|
||
| test("define CTE in CTE subquery") { | ||
| checkAnswer( | ||
| sql("with t2 as (with t1 as (select 1 as b, 2 as c) select b, c from t1) " + |
There was a problem hiding this comment.
can we use multi-line string with proper ident here? this test is almost unreadable...
|
Seems good, but not quite sure, cc @hvanhovell to confirm |
|
|
||
| test("define CTE in CTE subquery") { | ||
| checkAnswer( | ||
| sql("with t2 as (with t1 as (select 1 as b, 2 as c) select b, c from t1) " + |
|
@viirya does this improve performance? |
|
@hvanhovell This is for analyzer change and adds CTE in CTE feature. I don't expect there is performance improvement. |
|
I guess if using the same analyzed plan increases the chance to reuse exchange, then it may improve the performance. Anyway, it is not the purpose of this change. Because the analyzed subquery plan will be changed largely in optimization later, we cannot guarantee this. |
|
Test build #65630 has finished for PR 15146 at commit
|
|
Merging to master. Thanks! |
|
@hvanhovell @cloud-fan Thanks! |
4 participants
What changes were proposed in this pull request?
We substitute logical plan with CTE definitions in the analyzer rule CTESubstitution. A CTE definition can be used in the logical plan for multiple times, and its analyzed logical plan should be the same. We should not analyze CTE definitions multiple times when they are reused in the query.
By analyzing CTE definitions before substitution, we can support defining CTE in subquery.
How was this patch tested?
Jenkins tests.