[SPARK-12719][SQL] SQL generation support for Generate#11768
Closed
yhuai wants to merge 1 commit intoapache:masterfrom
yhuai:generate_1
Closed
[SPARK-12719][SQL] SQL generation support for Generate#11768yhuai wants to merge 1 commit intoapache:masterfrom yhuai:generate_1
yhuai wants to merge 1 commit intoapache:masterfrom
yhuai:generate_1
Conversation
|
Test build #53337 has finished for PR 11768 at commit
|
| // This only happens when we put UDTF in project list and there is no FROM clause. Because we | ||
| // always generate LATERAL VIEW for `Generate`, here we use a trick to put a dummy sub-query | ||
| // after FROM clause, so that we can generate a valid LATERAL VIEW SQL string. | ||
| s"(SELECT 1) ${SQLBuilder.newSubqueryName}" |
Contributor
Author
There was a problem hiding this comment.
Can you put a concrete example at here?
Contributor
Author
|
@cloud-fan I resolved the conflict and the commit at here. You can cherry-pick this commit to your original branch (and keep using that) or you can push update directly to my branch. |
| SQLBuilder.newSubqueryName, | ||
| "AS", | ||
| columnAliases | ||
| ) |
Contributor
Author
There was a problem hiding this comment.
This build call has many arguments. It will be good to put an example at here.
Contributor
Author
|
@cloud-fan Overall it looks good. I have left a few comments. |
|
Test build #53339 has finished for PR 11768 at commit
|
Contributor
Author
|
Closing it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR adds SQL generation support for Generate operator. It always converts Generate operator into LATERAL VIEW format as there are many limitations to put UDTF in project list.
This PR is based on #11658, please see the last commit to review the real changes.
Thanks @dilipbiswal for his initial work! Takes over #11596
How was this patch tested?
new tests in LogicalPlanToSQLSuite