Skip to content

Comments

[SPARK-28782][SQL] Generator support in aggregate expressions#25512

Closed
WeichenXu123 wants to merge 10 commits intoapache:masterfrom
WeichenXu123:explode_bug
Closed

[SPARK-28782][SQL] Generator support in aggregate expressions#25512
WeichenXu123 wants to merge 10 commits intoapache:masterfrom
WeichenXu123:explode_bug

Conversation

@WeichenXu123
Copy link
Contributor

@WeichenXu123 WeichenXu123 commented Aug 20, 2019 •
edited
Loading

What changes were proposed in this pull request?

Support generator in aggregate expressions.

In this PR, I check the aggregate logical plan, if its aggregateExpressions include generator, then convert this agg plan into "normal agg plan + generator plan + projection plan". I.e:

aggregate(with generator)
 |--child_plan

===>

project
  |--generator(resolved)
         |--aggregate
               |--child_plan

Why are the changes needed?

We should support sql like:

select explode(array(min(a), max(a))) from t

Does this PR introduce any user-facing change?

No

How was this patch tested?

Unit test added.

@SparkQA
Copy link

SparkQA commented Aug 20, 2019

Test build #109408 has finished for PR 25512 at commit 77fc638.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 21, 2019

Test build #109451 has finished for PR 25512 at commit 83129b3.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@WeichenXu123
Copy link
Contributor Author

WeichenXu123 commented Aug 21, 2019

Jenkins retest this please.

@SparkQA
Copy link

SparkQA commented Aug 21, 2019

Test build #109460 has finished for PR 25512 at commit 6ca1568.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 21, 2019

Test build #109453 has finished for PR 25512 at commit 6ca1568.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@WeichenXu123 WeichenXu123 changed the title [WIP][SPARK-28782][SQL] Support explode function on aggregate expressions [SPARK-28782][SQL] Support explode function on aggregate expressions Aug 27, 2019
@WeichenXu123
Copy link
Contributor Author

WeichenXu123 commented Aug 27, 2019

@cloud-fan Ready for review.

@SparkQA
Copy link

SparkQA commented Aug 27, 2019

Test build #109771 has finished for PR 25512 at commit 48565b6.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@WeichenXu123
Copy link
Contributor Author

WeichenXu123 commented Aug 27, 2019

Jenkins retest this please.

@SparkQA
Copy link

SparkQA commented Aug 27, 2019

Test build #109775 has finished for PR 25512 at commit 48565b6.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor

cloud-fan commented Aug 27, 2019

retest this please

@SparkQA
Copy link

SparkQA commented Aug 27, 2019

Test build #109803 has finished for PR 25512 at commit 48565b6.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@gatorsmile
Copy link
Member

gatorsmile commented Aug 27, 2019

@WeichenXu123 Update the PR description?

@WeichenXu123 WeichenXu123 changed the title [SPARK-28782][SQL] Support explode function on aggregate expressions [SPARK-28782][SQL] Generator support in aggregate expressions Aug 28, 2019
@WeichenXu123
Copy link
Contributor Author

WeichenXu123 commented Aug 28, 2019

@gatorsmile Done.

@WeichenXu123
Copy link
Contributor Author

WeichenXu123 commented Aug 28, 2019

Jenkins retest this please

@SparkQA
Copy link

SparkQA commented Aug 28, 2019

Test build #109842 has finished for PR 25512 at commit 48565b6.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 30, 2019

Test build #109935 has finished for PR 25512 at commit 1ad57a1.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 2, 2019

Test build #110017 has finished for PR 25512 at commit 207cd76.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 3, 2019

Test build #110040 has finished for PR 25512 at commit c4c3c60.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 4, 2019

Test build #110114 has finished for PR 25512 at commit a839035.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor

cloud-fan commented Sep 4, 2019

retest this please

@SparkQA
Copy link

SparkQA commented Sep 4, 2019

Test build #110121 has finished for PR 25512 at commit a839035.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan cloud-fan closed this in f8bc91f Sep 5, 2019
@cloud-fan
Copy link
Contributor

cloud-fan commented Sep 5, 2019

thanks, merging to master!

@WeichenXu123 WeichenXu123 deleted the explode_bug branch September 5, 2019 08:55
PavithraRamachandran pushed a commit to PavithraRamachandran/spark that referenced this pull request Sep 15, 2019
@WeichenXu123 @PavithraRamachandran
[SPARK-28782][SQL] Generator support in aggregate expressions
1810433 
### What changes were proposed in this pull request?

Support generator in aggregate expressions.

In this PR, I check the aggregate logical plan, if its aggregateExpressions include generator, then convert this agg plan into "normal agg plan + generator plan + projection plan". I.e:
```
aggregate(with generator)
 |--child_plan
```
===>
```
project
  |--generator(resolved)
         |--aggregate
               |--child_plan
```

### Why are the changes needed?

We should support sql like:
```
select explode(array(min(a), max(a))) from t
```

### Does this PR introduce any user-facing change?
No

### How was this patch tested?

Unit test added.

Closes apache#25512 from WeichenXu123/explode_bug.

Authored-by: WeichenXu <weichen.xu@databricks.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cloud-fan cloud-fan cloud-fan approved these changes

Assignees

No one assigned

Labels

SQL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@WeichenXu123 @SparkQA @cloud-fan @gatorsmile @dongjoon-hyun