[SPARK-30048][SQL] Enable aggregates with interval type values for RelationalGroupedDataset #26681
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…lationalGroupedDataset
Member
Author
|
cc @cloud-fan @maropu @HyukjinKwon @wangyum, thanks for reviewing. |
maropu
reviewed
Nov 26, 2019
| colNames.map { colName => | ||
| val namedExpr = df.resolve(colName) | ||
| if (!namedExpr.dataType.isInstanceOf[NumericType]) { | ||
| if (!TypeCollection.NumericAndInterval.acceptsType(namedExpr.dataType)) { |
Member
There was a problem hiding this comment.
Can you update the comment to make it more general one?
Member
Author
There was a problem hiding this comment.
thanks for your suggestion. Please check 。
cloud-fan
reviewed
Nov 26, 2019
| schema.fields.filter{ f => | ||
| TypeCollection.NumericAndInterval.acceptsType(f.dataType) | ||
| }.map { n => | ||
| queryExecution.analyzed.resolveQuoted(n.name, sparkSession.sessionState.analyzer.resolver).get |
Contributor
There was a problem hiding this comment.
nit: we can do
queryExecution.analyzed.output.filter { attr =>
TypeCollection.NumericAndInterval.acceptsType(attr.dataType)
}
|
Test build #114462 has finished for PR 26681 at commit
|
|
Test build #114469 has finished for PR 26681 at commit
|
|
Test build #114472 has finished for PR 26681 at commit
|
maropu
approved these changes
Nov 26, 2019
|
Test build #114499 has finished for PR 26681 at commit
|
|
Test build #114519 has finished for PR 26681 at commit
|
|
Test build #114731 has finished for PR 26681 at commit
|
Member
Contributor
|
thanks, merging to master! |
attilapiros
pushed a commit
to attilapiros/spark
that referenced
this pull request
Dec 6, 2019
…lationalGroupedDataset ### What changes were proposed in this pull request? Now the min/max/sum/avg are support for intervals, we should also enable it in RelationalGroupedDataset ### Why are the changes needed? API consistency improvement ### Does this PR introduce any user-facing change? yes, Dataset support min/max/sum/avg(mean) on intervals ### How was this patch tested? add ut Closes apache#26681 from yaooqinn/SPARK-30048. Authored-by: Kent Yao <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
cloud-fan
pushed a commit
that referenced
this pull request
Jan 19, 2020
### What changes were proposed in this pull request? As we are not going to follow ANSI to implement year-month and day-time interval types, it is weird to compare the year-month part to the day-time part for our current implementation of interval type now. Additionally, the current ordering logic comes from PostgreSQL where the implementation of the interval is messy. And we are not aiming PostgreSQL compliance at all. THIS PR will revert #26681 and #26337 ### Why are the changes needed? make interval type more future-proofing ### Does this PR introduce any user-facing change? there are new in 3.0, so no ### How was this patch tested? existing uts shall work Closes #27262 from yaooqinn/SPARK-30551. Authored-by: Kent Yao <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Now the min/max/sum/avg are support for intervals, we should also enable it in RelationalGroupedDataset
Why are the changes needed?
API consistency improvement
Does this PR introduce any user-facing change?
yes, Dataset support min/max/sum/avg(mean) on intervals
How was this patch tested?
add ut