Rewrite most aggregations as annotated functions#11477
Merged
dain merged 27 commits intotrinodb:masterfrom May 27, 2022
Merged
Conversation
Member
|
cc @radek-starburst @lukasz-stec @skrzypo987 |
electrum
reviewed
Apr 21, 2022
core/trino-main/src/main/java/io/trino/operator/aggregation/minmaxbyn/MaxByNStateFactory.java
Outdated
Show resolved
Hide resolved
561c20f to
e14d83a
Compare
e14d83a to
1a6dbd6
Compare
electrum
reviewed
May 25, 2022
Member
electrum
left a comment
There was a problem hiding this comment.
Commits through "Convert qdigest_agg and merge aggregations to annotated functions" look good
core/trino-main/src/main/java/io/trino/operator/aggregation/ChecksumAggregationFunction.java
Outdated
Show resolved
Hide resolved
electrum
reviewed
May 26, 2022
Member
electrum
left a comment
There was a problem hiding this comment.
Commits through "Convert avg(REAL) aggregation to annotated function" look good
core/trino-spi/src/test/java/io/trino/spi/function/TestScalarFunctionAdapter.java
Outdated
Show resolved
Hide resolved
core/trino-spi/src/test/java/io/trino/spi/function/TestScalarFunctionAdapter.java
Outdated
Show resolved
Hide resolved
core/trino-spi/src/test/java/io/trino/spi/function/TestScalarFunctionAdapter.java
Outdated
Show resolved
Hide resolved
...trino-main/src/main/java/io/trino/operator/aggregation/AggregationFromAnnotationsParser.java
Outdated
Show resolved
Hide resolved
...trino-main/src/main/java/io/trino/operator/aggregation/AggregationFromAnnotationsParser.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/operator/aggregation/state/InOutStateSerializer.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/operator/aggregation/state/StateCompiler.java
Outdated
Show resolved
Hide resolved
electrum
approved these changes
May 26, 2022
...trino-main/src/main/java/io/trino/operator/aggregation/AggregationFromAnnotationsParser.java
Outdated
Show resolved
Hide resolved
...trino-main/src/main/java/io/trino/operator/aggregation/AggregationFromAnnotationsParser.java
Outdated
Show resolved
Hide resolved
...trino-main/src/main/java/io/trino/operator/aggregation/AggregationFromAnnotationsParser.java
Outdated
Show resolved
Hide resolved
...trino-main/src/main/java/io/trino/operator/aggregation/AggregationFromAnnotationsParser.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/operator/annotations/TypeImplementationDependency.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/operator/aggregation/minmaxn/MaxNStateFactory.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/operator/aggregation/minmaxn/MinNStateFactory.java
Outdated
Show resolved
Hide resolved
...no-main/src/main/java/io/trino/operator/aggregation/minmaxbyn/MinByNAggregationFunction.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/operator/aggregation/minmaxbyn/MinByNStateFactory.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/metadata/SqlAggregationFunction.java
Outdated
Show resolved
Hide resolved
Similar to filtered aggregates, ordered aggregates can involved new symbols not in the original function call, and this optimizer does not support that model.
The typed states were added for old min/max_by aggregations, and these constructors are incompatible with the new generic state system.
a363a9c to
e3af9f6
Compare
e3af9f6 to
7c24aa0
Compare
ksobolew
reviewed
May 30, 2022
Comment on lines
+451
to
+459
| Optional<Class<?>> nativeContainerType = Arrays.stream(annotations) | ||
| .filter(SqlType.class::isInstance) | ||
| .map(SqlType.class::cast) | ||
| .findFirst() | ||
| .map(SqlType::nativeContainerType); | ||
| // Note: this cannot be done as a chain due to strange generic type mismatches | ||
| if (nativeContainerType.isPresent() && !nativeContainerType.get().equals(Object.class)) { | ||
| parameterType = nativeContainerType.get(); | ||
| } |
Contributor
There was a problem hiding this comment.
Challenge accepted ;)
Suggested change
| Optional<Class<?>> nativeContainerType = Arrays.stream(annotations) | |
| .filter(SqlType.class::isInstance) | |
| .map(SqlType.class::cast) | |
| .findFirst() | |
| .map(SqlType::nativeContainerType); | |
| // Note: this cannot be done as a chain due to strange generic type mismatches | |
| if (nativeContainerType.isPresent() && !nativeContainerType.get().equals(Object.class)) { | |
| parameterType = nativeContainerType.get(); | |
| } | |
| parameterType = Arrays.stream(annotations) | |
| .filter(SqlType.class::isInstance) | |
| .map(SqlType.class::cast) | |
| .findFirst() | |
| .<Class<?>>map(SqlType::nativeContainerType) | |
| .filter(not(Object.class::equals)) | |
| .orElse(parameterType); |
Member
|
@dain this does not split aggregation stats into smaller states, right? Hence there shouldn't be perf difference? |
Member
|
Here is the report with comparison before and after this change, looks like result are stable without any regressions (maybe even slightly better). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
NOTE: This is based on #11476, so skip the first few commits that overlap
Convert all of our aggregations to annotated functions (except for reduce which uses lambdas). To do this I extended the annotation system with:
Related issues, pull requests, and links
Documentation
(x) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.
Release notes
I'm not sure how much of this is visible to the SPI yet.
( ) No release notes entries required.
( ) Release notes entries required with the following suggested text: