Improve performance of aggregation operator#19425
Draft
fgwang7w wants to merge 8 commits intoprestodb:masterfrom
Draft
Improve performance of aggregation operator#19425fgwang7w wants to merge 8 commits intoprestodb:masterfrom
fgwang7w wants to merge 8 commits intoprestodb:masterfrom
Conversation
Cherry-pick of trinodb/trino@0a70468 co-authored-by: Karol Sobczak <karol.sobczak@karolsobczak.com>
Cherry-pick of trinodb/trino@301ff47 Co-authored-by: skrzypo987<krzysztof.skrzypczynski@starburstdata.com>
If the number of combinations of all dictionaries in a page is below certain number, we can store the results in a small array and reuse found groups Cherry-pick of trinodb/trino@ffd1ee8 Co-authored-by: skrzypo987<krzysztof.skrzypczynski@starburstdata.com>
For simplicity and tiny performance gain. Cherry-pick of trinodb/trino@7ec3bd0 Co-authored-by: skrzypo987 <krzysztof.skrzypczynski@starburstdata.com>
Cherry-pick of trinodb/trino@7ee53ea Co-authored-by: skrzypo987 <krzysztof.skrzypczynski@starburstdata.com>
Cherry-pick of trinodb/trino@27e0c32 Co-authored-by: skrzypo987<krzysztof.skrzypczynski@starburstdata.com>
Previously the hash table capacity was checked every row to see whether a rehash is needed. Now the input page is split into batches and it is assumed that every row in batch will create a new group (which is rarely the case) and rehashing is done in advance before processing. This may slightly increase memory footprint for small number of groups, however there is a tiny performance gain as the capacity is not checked every row. Cherry-pick of trinodb/trino@88cd492 Co-authored-by: skrzypo987<krzysztof.skrzypczynski@starburstdata.com>
|
There's an off-by-one error in the check that can cause a failure when the page is empty Cherry-pick of trinodb/trino@08db4fb Co-authored-by: Karol Sobczak <karol.sobczak@karolsobczak.com>
fgwang7w
force-pushed
the
optimizemultichannelgroupby
branch
from
3cb3d2a to
cfcbaae
Compare
Contributor
|
@tdcmeehan Do you know how we solve the CLA problems? |
Member
Author
|
@tdcmeehan @yingsu00 gentle ping. we still have CLA compliance issues unsolved and need community's support to figure out how to make it passed. thanks |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reduce large long[] memory usage and Improve Group-by performance
For memory optimization:
MultiChannelGroupByHash.e.g we are looking at 64MB of long[] bytes * 15 = 960MB that can be avoided for memory allocation
Cherry-pick of trinodb/trino#9514
Cherry-pick of trinodb/trino#10965
Cherry-pick of trinodb/trino#12336
Cherry-pick of trinodb/trino#12597
Test Result: (sample query from tpcds-q10 with multiple grouping sets)
Before:
Peak User Memory | 11.37MB
Peak Total Memory | 78.63MB
Elapsed Time | 7.68s
After:
Peak User Memory | 5.65MB
Peak Total Memory | 61.71MB
Elapsed Time | 2.08s
Performance test on TPC-H 1TB benchmark: