Optimize repeat function by creating runlength block#21984
Optimize repeat function by creating runlength block#21984pranjalssh merged 1 commit intoprestodb:masterfrom
Conversation
e69a0c5 to
d1b60d1
Compare
65504c0 to
c2911e4
Compare
There was a problem hiding this comment.
These two functions now succeeds because it will not exceed the memory limit after outputting RunLengthEncodedBlock.
|
Also can we do some similar for SEQUENCE - using literal encoder? |
There was a problem hiding this comment.
why does it need to be < 10K?
There was a problem hiding this comment.
I guess this is intentional to avoid generating too large array when this function was added. Since we never heard complaint about it, I think it's reasonable to keep this limit here.
presto-main/src/main/java/com/facebook/presto/operator/scalar/RepeatFunction.java
Outdated
Show resolved
Hide resolved
c501027
c501027 to
f72450c
Compare
There was a problem hiding this comment.
Add test for non constant function arguments
There was a problem hiding this comment.
Try non-const args here as well
|
Nit: suggest changing the first word of the release note entry from "Optimize" to "Improve" following the Order of Changes in the [Release Note Guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines. |
4f0b6f9 to
48f3c99
Compare
48f3c99 to
0bc19a1
Compare
Description
Currently repeat function implementation is to write the result block multiple times. However this is not efficient, since the result block is a repeat of the same value multiple times, we can create a RunLengthBlock which is more efficient, both in memory and construction cost.
Motivation and Context
To improve performance of repeat function.
Also the result block created by repeat function will have small memory size, and it will be evaluated during query compilation (otherwise it will be skipped as the result block is too big
presto/presto-main/src/main/java/com/facebook/presto/sql/planner/RowExpressionInterpreter.java
Line 776 in 6998a73
Impact
Improve performance for repeat function. repeat(1, 1000) shows 35% performance improvement.
Current:
After optimization
Test Plan
Add unit tests
Contributor checklist
Release Notes
Please follow release notes guidelines and fill in the release notes below.