Replace row-wise DefaultPagePartitioner with ColumnarPagePartitioner#11146
Replace row-wise DefaultPagePartitioner with ColumnarPagePartitioner#11146sopel39 merged 5 commits intotrinodb:masterfrom
Conversation
3b8d2fd to
caeb593
Compare
skrzypo987
left a comment
There was a problem hiding this comment.
LGTM % comments
Share some microbenchmark results with different null rates
There was a problem hiding this comment.
I don't think this comment is needed here
There was a problem hiding this comment.
this is copied from the original implementation, also IMO it should stay if it affects performance
There was a problem hiding this comment.
Move this method next to partitionPageByRow
There was a problem hiding this comment.
That is not always true. A block may have a giant dictionary and use only a small subset of values. Please add a note that this is only a heuristic
There was a problem hiding this comment.
we can't have giant dictionary blocks here as the dictionary size is limited by page size by the if below.
There was a problem hiding this comment.
nit: in the future we could reuse dictionaries between pages
There was a problem hiding this comment.
unless we are going to reuse them, its way easier to read if we have the values directly here with comments.
There was a problem hiding this comment.
If dictionary size is almost as big as the block size it will still be slower. Consider adding a coefficient here. Somewhere around 2 probably
There was a problem hiding this comment.
it's not slower. it's a lot faster (to my surprise)
dictionary
Benchmark (channelCount) (enableCompression) (nullRate) (partitionCount) (positionCount) (type) Mode Cnt Score Error Units
BenchmarkPartitionedOutputOperator.addPage 1 false 0.2 16 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_80_PERCENT avgt 10 575.579 ± 17.971 ms/op
BenchmarkPartitionedOutputOperator.addPage 1 false 0.2 16 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_100_PERCENT_MINUS_1 avgt 10 569.529 ± 29.685 ms/op
generic
Benchmark (channelCount) (enableCompression) (nullRate) (partitionCount) (positionCount) (type) Mode Cnt Score Error Units
BenchmarkPartitionedOutputOperator.addPage 1 false 0.2 16 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_80_PERCENT avgt 10 688.153 ± 24.111 ms/op
BenchmarkPartitionedOutputOperator.addPage 1 false 0.2 16 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_100_PERCENT_MINUS_1 avgt 10 690.021 ± 5.662 ms/op
There was a problem hiding this comment.
see createAppender above
...o-main/src/main/java/io/trino/operator/output/columnar/IsolatedPositionsAppenderFactory.java
Outdated
Show resolved
Hide resolved
caeb593 to
5bb111d
Compare
lukasz-stec
left a comment
There was a problem hiding this comment.
comments answered or addressed
There was a problem hiding this comment.
see createAppender above
There was a problem hiding this comment.
this is copied from the original implementation, also IMO it should stay if it affects performance
There was a problem hiding this comment.
unless we are going to reuse them, its way easier to read if we have the values directly here with comments.
There was a problem hiding this comment.
it's not slower. it's a lot faster (to my surprise)
dictionary
Benchmark (channelCount) (enableCompression) (nullRate) (partitionCount) (positionCount) (type) Mode Cnt Score Error Units
BenchmarkPartitionedOutputOperator.addPage 1 false 0.2 16 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_80_PERCENT avgt 10 575.579 ± 17.971 ms/op
BenchmarkPartitionedOutputOperator.addPage 1 false 0.2 16 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_100_PERCENT_MINUS_1 avgt 10 569.529 ± 29.685 ms/op
generic
Benchmark (channelCount) (enableCompression) (nullRate) (partitionCount) (positionCount) (type) Mode Cnt Score Error Units
BenchmarkPartitionedOutputOperator.addPage 1 false 0.2 16 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_80_PERCENT avgt 10 688.153 ± 24.111 ms/op
BenchmarkPartitionedOutputOperator.addPage 1 false 0.2 16 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_100_PERCENT_MINUS_1 avgt 10 690.021 ± 5.662 ms/op
There was a problem hiding this comment.
we can't have giant dictionary blocks here as the dictionary size is limited by page size by the if below.
|
We don't need the |
There was a problem hiding this comment.
Revert 6058b29, then we don't need this factory method
There was a problem hiding this comment.
I think this is wrong. You can check for FixedWithType and use the size to know if it's 1/2/4/8/16. This will remove many cases below, and handle IpAddressType that is missed. I think all of the SlicePositionsAppender checks can be replaced with VariableWidthType.
There was a problem hiding this comment.
I used FixedWithType and VariableWidthType. It's a lot cleaner. I also added Int96PositionsAppender
There was a problem hiding this comment.
No need for this interface, since there is only one implementation.
There was a problem hiding this comment.
removed PositionsAppenderFactory interface and renamed IsolatedPositionsAppenderFactory to PositionsAppenderFactory
There was a problem hiding this comment.
Initialize in the field declaration
There was a problem hiding this comment.
moved to the LocalExecutionPlanner due to 6058b29 revert
This reverts commit 6058b29
Static pollution can fail if a code path is chosen based on the benchmark parameters since it only pollutes one code path and not necessarily the one under test
5bb111d to
c93f998
Compare
lukasz-stec
left a comment
There was a problem hiding this comment.
I reverted 6058b29 and rebased on top of it.
There was a problem hiding this comment.
moved to the LocalExecutionPlanner due to 6058b29 revert
There was a problem hiding this comment.
I used FixedWithType and VariableWidthType. It's a lot cleaner. I also added Int96PositionsAppender
There was a problem hiding this comment.
removed PositionsAppenderFactory interface and renamed IsolatedPositionsAppenderFactory to PositionsAppenderFactory
There was a problem hiding this comment.
Can we keep it in io.trino.operator.output and call this class just PagePartitioner ?
There was a problem hiding this comment.
Yeah, makes sense.
It also then makes sense to move PositionsAppender class + factory
core/trino-main/src/main/java/io/trino/operator/output/PartitionedOutputOperator.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/operator/output/PartitionedOutputOperator.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
nit: in the future we could reuse dictionaries between pages
There was a problem hiding this comment.
remaind me again why we couldn't just isolate this guy (type.appendTo intuitively should be inlined since it's called from single isolated callsite). Please add comment in code.
There was a problem hiding this comment.
actually, we should isolate TypedPositionsAppender. It may or may not help, depending on jit decisions but it won't hurt. I'll fix that.
Why we can't use TypedPositionsAppender for everything? Because Type.appendTo method uses multiple Block types (Dict, RLe, normal) so the calls to Block.IsNull or Block.getLong are mega-morphic and thus Type method body uses virtual calls.
If the Type.appendTo body is inlined in the (isolated) PositionsAppender. appendTo loop, the the call site is not mega-morphic anymore and this works well, but this depends on the inlining decision that is influenced by other factors like compiled method size, byte code size, order of compilation (including the pollution state at the compilation time) etc. and in my experiments it was not happening, even for simple types.
There was a problem hiding this comment.
I double-checked the LongPositionsAppender vs isolated TypedPositionsAppender.
TypedPositionsAppender
Benchmark (channelCount) (enableCompression) (nullRate) (partitionCount) (positionCount) (type) Mode Cnt Score Error Units
BenchmarkPartitionedOutputOperator.addPage 1 false 0.2 16 8192 BIGINT avgt 10 1349.558 ± 16.764 ms/op
vs
LongPositionsAppender
Benchmark (channelCount) (enableCompression) (nullRate) (partitionCount) (positionCount) (type) Mode Cnt Score Error Units
BenchmarkPartitionedOutputOperator.addPage 1 false 0.2 16 8192 BIGINT avgt 10 1022.218 ± 17.139 ms/op
The reason for the perf difference is that although most of the AbstractLongType.appendTo is inlined into TypedPositionsAppender, there is still a virtual call to Block.isNull (see the jitwatch call chain screens below). The reason for the inline failure is <inline_fail reason="virtual call"/>
<parse method="1259" stamp="30.717" uses="26103.000000">
<observe total="4" count="4" trap="class_check"/>
<observe total="4" count="4" trap="bimorphic_or_optimized_type_check"/>
<bc code="185" bci="2"/>
<klass name="io.trino.spi.block.LongArrayBlock" flags="1" id="1266"/>
<klass name="io.trino.spi.block.DictionaryBlock" flags="1" id="1267"/>
<type name="boolean" id="1101"/>
<method bytes="0" name="isNull" flags="1025" holder="1246" arguments="1107" id="1260" iicount="1" return="1101"/>
<call receiver_count="22776" virtual="1" receiver2="1267" receiver2_count="8192" method="1260" inline="1" receiver="1266" count="39160" prof_factor="0.662378"/>
<inline_fail reason="virtual call"/>
The TypedPositionsAppender isolation is forced with code below, that is used per input Block class but still, there are DictionaryBlock calls there even though dictionary is not part of the benchmarked type (but used to pollute the profile)
private PositionsAppender createAppender(Type type)
{
Class<? extends PositionsAppender> appenderClass = findDedicatedAppenderClassFor(type);
if (***appenderClass == null || true***) {
return isolateTypeAppender(type);
}
return isolateAppender(appenderClass);
}
TypedPositionsAppender call chain

LongPositionsAppender call chain - mostly inlined, except for growCapacity

c93f998 to
39c2cf5
Compare
lukasz-stec
left a comment
There was a problem hiding this comment.
most comments addressed.
I'm waiting to rename ColumnarPagePartitioner to PagePartitioner and move the classes to io.trino.operator.output for most of the comments to be resolved to not lose conversation handles in github.
core/trino-main/src/main/java/io/trino/operator/output/PartitionedOutputOperator.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
actually, we should isolate TypedPositionsAppender. It may or may not help, depending on jit decisions but it won't hurt. I'll fix that.
Why we can't use TypedPositionsAppender for everything? Because Type.appendTo method uses multiple Block types (Dict, RLe, normal) so the calls to Block.IsNull or Block.getLong are mega-morphic and thus Type method body uses virtual calls.
If the Type.appendTo body is inlined in the (isolated) PositionsAppender. appendTo loop, the the call site is not mega-morphic anymore and this works well, but this depends on the inlining decision that is influenced by other factors like compiled method size, byte code size, order of compilation (including the pollution state at the compilation time) etc. and in my experiments it was not happening, even for simple types.
core/trino-main/src/main/java/io/trino/operator/output/PartitionedOutputOperator.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
I double-checked the LongPositionsAppender vs isolated TypedPositionsAppender.
TypedPositionsAppender
Benchmark (channelCount) (enableCompression) (nullRate) (partitionCount) (positionCount) (type) Mode Cnt Score Error Units
BenchmarkPartitionedOutputOperator.addPage 1 false 0.2 16 8192 BIGINT avgt 10 1349.558 ± 16.764 ms/op
vs
LongPositionsAppender
Benchmark (channelCount) (enableCompression) (nullRate) (partitionCount) (positionCount) (type) Mode Cnt Score Error Units
BenchmarkPartitionedOutputOperator.addPage 1 false 0.2 16 8192 BIGINT avgt 10 1022.218 ± 17.139 ms/op
The reason for the perf difference is that although most of the AbstractLongType.appendTo is inlined into TypedPositionsAppender, there is still a virtual call to Block.isNull (see the jitwatch call chain screens below). The reason for the inline failure is <inline_fail reason="virtual call"/>
<parse method="1259" stamp="30.717" uses="26103.000000">
<observe total="4" count="4" trap="class_check"/>
<observe total="4" count="4" trap="bimorphic_or_optimized_type_check"/>
<bc code="185" bci="2"/>
<klass name="io.trino.spi.block.LongArrayBlock" flags="1" id="1266"/>
<klass name="io.trino.spi.block.DictionaryBlock" flags="1" id="1267"/>
<type name="boolean" id="1101"/>
<method bytes="0" name="isNull" flags="1025" holder="1246" arguments="1107" id="1260" iicount="1" return="1101"/>
<call receiver_count="22776" virtual="1" receiver2="1267" receiver2_count="8192" method="1260" inline="1" receiver="1266" count="39160" prof_factor="0.662378"/>
<inline_fail reason="virtual call"/>
The TypedPositionsAppender isolation is forced with code below, that is used per input Block class but still, there are DictionaryBlock calls there even though dictionary is not part of the benchmarked type (but used to pollute the profile)
private PositionsAppender createAppender(Type type)
{
Class<? extends PositionsAppender> appenderClass = findDedicatedAppenderClassFor(type);
if (***appenderClass == null || true***) {
return isolateTypeAppender(type);
}
return isolateAppender(appenderClass);
}
TypedPositionsAppender call chain

LongPositionsAppender call chain - mostly inlined, except for growCapacity

There was a problem hiding this comment.
Let's add some comment here why isolateTypeAppender(type); is not sufficient for primitive types.
There was a problem hiding this comment.
Extended class comment. PTAL
There was a problem hiding this comment.
That should be Class<? extends Block>. You should extract block types from all children using io.trino.spi.block.Block#getChildren
There was a problem hiding this comment.
Do you mean List< Class<? extends Block>>? This version does not support complex types like RowType or MapType so this won't help.
For RowType support, this would probably be some kind of tree unless we don't care about the lower level nesting.
There was a problem hiding this comment.
what about handling mayHaveNull here and in other places?
There was a problem hiding this comment.
added separate commit (could be squshed if we decide to keep it). Seems to help relatively a lot in tpcds/tpch
cpoo_sf1000_orc_part_mayHaveNull.pdf
There was a problem hiding this comment.
remove. I dont think it matters. Branch prediction would work the same
There was a problem hiding this comment.
removed (dedicated PositionsAppenders are anyway in cases where this could have impact)
core/trino-main/src/main/java/io/trino/operator/output/PositionsAppender.java
Outdated
Show resolved
Hide resolved
|
mind automation. Either it's (new) flakynes or something is broken ( |
There was a problem hiding this comment.
return Optional.ofNullable(findDedicatedAppenderClassFor(type))
.map(this::isolateAppender)
.orElseGet(() -> isolateTypeAppender(type));Column-wise approach works in two stages. First it divides the input positions to output partitions. Then, for each partition and column, all values in the give column for the selected positions for the given partition are copied to the output buffer. Benchmark channelCount enableCompression nullRate partitionCount positionCount type baseline columnar columnar % BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 2 8192 BIGINT 1302.926 769.102 -40.97116797 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 2 8192 BIGINT_SKEWED_HASH 1936.168 1118.931 -42.20899219 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 2 8192 DICTIONARY_BIGINT 1206.206 650.893 -46.03799019 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 2 8192 RLE_BIGINT 952.319 479.22 -49.67862659 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 2 8192 BIGINT_PARTITION_CHANNEL_20_PERCENT 888.163 560.123 -36.93466177 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 2 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_20_PERCENT 1114.431 498.424 -55.27547242 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 2 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_50_PERCENT 1128.77 546.881 -51.55071449 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 2 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_80_PERCENT 1328.122 766.677 -42.27360137 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 2 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_100_PERCENT 1406.781 947 -32.6831966 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 2 8192 RLE_PARTITION_BIGINT 1780.045 572.681 -67.82772346 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 2 8192 RLE_PARTITION_NULL_BIGINT 6.281 2.675 -57.41124025 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 2 8192 LONG_DECIMAL 2524.384 1786.068 -29.24737282 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 2 8192 INTEGER 1357.94 770.557 -43.25544575 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 2 8192 SMALLINT 1376.242 776.705 -43.56334133 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 2 8192 BOOLEAN 1273.884 705.885 -44.58796876 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 2 8192 VARCHAR 3407.343 3008.769 -11.69750154 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 2 8192 ARRAY_BIGINT 599.536 472.668 -21.1610312 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 2 8192 ARRAY_VARCHAR 1637.497 1534.025 -6.31891234 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 2 8192 ARRAY_ARRAY_BIGINT 1958.92 1740.14 -11.16839891 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 2 8192 MAP_BIGINT_BIGINT 2623.346 1863.795 -28.95351967 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 2 8192 MAP_BIGINT_MAP_BIGINT_BIGINT 5665.363 5628.05 -0.6586162264 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 2 8192 ROW_BIGINT_BIGINT 710.468 503.575 -29.12066413 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 2 8192 ROW_ARRAY_BIGINT_ARRAY_BIGINT 1617.816 1462.798 -9.581930207 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 16 8192 BIGINT 1444.293 831.347 -42.43917266 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 16 8192 BIGINT_SKEWED_HASH 1478.507 1129.366 -23.61442996 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 16 8192 DICTIONARY_BIGINT 1058.473 598.475 -43.45864278 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 16 8192 RLE_BIGINT 987.504 529.149 -46.41550819 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 16 8192 BIGINT_PARTITION_CHANNEL_20_PERCENT 785.252 583.389 -25.70677948 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 16 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_20_PERCENT 1092.184 523.947 -52.02758876 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 16 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_50_PERCENT 1111.695 566.813 -49.01362334 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 16 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_80_PERCENT 1342.185 845.712 -36.98990825 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 16 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_100_PERCENT 1439.271 929.554 -35.4149427 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 16 8192 RLE_PARTITION_BIGINT 1752.947 586.828 -66.52334611 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 16 8192 RLE_PARTITION_NULL_BIGINT 6.81 2.701 -60.33773862 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 16 8192 LONG_DECIMAL 1829.298 1233.125 -32.5902614 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 16 8192 INTEGER 1443.725 741.839 -48.61632236 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 16 8192 SMALLINT 1295.63 872.862 -32.6303034 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 16 8192 BOOLEAN 1187.705 788.527 -33.60918747 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 16 8192 VARCHAR 3491.593 2954.383 -15.38581387 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 16 8192 ARRAY_BIGINT 752.269 458.261 -39.08282808 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 16 8192 ARRAY_VARCHAR 1970.084 1583.343 -19.6306858 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 16 8192 ARRAY_ARRAY_BIGINT 2062.881 1718.861 -16.67667694 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 16 8192 MAP_BIGINT_BIGINT 2325.006 1616.14 -30.48878153 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 16 8192 MAP_BIGINT_MAP_BIGINT_BIGINT 7450.894 6816.993 -8.507717329 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 16 8192 ROW_BIGINT_BIGINT 710.563 641.962 -9.6544571 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 16 8192 ROW_ARRAY_BIGINT_ARRAY_BIGINT 1666.879 1349.303 -19.05213276 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 256 8192 BIGINT 2127.969 1162.371 -45.37650689 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 256 8192 BIGINT_SKEWED_HASH 1744.494 1225.401 -29.75607827 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 256 8192 DICTIONARY_BIGINT 1165.121 795.747 -31.70263003 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 256 8192 RLE_BIGINT 1003.777 542.223 -45.98172702 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 256 8192 BIGINT_PARTITION_CHANNEL_20_PERCENT 911.041 620.684 -31.87090372 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 256 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_20_PERCENT 1067.159 509.559 -52.25088295 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 256 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_50_PERCENT 1128.943 586.323 -48.06442841 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 256 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_80_PERCENT 1339.027 841.835 -37.13084202 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 256 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_100_PERCENT 1104.041 1006.807 -8.807100461 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 256 8192 RLE_PARTITION_BIGINT 1731.372 609.43 -64.80074762 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 256 8192 RLE_PARTITION_NULL_BIGINT 7.116 3.624 -49.07251265 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 256 8192 LONG_DECIMAL 2633.272 1375.594 -47.76103646 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 256 8192 INTEGER 2009.192 1045.348 -47.97172197 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 256 8192 SMALLINT 1933.012 959.111 -50.38256358 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 256 8192 BOOLEAN 1207.247 753.839 -37.55718589 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 256 8192 VARCHAR 5700.359 3255.2 -42.89482469 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 256 8192 ARRAY_BIGINT 938.458 766.523 -18.3210117 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 256 8192 ARRAY_VARCHAR 2368.788 1904.359 -19.60618679 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 256 8192 ARRAY_ARRAY_BIGINT 2781.737 2155.849 -22.49989844 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 256 8192 MAP_BIGINT_BIGINT 2349.572 1943.247 -17.29357517 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 256 8192 MAP_BIGINT_MAP_BIGINT_BIGINT 7496.46 6325.078 -15.62580205 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 256 8192 ROW_BIGINT_BIGINT 1160.061 836.216 -27.91620441 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0 256 8192 ROW_ARRAY_BIGINT_ARRAY_BIGINT 2697.067 1822.442 -32.42874574 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 2 8192 BIGINT 1646.478 1270.516 -22.83431664 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 2 8192 BIGINT_SKEWED_HASH 1919.47 1646.059 -14.24408821 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 2 8192 DICTIONARY_BIGINT 1226.175 919.27 -25.02946154 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 2 8192 RLE_BIGINT 993.583 546.51 -44.99603959 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 2 8192 BIGINT_PARTITION_CHANNEL_20_PERCENT 942.063 765.762 -18.7143535 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 2 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_20_PERCENT 1079.083 692.897 -35.78834992 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 2 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_50_PERCENT 1095.834 698.804 -36.23085248 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 2 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_80_PERCENT 1326.996 981.428 -26.04137465 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 2 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_100_PERCENT 1387.114 1199.907 -13.496151 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 2 8192 RLE_PARTITION_BIGINT 1875.813 916.654 -51.13297541 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 2 8192 RLE_PARTITION_NULL_BIGINT 6.857 3.915 -42.90506052 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 2 8192 LONG_DECIMAL 2050.416 1582.807 -22.80556726 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 2 8192 INTEGER 1530.588 1162.388 -24.05611438 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 2 8192 SMALLINT 1516.703 1013.684 -33.1652934 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 2 8192 BOOLEAN 1631.464 1240.743 -23.94910338 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 2 8192 VARCHAR 3604.938 3643.791 1.077771657 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 2 8192 ARRAY_BIGINT 808.011 725.302 -10.23612302 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 2 8192 ARRAY_VARCHAR 1650.827 1505.909 -8.778509196 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 2 8192 ARRAY_ARRAY_BIGINT 2142.252 2048.516 -4.375582331 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 2 8192 MAP_BIGINT_BIGINT 1785.228 1689.082 -5.385642618 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 2 8192 MAP_BIGINT_MAP_BIGINT_BIGINT 4851.934 4461.389 -8.049264479 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 2 8192 ROW_BIGINT_BIGINT 837.225 661.3 -21.0128699 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 2 8192 ROW_ARRAY_BIGINT_ARRAY_BIGINT 1852.099 1772.665 -4.288863608 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 16 8192 BIGINT 1515.336 1041.617 -31.26164758 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 16 8192 BIGINT_SKEWED_HASH 1907.812 1727.84 -9.433424258 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 16 8192 DICTIONARY_BIGINT 1043.224 656.817 -37.03969617 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 16 8192 RLE_BIGINT 1012.265 597.133 -41.01020978 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 16 8192 BIGINT_PARTITION_CHANNEL_20_PERCENT 948.128 790.254 -16.65112727 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 16 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_20_PERCENT 1098.098 687.351 -37.40531355 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 16 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_50_PERCENT 1093.862 746.08 -31.79395573 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 16 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_80_PERCENT 1336.2 1016.64 -23.9155815 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 16 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_100_PERCENT 1377.588 1192.294 -13.45061078 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 16 8192 RLE_PARTITION_BIGINT 1861.171 916.732 -50.74434321 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 16 8192 RLE_PARTITION_NULL_BIGINT 6.917 4.044 -41.53534769 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 16 8192 LONG_DECIMAL 1806.737 1178.898 -34.74988335 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 16 8192 INTEGER 1492.28 1012.088 -32.17841156 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 16 8192 SMALLINT 1441.843 888.18 -38.3996732 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 16 8192 BOOLEAN 1382.287 831.613 -39.83789184 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 16 8192 VARCHAR 3161.59 2949.017 -6.723610588 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 16 8192 ARRAY_BIGINT 709.124 577.973 -18.49479076 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 16 8192 ARRAY_VARCHAR 1519.582 1438.652 -5.325806702 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 16 8192 ARRAY_ARRAY_BIGINT 2131.942 2087.158 -2.100619998 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 16 8192 MAP_BIGINT_BIGINT 1551.028 1293.686 -16.59170563 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 16 8192 MAP_BIGINT_MAP_BIGINT_BIGINT 4609.262 4658.954 1.078090158 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 16 8192 ROW_BIGINT_BIGINT 783.013 599.198 -23.4753446 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 16 8192 ROW_ARRAY_BIGINT_ARRAY_BIGINT 1696.338 1674.645 -1.278813538 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 256 8192 BIGINT 2017.965 1251.385 -37.98777481 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 256 8192 BIGINT_SKEWED_HASH 1970.866 1815.443 -7.886025737 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 256 8192 DICTIONARY_BIGINT 1328.739 851.467 -35.91916847 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 256 8192 RLE_BIGINT 997.416 626.612 -37.17646398 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 256 8192 BIGINT_PARTITION_CHANNEL_20_PERCENT 936.802 773.944 -17.38446331 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 256 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_20_PERCENT 1100.638 724.973 -34.13156733 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 256 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_50_PERCENT 1097.292 770.055 -29.8222351 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 256 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_80_PERCENT 1350.826 1051.77 -22.13875066 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 256 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_100_PERCENT 1390.028 1218.119 -12.36730483 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 256 8192 RLE_PARTITION_BIGINT 1901.381 980.588 -48.42759026 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 256 8192 RLE_PARTITION_NULL_BIGINT 7.888 5.282 -33.03752535 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 256 8192 LONG_DECIMAL 2578.739 1464.405 -43.21236077 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 256 8192 INTEGER 2020.733 1298.996 -35.71659393 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 256 8192 SMALLINT 1917.248 1129.529 -41.08592107 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 256 8192 BOOLEAN 1376.727 976.403 -29.07795082 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 256 8192 VARCHAR 5123.119 3301.373 -35.55931455 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 256 8192 ARRAY_BIGINT 937.792 835.528 -10.90476353 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 256 8192 ARRAY_VARCHAR 2061.161 1894.317 -8.094661213 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 256 8192 ARRAY_ARRAY_BIGINT 2522.042 2269.116 -10.02861967 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 256 8192 MAP_BIGINT_BIGINT 2142.333 1694.135 -20.92102395 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 256 8192 MAP_BIGINT_MAP_BIGINT_BIGINT 5453.217 5341.727 -2.044481267 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 256 8192 ROW_BIGINT_BIGINT 1102.601 808.415 -26.68109316 BenchmarkPartitionedOutputOperator.addPage 1 TRUE 0.2 256 8192 ROW_ARRAY_BIGINT_ARRAY_BIGINT 2345.838 1876.885 -19.99085188 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 2 8192 BIGINT 1328.014 782.884 -41.04851304 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 2 8192 BIGINT_SKEWED_HASH 1341.07 951.721 -29.03271268 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 2 8192 DICTIONARY_BIGINT 998.892 565.345 -43.40279029 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 2 8192 RLE_BIGINT 989.367 549.45 -44.46449093 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 2 8192 BIGINT_PARTITION_CHANNEL_20_PERCENT 818.097 566.987 -30.69440421 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 2 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_20_PERCENT 1021.362 458.36 -55.12266953 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 2 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_50_PERCENT 1021.036 521.528 -48.92168347 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 2 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_80_PERCENT 1032.333 569.624 -44.8216806 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 2 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_100_PERCENT 1016.136 637.31 -37.28103325 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 2 8192 RLE_PARTITION_BIGINT 1677.801 580.697 -65.38939958 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 2 8192 RLE_PARTITION_NULL_BIGINT 6.148 2.601 -57.69355888 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 2 8192 LONG_DECIMAL 1609.268 1069.346 -33.55078209 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 2 8192 INTEGER 1326.069 879.578 -33.67026904 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 2 8192 SMALLINT 1302.886 849.126 -34.82729878 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 2 8192 BOOLEAN 1300.278 801.554 -38.35518251 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 2 8192 VARCHAR 2818.775 2802.964 -0.5609174198 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 2 8192 ARRAY_BIGINT 556.795 493.318 -11.40042565 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 2 8192 ARRAY_VARCHAR 1463.227 1289.521 -11.87143212 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 2 8192 ARRAY_ARRAY_BIGINT 1934.333 1855.516 -4.074634512 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 2 8192 MAP_BIGINT_BIGINT 1459.303 1071.631 -26.56555904 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 2 8192 MAP_BIGINT_MAP_BIGINT_BIGINT 4571.217 4710.173 3.039803186 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 2 8192 ROW_BIGINT_BIGINT 639.188 507.635 -20.58126873 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 2 8192 ROW_ARRAY_BIGINT_ARRAY_BIGINT 1479.139 1363.726 -7.802714958 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 16 8192 BIGINT 1436.78 898.92 -37.43509793 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 16 8192 BIGINT_SKEWED_HASH 1334.748 1014.379 -24.00220866 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 16 8192 DICTIONARY_BIGINT 1065.134 622.869 -41.52200568 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 16 8192 RLE_BIGINT 1010.447 562.927 -44.28930958 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 16 8192 BIGINT_PARTITION_CHANNEL_20_PERCENT 799.241 627.254 -21.51879095 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 16 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_20_PERCENT 1039.015 458.651 -55.85713392 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 16 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_50_PERCENT 1042.339 551.893 -47.05244647 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 16 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_80_PERCENT 1030.947 606.992 -41.12287053 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 16 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_100_PERCENT 1028.115 667.305 -35.09432311 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 16 8192 RLE_PARTITION_BIGINT 1634.18 591.4 -63.81059614 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 16 8192 RLE_PARTITION_NULL_BIGINT 6.011 2.579 -57.09532524 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 16 8192 LONG_DECIMAL 1751.223 1306.027 -25.42200508 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 16 8192 INTEGER 1428.153 938.248 -34.30339747 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 16 8192 SMALLINT 1425.804 932.788 -34.5781047 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 16 8192 BOOLEAN 1276.056 897.798 -29.64274295 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 16 8192 VARCHAR 3239.605 3084.066 -4.801171748 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 16 8192 ARRAY_BIGINT 587.28 525.919 -10.4483381 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 16 8192 ARRAY_VARCHAR 1594.76 1464.104 -8.192831523 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 16 8192 ARRAY_ARRAY_BIGINT 2004.263 1892.049 -5.59876623 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 16 8192 MAP_BIGINT_BIGINT 1596.031 1391.183 -12.83483842 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 16 8192 MAP_BIGINT_MAP_BIGINT_BIGINT 5044.242 4569.822 -9.405179212 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 16 8192 ROW_BIGINT_BIGINT 732.507 537.054 -26.68274842 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 16 8192 ROW_ARRAY_BIGINT_ARRAY_BIGINT 1519.684 1454.59 -4.283390494 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 256 8192 BIGINT 2142.286 1153.369 -46.16176365 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 256 8192 BIGINT_SKEWED_HASH 1356.356 1017.215 -25.0038338 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 256 8192 DICTIONARY_BIGINT 1424.596 850.085 -40.32799474 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 256 8192 RLE_BIGINT 996.244 631.306 -36.63138749 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 256 8192 BIGINT_PARTITION_CHANNEL_20_PERCENT 820.281 635.856 -22.48314907 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 256 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_20_PERCENT 1052.514 516.466 -50.93024891 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 256 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_50_PERCENT 1105.52 580.888 -47.45567697 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 256 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_80_PERCENT 1041.5 621.096 -40.36524244 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 256 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_100_PERCENT 1037.605 704.192 -32.13294076 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 256 8192 RLE_PARTITION_BIGINT 1711.72 622.791 -63.61607039 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 256 8192 RLE_PARTITION_NULL_BIGINT 6.876 3.74 -45.60791158 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 256 8192 LONG_DECIMAL 2703.82 1629.277 -39.7416618 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 256 8192 INTEGER 2097.015 1232.037 -41.24805974 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 256 8192 SMALLINT 1970.243 1202.984 -38.94235381 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 256 8192 BOOLEAN 1300.563 955.816 -26.5075202 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 256 8192 VARCHAR 5433.172 3729.668 -31.35376535 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 256 8192 ARRAY_BIGINT 962.135 780.835 -18.84350949 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 256 8192 ARRAY_VARCHAR 2287.687 2069.558 -9.534914523 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 256 8192 ARRAY_ARRAY_BIGINT 2740.952 2093.304 -23.62857868 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 256 8192 MAP_BIGINT_BIGINT 2376.493 1755.927 -26.11267948 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 256 8192 MAP_BIGINT_MAP_BIGINT_BIGINT 7145.292 6860.392 -3.987240829 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 256 8192 ROW_BIGINT_BIGINT 1148.814 737.52 -35.80161802 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0 256 8192 ROW_ARRAY_BIGINT_ARRAY_BIGINT 2438.02 1730.604 -29.01600479 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 2 8192 BIGINT 1542.505 1141.967 -25.96672296 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 2 8192 BIGINT_SKEWED_HASH 1528.385 1200.545 -21.45009274 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 2 8192 DICTIONARY_BIGINT 1050.792 758.367 -27.82900898 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 2 8192 RLE_BIGINT 985.578 547.333 -44.46578556 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 2 8192 BIGINT_PARTITION_CHANNEL_20_PERCENT 877.957 675.363 -23.0756176 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 2 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_20_PERCENT 1074.513 589.388 -45.14836023 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 2 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_50_PERCENT 1044.045 631.848 -39.48076951 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 2 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_80_PERCENT 1040.564 627.667 -39.68011578 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 2 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_100_PERCENT 1029.127 779.784 -24.22859375 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 2 8192 RLE_PARTITION_BIGINT 1850.428 867.953 -53.09447328 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 2 8192 RLE_PARTITION_NULL_BIGINT 6.78 3.874 -42.86135693 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 2 8192 LONG_DECIMAL 1681.209 1185.421 -29.48996823 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 2 8192 INTEGER 1486.483 1035.678 -30.32695295 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 2 8192 SMALLINT 1393.511 953.533 -31.57334244 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 2 8192 BOOLEAN 1445.677 991.316 -31.42894298 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 2 8192 VARCHAR 2870.282 2633.568 -8.247064226 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 2 8192 ARRAY_BIGINT 703.224 558.767 -20.54210323 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 2 8192 ARRAY_VARCHAR 1255.166 1174.679 -6.412458591 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 2 8192 ARRAY_ARRAY_BIGINT 2006.519 1945.431 -3.044476529 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 2 8192 MAP_BIGINT_BIGINT 1439.195 1309.613 -9.003783365 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 2 8192 MAP_BIGINT_MAP_BIGINT_BIGINT 3808.236 3492.41 -8.293236028 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 2 8192 ROW_BIGINT_BIGINT 741.837 564.075 -23.96240684 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 2 8192 ROW_ARRAY_BIGINT_ARRAY_BIGINT 1549.318 1512.847 -2.354003503 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 16 8192 BIGINT 1466.336 979.943 -33.17063756 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 16 8192 BIGINT_SKEWED_HASH 1574.268 1373.047 -12.78187704 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 16 8192 DICTIONARY_BIGINT 1023.141 655.652 -35.91772786 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 16 8192 RLE_BIGINT 980.678 568.113 -42.06936426 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 16 8192 BIGINT_PARTITION_CHANNEL_20_PERCENT 882.148 728.543 -17.41261104 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 16 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_20_PERCENT 1034.492 650.021 -37.16519799 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 16 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_50_PERCENT 1027.63 622.823 -39.392291 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 16 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_80_PERCENT 1019.902 658.343 -35.4503668 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 16 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_100_PERCENT 1039.249 796.849 -23.32453531 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 16 8192 RLE_PARTITION_BIGINT 1838.673 900.44 -51.02772489 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 16 8192 RLE_PARTITION_NULL_BIGINT 6.892 3.91 -43.26755659 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 16 8192 LONG_DECIMAL 1768.704 1141.687 -35.45064635 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 16 8192 INTEGER 1478.627 889.719 -39.82802965 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 16 8192 SMALLINT 1431.79 858.757 -40.02214012 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 16 8192 BOOLEAN 1347.694 940.269 -30.23126912 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 16 8192 VARCHAR 2971.862 2761.666 -7.072872159 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 16 8192 ARRAY_BIGINT 725.752 694.491 -4.307394261 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 16 8192 ARRAY_VARCHAR 1340 1199.433 -10.49007463 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 16 8192 ARRAY_ARRAY_BIGINT 1997.005 1884.82 -5.61766245 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 16 8192 MAP_BIGINT_BIGINT 1496.776 1322.185 -11.66447084 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 16 8192 MAP_BIGINT_MAP_BIGINT_BIGINT 4043.786 3725.813 -7.863249935 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 16 8192 ROW_BIGINT_BIGINT 794.52 576.589 -27.42926547 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 16 8192 ROW_ARRAY_BIGINT_ARRAY_BIGINT 1608.529 1471.911 -8.493350135 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 256 8192 BIGINT 1983.131 1426.8 -28.05316442 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 256 8192 BIGINT_SKEWED_HASH 1688.429 1351.925 -19.93000594 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 256 8192 DICTIONARY_BIGINT 1344.609 848.275 -36.91288694 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 256 8192 RLE_BIGINT 989.584 617.285 -37.62176834 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 256 8192 BIGINT_PARTITION_CHANNEL_20_PERCENT 873.076 756.496 -13.35278945 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 256 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_20_PERCENT 1047.469 679.288 -35.14958438 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 256 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_50_PERCENT 1056.427 675.651 -36.04375882 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 256 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_80_PERCENT 1041.838 681.072 -34.62784041 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 256 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_100_PERCENT 1033.718 816.641 -20.99963433 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 256 8192 RLE_PARTITION_BIGINT 1850.437 933.937 -49.52884103 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 256 8192 RLE_PARTITION_NULL_BIGINT 7.586 5.209 -31.33403638 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 256 8192 LONG_DECIMAL 2475.471 1443.022 -41.70717411 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 256 8192 INTEGER 2011.162 1267.454 -36.97902009 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 256 8192 SMALLINT 1908.761 1130.535 -40.77126471 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 256 8192 BOOLEAN 1361.54 870.647 -36.0542474 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 256 8192 VARCHAR 4839.889 3281.086 -32.20741219 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 256 8192 ARRAY_BIGINT 893.161 808.889 -9.435252995 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 256 8192 ARRAY_VARCHAR 1912.328 1659.965 -13.19663782 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 256 8192 ARRAY_ARRAY_BIGINT 2296.046 2055.136 -10.49238561 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 256 8192 MAP_BIGINT_BIGINT 2138.706 1742.191 -18.53994892 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 256 8192 MAP_BIGINT_MAP_BIGINT_BIGINT 5432.889 5140.89 -5.374654259 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 256 8192 ROW_BIGINT_BIGINT 1082.421 771.412 -28.73272045 BenchmarkPartitionedOutputOperator.addPage 1 FALSE 0.2 256 8192 ROW_ARRAY_BIGINT_ARRAY_BIGINT 2214.725 1695.168 -23.45921051 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 2 8192 BIGINT 1863.551 1148.084 -38.39267077 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 2 8192 BIGINT_SKEWED_HASH 2207.817 1657.039 -24.94672339 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 2 8192 DICTIONARY_BIGINT 1656.253 1055.973 -36.24325511 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 2 8192 RLE_BIGINT 1363.576 688.575 -49.50226463 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 2 8192 BIGINT_PARTITION_CHANNEL_20_PERCENT 1228.092 873.406 -28.88106103 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 2 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_20_PERCENT 1510.711 762.043 -49.55732764 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 2 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_50_PERCENT 1534.493 836.388 -45.49417951 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 2 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_80_PERCENT 1702.515 1143.947 -32.80840404 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 2 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_100_PERCENT 1800.282 1265.429 -29.70940108 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 2 8192 RLE_PARTITION_BIGINT 2364.895 899.545 -61.96258185 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 2 8192 RLE_PARTITION_NULL_BIGINT 10.29 3.816 -62.9154519 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 2 8192 LONG_DECIMAL 4256.45 3758.952 -11.68809689 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 2 8192 INTEGER 1786.379 1131.209 -36.67586777 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 2 8192 SMALLINT 1961.655 1113.842 -43.21927148 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 2 8192 BOOLEAN 1723.534 995.049 -42.26693526 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 2 8192 VARCHAR 6947.303 6214.733 -10.54466748 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 2 8192 ARRAY_BIGINT 1032.977 978.331 -5.290146828 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 2 8192 ARRAY_VARCHAR 3319.275 3043.291 -8.31458677 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 2 8192 ARRAY_ARRAY_BIGINT 4007.356 3653.61 -8.827416381 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 2 8192 MAP_BIGINT_BIGINT 3452.737 3560.543 3.122334542 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 2 8192 MAP_BIGINT_MAP_BIGINT_BIGINT 11520.933 11599.765 0.6842501384 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 2 8192 ROW_BIGINT_BIGINT 1131.367 1024.381 -9.45634794 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 2 8192 ROW_ARRAY_BIGINT_ARRAY_BIGINT 3361.872 2708.01 -19.44934251 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 16 8192 BIGINT 2029.292 1300.127 -35.93199007 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 16 8192 BIGINT_SKEWED_HASH 2229.439 1514.66 -32.06093551 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 16 8192 DICTIONARY_BIGINT 1548.21 879.077 -43.21978285 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 16 8192 RLE_BIGINT 1347.258 741.15 -44.98826505 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 16 8192 BIGINT_PARTITION_CHANNEL_20_PERCENT 1266.302 890.737 -29.65840692 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 16 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_20_PERCENT 1519.514 787.816 -48.15342274 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 16 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_50_PERCENT 1583.208 869.901 -45.05453484 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 16 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_80_PERCENT 1702.761 1149.379 -32.49909999 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 16 8192 BIGINT_DICTIONARY_PARTITION_CHANNEL_100_PERCENT 1788.277 1317.585 -26.32097824 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 16 8192 RLE_PARTITION_BIGINT 2383.934 1007.899 -57.72118691 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 16 8192 RLE_PARTITION_NULL_BIGINT 8.474 3.914 -53.81165919 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 16 8192 LONG_DECIMAL 2582.467 1945.5 -24.66505864 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 16 8192 INTEGER 1930.602 1167.063 -39.54927012 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 16 8192 SMALLINT 1964.057 1166.88 -40.58828232 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 16 8192 BOOLEAN 1710.449 1107.672 -35.24086366 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 16 8192 VARCHAR 6951.744 5737.327 -17.46924225 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 16 8192 ARRAY_BIGINT 1584.935 792.682 -49.98646632 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 16 8192 ARRAY_VARCHAR 3400.526 3122.428 -8.178087743 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 16 8192 ARRAY_ARRAY_BIGINT 3977.859 3871.471 -2.674504049 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 16 8192 MAP_BIGINT_BIGINT 3416.368 3007.709 -11.96179686 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 16 8192 MAP_BIGINT_MAP_BIGINT_BIGINT 12101.568 12336.407 1.94056671 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 16 8192 ROW_BIGINT_BIGINT 1167.105 961.313 -17.63268943 BenchmarkPartitionedOutputOperator.addPage 2 TRUE 0 16 8192 ROW_ARRAY_BIGINT_ARRAY_BIGINT 3696.905 3036.086 -17.87492511 BenchmarkPartitionedOu…
lukasz-stec
left a comment
There was a problem hiding this comment.
mayHaveNull commit squashed. The rest of the comments were also addressed.
core/trino-main/src/main/java/io/trino/operator/output/PositionsAppender.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
removed (dedicated PositionsAppenders are anyway in cases where this could have impact)
There was a problem hiding this comment.
lgtm
In the future, we might consider moving io.trino.spi.type.Type#appendTo to TypeManager which would allow us to overload different batch appendTo methods.
For example, we could have:
@BatchOperator(APPEND_TO)
void appendTo(@SqlType(StandardTypes.BIGINT) LongArrayBlock block, BlockBuilder builder) {
...
}
@BatchOperator(APPEND_TO)
void appendTo(@SqlType(StandardTypes.BIGINT) RunLengthEncoded block, BlockBuilder builder) {
...
}
|
I also ran columnar page partitioner benchmarks on x86 and the gains are even better
|
Description
Add PagePartitioner implementation that copies pages column-wise.
The implementation switches to the old row-wise approach for cases where it's not beneficial to go column-wise.
Benchmarks
# Tpch/tpch
sf1000 orc partitioned
cpoo_sf1000_orc_part.pdf
JMH
improvement
core query engine.,
PartitionedOutputOperator.Improved partitioned exchange by going column-wise instead of row-wise.
Related issues, pull requests, and links
Documentation
(X ) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.
Release notes
( ) No release notes entries required.
(X ) Release notes entries required with the following suggested text:
Improved partitioned exchange by going column-wise instead of row-wise.