Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Aggregator support batch serialize #9777

Open
wants to merge 41 commits into
base: master
Choose a base branch
from
Open

Aggregator support batch serialize #9777

wants to merge 41 commits into from

Conversation

guo-shaoge
Copy link
Contributor

@guo-shaoge guo-shaoge commented Jan 9, 2025 •
edited
Loading

What problem does this PR solve?

Issue Number: close #9692

Problem Summary: Reduce virtual function call for key_serialize and key_string.

  1. For key_serialized, batch-wise serialization/deserialization can reduce virtual function call (Also enable prefetch for key_serialized, because we can batch get hash after this pr)
  2. For key_string, batch-wise sortKey can reduce virtual function call.

Workload: tpch-50g
Queries: same with #9679

  # --q5 Q3-1: key_serialized as group by method;  33/300M; HashMap with StringRef key
  "select /*+ mpp_1phase_agg() */ sum(l_discount), l_returnflag from lineitem group by l_returnflag, l_discount;"
  # --q6 Q3-2: key_serialized as group by method; 77M/300M; HashMap with StringRef key
  "select /*+ mpp_1phase_agg() */ sum(l_discount) as csum, l_returnflag from lineitem group by l_returnflag, l_discount, l_extendedprice having csum > 100;"


  # --q7 Q4-1: two_keys_num64_strbinpadding: 21/300M; HashMap with StringRef key
  "select /*+ mpp_1phase_agg() */ sum(l_discount) from lineitem group by l_returnflag, L_LINENUMBER;"
  # --q8 Q4-2: two_keys_num64_strbinpadding; 29.9M/300M; HashMap with StringRef key
  "select /*+ mpp_1phase_agg() */ sum(l_discount) as csum, l_partkey from lineitem group by l_returnflag, l_partkey having csum > 100;"
Query nightly-20250205 opt-only_batch opt-batch+prefetch rate-opt_only_batch rate-opt_batch+prefetch
Q3-1 1.79 1.54 1.58 13.97% 11.73%
Q3-2 7.45 6.13 4.9 17.72% 34.23%
Q4-1 2 2.07 2.09 -3.50% -4.50%
Q4-2 5.34 4.48 3.57 16.10% 33.15%

Workload: clickbench
Queries: https://github.com/ClickHouse/ClickBench/blob/fdfdb5d94f2a668dce1f63d55498aa34510e4c9c/clickhouse/queries.sql#L11

Query nightly-20250205 opt-only_batch opt-batch+prefetch rate-opt_only_batch rate-opt_batch+prefetch
q10 242.6 234.7 233 3.26% 3.96%
q11 269.3 264.6 251.2 1.75% 6.72%
q13 879.1 851.2 830.1 3.17% 5.57%
q14 662.4 620.9 634 6.27% 4.29%
q16 1.53 1.42 1.4 7.19% 8.50%
q17 1.4 1.33 1.26 5.00% 10.00%
q18 4.55 3.81 3.59 16.26% 21.10%

NOTE:

  1. nightly-20250205 commit: fe563a1; opt-batch commit: f20224a
  2. For Q4-1/Q4-2, nightly key is two_keys_num64_strbinpadding. opt key is key_serialized.

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

@ti-chi-bot ti-chi-bot bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. labels Jan 9, 2025
@ti-chi-bot ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jan 9, 2025

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from guo-shaoge, ensuring that each of them provides their approval before proceeding. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed do-not-merge/needs-linked-issue labels Jan 9, 2025
@guo-shaoge guo-shaoge force-pushed the hashagg_batch_serialize branch 4 times, most recently from ff696d3 to 8bbb062 Compare January 14, 2025 16:06
@ti-chi-bot ti-chi-bot bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jan 22, 2025
@guo-shaoge
tmp save
f6bb074 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
refine
51e24d8 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
plan 1 code
543f1b8 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
refine
370bf0a 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
remove changes in count
a13008f 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
refine
817854b 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
refine
d9aeae7 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
refine
eb01d9e 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
refine
62fbbed 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
refine
0ce2f53 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
refine
c4c0eb9 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge guo-shaoge force-pushed the hashagg_batch_serialize branch from 30a5b1c to 241537d Compare January 29, 2025 03:57
@ti-chi-bot ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 29, 2025
@guo-shaoge guo-shaoge force-pushed the hashagg_batch_serialize branch 3 times, most recently from fc95526 to 3354b7f Compare February 4, 2025 04:59
@guo-shaoge
fix
4681869 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
fix collator
3681aac 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
fix
aa32c12 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
fix
63870c4 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
refine
d7b86f7 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
refine
7a68832 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
hashagg batch
fdac3cd 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge guo-shaoge force-pushed the hashagg_batch_serialize branch from 3bb7660 to fdac3cd Compare February 17, 2025 09:31
@guo-shaoge
fmt
3d2b2f1 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
fix
f94d6cb 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
add nullmap for column decimal cout
5fe79b6 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
fix
ef3953b 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
more unit test
d203047 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
fix
f27e192 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
refine
2218337 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
fmt
5396f00 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
fix
59925ee 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
Merge branch 'batch_serialize_refine' of github.com:guo-shaoge/tiflas…
f20224a 
…h into hashagg_batch_serialize
@guo-shaoge
refine
f33c290 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
Merge branch 'master' into hashagg_batch_serialize
099b1e3
@guo-shaoge
refine
c51fd3c 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge
refine
a2f168e 
Signed-off-by: guo-shaoge <[email protected]>
@guo-shaoge guo-shaoge force-pushed the hashagg_batch_serialize branch from bb98d79 to a2f168e Compare February 21, 2025 06:06
@guo-shaoge guo-shaoge requested a review from yibin87 February 24, 2025 02:47
yibin87
yibin87 reviewed Feb 25, 2025
View reviewed changes
dbms/src/Common/ColumnsHashing.h Outdated
class KeyStringBatchHandlerBase
{
private:
size_t batch_row_idx = 0;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe renamed to processed_row_count to make it easier to understand.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

yibin87
yibin87 reviewed Feb 25, 2025
View reviewed changes
dbms/src/Common/ColumnsHashing.h Outdated
class KeySerializedBatchHandlerBase
{
private:
size_t batch_row_idx = 0;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

dbms/src/Common/ColumnsHashing.h
RUNTIME_CHECK(max_batch_size >= 256);
batch_row_idx = start_row;
sort_key_containers.resize(max_batch_size);
batch_rows.reserve(max_batch_size);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A little strange here, for one uses resize while the other uses reserve.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

batch_rows will be resize each time prepareNextBatchType is called.

dbms/src/Common/ColumnsHashing.h
void init(size_t start_row, const ColumnRawPtrs & key_columns, const TiDB::TiDBCollators & collators)
{
batch_row_idx = start_row;
byte_size.resize_fill_zero(key_columns[0]->size());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we already have start_row here, do we need to resize to total size of key_columns[0]?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

countSerializeByteSizeForCmp doesn't have start_row parameter. So we have to resize byte_size to the total size.

dbms/src/Common/ColumnsHashing.h
byte_size.resize_fill_zero(key_columns[0]->size());
RUNTIME_CHECK(!byte_size.empty());
for (size_t i = 0; i < key_columns.size(); ++i)
key_columns[i]->countSerializeByteSizeForCmp(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From the following code, each non-null, non-collation string column will add sizeof(UINT32) to byte_size, not sure for multiple string columns, do we need to add this for each?

tiflash/dbms/src/Columns/ColumnString.cpp

Line 582 in 94402d5

byte_size[i] += sizeof(UInt32) + sizeAt(i);

Copy link
Contributor

@yibin87 yibin87 Feb 25, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I figure out, we do need to add for each, as a type of separator here.

yibin87
yibin87 reviewed Feb 25, 2025
View reviewed changes
dbms/src/Common/ColumnsHashing.h
&sort_key_container);

for (size_t i = 0; i < cur_batch_size; ++i)
real_byte_size[i] = pos[i] - ori_pos[i];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When will real_byte_size differ from byte_size?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the column string, the currently pre-allocated memory space is based on the maximum space that a UTF-8 character can occupy. However, in practice, many characters do not require such a large amount of space. Therefore, byte_size[i] for the column string represents the maximum occupied space, rather than the actual space used.

dbms/src/Interpreters/Aggregator.cpp
aggregates_pool,
sort_key_containers);

if constexpr (enable_prefetch)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it expected that hashvals not updated when enable_prefetch is false?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. When enable_prefetch is false and batch_get_key_holder is true, will use emplaceKey(method, state, key_holder) to emplace into HashTable. So no need to init hashvals

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hashvals is only for prefetch to use. so no need to compute hashval for other situations

@guo-shaoge
processed_row_idx
3df2c92 
Signed-off-by: guo-shaoge <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yibin87 yibin87 yibin87 left review comments

@windtalker windtalker Awaiting requested review from windtalker

Assignees
No one assigned
Labels
release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Aggregator support batch allocate memory for key_serialized
2 participants
@guo-shaoge @yibin87