[Enhancement] Optimize memory usage of primary key table large load #12068
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sevev
changed the title
sevev
changed the title
sevev
force-pushed
the
reduce_mem_usage_in_apply
branch
from
f42c800
to
884e06b
Compare
sevev
changed the title
chaoyli
reviewed
Oct 13, 2022
sevev
force-pushed
the
reduce_mem_usage_in_apply
branch
2 times, most recently
from
934b49a
to
fdbca36
Compare
chaoyli
reviewed
Nov 3, 2022
sevev
changed the title
sevev
force-pushed
the
reduce_mem_usage_in_apply
branch
from
c93d660
to
be09f5a
Compare
sevev
changed the title
|
run starrocks_be_unittest |
chaoyli
previously approved these changes
Nov 9, 2022
|
clang-tidy review says "All clean, LGTM! 👍" |
sevev
force-pushed
the
reduce_mem_usage_in_apply
branch
from
4bca6bf
to
e58c601
Compare
|
run starrocks_admit_test |
decster
approved these changes
Nov 21, 2022
|
run starrocks_admit_test |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
@mergify backport branch-2.5 |
|
Kudos, SonarCloud Quality Gate passed!
|
mergify bot
pushed a commit
that referenced
this pull request
Nov 21, 2022
…12068) Currently, RowsetUpdateState::load will preload all segments primary keys into memory, if the load(rowset) is very large, it will use a lot of memory during the commit or apply phrase. For large load(rowset), we don't preload all segment's primary keys but process segment by segment, which can reduce the memory usage during apply. It is important to note that the limitation is a soft limit because we can't tolerate the failure to apply, so memory usage may still exceed the limitation. In my test env, one BE with two HDD, using Broker load, create a table with persistent index: use tpcds to test create table sql, using broker load: CREATE TABLE `store_sales` ( `ss_item_sk` bigint(20) NOT NULL COMMENT "", `ss_ticket_number` bigint(20) NOT NULL COMMENT "", `ss_sold_date_sk` bigint(20) NULL COMMENT "", `ss_sold_time_sk` bigint(20) NULL COMMENT "", `ss_customer_sk` bigint(20) NULL COMMENT "", `ss_cdemo_sk` bigint(20) NULL COMMENT "", `ss_hdemo_sk` bigint(20) NULL COMMENT "", `ss_addr_sk` bigint(20) NULL COMMENT "", `ss_store_sk` bigint(20) NULL COMMENT "", `ss_promo_sk` bigint(20) NULL COMMENT "", `ss_quantity` bigint(20) NULL COMMENT "", `ss_wholesale_cost` decimal64(7, 2) NULL COMMENT "", `ss_list_price` decimal64(7, 2) NULL COMMENT "", `ss_sales_price` decimal64(7, 2) NULL COMMENT "", `ss_ext_discount_amt` decimal64(7, 2) NULL COMMENT "", `ss_ext_sales_price` decimal64(7, 2) NULL COMMENT "", `ss_ext_wholesale_cost` decimal64(7, 2) NULL COMMENT "", `ss_ext_list_price` decimal64(7, 2) NULL COMMENT "", `ss_ext_tax` decimal64(7, 2) NULL COMMENT "", `ss_coupon_amt` decimal64(7, 2) NULL COMMENT "", `ss_net_paid` decimal64(7, 2) NULL COMMENT "", `ss_net_paid_inc_tax` decimal64(7, 2) NULL COMMENT "", `ss_net_profit` decimal64(7, 2) NULL COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`ss_item_sk`, `ss_ticket_number`) COMMENT "OLAP" DISTRIBUTED BY HASH(`ss_item_sk`, `ss_ticket_number`) BUCKETS 2 PROPERTIES ( "replication_num" = "1", "in_memory" = "false", "storage_format" = "DEFAULT", "enable_persistent_index" = "true", "compression" = "LZ4" ); PrimaryKey Length RowNum BucketNum Load time(s) Apply time(ms) Peak Memory usage(GB) Note 16 Bytes 864001869 2 7643 355200 25.03 branch-opt 16 Bytes 864001869 2 7591 348465 46.45 branch-main 16 Bytes 864001869 100 7194 32705 25.11 branch-opt 16 Bytes 864001869 100 7104 30705 43.14 branch-main Note there are still some scenarios we don't resolve in this pr: In the partial update, the read column data maybe very large and we don't resolve it in this pr We still need to load all primary key into L0 of persistent index first which maybe cause OOM (cherry picked from commit 6c68734) # Conflicts: # be/src/storage/rowset_update_state.cpp
✅ Backports have been created
|
|
@mergify backport branch-2.5 |
✅ Backports have been created
|
sevev
added a commit
to sevev/starrocks
that referenced
this pull request
Nov 21, 2022
…tarRocks#12068) Currently, RowsetUpdateState::load will preload all segments primary keys into memory, if the load(rowset) is very large, it will use a lot of memory during the commit or apply phrase. For large load(rowset), we don't preload all segment's primary keys but process segment by segment, which can reduce the memory usage during apply. It is important to note that the limitation is a soft limit because we can't tolerate the failure to apply, so memory usage may still exceed the limitation. In my test env, one BE with two HDD, using Broker load, create a table with persistent index: use tpcds to test create table sql, using broker load: CREATE TABLE `store_sales` ( `ss_item_sk` bigint(20) NOT NULL COMMENT "", `ss_ticket_number` bigint(20) NOT NULL COMMENT "", `ss_sold_date_sk` bigint(20) NULL COMMENT "", `ss_sold_time_sk` bigint(20) NULL COMMENT "", `ss_customer_sk` bigint(20) NULL COMMENT "", `ss_cdemo_sk` bigint(20) NULL COMMENT "", `ss_hdemo_sk` bigint(20) NULL COMMENT "", `ss_addr_sk` bigint(20) NULL COMMENT "", `ss_store_sk` bigint(20) NULL COMMENT "", `ss_promo_sk` bigint(20) NULL COMMENT "", `ss_quantity` bigint(20) NULL COMMENT "", `ss_wholesale_cost` decimal64(7, 2) NULL COMMENT "", `ss_list_price` decimal64(7, 2) NULL COMMENT "", `ss_sales_price` decimal64(7, 2) NULL COMMENT "", `ss_ext_discount_amt` decimal64(7, 2) NULL COMMENT "", `ss_ext_sales_price` decimal64(7, 2) NULL COMMENT "", `ss_ext_wholesale_cost` decimal64(7, 2) NULL COMMENT "", `ss_ext_list_price` decimal64(7, 2) NULL COMMENT "", `ss_ext_tax` decimal64(7, 2) NULL COMMENT "", `ss_coupon_amt` decimal64(7, 2) NULL COMMENT "", `ss_net_paid` decimal64(7, 2) NULL COMMENT "", `ss_net_paid_inc_tax` decimal64(7, 2) NULL COMMENT "", `ss_net_profit` decimal64(7, 2) NULL COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`ss_item_sk`, `ss_ticket_number`) COMMENT "OLAP" DISTRIBUTED BY HASH(`ss_item_sk`, `ss_ticket_number`) BUCKETS 2 PROPERTIES ( "replication_num" = "1", "in_memory" = "false", "storage_format" = "DEFAULT", "enable_persistent_index" = "true", "compression" = "LZ4" ); PrimaryKey Length RowNum BucketNum Load time(s) Apply time(ms) Peak Memory usage(GB) Note 16 Bytes 864001869 2 7643 355200 25.03 branch-opt 16 Bytes 864001869 2 7591 348465 46.45 branch-main 16 Bytes 864001869 100 7194 32705 25.11 branch-opt 16 Bytes 864001869 100 7104 30705 43.14 branch-main Note there are still some scenarios we don't resolve in this pr: In the partial update, the read column data maybe very large and we don't resolve it in this pr We still need to load all primary key into L0 of persistent index first which maybe cause OOM
11 tasks
chaoyli
pushed a commit
that referenced
this pull request
Nov 27, 2022
…12068) (#13744) Currently, RowsetUpdateState::load will preload all segments primary keys into memory, if the load(rowset) is very large, it will use a lot of memory during the commit or apply phrase. For large load(rowset), we don't preload all segment's primary keys but process segment by segment, which can reduce the memory usage during apply. It is important to note that the limitation is a soft limit because we can't tolerate the failure to apply, so memory usage may still exceed the limitation. In my test env, one BE with two HDD, using Broker load, create a table with persistent index: use tpcds to test create table sql, using broker load: CREATE TABLE `store_sales` ( `ss_item_sk` bigint(20) NOT NULL COMMENT "", `ss_ticket_number` bigint(20) NOT NULL COMMENT "", `ss_sold_date_sk` bigint(20) NULL COMMENT "", `ss_sold_time_sk` bigint(20) NULL COMMENT "", `ss_customer_sk` bigint(20) NULL COMMENT "", `ss_cdemo_sk` bigint(20) NULL COMMENT "", `ss_hdemo_sk` bigint(20) NULL COMMENT "", `ss_addr_sk` bigint(20) NULL COMMENT "", `ss_store_sk` bigint(20) NULL COMMENT "", `ss_promo_sk` bigint(20) NULL COMMENT "", `ss_quantity` bigint(20) NULL COMMENT "", `ss_wholesale_cost` decimal64(7, 2) NULL COMMENT "", `ss_list_price` decimal64(7, 2) NULL COMMENT "", `ss_sales_price` decimal64(7, 2) NULL COMMENT "", `ss_ext_discount_amt` decimal64(7, 2) NULL COMMENT "", `ss_ext_sales_price` decimal64(7, 2) NULL COMMENT "", `ss_ext_wholesale_cost` decimal64(7, 2) NULL COMMENT "", `ss_ext_list_price` decimal64(7, 2) NULL COMMENT "", `ss_ext_tax` decimal64(7, 2) NULL COMMENT "", `ss_coupon_amt` decimal64(7, 2) NULL COMMENT "", `ss_net_paid` decimal64(7, 2) NULL COMMENT "", `ss_net_paid_inc_tax` decimal64(7, 2) NULL COMMENT "", `ss_net_profit` decimal64(7, 2) NULL COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`ss_item_sk`, `ss_ticket_number`) COMMENT "OLAP" DISTRIBUTED BY HASH(`ss_item_sk`, `ss_ticket_number`) BUCKETS 2 PROPERTIES ( "replication_num" = "1", "in_memory" = "false", "storage_format" = "DEFAULT", "enable_persistent_index" = "true", "compression" = "LZ4" ); PrimaryKey Length RowNum BucketNum Load time(s) Apply time(ms) Peak Memory usage(GB) Note 16 Bytes 864001869 2 7643 355200 25.03 branch-opt 16 Bytes 864001869 2 7591 348465 46.45 branch-main 16 Bytes 864001869 100 7194 32705 25.11 branch-opt 16 Bytes 864001869 100 7104 30705 43.14 branch-main Note there are still some scenarios we don't resolve in this pr: In the partial update, the read column data maybe very large and we don't resolve it in this pr We still need to load all primary key into L0 of persistent index first which maybe cause OOM
This was referenced Nov 28, 2022
chaoyli
pushed a commit
that referenced
this pull request
Dec 5, 2022
This bug is introduced by #12068. To reduce the memory usage during apply, we don't preload all segments' primary keys and process segment by segment(#12068). ``` .... uint32_t max_rowset_id = *std::max_element(info->inputs.begin(), info->inputs.end()); Rowset* rowset = _get_rowset(max_rowset_id).get(); ..... for (size_t i = 0; i < _compaction_state->pk_cols.size(); i++) { // the rowset is not what we should load if (st = _compaction_state->load_segments(rowset, i); !st.ok()) { manager->index_cache().release(index_entry); _compaction_state.reset(); std::string msg = strings::Substitute("_apply_compaction_commit error: load compaction state failed: $0 $1", st.to_string(), debug_string()); LOG(ERROR) << msg; _set_error(msg); return; } ..... } ``` As the above code shown, we will load one segment for a loop. However, the `rowset` is not the output rowset after compaction but one of the input rowsets, so we may get an unexpected error.
mergify bot
pushed a commit
that referenced
this pull request
Dec 5, 2022
This bug is introduced by #12068. To reduce the memory usage during apply, we don't preload all segments' primary keys and process segment by segment(#12068). ``` .... uint32_t max_rowset_id = *std::max_element(info->inputs.begin(), info->inputs.end()); Rowset* rowset = _get_rowset(max_rowset_id).get(); ..... for (size_t i = 0; i < _compaction_state->pk_cols.size(); i++) { // the rowset is not what we should load if (st = _compaction_state->load_segments(rowset, i); !st.ok()) { manager->index_cache().release(index_entry); _compaction_state.reset(); std::string msg = strings::Substitute("_apply_compaction_commit error: load compaction state failed: $0 $1", st.to_string(), debug_string()); LOG(ERROR) << msg; _set_error(msg); return; } ..... } ``` As the above code shown, we will load one segment for a loop. However, the `rowset` is not the output rowset after compaction but one of the input rowsets, so we may get an unexpected error. (cherry picked from commit 287f82a)
wanpengfei-git
pushed a commit
that referenced
this pull request
Dec 8, 2022
This bug is introduced by #12068. To reduce the memory usage during apply, we don't preload all segments' primary keys and process segment by segment(#12068). ``` .... uint32_t max_rowset_id = *std::max_element(info->inputs.begin(), info->inputs.end()); Rowset* rowset = _get_rowset(max_rowset_id).get(); ..... for (size_t i = 0; i < _compaction_state->pk_cols.size(); i++) { // the rowset is not what we should load if (st = _compaction_state->load_segments(rowset, i); !st.ok()) { manager->index_cache().release(index_entry); _compaction_state.reset(); std::string msg = strings::Substitute("_apply_compaction_commit error: load compaction state failed: $0 $1", st.to_string(), debug_string()); LOG(ERROR) << msg; _set_error(msg); return; } ..... } ``` As the above code shown, we will load one segment for a loop. However, the `rowset` is not the output rowset after compaction but one of the input rowsets, so we may get an unexpected error. (cherry picked from commit 287f82a)
decster
pushed a commit
that referenced
this pull request
Dec 26, 2022
We have partially optimized the primary key model for large import memory usage in this pr(#12068), but the enhancement doesn't work if the load is partial update. And we also need a lot of memory if you do a large number of partial updates in one transaction. So this pr will try to reduce the memory usage of large partial update. There are two reasons for large memory usage during partial column updates: 1. The first one is that updating a few columns may increase the segment file size and we need to load all data of segment into memory which will cost a lot of memory. 2. The second one is that doing partial update requires reading data from other columns into memory, which can take up a lot of memory if the table has many columns. In order to reduce memory usage, the following two adjustments are made: 1. The first one is to estimate the length of the updated partial columns in each row when importing data, thus reducing the size of the segment file 2. The second one is not to load all the data of the rowset into memory at once, but to load them one by one according to the segment. In my test env, one BE with two HDD, using StreamLoad, create a table with 65 column, 20 buckets: ``` CREATE TABLE `partial_test` ( `col_1` bigint(20) NOT NULL COMMENT "", `col_2` bigint(20) NOT NULL COMMENT "", `col_3` bigint(20) NOT NULL COMMENT "", `col_4` varchar(150) NOT NULL COMMENT "", `col_5` varchar(150) NOT NULL COMMENT "", `col_6` varchar(150) NULL COMMENT "", `col_7` varchar(150) NULL COMMENT "", `col_8` varchar(1024) NULL COMMENT "", `col_9` varchar(120) NULL COMMENT "", `col_10` varchar(60) NULL COMMENT "", `col_11` varchar(10) NULL COMMENT "", `col_12` varchar(120) NULL COMMENT "", `col_13` varchar(524) NULL COMMENT "", `col_14` varchar(100) NULL COMMENT "", `col_15` varchar(150) NULL COMMENT "", `col_16` varchar(150) NULL COMMENT "", `col_17` varchar(150) NULL COMMENT "", `col_18` bigint(20) NULL COMMENT "", `col_19` varchar(500) NULL COMMENT "", `col_20` varchar(150) NULL COMMENT "", `col_21` tinyint(4) NULL COMMENT "", `col_22` int(11) NULL COMMENT "", `col_23` varchar(524) NULL COMMENT "", `col_24` bigint(20) NULL COMMENT "", `col_25` bigint(20) NULL COMMENT "", `col_26` varchar(8) NULL COMMENT "", `col_27` decimal64(18, 6) NULL COMMENT "", `col_28` decimal64(18, 6) NULL COMMENT "", `col_29` decimal64(18, 6) NULL COMMENT "", `col_30` decimal64(18, 6) NULL COMMENT "", `col_31` decimal64(18, 6) NULL COMMENT "", `col_32` decimal64(18, 6) NULL COMMENT "", `col_33` bigint(20) NULL COMMENT "", `col_34` decimal64(18, 6) NULL COMMENT "", `col_35` varchar(8) NULL COMMENT "", `col_36` decimal64(18, 6) NULL COMMENT "", `col_37` decimal64(18, 6) NULL COMMENT "", `col_38` varchar(8) NULL COMMENT "", `col_39` decimal64(18, 6) NULL COMMENT "", `col_40` decimal64(18, 6) NULL COMMENT "", `col_41` varchar(8) NULL COMMENT "", `col_42` decimal64(18, 6) NULL COMMENT "", `col_43` decimal64(18, 6) NULL COMMENT "", `col_44` decimal64(18, 6) NULL COMMENT "", `col_45` decimal64(18, 6) NULL COMMENT "", `col_46` int(11) NULL COMMENT "", `col_47` int(11) NOT NULL COMMENT "", `col_48` tinyint(4) NULL COMMENT "", `col_49` varchar(200) NULL COMMENT "", `col_50` tinyint(4) NULL COMMENT "", `col_51` varchar(200) NULL COMMENT "", `col_52` varchar(10) NULL COMMENT "", `col_53` tinyint(4) NULL COMMENT "", `col_54` tinyint(4) NULL COMMENT "", `col_55` varchar(150) NULL COMMENT "", `col_56` varchar(150) NULL COMMENT "", `col_57` varchar(500) NULL COMMENT "", `col_58` tinyint(4) NULL COMMENT "", `col_59` varchar(100) NULL COMMENT "", `col_60` varchar(150) NULL COMMENT "", `col_61` varchar(150) NULL COMMENT "", `col_62` varchar(150) NULL COMMENT "", `col_63` varchar(150) NULL COMMENT "", `col_64` datetime NULL COMMENT "", `col_65` datetime NULL COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`col_1`, `col_2`, `col_3`) COMMENT "OLAP" DISTRIBUTED BY HASH(`col_1`, `col_2`) BUCKETS 20 PROPERTIES ( "replication_num" = "1", "in_memory" = "false", "storage_format" = "V2", "enable_persistent_index" = "true", "compression" = "LZ4" ); ``` |PrimaryKey Length| RowNum|BucketNum| Column Num| Partial ColumnNum | PartialUpdate RowsNum| Load time(s)| Apply time(ms)| Peak UpdateMemory usage | Note | |---------------------|----------|------------|----------------|--------------------|------------------------------|----|-----|-----|----| |12 Bytes| 300M | 20 | 65 | 5 | 100M | 135261 | 106693 | 78.9G | branch-main | |12 Bytes| 300M | 20 | 65 | 5 | 100M | 166449| 149870 | 10.3G | branch-opt | |12 Bytes| 300M | 20 | 65 | 5 | 100K | 2078 | 529 | 60.1M | branch-main | |12 Bytes| 300M | 20 | 65 | 5 | 100K | 2211 | 541 | 60.2M | branch-opt |
mergify bot
pushed a commit
that referenced
this pull request
Dec 29, 2022
We have partially optimized the primary key model for large import memory usage in this pr(#12068), but the enhancement doesn't work if the load is partial update. And we also need a lot of memory if you do a large number of partial updates in one transaction. So this pr will try to reduce the memory usage of large partial update. There are two reasons for large memory usage during partial column updates: 1. The first one is that updating a few columns may increase the segment file size and we need to load all data of segment into memory which will cost a lot of memory. 2. The second one is that doing partial update requires reading data from other columns into memory, which can take up a lot of memory if the table has many columns. In order to reduce memory usage, the following two adjustments are made: 1. The first one is to estimate the length of the updated partial columns in each row when importing data, thus reducing the size of the segment file 2. The second one is not to load all the data of the rowset into memory at once, but to load them one by one according to the segment. In my test env, one BE with two HDD, using StreamLoad, create a table with 65 column, 20 buckets: ``` CREATE TABLE `partial_test` ( `col_1` bigint(20) NOT NULL COMMENT "", `col_2` bigint(20) NOT NULL COMMENT "", `col_3` bigint(20) NOT NULL COMMENT "", `col_4` varchar(150) NOT NULL COMMENT "", `col_5` varchar(150) NOT NULL COMMENT "", `col_6` varchar(150) NULL COMMENT "", `col_7` varchar(150) NULL COMMENT "", `col_8` varchar(1024) NULL COMMENT "", `col_9` varchar(120) NULL COMMENT "", `col_10` varchar(60) NULL COMMENT "", `col_11` varchar(10) NULL COMMENT "", `col_12` varchar(120) NULL COMMENT "", `col_13` varchar(524) NULL COMMENT "", `col_14` varchar(100) NULL COMMENT "", `col_15` varchar(150) NULL COMMENT "", `col_16` varchar(150) NULL COMMENT "", `col_17` varchar(150) NULL COMMENT "", `col_18` bigint(20) NULL COMMENT "", `col_19` varchar(500) NULL COMMENT "", `col_20` varchar(150) NULL COMMENT "", `col_21` tinyint(4) NULL COMMENT "", `col_22` int(11) NULL COMMENT "", `col_23` varchar(524) NULL COMMENT "", `col_24` bigint(20) NULL COMMENT "", `col_25` bigint(20) NULL COMMENT "", `col_26` varchar(8) NULL COMMENT "", `col_27` decimal64(18, 6) NULL COMMENT "", `col_28` decimal64(18, 6) NULL COMMENT "", `col_29` decimal64(18, 6) NULL COMMENT "", `col_30` decimal64(18, 6) NULL COMMENT "", `col_31` decimal64(18, 6) NULL COMMENT "", `col_32` decimal64(18, 6) NULL COMMENT "", `col_33` bigint(20) NULL COMMENT "", `col_34` decimal64(18, 6) NULL COMMENT "", `col_35` varchar(8) NULL COMMENT "", `col_36` decimal64(18, 6) NULL COMMENT "", `col_37` decimal64(18, 6) NULL COMMENT "", `col_38` varchar(8) NULL COMMENT "", `col_39` decimal64(18, 6) NULL COMMENT "", `col_40` decimal64(18, 6) NULL COMMENT "", `col_41` varchar(8) NULL COMMENT "", `col_42` decimal64(18, 6) NULL COMMENT "", `col_43` decimal64(18, 6) NULL COMMENT "", `col_44` decimal64(18, 6) NULL COMMENT "", `col_45` decimal64(18, 6) NULL COMMENT "", `col_46` int(11) NULL COMMENT "", `col_47` int(11) NOT NULL COMMENT "", `col_48` tinyint(4) NULL COMMENT "", `col_49` varchar(200) NULL COMMENT "", `col_50` tinyint(4) NULL COMMENT "", `col_51` varchar(200) NULL COMMENT "", `col_52` varchar(10) NULL COMMENT "", `col_53` tinyint(4) NULL COMMENT "", `col_54` tinyint(4) NULL COMMENT "", `col_55` varchar(150) NULL COMMENT "", `col_56` varchar(150) NULL COMMENT "", `col_57` varchar(500) NULL COMMENT "", `col_58` tinyint(4) NULL COMMENT "", `col_59` varchar(100) NULL COMMENT "", `col_60` varchar(150) NULL COMMENT "", `col_61` varchar(150) NULL COMMENT "", `col_62` varchar(150) NULL COMMENT "", `col_63` varchar(150) NULL COMMENT "", `col_64` datetime NULL COMMENT "", `col_65` datetime NULL COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`col_1`, `col_2`, `col_3`) COMMENT "OLAP" DISTRIBUTED BY HASH(`col_1`, `col_2`) BUCKETS 20 PROPERTIES ( "replication_num" = "1", "in_memory" = "false", "storage_format" = "V2", "enable_persistent_index" = "true", "compression" = "LZ4" ); ``` |PrimaryKey Length| RowNum|BucketNum| Column Num| Partial ColumnNum | PartialUpdate RowsNum| Load time(s)| Apply time(ms)| Peak UpdateMemory usage | Note | |---------------------|----------|------------|----------------|--------------------|------------------------------|----|-----|-----|----| |12 Bytes| 300M | 20 | 65 | 5 | 100M | 135261 | 106693 | 78.9G | branch-main | |12 Bytes| 300M | 20 | 65 | 5 | 100M | 166449| 149870 | 10.3G | branch-opt | |12 Bytes| 300M | 20 | 65 | 5 | 100K | 2078 | 529 | 60.1M | branch-main | |12 Bytes| 300M | 20 | 65 | 5 | 100K | 2211 | 541 | 60.2M | branch-opt | (cherry picked from commit 545b7be) # Conflicts: # be/src/storage/memtable.h # be/src/storage/rowset_update_state.cpp # be/src/storage/rowset_update_state.h # be/src/storage/tablet_updates.cpp
sevev
added a commit
to sevev/starrocks
that referenced
this pull request
Dec 29, 2022
We have partially optimized the primary key model for large import memory usage in this pr(StarRocks#12068), but the enhancement doesn't work if the load is partial update. And we also need a lot of memory if you do a large number of partial updates in one transaction. So this pr will try to reduce the memory usage of large partial update. There are two reasons for large memory usage during partial column updates: 1. The first one is that updating a few columns may increase the segment file size and we need to load all data of segment into memory which will cost a lot of memory. 2. The second one is that doing partial update requires reading data from other columns into memory, which can take up a lot of memory if the table has many columns. In order to reduce memory usage, the following two adjustments are made: 1. The first one is to estimate the length of the updated partial columns in each row when importing data, thus reducing the size of the segment file 2. The second one is not to load all the data of the rowset into memory at once, but to load them one by one according to the segment. In my test env, one BE with two HDD, using StreamLoad, create a table with 65 column, 20 buckets: ``` CREATE TABLE `partial_test` ( `col_1` bigint(20) NOT NULL COMMENT "", `col_2` bigint(20) NOT NULL COMMENT "", `col_3` bigint(20) NOT NULL COMMENT "", `col_4` varchar(150) NOT NULL COMMENT "", `col_5` varchar(150) NOT NULL COMMENT "", `col_6` varchar(150) NULL COMMENT "", `col_7` varchar(150) NULL COMMENT "", `col_8` varchar(1024) NULL COMMENT "", `col_9` varchar(120) NULL COMMENT "", `col_10` varchar(60) NULL COMMENT "", `col_11` varchar(10) NULL COMMENT "", `col_12` varchar(120) NULL COMMENT "", `col_13` varchar(524) NULL COMMENT "", `col_14` varchar(100) NULL COMMENT "", `col_15` varchar(150) NULL COMMENT "", `col_16` varchar(150) NULL COMMENT "", `col_17` varchar(150) NULL COMMENT "", `col_18` bigint(20) NULL COMMENT "", `col_19` varchar(500) NULL COMMENT "", `col_20` varchar(150) NULL COMMENT "", `col_21` tinyint(4) NULL COMMENT "", `col_22` int(11) NULL COMMENT "", `col_23` varchar(524) NULL COMMENT "", `col_24` bigint(20) NULL COMMENT "", `col_25` bigint(20) NULL COMMENT "", `col_26` varchar(8) NULL COMMENT "", `col_27` decimal64(18, 6) NULL COMMENT "", `col_28` decimal64(18, 6) NULL COMMENT "", `col_29` decimal64(18, 6) NULL COMMENT "", `col_30` decimal64(18, 6) NULL COMMENT "", `col_31` decimal64(18, 6) NULL COMMENT "", `col_32` decimal64(18, 6) NULL COMMENT "", `col_33` bigint(20) NULL COMMENT "", `col_34` decimal64(18, 6) NULL COMMENT "", `col_35` varchar(8) NULL COMMENT "", `col_36` decimal64(18, 6) NULL COMMENT "", `col_37` decimal64(18, 6) NULL COMMENT "", `col_38` varchar(8) NULL COMMENT "", `col_39` decimal64(18, 6) NULL COMMENT "", `col_40` decimal64(18, 6) NULL COMMENT "", `col_41` varchar(8) NULL COMMENT "", `col_42` decimal64(18, 6) NULL COMMENT "", `col_43` decimal64(18, 6) NULL COMMENT "", `col_44` decimal64(18, 6) NULL COMMENT "", `col_45` decimal64(18, 6) NULL COMMENT "", `col_46` int(11) NULL COMMENT "", `col_47` int(11) NOT NULL COMMENT "", `col_48` tinyint(4) NULL COMMENT "", `col_49` varchar(200) NULL COMMENT "", `col_50` tinyint(4) NULL COMMENT "", `col_51` varchar(200) NULL COMMENT "", `col_52` varchar(10) NULL COMMENT "", `col_53` tinyint(4) NULL COMMENT "", `col_54` tinyint(4) NULL COMMENT "", `col_55` varchar(150) NULL COMMENT "", `col_56` varchar(150) NULL COMMENT "", `col_57` varchar(500) NULL COMMENT "", `col_58` tinyint(4) NULL COMMENT "", `col_59` varchar(100) NULL COMMENT "", `col_60` varchar(150) NULL COMMENT "", `col_61` varchar(150) NULL COMMENT "", `col_62` varchar(150) NULL COMMENT "", `col_63` varchar(150) NULL COMMENT "", `col_64` datetime NULL COMMENT "", `col_65` datetime NULL COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`col_1`, `col_2`, `col_3`) COMMENT "OLAP" DISTRIBUTED BY HASH(`col_1`, `col_2`) BUCKETS 20 PROPERTIES ( "replication_num" = "1", "in_memory" = "false", "storage_format" = "V2", "enable_persistent_index" = "true", "compression" = "LZ4" ); ``` |PrimaryKey Length| RowNum|BucketNum| Column Num| Partial ColumnNum | PartialUpdate RowsNum| Load time(s)| Apply time(ms)| Peak UpdateMemory usage | Note | |---------------------|----------|------------|----------------|--------------------|------------------------------|----|-----|-----|----| |12 Bytes| 300M | 20 | 65 | 5 | 100M | 135261 | 106693 | 78.9G | branch-main | |12 Bytes| 300M | 20 | 65 | 5 | 100M | 166449| 149870 | 10.3G | branch-opt | |12 Bytes| 300M | 20 | 65 | 5 | 100K | 2078 | 529 | 60.1M | branch-main | |12 Bytes| 300M | 20 | 65 | 5 | 100K | 2211 | 541 | 60.2M | branch-opt |
16 tasks
wanpengfei-git
pushed a commit
that referenced
this pull request
Jan 9, 2023
We have partially optimized the primary key model for large import memory usage in this pr(#12068), but the enhancement doesn't work if the load is partial update. And we also need a lot of memory if you do a large number of partial updates in one transaction. So this pr will try to reduce the memory usage of large partial update. There are two reasons for large memory usage during partial column updates: 1. The first one is that updating a few columns may increase the segment file size and we need to load all data of segment into memory which will cost a lot of memory. 2. The second one is that doing partial update requires reading data from other columns into memory, which can take up a lot of memory if the table has many columns. In order to reduce memory usage, the following two adjustments are made: 1. The first one is to estimate the length of the updated partial columns in each row when importing data, thus reducing the size of the segment file 2. The second one is not to load all the data of the rowset into memory at once, but to load them one by one according to the segment. In my test env, one BE with two HDD, using StreamLoad, create a table with 65 column, 20 buckets: ``` CREATE TABLE `partial_test` ( `col_1` bigint(20) NOT NULL COMMENT "", `col_2` bigint(20) NOT NULL COMMENT "", `col_3` bigint(20) NOT NULL COMMENT "", `col_4` varchar(150) NOT NULL COMMENT "", `col_5` varchar(150) NOT NULL COMMENT "", `col_6` varchar(150) NULL COMMENT "", `col_7` varchar(150) NULL COMMENT "", `col_8` varchar(1024) NULL COMMENT "", `col_9` varchar(120) NULL COMMENT "", `col_10` varchar(60) NULL COMMENT "", `col_11` varchar(10) NULL COMMENT "", `col_12` varchar(120) NULL COMMENT "", `col_13` varchar(524) NULL COMMENT "", `col_14` varchar(100) NULL COMMENT "", `col_15` varchar(150) NULL COMMENT "", `col_16` varchar(150) NULL COMMENT "", `col_17` varchar(150) NULL COMMENT "", `col_18` bigint(20) NULL COMMENT "", `col_19` varchar(500) NULL COMMENT "", `col_20` varchar(150) NULL COMMENT "", `col_21` tinyint(4) NULL COMMENT "", `col_22` int(11) NULL COMMENT "", `col_23` varchar(524) NULL COMMENT "", `col_24` bigint(20) NULL COMMENT "", `col_25` bigint(20) NULL COMMENT "", `col_26` varchar(8) NULL COMMENT "", `col_27` decimal64(18, 6) NULL COMMENT "", `col_28` decimal64(18, 6) NULL COMMENT "", `col_29` decimal64(18, 6) NULL COMMENT "", `col_30` decimal64(18, 6) NULL COMMENT "", `col_31` decimal64(18, 6) NULL COMMENT "", `col_32` decimal64(18, 6) NULL COMMENT "", `col_33` bigint(20) NULL COMMENT "", `col_34` decimal64(18, 6) NULL COMMENT "", `col_35` varchar(8) NULL COMMENT "", `col_36` decimal64(18, 6) NULL COMMENT "", `col_37` decimal64(18, 6) NULL COMMENT "", `col_38` varchar(8) NULL COMMENT "", `col_39` decimal64(18, 6) NULL COMMENT "", `col_40` decimal64(18, 6) NULL COMMENT "", `col_41` varchar(8) NULL COMMENT "", `col_42` decimal64(18, 6) NULL COMMENT "", `col_43` decimal64(18, 6) NULL COMMENT "", `col_44` decimal64(18, 6) NULL COMMENT "", `col_45` decimal64(18, 6) NULL COMMENT "", `col_46` int(11) NULL COMMENT "", `col_47` int(11) NOT NULL COMMENT "", `col_48` tinyint(4) NULL COMMENT "", `col_49` varchar(200) NULL COMMENT "", `col_50` tinyint(4) NULL COMMENT "", `col_51` varchar(200) NULL COMMENT "", `col_52` varchar(10) NULL COMMENT "", `col_53` tinyint(4) NULL COMMENT "", `col_54` tinyint(4) NULL COMMENT "", `col_55` varchar(150) NULL COMMENT "", `col_56` varchar(150) NULL COMMENT "", `col_57` varchar(500) NULL COMMENT "", `col_58` tinyint(4) NULL COMMENT "", `col_59` varchar(100) NULL COMMENT "", `col_60` varchar(150) NULL COMMENT "", `col_61` varchar(150) NULL COMMENT "", `col_62` varchar(150) NULL COMMENT "", `col_63` varchar(150) NULL COMMENT "", `col_64` datetime NULL COMMENT "", `col_65` datetime NULL COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`col_1`, `col_2`, `col_3`) COMMENT "OLAP" DISTRIBUTED BY HASH(`col_1`, `col_2`) BUCKETS 20 PROPERTIES ( "replication_num" = "1", "in_memory" = "false", "storage_format" = "V2", "enable_persistent_index" = "true", "compression" = "LZ4" ); ``` |PrimaryKey Length| RowNum|BucketNum| Column Num| Partial ColumnNum | PartialUpdate RowsNum| Load time(s)| Apply time(ms)| Peak UpdateMemory usage | Note | |---------------------|----------|------------|----------------|--------------------|------------------------------|----|-----|-----|----| |12 Bytes| 300M | 20 | 65 | 5 | 100M | 135261 | 106693 | 78.9G | branch-main | |12 Bytes| 300M | 20 | 65 | 5 | 100M | 166449| 149870 | 10.3G | branch-opt | |12 Bytes| 300M | 20 | 65 | 5 | 100K | 2078 | 529 | 60.1M | branch-main | |12 Bytes| 300M | 20 | 65 | 5 | 100K | 2211 | 541 | 60.2M | branch-opt |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this：
Which issues of this PR fixes ：
Fixes #9344
Problem Summary(Required) ：
Currently, RowsetUpdateState::load will preload all segments primary keys into memory, if the load(rowset) is very large, it will use a lot of memory during the commit or apply phrase.
For large load(rowset), we don't preload all segment's primary keys but process segment by segment, which can reduce the memory usage during apply.
It is important to note that the limitation is a soft limit because we can't tolerate the failure to apply, so memory usage may still exceed the limitation.
In my test env, one BE with two HDD, using Broker load, create a table with persistent index:
use tpcds to test
create table sql, using broker load:
Note there are still some scenarios we don't resolve in this pr:
L0of persistent index first which maybe cause OOMChecklist: