[Feature] Automatically specifying tablet when create table #10614

chaoyli · 2022-08-30T13:43:06Z

What type of PR is this：

Which issues of this PR fixes ：

Fixes #
Specifying the tablet num when creating a table is frustrating.
Many users can not know how to give the right hint.
The pull request tries to eliminate the pain by automatically specifying the num.

If the table is created from scratch, the tablet num will be
speculated according to the number of backends.

CREATE TABLE `test_table2` (
    `pk1` bigint(20) NOT NULL COMMENT "",
    `pk2` date NOT NULL COMMENT "",
    `v0` varchar(65533) NOT NULL COMMENT ""
) ENGINE=OLAP
PRIMARY KEY(`pk1`, `pk2`)
DISTRIBUTED BY HASH(`pk1`)

 if (backendNum <= 3) {
    bucketNum = 2 * backendNum;
 } else if (backendNum <= 6) {
    bucketNum = (int) 1.5 * backendNum;
 } else if (backendNum <= 12) {
    bucketNum = 12;
 }  else {
    bucketNum = Math.min(backendNum, 48);
 }

Add partition will use the last five partition sizes to calculate the tablet num.
```
ALTER TABLE test_table2 add partition p20220811 values less than("2022-08-11")
```
The num = total_size_of_last_five_partition / 1GB

Problem Summary(Required) ：

Checklist:

I have added test cases for my bug fix or my new feature
I have added user document for my new feature or new function

fe/fe-core/src/main/cup/sql_parser.cup

fe/fe-core/src/main/java/com/starrocks/server/LocalMetastore.java

howrocks · 2022-09-08T05:40:20Z

fe/fe-core/src/main/java/com/starrocks/server/LocalMetastore.java

+            // The bucketNum will be given a small multiplier factor for small backends.
+            if (backendNum <= recentPartitionNum) {
+                avgBucketNum = 2 * backendNum;
+            } else if (backendNum <= 6) {


1.5x and 1.2x may cause very unbalanced load between BEs when PoC testing.

The PoC will only have the same backends. So I think the bucket is enough. And also the PoC backend will not be installed many nodes

fe/fe-core/src/main/java/com/starrocks/catalog/OlapTable.java

fe/fe-core/src/main/java/com/starrocks/server/LocalMetastore.java

fe/fe-core/src/main/cup/sql_parser.cup

fe/fe-core/src/main/java/com/starrocks/server/LocalMetastore.java

chaoyli · 2022-09-09T11:44:32Z

run starrocks_fe_unittest

Specfiy the tablet num when create table is very noisy. Many users can not know how to give a right hint. The pull request try to eliminate the pain to specify the num. 1. If the table is created from scratch, the tablet num will be equal to the number of backends. ``` CREATE TABLE `test_table2` ( `pk1` bigint(20) NOT NULL COMMENT "", `pk2` date NOT NULL COMMENT "", `v0` varchar(65533) NOT NULL COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`pk1`, `pk2`) DISTRIBUTED BY HASH(`pk1`) ``` 2. Add partition will use the last five partition size to calculate the tablet num. ``` ALTER TABLE test_table2 add partition p20220811 values less than("2022-08-11") ``` The num = total_size_of_last_five_partition / 10GB

sonarcloud · 2022-09-15T03:11:27Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

0.0% Coverage
0.0% Duplication

wanpengfei-git · 2022-09-15T03:17:54Z

[FE PR Coverage Check]

😞 fail : 24 / 51 (47.06%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/catalog/OlapTable.java	0	4	00.00%	[889, 890, 893, 896]
🔵	com/starrocks/catalog/RandomDistributionInfo.java	0	2	00.00%	[62, 63]
🔵	com/starrocks/catalog/DistributionInfo.java	0	1	00.00%	[71]
🔵	com/starrocks/server/LocalMetastore.java	19	39	48.72%	[949, 953, 967, 968, 969, 970, 971, 973, 975, 976, 977, 981, 982, 983, 984, 986, 987, 988, 990, 1017]
🔵	com/starrocks/sql/parser/AstBuilder.java	1	1	100.00%	[]
🔵	com/starrocks/analysis/HashDistributionDesc.java	1	1	100.00%	[]
🔵	com/starrocks/catalog/HashDistributionInfo.java	3	3	100.00%	[]

wanpengfei-git · 2022-09-15T05:33:31Z

run starrocks_admit_test

nshangyiming · 2022-09-15T07:40:54Z

Shall we update the corresponding documentation?

…et num (#11337) In this PR [Feature] Automatically specifying tablet when create table #10614, SR set default bucket num from 10 to 0, this would result in create mv failed when user not set bucket num. according to that PR, we use be numner to calculate bucket num SR do not support create mv on view, so we need to disable it at analyze stage

chaoyli force-pushed the main branch 3 times, most recently from fc4f7ae to 8317546 Compare August 30, 2022 13:49

chaoyli changed the title ~~[Feature] Automatically create tablet when create table~~ [Feature] Automatically specifying tablet when create table Aug 30, 2022

howrocks reviewed Aug 31, 2022

View reviewed changes

fe/fe-core/src/main/cup/sql_parser.cup Outdated Show resolved Hide resolved

howrocks reviewed Aug 31, 2022

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/server/LocalMetastore.java Outdated Show resolved Hide resolved

fe/fe-core/src/main/java/com/starrocks/server/LocalMetastore.java Outdated Show resolved Hide resolved

howrocks reviewed Sep 3, 2022

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/server/LocalMetastore.java Outdated Show resolved Hide resolved

chaoyli force-pushed the main branch 2 times, most recently from 8eed501 to ae0476f Compare September 7, 2022 13:28

howrocks reviewed Sep 8, 2022

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/server/LocalMetastore.java Outdated Show resolved Hide resolved

howrocks reviewed Sep 8, 2022

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/catalog/OlapTable.java Outdated Show resolved Hide resolved

howrocks reviewed Sep 8, 2022

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/server/LocalMetastore.java Outdated Show resolved Hide resolved

howrocks reviewed Sep 8, 2022

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/server/LocalMetastore.java Show resolved Hide resolved

chaoyli force-pushed the main branch from ae0476f to 7c22f9f Compare September 9, 2022 07:50

trueeyu reviewed Sep 9, 2022

View reviewed changes

fe/fe-core/src/main/cup/sql_parser.cup Show resolved Hide resolved

trueeyu reviewed Sep 9, 2022

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/server/LocalMetastore.java Show resolved Hide resolved

trueeyu reviewed Sep 9, 2022

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/server/LocalMetastore.java Outdated Show resolved Hide resolved

chaoyli force-pushed the main branch 3 times, most recently from d5a5b62 to 10ba730 Compare September 9, 2022 09:01

trueeyu previously approved these changes Sep 9, 2022

View reviewed changes

chaoyli dismissed trueeyu’s stale review via 7fc71a3 September 9, 2022 11:26

chaoyli force-pushed the main branch from 10ba730 to 7fc71a3 Compare September 9, 2022 11:26

trueeyu previously approved these changes Sep 9, 2022

View reviewed changes

chaoyli dismissed trueeyu’s stale review via 4ca06ff September 14, 2022 07:10

chaoyli force-pushed the main branch 3 times, most recently from d4f5b7b to 000bb72 Compare September 14, 2022 12:02

chaoyli added 4 commits September 15, 2022 10:45

[Feature] Automatically create tablet when create table

abcadf2

[Feature] Automatically create tablet when create table

5bdc2b7

[Feature] Automatically create tablet when create table

3338273

chaoyli force-pushed the main branch from 000bb72 to 3338273 Compare September 15, 2022 03:01

trueeyu approved these changes Sep 15, 2022

View reviewed changes

Astralidea approved these changes Sep 15, 2022

View reviewed changes

wanpengfei-git added the Approved Ready to merge label Sep 15, 2022

chaoyli enabled auto-merge (squash) September 15, 2022 05:57

chaoyli merged commit 8c19e6b into StarRocks:main Sep 15, 2022

Youngwb mentioned this pull request Sep 17, 2022

[BugFix] Fix unknown error when create materialized view without bucket num #11337

Merged

9 tasks

meegoo mentioned this pull request Aug 16, 2024

[BugFix] Fix alter table distribution npe since it is not used #49906

Merged

26 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Automatically specifying tablet when create table #10614

[Feature] Automatically specifying tablet when create table #10614

chaoyli commented Aug 30, 2022 •

edited

Loading

howrocks Sep 8, 2022

chaoyli Sep 9, 2022

chaoyli commented Sep 9, 2022

sonarcloud bot commented Sep 15, 2022

wanpengfei-git commented Sep 15, 2022

wanpengfei-git commented Sep 15, 2022

nshangyiming commented Sep 15, 2022

[Feature] Automatically specifying tablet when create table #10614

[Feature] Automatically specifying tablet when create table #10614

Conversation

chaoyli commented Aug 30, 2022 • edited Loading

What type of PR is this：

Which issues of this PR fixes ：

Problem Summary(Required) ：

Checklist:

howrocks Sep 8, 2022

Choose a reason for hiding this comment

chaoyli Sep 9, 2022

Choose a reason for hiding this comment

chaoyli commented Sep 9, 2022

sonarcloud bot commented Sep 15, 2022

wanpengfei-git commented Sep 15, 2022

[FE PR Coverage Check]

file detail

wanpengfei-git commented Sep 15, 2022

nshangyiming commented Sep 15, 2022

chaoyli commented Aug 30, 2022 •

edited

Loading