Add dynamically adjust the upper limit of the data that each layer configure option. #497

WyattJia · 2022-02-24T14:02:44Z

Levels' Target Size level_compaction_dynamic_level_bytes
Reference issue: [NEW] Supports to configure the rocksdb advance option max_bytes_for_level_base #488

Please check and review, thanks.

…eatures.

Update Rocksdb options, enable level_compaction_dynamic_level_bytes f…

git-hulk · 2022-02-24T15:38:53Z

Cool, thanks for your contribution.

git-hulk · 2022-02-24T15:53:12Z

@WyattJia I have a look at the PR, overall was good to me. But we need to fix those lint issues first:

src/config.cc:152:  Lines should be <= 120 characters long  [whitespace/line_length] [2]
src/config.cc:153:  Lines should be <= 120 characters long  [whitespace/line_length] [2]
src/config.cc:427:  Line ends in whitespace.  Consider deleting these extra spaces.  [whitespace/end_of_line] [4]
src/config.cc:428:  Lines should be <= 120 characters long  [whitespace/line_length] [2]

WyattJia · 2022-02-25T01:17:59Z

@WyattJia I have a look at the PR, overall was good to me. But we need to fix those lint issues first:

src/config.cc:152:  Lines should be <= 120 characters long  [whitespace/line_length] [2]
src/config.cc:153:  Lines should be <= 120 characters long  [whitespace/line_length] [2]
src/config.cc:427:  Line ends in whitespace.  Consider deleting these extra spaces.  [whitespace/end_of_line] [4]
src/config.cc:428:  Lines should be <= 120 characters long  [whitespace/line_length] [2]

Copy that, I've formated and pushed code, thanks for your review.

kvrocks.conf

src/config.h

src/config.cc

kvrocks.conf

Co-authored-by: Myth <[email protected]>

caipengbo

Good job!

src/config.cc

tests/config_test.cc

src/config.cc

.gitignore

Co-authored-by: Wang Yuan <[email protected]>

src/config.cc

kvrocks.conf

git-hulk · 2022-03-06T08:01:47Z

cool, also need to multiple the MiB in config's callback like RocksDB.write_buffer_size

https://github.com/KvrocksLabs/kvrocks/blob/unstable/src/config.cc#L346

WyattJia · 2022-03-06T08:17:02Z

cool, also need to multiple the MiB in config's callback like RocksDB.write_buffer_size

https://github.com/KvrocksLabs/kvrocks/blob/unstable/src/config.cc#L346

Done and thank you for your reminder.

git-hulk · 2022-03-06T09:06:26Z

@WyattJia Cool, really thanks for your contribution and patient.

git-hulk · 2022-03-06T09:08:47Z

@ShooterIT Please take a look again

ShooterIT

LGTM

ShooterIT · 2022-04-28T11:16:22Z

src/config.cc

+        if (!RocksDB.level_compaction_dynamic_level_bytes) {
+          return Status(Status::NotOK, errNotSetLevelCompactionDynamicLevelBytes);
+        }


this judgement is necessary? a mistake?

ShooterIT · 2022-04-28T11:19:36Z

kvrocks.conf

+# The total file size of level-1 sst.
+#
+# Default: 256 M
+rocksdb.max_bytes_for_level_base 256


since this config has bytes word, we should use byte unit, right? or we rename it to level_base_size? maybe rocksdb name is better?

This commit is not released in newest version, we have chances to change.

Yeah, I also think bytes is better.

ShooterIT · 2022-04-28T11:20:29Z

@WyattJia have a look?

WyattJia added 2 commits February 24, 2022 18:14

Update Rocksdb options, enable level_compaction_dynamic_level_bytes f…

b3e1fb3

…eatures.

Merge pull request #1 from WyattJia/max_bytes_for_level_base

bd2e708

Update Rocksdb options, enable level_compaction_dynamic_level_bytes f…

git-hulk requested review from ShooterIT and git-hulk February 24, 2022 15:39

git-hulk added the feature type new feature label Feb 24, 2022

Format config.cc.

eacb3ea

Lint config.cc, remove useless comma.

bf8c93f

caipengbo reviewed Feb 25, 2022

View reviewed changes

kvrocks.conf Outdated Show resolved Hide resolved

src/config.h Show resolved Hide resolved

src/config.cc Outdated Show resolved Hide resolved

git-hulk reviewed Feb 25, 2022

View reviewed changes

src/config.cc Outdated Show resolved Hide resolved

src/config.cc Outdated Show resolved Hide resolved

WyattJia added 2 commits February 25, 2022 14:27

Update conf, rewrite levels target size option comments.

8aa0025

Update level_compaction_dynamic_level_bytes config load.

d156b9c

caipengbo previously approved these changes Feb 25, 2022

View reviewed changes

kvrocks.conf Outdated Show resolved Hide resolved

kvrocks.conf Outdated Show resolved Hide resolved

kvrocks.conf Outdated Show resolved Hide resolved

Update kvrocks.conf

d5aa2f5

Co-authored-by: Myth <[email protected]>

WyattJia dismissed caipengbo’s stale review via d5aa2f5 February 25, 2022 06:36

WyattJia and others added 2 commits February 25, 2022 14:36

Update kvrocks.conf

d432188

Co-authored-by: Myth <[email protected]>

Update kvrocks.conf

335c033

Co-authored-by: Myth <[email protected]>

WyattJia changed the title ~~Add rocksdb level_compaction_dynamic_level_bytes option configuration support.~~ Add dynamically adjust the upper limit of the data that each layer configure option. Feb 25, 2022

caipengbo previously approved these changes Feb 25, 2022

View reviewed changes

Update levels target size option comments.

612e690

WyattJia dismissed caipengbo’s stale review via 612e690 February 25, 2022 06:51

WyattJia added 3 commits February 25, 2022 15:02

Update levels target size option comments.

de62f14

Update level_compaction_dynamic_level_bytes config load.

dceca92

Fix level_compaction_dynamic_level_bytes conf value error.

d78c3df

WyattJia requested a review from git-hulk February 25, 2022 08:42

Lint config.cc

14a728c

git-hulk reviewed Feb 25, 2022

View reviewed changes

src/config.cc Outdated Show resolved Hide resolved

Update src/config.cc

d4553b4

WyattJia added 2 commits February 25, 2022 19:37

Fix configura option unit error.

27df4cb

Revert blob_file_size default unit error.

30646b7

git-hulk reviewed Feb 25, 2022

View reviewed changes

tests/config_test.cc Outdated Show resolved Hide resolved

Update tests/config_test.cc

31e8773

git-hulk reviewed Feb 25, 2022

View reviewed changes

src/config.cc Outdated Show resolved Hide resolved

Update src/config.cc

278a3c1

git-hulk reviewed Feb 25, 2022

View reviewed changes

src/config.cc Outdated Show resolved Hide resolved

Update src/config.cc

c73f2cf

git-hulk previously approved these changes Feb 26, 2022

View reviewed changes

ShooterIT reviewed Feb 28, 2022

View reviewed changes

.gitignore Outdated Show resolved Hide resolved

Update .gitignore

31eb562

Co-authored-by: Wang Yuan <[email protected]>

WyattJia dismissed git-hulk’s stale review via 31eb562 February 28, 2022 01:53

ShooterIT reviewed Feb 28, 2022

View reviewed changes

src/config.cc Outdated Show resolved Hide resolved

kvrocks.conf Outdated Show resolved Hide resolved

WyattJia added 4 commits February 28, 2022 10:27

Update config.cc

d2f7ad2

Update kvrocks.conf

4fc9bbf

Update kvrocks.conf

a35ad25

Update max_bytes_for_level_base unit bytes to MiB.

b695d71

Fix config.cc, max_bytes_for_level_base unit bytes to MiB.

004c97e

ShooterIT approved these changes Mar 6, 2022

View reviewed changes

git-hulk approved these changes Mar 6, 2022

View reviewed changes

git-hulk merged commit 84cdbc7 into apache:unstable Mar 6, 2022

ShooterIT reviewed Apr 28, 2022

View reviewed changes

git-hulk mentioned this pull request May 28, 2022

Change the unit of the rocksdb options #612

Closed

2 tasks

tisonkun mentioned this pull request Mar 18, 2023

[NEW] Supports to configure the rocksdb advance option max_bytes_for_level_base #488

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add dynamically adjust the upper limit of the data that each layer configure option. #497

Add dynamically adjust the upper limit of the data that each layer configure option. #497

WyattJia commented Feb 24, 2022

git-hulk commented Feb 24, 2022

git-hulk commented Feb 24, 2022 •

edited

Loading

WyattJia commented Feb 25, 2022

caipengbo left a comment

git-hulk commented Mar 6, 2022

WyattJia commented Mar 6, 2022

git-hulk commented Mar 6, 2022

git-hulk commented Mar 6, 2022

ShooterIT left a comment

ShooterIT Apr 28, 2022

ShooterIT Apr 28, 2022

caipengbo Apr 28, 2022

ShooterIT commented Apr 28, 2022

Add dynamically adjust the upper limit of the data that each layer configure option. #497

Add dynamically adjust the upper limit of the data that each layer configure option. #497

Conversation

WyattJia commented Feb 24, 2022

git-hulk commented Feb 24, 2022

git-hulk commented Feb 24, 2022 • edited Loading

WyattJia commented Feb 25, 2022

caipengbo left a comment

Choose a reason for hiding this comment

git-hulk commented Mar 6, 2022

WyattJia commented Mar 6, 2022

git-hulk commented Mar 6, 2022

git-hulk commented Mar 6, 2022

ShooterIT left a comment

Choose a reason for hiding this comment

ShooterIT Apr 28, 2022

Choose a reason for hiding this comment

ShooterIT Apr 28, 2022

Choose a reason for hiding this comment

caipengbo Apr 28, 2022

Choose a reason for hiding this comment

ShooterIT commented Apr 28, 2022

git-hulk commented Feb 24, 2022 •

edited

Loading