[Enhancement] Dump a new snapshot if persistent index keep too many WAL #22140
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: zhangqiang <[email protected]>
sevev
changed the title
chaoyli
reviewed
Apr 24, 2023
Signed-off-by: zhangqiang <[email protected]>
decster
approved these changes
May 9, 2023
chaoyli
approved these changes
May 10, 2023
abc982627271
pushed a commit
to abc982627271/starrocks
that referenced
this pull request
May 11, 2023
…AL (StarRocks#22140) Persistent Index for the primary key table will append wal log into `L0` index file if the following conditions are met: 1. The memory usage of `L0` is less than 10% of `L1` usage 3. The memory usage of `L0` is greater than max snapshot size(default is 16MB) So the `L0` index file may keep growing, and keep appending WAL. And this PR(StarRocks#12862) triggers flush or merge compaction if the `L0` file size is greater than 200MB to avoid the growth of `L0` index file. However, if the file size of `L1` is very large(e.g. 4GB) and the `L0` memory usage is less than `L1` 10% usage but greater than 200MB(e.g. 256MB), we will always flush a new snapshot(about 256MB) even if we only update one record which causes a lot of disk IO. So this pr change the check and dump a new snapshot if the `L0` index file keep too many wal logs.
|
@mergify backport branch-3.0 |
✅ Backports have been created
|
mergify bot
pushed a commit
that referenced
this pull request
May 22, 2023
…AL (#22140) Persistent Index for the primary key table will append wal log into `L0` index file if the following conditions are met: 1. The memory usage of `L0` is less than 10% of `L1` usage 3. The memory usage of `L0` is greater than max snapshot size(default is 16MB) So the `L0` index file may keep growing, and keep appending WAL. And this PR(#12862) triggers flush or merge compaction if the `L0` file size is greater than 200MB to avoid the growth of `L0` index file. However, if the file size of `L1` is very large(e.g. 4GB) and the `L0` memory usage is less than `L1` 10% usage but greater than 200MB(e.g. 256MB), we will always flush a new snapshot(about 256MB) even if we only update one record which causes a lot of disk IO. So this pr change the check and dump a new snapshot if the `L0` index file keep too many wal logs. (cherry picked from commit 817aa23)
|
@mergify backport branch-2.5 |
✅ Backports have been created
|
mergify bot
pushed a commit
that referenced
this pull request
May 22, 2023
…AL (#22140) Persistent Index for the primary key table will append wal log into `L0` index file if the following conditions are met: 1. The memory usage of `L0` is less than 10% of `L1` usage 3. The memory usage of `L0` is greater than max snapshot size(default is 16MB) So the `L0` index file may keep growing, and keep appending WAL. And this PR(#12862) triggers flush or merge compaction if the `L0` file size is greater than 200MB to avoid the growth of `L0` index file. However, if the file size of `L1` is very large(e.g. 4GB) and the `L0` memory usage is less than `L1` 10% usage but greater than 200MB(e.g. 256MB), we will always flush a new snapshot(about 256MB) even if we only update one record which causes a lot of disk IO. So this pr change the check and dump a new snapshot if the `L0` index file keep too many wal logs. (cherry picked from commit 817aa23)
sevev
added a commit
to sevev/starrocks
that referenced
this pull request
May 23, 2023
…AL (StarRocks#22140) Persistent Index for the primary key table will append wal log into `L0` index file if the following conditions are met: 1. The memory usage of `L0` is less than 10% of `L1` usage 3. The memory usage of `L0` is greater than max snapshot size(default is 16MB) So the `L0` index file may keep growing, and keep appending WAL. And this PR(StarRocks#12862) triggers flush or merge compaction if the `L0` file size is greater than 200MB to avoid the growth of `L0` index file. However, if the file size of `L1` is very large(e.g. 4GB) and the `L0` memory usage is less than `L1` 10% usage but greater than 200MB(e.g. 256MB), we will always flush a new snapshot(about 256MB) even if we only update one record which causes a lot of disk IO. So this pr change the check and dump a new snapshot if the `L0` index file keep too many wal logs. Signed-off-by: zhangqiang <[email protected]>
sevev
added a commit
to sevev/starrocks
that referenced
this pull request
May 23, 2023
…AL (StarRocks#22140) Persistent Index for the primary key table will append wal log into `L0` index file if the following conditions are met: 1. The memory usage of `L0` is less than 10% of `L1` usage 3. The memory usage of `L0` is greater than max snapshot size(default is 16MB) So the `L0` index file may keep growing, and keep appending WAL. And this PR(StarRocks#12862) triggers flush or merge compaction if the `L0` file size is greater than 200MB to avoid the growth of `L0` index file. However, if the file size of `L1` is very large(e.g. 4GB) and the `L0` memory usage is less than `L1` 10% usage but greater than 200MB(e.g. 256MB), we will always flush a new snapshot(about 256MB) even if we only update one record which causes a lot of disk IO. So this pr change the check and dump a new snapshot if the `L0` index file keep too many wal logs. Signed-off-by: zhangqiang <[email protected]>
sevev
added a commit
to sevev/starrocks
that referenced
this pull request
May 23, 2023
…AL (StarRocks#22140) Persistent Index for the primary key table will append wal log into `L0` index file if the following conditions are met: 1. The memory usage of `L0` is less than 10% of `L1` usage 3. The memory usage of `L0` is greater than max snapshot size(default is 16MB) So the `L0` index file may keep growing, and keep appending WAL. And this PR(StarRocks#12862) triggers flush or merge compaction if the `L0` file size is greater than 200MB to avoid the growth of `L0` index file. However, if the file size of `L1` is very large(e.g. 4GB) and the `L0` memory usage is less than `L1` 10% usage but greater than 200MB(e.g. 256MB), we will always flush a new snapshot(about 256MB) even if we only update one record which causes a lot of disk IO. So this pr change the check and dump a new snapshot if the `L0` index file keep too many wal logs. Signed-off-by: zhangqiang <[email protected]>
sevev
added a commit
to sevev/starrocks
that referenced
this pull request
May 23, 2023
…AL (StarRocks#22140) Persistent Index for the primary key table will append wal log into `L0` index file if the following conditions are met: 1. The memory usage of `L0` is less than 10% of `L1` usage 3. The memory usage of `L0` is greater than max snapshot size(default is 16MB) So the `L0` index file may keep growing, and keep appending WAL. And this PR(StarRocks#12862) triggers flush or merge compaction if the `L0` file size is greater than 200MB to avoid the growth of `L0` index file. However, if the file size of `L1` is very large(e.g. 4GB) and the `L0` memory usage is less than `L1` 10% usage but greater than 200MB(e.g. 256MB), we will always flush a new snapshot(about 256MB) even if we only update one record which causes a lot of disk IO. So this pr change the check and dump a new snapshot if the `L0` index file keep too many wal logs. Signed-off-by: zhangqiang <[email protected]>
Moonm3n
pushed a commit
to Moonm3n/starrocks
that referenced
this pull request
May 23, 2023
…AL (StarRocks#22140) Persistent Index for the primary key table will append wal log into `L0` index file if the following conditions are met: 1. The memory usage of `L0` is less than 10% of `L1` usage 3. The memory usage of `L0` is greater than max snapshot size(default is 16MB) So the `L0` index file may keep growing, and keep appending WAL. And this PR(StarRocks#12862) triggers flush or merge compaction if the `L0` file size is greater than 200MB to avoid the growth of `L0` index file. However, if the file size of `L1` is very large(e.g. 4GB) and the `L0` memory usage is less than `L1` 10% usage but greater than 200MB(e.g. 256MB), we will always flush a new snapshot(about 256MB) even if we only update one record which causes a lot of disk IO. So this pr change the check and dump a new snapshot if the `L0` index file keep too many wal logs. Signed-off-by: Moonm3n <[email protected]>
decster
pushed a commit
that referenced
this pull request
May 25, 2023
…nt index keep too many WAL (#22140) (#23922) Persistent Index for the primary key table will append wal log into `L0` index file if the following conditions are met: 1. The memory usage of `L0` is less than 10% of `L1` usage 3. The memory usage of `L0` is greater than max snapshot size(default is 16MB) So the `L0` index file may keep growing, and keep appending WAL. And this PR(#12862) triggers flush or merge compaction if the `L0` file size is greater than 200MB to avoid the growth of `L0` index file. However, if the file size of `L1` is very large(e.g. 4GB) and the `L0` memory usage is less than `L1` 10% usage but greater than 200MB(e.g. 256MB), we will always flush a new snapshot(about 256MB) even if we only update one record which causes a lot of disk IO. So this pr change the check and dump a new snapshot if the `L0` index file keep too many wal logs. Signed-off-by: zhangqiang <[email protected]>
decster
pushed a commit
that referenced
this pull request
May 25, 2023
…nt index keep too many WAL (#22140) (#23920) Persistent Index for the primary key table will append wal log into `L0` index file if the following conditions are met: 1. The memory usage of `L0` is less than 10% of `L1` usage 3. The memory usage of `L0` is greater than max snapshot size(default is 16MB) So the `L0` index file may keep growing, and keep appending WAL. And this PR(#12862) triggers flush or merge compaction if the `L0` file size is greater than 200MB to avoid the growth of `L0` index file. However, if the file size of `L1` is very large(e.g. 4GB) and the `L0` memory usage is less than `L1` 10% usage but greater than 200MB(e.g. 256MB), we will always flush a new snapshot(about 256MB) even if we only update one record which causes a lot of disk IO. So this pr change the check and dump a new snapshot if the `L0` index file keep too many wal logs. Signed-off-by: zhangqiang <[email protected]>
numbernumberone
pushed a commit
to numbernumberone/starrocks
that referenced
this pull request
May 31, 2023
…AL (StarRocks#22140) Persistent Index for the primary key table will append wal log into `L0` index file if the following conditions are met: 1. The memory usage of `L0` is less than 10% of `L1` usage 3. The memory usage of `L0` is greater than max snapshot size(default is 16MB) So the `L0` index file may keep growing, and keep appending WAL. And this PR(StarRocks#12862) triggers flush or merge compaction if the `L0` file size is greater than 200MB to avoid the growth of `L0` index file. However, if the file size of `L1` is very large(e.g. 4GB) and the `L0` memory usage is less than `L1` 10% usage but greater than 200MB(e.g. 256MB), we will always flush a new snapshot(about 256MB) even if we only update one record which causes a lot of disk IO. So this pr change the check and dump a new snapshot if the `L0` index file keep too many wal logs.
abc982627271
pushed a commit
to abc982627271/starrocks
that referenced
this pull request
Jun 5, 2023
…AL (StarRocks#22140) Persistent Index for the primary key table will append wal log into `L0` index file if the following conditions are met: 1. The memory usage of `L0` is less than 10% of `L1` usage 3. The memory usage of `L0` is greater than max snapshot size(default is 16MB) So the `L0` index file may keep growing, and keep appending WAL. And this PR(StarRocks#12862) triggers flush or merge compaction if the `L0` file size is greater than 200MB to avoid the growth of `L0` index file. However, if the file size of `L1` is very large(e.g. 4GB) and the `L0` memory usage is less than `L1` 10% usage but greater than 200MB(e.g. 256MB), we will always flush a new snapshot(about 256MB) even if we only update one record which causes a lot of disk IO. So this pr change the check and dump a new snapshot if the `L0` index file keep too many wal logs.
southernriver
pushed a commit
to southernriver/starrocks
that referenced
this pull request
Nov 28, 2023
…AL (StarRocks#22140) Persistent Index for the primary key table will append wal log into `L0` index file if the following conditions are met: 1. The memory usage of `L0` is less than 10% of `L1` usage 3. The memory usage of `L0` is greater than max snapshot size(default is 16MB) So the `L0` index file may keep growing, and keep appending WAL. And this PR(StarRocks#12862) triggers flush or merge compaction if the `L0` file size is greater than 200MB to avoid the growth of `L0` index file. However, if the file size of `L1` is very large(e.g. 4GB) and the `L0` memory usage is less than `L1` 10% usage but greater than 200MB(e.g. 256MB), we will always flush a new snapshot(about 256MB) even if we only update one record which causes a lot of disk IO. So this pr change the check and dump a new snapshot if the `L0` index file keep too many wal logs. (cherry picked from commit 817aa23)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this：
Which issues of this PR fixes ：
Fixes #
Problem Summary(Required) ：
Persistent Index for the primary key table will append wal log into
L0index file if the following conditions are met:L0is less than 10% ofL1usageL0is greater than max snapshot size(default is 16MB)So the
L0index file may keep growing, and keep appending WAL. And this PR(#12862) triggers flush or merge compaction if theL0file size is greater than 200MB to avoid the growth ofL0index file.However, if the file size of
L1is very large(e.g. 4GB) and theL0memory usage is less thanL110% usage but greater than 200MB(e.g. 256MB), we will always flush a new snapshot(about 256MB) even if we only update one record which causes a lot of disk IO.So this pr change the check and dump a new snapshot if the
L0index file keep too many wal logs.Checklist:
Bugfix cherry-pick branch check: