Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(compaction): Use separate compactors for L0, L1 #1466

Merged
merged 7 commits into from
Aug 19, 2020
Merged

fix(compaction): Use separate compactors for L0, L1 #1466

merged 7 commits into from
Aug 19, 2020

Conversation

jarifibrahim
Copy link
Contributor

@jarifibrahim jarifibrahim commented Aug 19, 2020 •
edited by manishrjain
Loading

Related to #1459

This PR contains the following changes to compactions

  • Use a separate thread for compacting Level 0 and 1 and a separate one for other levels
  • Pick levels to compact based on score.
  • Stall Level 0 if compactions cannot keep up (we had added this in Do not stall Level 0 and 1 #1186)
  • Limit the number of open table builders to 5 in compactions.

This change is Reviewable

@jarifibrahim jarifibrahim mentioned this pull request Aug 19, 2020
Merged
manishrjain
manishrjain approved these changes Aug 19, 2020
View reviewed changes
Copy link
Contributor

@manishrjain manishrjain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 3 of 5 files at r1, 3 of 3 files at r2.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on @ashish-goswami)

@jarifibrahim jarifibrahim merged commit 0b8eb4c into master Aug 19, 2020
@jarifibrahim jarifibrahim deleted the ibrahim/compaction-change branch August 19, 2020 13:12
jarifibrahim pushed a commit that referenced this pull request Aug 19, 2020
Improvements: Manual Memory allocation via Calloc (#1459)
5c1b4c8 
This PR contains a  bunch of improvements
1. This PR reverts the following reverts (adds them back)
    - Revert "Compress/Encrypt Blocks in the background  6001230
    - Revert "Buffer pool for decompression 800305e
   - Revert "fix: Fix race condition in block.incRef 63d9309
    - Revert "add assert to check integer overflow for table size e0d058c
2. Calloc and Free for memory allocation
    - The block buffers for decompression and encryption are now allocated using calloc and free.
      The `y/calloc.go` file contains the code.

The compaction changes are moved to a separate PR for easier cherry-picking
#1466
jarifibrahim pushed a commit that referenced this pull request Aug 20, 2020
fix(compaction): Use separate compactors for L0, L1 (#1466)
6f27a92 
Related to #1459

This PR contains the following changes to compactions
- Use a separate thread for compacting Level 0 and 1 and a separate one for other levels
- Pick levels to compact based on score.
- Stall Level 0 if compactions cannot keep up (we had added this in #1186)
- Limit the number of open table builders to 5 in compactions.

(cherry picked from commit 0b8eb4c)
jarifibrahim pushed a commit that referenced this pull request Aug 25, 2020
fix(compaction): Use separate compactors for L0, L1 (#1466) (#1468)
6d05358 
Related to #1459

This PR contains the following changes to compactions
- Use a separate thread for compacting Level 0 and 1 and a separate one for other levels
- Pick levels to compact based on score.
- Stall Level 0 if compactions cannot keep up (we had added this in #1186)
- Limit the number of open table builders to 5 in compactions.

(cherry picked from commit 0b8eb4c)
@jarifibrahim jarifibrahim mentioned this pull request Sep 28, 2020
Closed
jarifibrahim pushed a commit that referenced this pull request Oct 2, 2020
fix(compaction): Use separate compactors for L0, L1 (#1466)
082f287 
Related to #1459

This PR contains the following changes to compactions
- Use a separate thread for compacting Level 0 and 1 and a separate one for other levels
- Pick levels to compact based on score.
- Stall Level 0 if compactions cannot keep up (we had added this in #1186)
- Limit the number of open table builders to 5 in compactions.
jarifibrahim pushed a commit that referenced this pull request Oct 2, 2020
Improvements: Manual Memory allocation via Calloc (#1459)
1abe0a8 
This PR contains a  bunch of improvements
1. This PR reverts the following reverts (adds them back)
    - Revert "Compress/Encrypt Blocks in the background  6001230
    - Revert "Buffer pool for decompression 800305e
   - Revert "fix: Fix race condition in block.incRef 63d9309
    - Revert "add assert to check integer overflow for table size e0d058c
2. Calloc and Free for memory allocation
    - The block buffers for decompression and encryption are now allocated using calloc and free.
      The `y/calloc.go` file contains the code.

The compaction changes are moved to a separate PR for easier cherry-picking
#1466
manishrjain pushed a commit that referenced this pull request Oct 15, 2020
@manishrjain
fix(compaction): Use separate compactors for L0, L1 (#1466)
dab2927 
Related to #1459

This PR contains the following changes to compactions
- Use a separate thread for compacting Level 0 and 1 and a separate one for other levels
- Pick levels to compact based on score.
- Stall Level 0 if compactions cannot keep up (we had added this in #1186)
- Limit the number of open table builders to 5 in compactions.
danielmai added a commit to hypermodeinc/dgraph that referenced this pull request Oct 15, 2020
@danielmai
Update Badger to hypermodeinc/badger@dab2927.
d665f5a 
Include cherry-pick for the following change:

Author: Ibrahim Jarif <[email protected]>
Date:   Wed Aug 19 18:42:06 2020 +0530

    fix(compaction): Use separate compactors for L0, L1 (hypermodeinc/badger#1466)
manishrjain pushed a commit to outcaste-io/outserv that referenced this pull request Jul 6, 2022
Improvements: Manual Memory allocation via Calloc (#1459)
4a91ab4 
This PR contains a  bunch of improvements
1. This PR reverts the following reverts (adds them back)
    - Revert "Compress/Encrypt Blocks in the background  hypermodeinc/badger@8ad3232
    - Revert "Buffer pool for decompression hypermodeinc/badger@d37ba0b
   - Revert "fix: Fix race condition in block.incRef hypermodeinc/badger@2ddc993
    - Revert "add assert to check integer overflow for table size hypermodeinc/badger@b3881c7
2. Calloc and Free for memory allocation
    - The block buffers for decompression and encryption are now allocated using calloc and free.
      The `y/calloc.go` file contains the code.

The compaction changes are moved to a separate PR for easier cherry-picking
hypermodeinc/badger#1466
mYmNeo added a commit to mYmNeo/badger that referenced this pull request Jan 16, 2023
@mYmNeo
Use separate compactors for L0, L1 (hypermodeinc#1466) (hypermodeinc#…
2469606 
…1468)

Signed-off-by: thomassong <[email protected]>
mYmNeo added a commit to mYmNeo/badger that referenced this pull request Feb 13, 2023
@mYmNeo
Use separate compactors for L0, L1 (hypermodeinc#1466) (hypermodeinc#…
aa66570 
…1468)

Signed-off-by: thomassong <[email protected]>
mYmNeo added a commit to mYmNeo/badger that referenced this pull request Sep 14, 2024
@mYmNeo
Use separate compactors for L0, L1 (hypermodeinc#1466) (hypermodeinc#…
d85cfad 
…1468)

Signed-off-by: thomassong <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@manishrjain manishrjain manishrjain approved these changes

@ashish-goswami ashish-goswami Awaiting requested review from ashish-goswami

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jarifibrahim @manishrjain