-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
VLOGs total size ~1gb with no stored data on default config #1297
Comments
@jsign For badger, a As for the vlog GC, the vlog GC will never delete the lastest vlog file which is actively being written. That's why you see 1 vlog file left after GC. You can try to reduce the Also, please note that the default value of |
@jarifibrahim, thanks for your fast reply. Considering that, and other flags I tested in other scenarios in that program, the following configuration seems to make the trick for controlling the total size of files on disk:
As a particular example, running the setup:
Yielded to:
To repeat, with the default config the same run has:
Sounds reasonable to you? |
EDIT Do not use numVersionsToKeep=0. See PR #1300
@jsign The numbers look fine to me but I have two questions. Why do you need to reduce the number of level zero tables? The level zero tables are kept in memory (you can disable it by And why do you reduce the level 0 stall? The default is 10 tables which means that if your level 0 has more than 10 tables, we will stall the writes. This is done to reduce the memory usage. |
@jarifibrahim , thanks for pointing that out, avoiding Thanks! |
…age collection fixes ipfs#54 settings are by courtesy of @jsign, see [his post here](dgraph-io/badger#1297 (comment))
What version of Go are you using (
go version
)?What version of Badger are you using?
github.com/dgraph-io/badger/v2 v2.0.1-rc1.0.20200409094109-809725940698
Does this issue reproduce with the latest master?
Yes.
What are the hardware specifications of the machine (RAM, OS, Disk)?
RAM 8gb, latest Catalina, 512gb
What did you do?
I created a GH repo to reproduce the problem:
git clone https://github.com/jsign/go-badger2-size.git
go run main.go
The program tries different configurations to see how they affect the final SST and VLOG total sizes.
Some points about the program:
0.01
rate until it returnsErrNoRewrite
.The above flow runs for different scenarios described as
opts
used to open a DB.There're four scenarios that run in parallel. Running them concurrently is OK since I'm not testing for performance, only wanting to inspect files. Might take <5min to run.
What did you expect to see?
The process creates a million keys and then deletes all of them. I'd expect some config might achieve the total size to be ~negligible.
What did you see instead?
The output of the program (stripping badger logs):
Above sizes are in KB.
In the default config, the total size is greater than 1gb.
On an agressive setup is ~600mb.
To be clear, the DB has no stored data but is allocating 600mb in the best scenario.
I'm looking for a configuration that does this without stopping the DB. Is there any possible configuration that can make the DB have ~negligible size if it isn't storing any keys?
The text was updated successfully, but these errors were encountered: