-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make disk IO and processing concurrent #1244
Closed
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
After testing it out a bit more, just doing the decr() and releasing the PLs at the earliest gives us most of the benefits. So we don't have to use channels to fetch the PLs separately. Continuing this in #1255. |
jarifibrahim
pushed a commit
that referenced
this pull request
Mar 16, 2020
Important changes ``` - Changes to overlap check in compaction. - Remove 'this entry should've been caught' log. - Changes to write stalling on levels 0 and 1. - Compression is disabled by default in Badger. - Bloom filter caching in a separate ristretto cache. - Compression/Encryption in background. - Disable cache by default in badger. ``` The following new changes are being added from badger `git log ab4352b00a17...91c31ebe8c22` ``` 91c31eb Disable cache by default (#1257) eaf64c0 Add separate cache for bloom filters (#1260) 1bcbefc Add BypassDirLock option (#1243) c6c1e5e Add support for watching nil prefix in subscribe API (#1246) b13b927 Compress/Encrypt Blocks in the background (#1227) bdb2b13 fix changelog for v2.0.2 (#1244) 8dbc982 Add Dkron to README (#1241) 3d95b94 Remove coveralls from Travis Build(#1219) 5b4c0a6 Fix ValueThreshold for in-memory mode (#1235) 617ed7c Initialize vlog before starting compactions in db.Open (#1226) e908818 Update CHANGELOG for Badger 2.0.2 release. (#1230) bce069c Fix int overflow for 32bit (#1216) e029e93 Remove ExampleDB_Subscribe Test (#1214) 8734e3a Add missing package to README for badger.NewEntry (#1223) 78d405a Replace t.Fatal with require.NoError in tests (#1213) c51748e Fix flaky TestPageBufferReader2 test (#1210) eee1602 Change else-if statements to idiomatic switch statements. (#1207) 3e25d77 Rework concurrency semantics of valueLog.maxFid (#1184) (#1187) 4676ca9 Add support for caching bloomfilters (#1204) c3333a5 Disable compression and set ZSTD Compression Level to 1 (#1191) 0acb3f6 Fix L0/L1 stall test (#1201) 7e5a956 Support disabling the cache completely. (#1183) (#1185) 82381ac Update ristretto to version 8f368f2 (#1195) 3747be5 Improve write stalling on level 0 and 1 5870b7b Run all tests on CI (#1189) 01a00cb Add Jaegar to list of projects (#1192) 9d6512b Use fastRand instead of locked-rand in skiplist (#1173) 2698bfc Avoid sync in inmemory mode (#1190) 2a90c66 Remove the 'this entry should've caught' log from value.go (#1170) 0a06173 Fix checkOverlap in compaction (#1166) 0f2e629 Fix windows build (#1177) 03af216 Fix commit sha for WithInMemory in CHANGELOG. (#1172) 23a73cd Update CHANGELOG for v2.0.1 release. (#1181) 465f28a Cast sz to uint32 to fix compilation on 32 bit (#1175) ea01d38 Rename option builder from WithInmemory to WithInMemory. (#1169) df99253 Remove ErrGCInMemoryMode in CHANGELOG. (#1171) 8dfdd6d Adding changes for 2.0.1 so far (#1168) ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In
processTask
fetching the values and processing it were sequential which was slow when we had huge UID lists.For example, this query on SuperUser data (50M rdfs) has ~350K UIDs which are of the type
Question
The processing time reduces from
~20s
to~9s
when we makeGetOrCreate
and the processing concurrent.This change is