Decrease tick interval to 50 ms. #1191
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is a stop-gap. For whatever reason, we aren't replaying log entries any faster than one entry per tick. This makes replaying the log be about 15x slower than loading the data initially cost (when running locally on a decent laptop). So, reducing the tick interval by a factor of 20 seems like it'll suffice, for that hardware.
|
Closing. I want to look at this further. It looks like it might be tied to the heartbeat interval rather than the Tick rate. |
jarifibrahim
pushed a commit
that referenced
this pull request
Mar 16, 2020
Important changes ``` - Changes to overlap check in compaction. - Remove 'this entry should've been caught' log. - Changes to write stalling on levels 0 and 1. - Compression is disabled by default in Badger. - Bloom filter caching in a separate ristretto cache. - Compression/Encryption in background. - Disable cache by default in badger. ``` The following new changes are being added from badger `git log ab4352b00a17...91c31ebe8c22` ``` 91c31eb Disable cache by default (#1257) eaf64c0 Add separate cache for bloom filters (#1260) 1bcbefc Add BypassDirLock option (#1243) c6c1e5e Add support for watching nil prefix in subscribe API (#1246) b13b927 Compress/Encrypt Blocks in the background (#1227) bdb2b13 fix changelog for v2.0.2 (#1244) 8dbc982 Add Dkron to README (#1241) 3d95b94 Remove coveralls from Travis Build(#1219) 5b4c0a6 Fix ValueThreshold for in-memory mode (#1235) 617ed7c Initialize vlog before starting compactions in db.Open (#1226) e908818 Update CHANGELOG for Badger 2.0.2 release. (#1230) bce069c Fix int overflow for 32bit (#1216) e029e93 Remove ExampleDB_Subscribe Test (#1214) 8734e3a Add missing package to README for badger.NewEntry (#1223) 78d405a Replace t.Fatal with require.NoError in tests (#1213) c51748e Fix flaky TestPageBufferReader2 test (#1210) eee1602 Change else-if statements to idiomatic switch statements. (#1207) 3e25d77 Rework concurrency semantics of valueLog.maxFid (#1184) (#1187) 4676ca9 Add support for caching bloomfilters (#1204) c3333a5 Disable compression and set ZSTD Compression Level to 1 (#1191) 0acb3f6 Fix L0/L1 stall test (#1201) 7e5a956 Support disabling the cache completely. (#1183) (#1185) 82381ac Update ristretto to version 8f368f2 (#1195) 3747be5 Improve write stalling on level 0 and 1 5870b7b Run all tests on CI (#1189) 01a00cb Add Jaegar to list of projects (#1192) 9d6512b Use fastRand instead of locked-rand in skiplist (#1173) 2698bfc Avoid sync in inmemory mode (#1190) 2a90c66 Remove the 'this entry should've caught' log from value.go (#1170) 0a06173 Fix checkOverlap in compaction (#1166) 0f2e629 Fix windows build (#1177) 03af216 Fix commit sha for WithInMemory in CHANGELOG. (#1172) 23a73cd Update CHANGELOG for v2.0.1 release. (#1181) 465f28a Cast sz to uint32 to fix compilation on 32 bit (#1175) ea01d38 Rename option builder from WithInmemory to WithInMemory. (#1169) df99253 Remove ErrGCInMemoryMode in CHANGELOG. (#1171) 8dfdd6d Adding changes for 2.0.1 so far (#1168) ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a stop-gap. For whatever reason, we aren't replaying log entries any faster than one entry per tick. This makes replaying the log be about 15x slower than loading the data initially cost (when running locally on a Thinkpad W520). So, reducing the tick interval by a factor of 20 seems like it'll make the Raft tick frequency not be the bottleneck.
In general this won't be true, for example, if we did the initial writes more slowly, then they'd likewise be forwarded to other nodes more slowly (with fewer writes per raft index). This generally doesn't fix the overall problem.
Connected to #1180.
This change is