fix(indexes): efficient rocksdb mempool-tips initialization#938
Merged
fix(indexes): efficient rocksdb mempool-tips initialization#938
Conversation
3f52b0a to
02fa789
Compare
02fa789 to
bbcb9bf
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #938 +/- ##
==========================================
+ Coverage 84.91% 85.04% +0.12%
==========================================
Files 314 314
Lines 23901 23919 +18
Branches 3609 3614 +5
==========================================
+ Hits 20296 20342 +46
+ Misses 2899 2870 -29
- Partials 706 707 +1 ☔ View full report in Codecov by Sentry. |
msbrogli
requested changes
Feb 23, 2024
bbcb9bf to
053da25
Compare
1 task
053da25 to
224f7f7
Compare
glevco
approved these changes
Mar 21, 2024
224f7f7 to
d263ea3
Compare
d263ea3 to
4705d5c
Compare
4705d5c to
b1c4bec
Compare
b1c4bec to
12304da
Compare
msbrogli
approved these changes
Jul 24, 2024
2 tasks
12304da to
bd8c2aa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
When we made sync-v2 always available, even when not enabled, it implied having to initialize the mempool-tips index, but its RocksDB implementation had been disabled for having a very slow initialization but using a memory index has the downside of always initializing the index on every start.
The mempool is defined as the set of transactions that don't have a
first_blockand are not voided, the mempool-tips index only stores only the mempool transactions that are "tips" (in the sense that all the other transactions can be reached by traversing to the left), the practical definition is the set of all transactions that don't have afirst_block, don't have a non-voided child and don't have a non-voided spent_output (that is, they aren't literal "tips" because they can have children or spent outputs, as long as those are voided).Before this PR, the initialization was implemented as simply processing all transactions in topological order the same way they would be if a node received them from the network, with the disadvantage that the it would almost always cache miss due to how the update has to load transactions "to the right" of the iteration loop, making it very slow.
This PR simplifies the loading by looking at the
first_block,children,spent_outputsandvoided_bymetadata fields to determine if a transaction is in the mempool and only adding it to the index if it is, instead of adding and removing transactions.Acceptance Criteria
MemoryTipsIndex.init_loop_stepby restricting the scope and making it only add transactions without a first_block, children or spent outputs.Checklist
master, confirm this code is production-ready and can be included in future releases as soon as it gets merged