Ordered account insertion into merkle trie (I)#1697
Conversation
ghost
left a comment
There was a problem hiding this comment.
Added some minor comments. Question: how efficient is using sql's orderby / indexing as a sorting algorithm?
I'm not sure that we have a good alternative. We don't have enough memory to load it all in anyway. |
algonautshant
left a comment
There was a problem hiding this comment.
Looks good.
I feel like this can be simplified. So I need to understand why all these cases are needed.
| iterator.step = oaiStepCreateOrderingAccountIndex | ||
| return | ||
| } | ||
| if iterator.step == oaiStepCreateOrderingAccountIndex { |
There was a problem hiding this comment.
Why this step needs to return? Can't it directly call Next to advance to oaiStepSelectFromOrderedTable?
algonautshant
left a comment
There was a problem hiding this comment.
Looks great.
Thanks for the changes.
Just one comment about closing the iterator.
algorandskiy
left a comment
There was a problem hiding this comment.
LGTM, not sure although why do we need such level of granularity/steps within the Next() function
Refactor merkle trie rebuild to use ordered accounts insertions
Summary
Inserting ordered hashes into the merkle trie improve its insertion speed dramatically. The goal of this change is to modify the insertion that doing so that these would always be in-order.
We currently have two such use cases :
This PR addresses the first part, using an ordered accounts hashes to be inserted into the trie.
The new
orderedAccountsIteris intended to replace theencodedAccountsBatchIter, but that would only be possible on the subsequent PR.Performance Plan
I added a benchmark to test the gains:
The 6M accounts were chosen since it's roughly the number of accounts we currently have on mainnet.
Test Plan
Existing tests already cover the functional portion of this PR.