Vacuum sqlite database at startup & after transactions preload #1771
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Alenar
force-pushed
the
djo/vacuum_cardano_tx_signer_db
branch
from
d30684f to
574224a
Compare
dlachaume
approved these changes
Jun 24, 2024
jpraynaud
approved these changes
Jun 24, 2024
Alenar
force-pushed
the
djo/vacuum_cardano_tx_signer_db
branch
from
78e4efe to
f1e3b10
Compare
Alenar
force-pushed
the
djo/vacuum_cardano_tx_signer_db
branch
from
f1e3b10 to
4edcccf
Compare
As it doesn't do any async op underhood.
To execute a sqlite vacuum after a transaction importer work.
…oader Since it will be the one that will make the db size growing the most and we wan't to avoid locking the database (as vacuum needs to) in normal operations.
to limit duplication between aggregator & signer
As it can be very large, contrary to the signer were the db is pruned regularly.
* Mithril-aggregator from `0.5.25` to `0.5.26` * Mithril-signer from `0.2.148` to `0.2.149` * Mithril-persistence from `0.2.9` to `0.4.10`
Alenar
force-pushed
the
djo/vacuum_cardano_tx_signer_db
branch
from
4edcccf to
d1f884a
Compare
Alenar
force-pushed
the
djo/vacuum_cardano_tx_signer_db
branch
from
a85e9ec to
335e28d
Compare
12 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Content
This PR:
mithril-aggregator&mithril-signerVacuum their databases at startup (this can take a while so a log message is print).mithril-signerso it execute a Vacuum after each chunk imported.We do that since we observed that auto-vacuum isn't executed until the whole import is finished (maybe because the db is a lot busy in this phase) so a manual operation is needed.
Pre-submit checklist
Comments
After merging #1766 we observed that the SQLite database file size did not decrease after pruning transactions in signer. This is due to SQLite not automatically reclaiming free pages by itself, we need to use the
vacuumoperation to do that.Note: Vacuum reconstruct the SQLite by creating a new file, that mean that it needs an exclusive lock on the database so we can't use it after every deletion carelessly.
For everyday task we enable auto-vacuum that makes SQLite handle it by itself but a the cost of greater fragmentation.
Given this drawback we only enable it on Cardano transactions databases since they are the only one were removal of data occurs on large scale, otherwise simply executing Vacuum at startup is enough.
Issue(s)
Relates to #1766