feat(pruner, primitives): move prune batch sizes to ChainSpec#4318
feat(pruner, primitives): move prune batch sizes to ChainSpec#4318
ChainSpec#4318Conversation
This reverts commit df14e1e.
shekhirin
changed the title
ChainSpec
shekhirin
force-pushed
the
alexey/chain-spec-prune-batch-size
branch
from
dd1f651 to
8c06a5d
Compare
shekhirin
changed the title
ChainSpecChainSpec
| /// These settings are sufficient to prune more data than generated with each new block. | ||
| pub const fn mainnet() -> Self { | ||
| Self { | ||
| receipts: 250, | ||
| transaction_lookup: 250, | ||
| transaction_senders: 250, | ||
| account_history: 1000, |
There was a problem hiding this comment.
assuming these are heuristics/made up?
There was a problem hiding this comment.
receipts,transaction_lookupandtransaction_senders– basically transactions per block which currently sits at 150 avg, occasionally jumping up to 250account_history–AccountChangeSettable on avg grows for 200 entries per block,AccountHistoryis cleared up based on the results of clearing the changesetsstorage_history–StorageChangeSettable on avg grows for 500 entries per block,StorageHistoryis cleared up based on the results of clearing the changesets
I just added a new DB metric #4316 which will allow to measure these values more accurately, as I was gathering the data just snapshotting the db stats
| base_fee_params: BaseFeeParams::ethereum(), | ||
| prune_batch_sizes: PruneBatchSizes::mainnet(), |
joshieDo
left a comment
joshieDo
left a comment
There was a problem hiding this comment.
lgtm,
maybe it the future it can be self adaptive, since i guess it also depends on the machine running it
how did it perform on the last run?
- did it start losing track of the tip? reduce batch size.
- did it have "time to spare"? increase batch size
you're right, imagine it takes 5s to commit a new head and you also need to run the pruner in the time before the next FCU.
yep, something like this should work. We can also be dependent on the number of transactions in the previous block for receipts/tx senders/tx lookup. |
…lexey/chain-spec-prune-batch-size
shekhirin
force-pushed
the
alexey/chain-spec-prune-batch-size
branch
from
2f4561d to
2d40ea1
Compare
shekhirin
force-pushed
the
alexey/chain-spec-prune-batch-size
branch
from
005c3e1 to
3e511ca
Compare
Codecov Report
... and 7 files with indirect coverage changes
Flags with carried forward coverage won't be shown. Click here to find out more.
|
3 participants
We need to set pruner batch sizes depending on the network the node is running. The reason for this is that mainnet has more transactions per block than testnets, hence more receipts, tx senders, tx lookup entries and changesets are inserted in the database. We need to prune more data than the amount of new incoming data, but also not too much because otherwise the problems described in #4246 will arise.