db/store full (forked) blocks cleanup

[antiochp]

We compact/prune the blocks db periodically to remove historical full blocks from the db older than 1 week (unless running in archive mode).

But - we do this based on the current chain by traversing back from the head of the chain.

So if we see any short lived forks or reorgs we will never actually clean these forked blocks up and they will remain in the db indefinitely.

i.e. if we have the following blocks in the chain -

B1 <- B2 <- B3 etc.

We will never remove any forked blocks, say B2' for example.

[192-sean]

Where would I start if I were to tackle this.

[antiochp]

Right now we traverse back through historical blocks and then go to the db for each one to be removed. But I think a better approach may be start at the db and simply iterate over the blocks in the db, removing any that are candidates for removal.

This might be a good introduction to the lower level db access stuff, so I'd start looking around our lmdb code to see how we might use an iterator over full blocks. We do something similar in the peers db (via an iterator).

[miketang84]

I would like to process this.

[miketang84]

Hi, @antiochp I have the following questions:

If we clean old blocks by iterating db directly, how we check which block is at a short-lived fork? or even a fork?

And, should we remove all fork blocks from db?

And I proposed a request on #2807 , does it walk the right way?

Thank you.

[hayeah]

My understanding is that instead of using batch.get_previous_header(&current) to iterate through the blocks, you should access the underlying store of the Batch type (which is a LMDB instance)

/// An atomic batch in which all changes can be committed all at once or
/// discarded on error.
pub struct Batch<'a> {
	db: store::Batch<'a>,
}

Create a LMDB iterator to access ALL the blocks, regardless of whether the block is in the chain or not.

Then we need to have an efficient way to check whether a particular block exists on the chain (if not, remove it).

To check whether a block is on chain, one naive way is to iterate through the chain (using get_previous_header), and save block hashes in a map. The CUT_THROUGH_HORIZON is ~10k blocks, so we'll have to keep 10k entries in memory for the compaction process.

[antiochp]

Then we need to have an efficient way to check whether a particular block exists on the chain (if not, remove it).

chain.is_on_current_chain()

This checks the header against the header at that height based on our current chain - if they match then we're good (header at height h is the one we think it is).

[miketang84]

hi, #2810 CI prompted:

running 5 tests
test hard_forks ... ok
test next_target_adjustment ... ok
test test_secondary_pow_ratio ... ok
test adjustment_scenarios ... test adjustment_scenarios has been running for over 60 seconds
test test_secondary_pow_scale ... test test_secondary_pow_scale has been running for over 60 seconds

No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received

What is the meaning of it? Because in my local device, there are no these timeout error. How should I resolve this consensus timeout?

[chisa0a]

@daogangtang had a similar issue on a resource constrained device. If you have the ability to increase the memory available on the CI box, that might help.

Another option is to refactor the tests to consume less resources, which is being partly addressed by #2813 (IIUC).

[miketang84]

@chisa0a Yes, thank you, I repost a PR #2815 , and it pass now.

[antiochp]

Resolved in #2815. 🚀