feature: ancient blockchain pruner

[jsvisa]

Description

Implement historical blockchain data pruning #718

Changes

• Bugfix (non-breaking change that solves an issue)
• Hotfix (change that solves an urgent issue, and requires immediate attention)
• New feature (non-breaking change that adds functionality)
• Breaking change (change that is not backwards-compatible and/or changes current functionality)
• Changes only for a subset of nodes

Breaking changes

Please complete this section if any breaking changes have been made, otherwise delete it

Nodes audience

In case this PR includes changes that must be applied only to a subset of nodes, please specify how you handled it (e.g. by adding a flag with a default value...)

Checklist

• I have added at least 2 reviewer or the whole pos-v1 team
• I have added sufficient documentation in code
• I will be resolving comments - if any - by pushing each fix in a separate commit and linking the commit hash in the comment reply

Cross repository changes

• This PR requires changes to heimdall
  • In case link the PR here:
• This PR requires changes to matic-cli
  • In case link the PR here:

Testing

• I have added unit tests
• I have added tests to CI
• I have tested this code manually on local environment
• I have tested this code manually on remote devnet using express-cli
• I have tested this code manually on mumbai
• I have created new e2e tests into express-cli

Manual tests

Please complete this section with the steps you performed if you ran manual tests for this functionality, otherwise delete it

Additional comments

Ancient Data Prune

A new tool is introduced to prune ancient undesired block data, it will discard block, receipt, header in the ancient db to save space.
Example: ./build/bin/bor snapshot prune-block -datadir ./node -check-snapshot-with-mpt -block-amount-reserved 4096

[JekaMas]

Is it cherry-picking from BSC repository? If it is we need to check licences.

[jsvisa]

@JekaMas Yes, most of the code are cherry-picked from bsc, mainly those PR:

• [R4R]offline block prune bnb-chain/bsc#543
• [R4R]db: freezer batch compatible offline prunblock command bnb-chain/bsc#1005
• [R4R]fix pruner block tool bug, add some check logic bnb-chain/bsc#817

[jsvisa]

I'll test this feature in our infra for a while, and after that will make this pr as ready.

[petejkim]

@jsvisa very nice!

Is it cherry-picking from BSC repository? If it is we need to check licences.

@JekaMas The license is the same as geth and bor (LGPL/GPL 3.0).

[jsvisa]

It's ready for review, it's there anyone who can help to review this PR, thanks.

[Raneet10]

Hi @jsvisa , thanks for raising the PR! We haven't had the bandwidth to review but we shall soon.

[0x090909]

Any news? is this a working feature in the PR repository?

[jsvisa]

Any news? is this a working feature in the PR repository?

If you also need this feature, you can try this branch https://github.com/jsvisa/bor/tree/block-pruner

[0x090909]

Any news? is this a working feature in the PR repository?

If you also need this feature, you can try this branch https://github.com/jsvisa/bor/tree/block-pruner

Hey man yes I need it, I tried to use it, but unfortunatelly after I do prune the ancient blocks, my snapshot kinda gets screwed up and then the ancient directory is eliminated completely.

Afterwards if I try to start the node, bor gives an error: says please specify the correct ancient directory since it has been moved.

Can you provide exact command you used with your snapshot?

I used the following:

./bor_patched snapshot prune-block --datadir ./bor-mainnet/data/

Also I noticed, if I use bor_patched for whatever reason, the pruner creates a directory called bor_patched inside the snapshot, is it correct?

[jsvisa]

Also I noticed, if I use bor_patched for whatever reason, the pruner creates a directory called bor_patched inside the snapshot, is it correct?

No, you can't rename the binary name, else the directory will be misused. (just naming as bor)

one more thing, after you prune the old ancient, you also need to run the node with the patched bor binary. good luck.

[0x090909]

Also I noticed, if I use bor_patched for whatever reason, the pruner creates a directory called bor_patched inside the snapshot, is it correct?

No, you can't rename the binary name, else the directory will be misused. (just naming as bor)

one more thing, after you prune the old ancient, you also need to run the node with the patched bor binary. good luck.

Okay thank you, Ill try again.

[0x090909]

@jsvisa Hey man it works!
I spent 2 days in downloading snapshots, and at 4th attempt it worked, cant beleive its was only because of the executable name 🤦🤦🤦🤦🤦
Thanks

[jsvisa]

@jsvisa Hey man it works! I spent 2 days in downloading snapshots, and at 4th attempt it worked, cant beleive its was only because of the executable name 🤦🤦🤦🤦🤦 Thanks

yeah, this is a design from go-ethereum

[kaber2]

Given that this obviously useful feature has been dragging out for a year now, I'd like to add a datapoint to the argument of data availability. Right now, there is no data availability at all. I'm currently trying to sync a couple of tens of servers from your latest outdated snapshot, which is two month old, and the ETA just for 1.5 million blocks is between 80 and 270 hours, going up and down randomly. So this is in my opinion a very weak argument, it has been factually impossible to sync Polygon from scratch for a long time.

As an aside, you might want to seriously get working on reducing further bloat, as an easy starting point, I'd suggest to finally remove the completely useless and long time deprecated LogFeeTransfer() event that is bloating the state for every single transaction.

[kaber2]

I might add, presuming to speak for all node operators, we have had it with your sloppy way of running this chain. BSC used to be the dog pile of blockchains, but they got their act together, while Polygon is bloating like crazy, taking ages to include useful features for node operators, doing sloppy merges and making life more difficult for everyone by mindlessly reformatting upstream code, deep reorging constantly, etc. Get your act together, or this will blow up soon.

[manav2401]

@kaber2 @0x090909, apologies for the long delay. We understand the concern of increased disk usage and are internally convinced for inclusion of this feature. We're waiting for the final governance call (as mentioned here) before merging this fully. Meanwhile, we'll make sure that the PR is updated with latest changes. Thanks!

[0x090909]

@kaber2 you can use their snapshots, work good and they update it daily.

https://snapshots.vaultstaking.com/

[0x090909]

Guyssss!!!

My storage is screaming! please

[0x090909]

@manav2401 updates?

[vicmik]

Hi.

Here is data for a sentry node that I pruned 45 days ago.
Freshly pruned:

• Total datadir: 2.7 TB
• Ancient dir: 1.8 TB (67%)

45 days later:

• Total datadir: 3.4 TB
• Ancient dir: 1.9 TB (55%)

Conclusion: the disk usage can be reduced by 50% by pruning the whole database (including ancient) every 2 months.

PS: Moving ancient to a cheaper storage may seem like a good idea but it is often impossible or complex. Not every server provider has it as a standard option to add a 4 TB HDD to the server. Configuring this over a network is probably possible but increases complexity in setting up a node.

[0x090909]

@vicmik I did try to store ancient in an external HDD, but unfortunatelly there are also storage issues and sometimes the db gets corrupted out of nowhere, while having the whole db on a NVME works always flawlessly

[KeM1aL]

I am using the branch jsvisa:block-pruner but the prune does not delete the data right ? It just move them to ancient folder (and compress).

Am I missing something ?

[jsvisa]

I'm rebasing the code to the last develop branch right now, maybe 1 or 2 days to finish it.

[paulgoleary]

Deleted comment with abusive / unprofessional language.

@jsvisa thanks for your help with the rebase.

We appreciate everyone's patience and attention to this issue. We will review priority asap.

[jsvisa]

Hello everyone, the rebasing process was done, but my local env doesn't have a large SSD to store the matic mainnet data, does anyone interested to verify the prune-block process?

[0x090909]

I can do that later

[0x090909]

Im building the block-pruner branch but I get this error:

I did go mod tidy and make clean, but problem persists

[jsvisa]

Im building the block-pruner branch but I get this error:

I did go mod tidy and make clean, but problem persists

I can build it with Golang v1.22.0 in Linux and macOS. you can try with 1.22.x or higher

[0x090909]

this is the version I have:

go version go1.22.1 linux/amd64

[0x090909]

I managed to compile the project, I was lacking gcc on my server.

It seems the prune was done successfully.

[0x090909]

I started the node and its syncing again, it worksss!!!! @jsvisa GREAT JOB!!!!!!!!1

[0x090909]

New storage space occupancy

[jsvisa]

I started the node and its syncing again, it worksss!!!! @jsvisa GREAT JOB!!!!!!!!1

Good job, Could you please paste the instructions below for others to follow up

[0x090909]

INSTRUCTIONS TO PRUNE ANCIENT DATA

1. Install go 1.20+ and gcc
2. clone git clone https://github.com/jsvisa/bor.git && cd bor
3. run the command make bor
4. put the executable from bor/build/bin/bor into your path dir
5. stop the node
6. finally run the prune command bor snapshot prune-block -block-amount-reserved=0 -datadir DATA_DIR
7. restart the node, with the new executable of bor. If you start it with the old executable its not able to open the ancient dir.

[0x090909]

@manav2401 @ssandeep when are you planning to merge?

[manav2401]

@0x090909 I will be merging this by end of this week. I will be testing this on internal nodes. Also, I am not sure how but a develop merge caused all the conflicts to reappear here. Will be fixing that.

[manav2401]

Hey folks, looks like the git history of this PR is corrupted and I can't seem to push a merge commit resolving conflicts to this branch. Hence, I have created a new PR with the same set of commits as in this PR instead. Please follow that going forward for updates and further discussion if any. Really appreciate everyone's help. Closing this for now. Thanks!

#1216