cmd/geth, eth, core: snapshot dump + unify with trie dump

[holiman]

Supersedes #22787

This PR adds support for doing dumps from the snapshot. This is useful for two purposes:

• For general chain investigations, it's faster to dump data using the snapshot than iterating the trie.
• For checking the trie against the snapshot, it's possible to dump out parts of the state from either, and compare the two.

There are a few differences, however.

• For snapshot dump, I haven't added the conversion from hashed form to preimages. Mostly because since we don't (by default) store preimages any longer, I don't think it makes sense to do an extra db lookup for every hash. However, I didn't remove the address lookup from the trie dump, to not destroy something.
• For snapshot dump, it only ever does iterative (jsonlines, not json) output.
• For trie dump, I changed the default to be iterative
• Both commands now take a start
• Both commands defaults to latest block if no args are given
• Also added progress reporting on both

Opening this in draft, will test it a bit more, add some performance numbers.

Examples:

Trie dump:

[user@work go-ethereum]$ ./build/bin/geth --ropsten  dump --start 0xfe6a58207750197f48cb90864096850259845c2c8e90c74433325c0b144bf8bb
INFO [05-01|20:24:33.872] Maximum peer count                       ETH=50 LES=0 total=50
WARN [05-01|20:24:33.872] Using the deprecated `testnet` datadir. Future versions will store the Ropsten chain in `ropsten`. 
INFO [05-01|20:24:33.872] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [05-01|20:24:33.873] Set global gas cap                       cap=25,000,000
INFO [05-01|20:24:33.873] Allocated cache and file handles         database=/home/user/.ethereum/ropsten/geth/chaindata cache=512.00MiB handles=262,144 readonly=true
INFO [05-01|20:24:33.912] Opened ancient database                  database=/home/user/.ethereum/ropsten/geth/chaindata/ancient readonly=true
INFO [05-01|20:24:33.913] Dump options                             block=0 hash=0x41941023680923e0fe4d74a34bdac8141f2540e3ae90623718e47d66d1ca4a2d code=true storage=true start=0xfe6a58207750197f48cb90864096850259845c2c8e90c74433325c0b144bf8bb
{"root":"0x217b0bbcfb72e2d57e28f33cb361b9983513177755dc3f33ce3e7022ed62b77b"}
{"balance":"0","nonce":0,"root":"56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421","codeHash":"c5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470","address":"0x000000000000000000000000000000000000007f","key":"0xfe6a58207750197f48cb90864096850259845c2c8e90c74433325c0b144bf8bb"}
{"balance":"0","nonce":0,"root":"56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421","codeHash":"c5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470","address":"0x000000000000000000000000000000000000000d","key":"0xfffa0eae268038cfa984647a1d0635beb86eda9fb7b500688f3189520cfa9ee5"}
INFO [05-01|20:24:33.913] Trie iteration complete                  accounts=2 elapsed="215.961µs"

Snapshot dump:

[user@work go-ethereum]$ ./build/bin/geth --ropsten  snapshot dump --start 0xfe6a58207750197f48cb90864096850259845c2c8e90c74433325c0b144bf8bb
INFO [05-01|20:25:02.252] Maximum peer count                       ETH=50 LES=0 total=50
WARN [05-01|20:25:02.253] Using the deprecated `testnet` datadir. Future versions will store the Ropsten chain in `ropsten`. 
INFO [05-01|20:25:02.253] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [05-01|20:25:02.253] Set global gas cap                       cap=25,000,000
INFO [05-01|20:25:02.253] Allocated cache and file handles         database=/home/user/.ethereum/ropsten/geth/chaindata cache=512.00MiB handles=262,144 readonly=true
INFO [05-01|20:25:02.272] Opened ancient database                  database=/home/user/.ethereum/ropsten/geth/chaindata/ancient readonly=true
INFO [05-01|20:25:02.272] Dump options                             block=0 hash=0x41941023680923e0fe4d74a34bdac8141f2540e3ae90623718e47d66d1ca4a2d code=true storage=true start=0xfe6a58207750197f48cb90864096850259845c2c8e90c74433325c0b144bf8bb
INFO [05-01|20:25:02.272] Snapshot iteration started               root=217b0b..62b77b
{"root":"0x217b0bbcfb72e2d57e28f33cb361b9983513177755dc3f33ce3e7022ed62b77b"}
{"balance":"0","nonce":0,"root":"56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421","codeHash":"c5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470","key":"0xfe6a58207750197f48cb90864096850259845c2c8e90c74433325c0b144bf8bb"}
{"balance":"0","nonce":0,"root":"56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421","codeHash":"c5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470","key":"0xfffa0eae268038cfa984647a1d0635beb86eda9fb7b500688f3189520cfa9ee5"}
INFO [05-01|20:25:02.273] Snapshot iteration complete              accounts=2 elapsed="161.081µs"

[holiman]

Dumping from the mainnet trie (was performed before the changes in this PR) :

$ ./build/bin/geth dump --iterative --nostorage --nocode --incompletes 12295139 
INFO [05-01|10:59:58.396] Maximum peer count                       ETH=50 LES=0 total=50
INFO [05-01|10:59:58.396] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [05-01|10:59:58.397] Set global gas cap                       cap=25,000,000
INFO [05-01|10:59:58.397] Allocated cache and file handles         database=/home/martin/.ethereum/geth/chaindata cache=512.00MiB handles=524,288 readonly=true
INFO [05-01|10:59:58.603] Opened ancient database                  database=/home/martin/.ethereum/geth/chaindata/ancient readonly=true
WARN [05-02|01:24:03.683] Dump incomplete due to missing preimages missing=128,454,104

Iterating the trie (no code nor storage lookup) took around ~14h. This one also checks the db for any preimages, which does add an extra lookup per account.

Same with snapshot dump:

$  ./build/bin/geth snapshot dump --nostorage --nocode 
INFO [05-02|20:24:57.934] Maximum peer count                       ETH=50 LES=0 total=50
INFO [05-02|20:24:57.934] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [05-02|20:24:57.935] Set global gas cap                       cap=25,000,000
INFO [05-02|20:24:57.935] Allocated cache and file handles         database=/home/martin/.ethereum/geth/chaindata cache=512.00MiB handles=524,288 readonly=true
INFO [05-02|20:25:00.574] Opened ancient database                  database=/home/martin/.ethereum/geth/chaindata/ancient readonly=true
INFO [05-02|20:25:00.646] Dump options                             block=12,295,139 hash=0x423ccad3ff8937e1efeee7597fa392e1e766592bed23a0de36c61e4f36e43e35 code=false storage=false start=0x00
INFO [05-02|20:25:00.815] Snapshot iteration started               root=a4f797..e2a994
INFO [05-02|20:25:10.815] Snapshot iteration in progress           at=04d739..80db86 accounts=2,429,369 elapsed=10.000s
[...]
INFO [05-02|20:33:30.888] Snapshot iteration in progress           at=fc0a83..2812af accounts=126,473,596 elapsed=8m30.072s
INFO [05-02|20:33:39.221] Snapshot iteration complete              accounts=128,462,996 elapsed=8m38.406s

Results (after I grepped out what I was looking for) are identical.

Performance

So, performance-wise, it's ~9m instead of 14h. 👀

[holiman]

Updated

[user@work go-ethereum]$ ./build/bin/geth --ropsten snapshot  dump --start 0x0011223344556677889900112233445566778899 --limit 2
INFO [05-11|12:49:20.813] Maximum peer count                       ETH=50 LES=0 total=50
WARN [05-11|12:49:20.813] Using the deprecated `testnet` datadir. Future versions will store the Ropsten chain in `ropsten`. 
INFO [05-11|12:49:20.813] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [05-11|12:49:20.814] Set global gas cap                       cap=25,000,000
INFO [05-11|12:49:20.814] Allocated cache and file handles         database=/home/user/.ethereum/ropsten/geth/chaindata cache=512.00MiB handles=262,144 readonly=true
INFO [05-11|12:49:20.825] Opened ancient database                  database=/home/user/.ethereum/ropsten/geth/chaindata/ancient readonly=true
INFO [05-11|12:49:20.825] Converting start-address to hash         address=0x0011223344556677889900112233445566778899 hash=0x6a9d3d525117c5aeb6609932889835d9634df9acb5cffe1b8f1aac612d1b2676
INFO [05-11|12:49:20.825] Dump options                             block=0 hash=0x41941023680923e0fe4d74a34bdac8141f2540e3ae90623718e47d66d1ca4a2d code=true storage=true start=0x6a9d3d525117c5aeb6609932889835d9634df9acb5cffe1b8f1aac612d1b2676 limit=2
INFO [05-11|12:49:20.825] Snapshot iteration started               root=217b0b..62b77b
{"root":"0x217b0bbcfb72e2d57e28f33cb361b9983513177755dc3f33ce3e7022ed62b77b"}
{"balance":"0","nonce":0,"root":"56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421","codeHash":"c5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470","key":"0x6aefc729e28e1bbd2e63dd0803314deb808ab044909af882b7935dd05f5939a4"}
{"balance":"0","nonce":0,"root":"56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421","codeHash":"c5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470","key":"0x6b2c03103b55a45cc1e4550889a9396e7e331b5cf1729e4871ff4dee701139e9"}
INFO [05-11|12:49:20.826] Snapshot iteration complete              accounts=2 elapsed="118.893µs"

[user@work go-ethereum]$ ./build/bin/geth --ropsten   dump --start 0x0011223344556677889900112233445566778899 --limit 2
INFO [05-11|12:49:23.637] Maximum peer count                       ETH=50 LES=0 total=50
WARN [05-11|12:49:23.637] Using the deprecated `testnet` datadir. Future versions will store the Ropsten chain in `ropsten`. 
INFO [05-11|12:49:23.637] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [05-11|12:49:23.637] Set global gas cap                       cap=25,000,000
INFO [05-11|12:49:23.638] Allocated cache and file handles         database=/home/user/.ethereum/ropsten/geth/chaindata cache=512.00MiB handles=262,144 readonly=true
INFO [05-11|12:49:23.653] Opened ancient database                  database=/home/user/.ethereum/ropsten/geth/chaindata/ancient readonly=true
INFO [05-11|12:49:23.653] Converting start-address to hash         address=0x0011223344556677889900112233445566778899 hash=0x6a9d3d525117c5aeb6609932889835d9634df9acb5cffe1b8f1aac612d1b2676
INFO [05-11|12:49:23.654] Dump options                             block=0 hash=0x41941023680923e0fe4d74a34bdac8141f2540e3ae90623718e47d66d1ca4a2d code=true storage=true start=0x6a9d3d525117c5aeb6609932889835d9634df9acb5cffe1b8f1aac612d1b2676 limit=2
{"root":"0x217b0bbcfb72e2d57e28f33cb361b9983513177755dc3f33ce3e7022ed62b77b"}
{"balance":"0","nonce":0,"root":"56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421","codeHash":"c5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470","address":"0x0000000000000000000000000000000000000049","key":"0x6aefc729e28e1bbd2e63dd0803314deb808ab044909af882b7935dd05f5939a4"}
{"balance":"0","nonce":0,"root":"56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421","codeHash":"c5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470","address":"0x00000000000000000000000000000000000000da","key":"0x6b2c03103b55a45cc1e4550889a9396e7e331b5cf1729e4871ff4dee701139e9"}
INFO [05-11|12:49:23.654] Trie iteration complete                  accounts=2 elapsed="241.032µs"