feat: reth db diff

[Rjected]

This adds a command, reth db diff, which is able to diff two reth databases, either a full DB diff or table-by-table.

Command line output example:

dan@Dans-MacBook-Pro-4 ~/p/reth (dan/reth-db-diff)> cargo r db diff --datadir ~/first-diff-datadir/ --secondary-datadir ~/second-diff-datadir/ --output ~/diff-results
   Compiling reth v0.1.0-alpha.4 (/Users/dan/projects/reth/bin/reth)
    Finished dev [unoptimized + debuginfo] target(s) in 12.87s
     Running `target/debug/reth db diff --datadir /Users/dan/first-diff-datadir/ --secondary-datadir /Users/dan/second-diff-datadir/ --output /Users/dan/diff-results`
2023-07-27T20:48:40.724516Z  INFO reth::db::diff: Analyzing table CanonicalHeaders...
2023-07-27T20:48:40.837339Z  INFO reth::db::diff: Done analyzing table CanonicalHeaders!
2023-07-27T20:48:40.837427Z  INFO reth::db::diff:
2023-07-27T20:48:40.837654Z  INFO reth::db::diff: Diff results for CanonicalHeaders:
2023-07-27T20:48:40.838173Z  INFO reth::db::diff: No discrepancies found in table CanonicalHeaders
2023-07-27T20:48:40.838369Z  INFO reth::db::diff: No extra elements found in table CanonicalHeaders
2023-07-27T20:48:40.838391Z  INFO reth::db::diff: Writing diff results for CanonicalHeaders to CanonicalHeaders.txt...
2023-07-27T20:48:40.839062Z  INFO reth::db::diff: Done writing diff results for CanonicalHeaders to /Users/dan/diff-results/CanonicalHeaders.txt
2023-07-27T20:48:40.839219Z  INFO reth::db::diff: Analyzing table HeaderTD...
2023-07-27T20:48:41.018847Z  INFO reth::db::diff: Done analyzing table HeaderTD!

Example of extra-element output (no discrepancies but a table has an extra element):

dan@Dans-MacBook-Pro-4 ~/p/reth (dan/reth-db-diff)> cat ~/diff-results/HeaderNumbers.txt
Diff results for HeaderNumbers
No discrepancies found in table HeaderNumbers
Found 10 extra elements in table HeaderNumbers
Extra elements:
Second { key: 0x0c1cd2b0f6213161986986a707bfa7280e9cfd31fd692b4e0acfb56339da9bc9, value: 100004 }
Second { key: 0xa93c3b8a437a216e241677827ad858e123024d0ec349f25be91f716f00760995, value: 100008 }
Second { key: 0x5c46eb47b3fd34d042768cbdecbe4c444c75704dac63e7b2aa870b4b7cbc78ef, value: 100010 }
Second { key: 0x0ba42c57b9c7317d81427594686fbb07387cffb4214ad5263574dd0e0764ed29, value: 100001 }
Second { key: 0xf96bc97790728fb5314fb5888244b0d6b7b6c527b407e2bbfd9f152a207bb144, value: 100007 }
Second { key: 0xf3784b63e43b22bc6ad90aad72e1ca55746a8e8b9a2539bc3671d894a39176fa, value: 100005 }
Second { key: 0xa36459a7b00deba23d1e2f51d1f87a4f6906caf8da443a23e058bfb7e2ac6656, value: 100002 }
Second { key: 0xf2f251f7f88622c6cae1bc963cf8f7ffd9a64412f81166f8b719d702973b8c3c, value: 100009 }
Second { key: 0x72e982b65e527aaca19e60a3da0bddd1070f2f76bba2334c8931308674cd5bff, value: 100003 }
Second { key: 0x34eec12e2f39abc569e8eb97177d855df19f4716ad472bc0f3b90231eb51e3cf, value: 100006 }

And discrepancy output:

dan@Dans-MacBook-Pro-4 ~/p/reth (dan/reth-db-diff)> cat ~/diff-results/HashedAccount.txt
Diff results for HashedAccount
Found 1 discrepancies in table HashedAccount
No extra elements found in table HashedAccount
Discrepancies:
TableDiffElement { key: 0x623afa54b237ecdf128faa469966051cff88b7dbdb0aec8062ef3fb218e22491, first: Account { nonce: 1, balance: 0x000000000000000000000000000000000000000000002a2dc2590a968a8bc1c8_U256, bytecode_hash: None }, second: Account { nonce: 1, balance: 0x000000000000000000000000000000000000000000002a2ed7e7509f9e5bc1c8_U256, bytecode_hash: None } }

[joshieDo]

Tx<'_, RO, NoWriteMap>,
mdbx::{tx::Tx, NoWriteMap, RO},

Could this be replaced by some form of DbTx ? Ideally mdbx shouldn't be leaked into other crates, unless reaally necessary

[Rjected]

Could this be replaced by some form of DbTx ? Ideally mdbx shouldn't be leaked into other crates, unless reaally necessary

that makes sense, changed it to impl DbTx<'a>, mdbx imports aren't needed now

[codecov]

Codecov Report

Merging #3917 (bbf71a8) into main (369b9a7) will decrease coverage by 0.41%.
Report is 12 commits behind head on main.
The diff coverage is 2.03%.

Files Changed	Coverage Δ
bin/reth/src/db/mod.rs	7.27% <0.00%> (-0.28%)	⬇️
crates/storage/db/src/tables/models/sharded_key.rs	85.71% <0.00%> (-14.29%)	⬇️
bin/reth/src/db/diff.rs	0.53% <0.53%> (ø)
crates/primitives/src/prune/part.rs	100.00% <100.00%> (ø)
crates/storage/db/src/tables/models/accounts.rs	95.45% <100.00%> (ø)
...torage/db/src/tables/models/storage_sharded_key.rs	100.00% <100.00%> (ø)

... and 30 files with indirect coverage changes

Flag	Coverage Δ
integration-tests	16.29% <0.00%> (-0.09%)	⬇️
unit-tests	64.05% <2.03%> (-0.39%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
reth binary	25.24% <0.52%> (-1.92%)	⬇️
blockchain tree	83.04% <ø> (ø)
pipeline	89.82% <ø> (ø)
storage (db)	74.27% <33.33%> (-0.03%)	⬇️
trie	94.70% <ø> (ø)
txpool	45.40% <ø> (-0.34%)	⬇️
networking	77.63% <ø> (-0.02%)	⬇️
rpc	58.50% <ø> (+0.15%)	⬆️
consensus	64.46% <ø> (ø)
revm	33.08% <ø> (-0.61%)	⬇️
payload builder	6.61% <ø> (ø)
primitives	87.81% <100.00%> (-0.02%)	⬇️

[mattsse]

I like this feature,

impl is well documented.

[mattsse]

I'd like some docs about what diff exactly does behind the scenes

[Rjected]

just added a comment that is hopefully sufficient

[mattsse]

we call this a lot with all tables, which likely generates a lot of code due to how big the functions is.

we make it less generic over T, but we still new value etc.

so probably not worth the effort imo

[Rjected]

yeah, because cursor_read is generic over T, I'm not sure this can be simplified much further, even in code size, especially since we want to do this for every possible table