feat: add `reth db snapshot transactions | receipts` commands by joshieDo · Pull Request #5007 · paradigmxyz/reth

joshieDo · 2023-10-13T00:13:56Z

moved the read + parallel hash calculation of the transaction table used on TxLookupStage ( cc @onbjerg @mattsse )
Adds SegmentHeader as a NippyJar header. Currently holds the block and transaction range of the snapshot.
adds reth db snapshot transactions command
adds reth db snapshot receipts command

codecov · 2023-10-13T00:39:51Z

Codecov Report

Merging #5007 (e5e71be) into main (4dc15c3) will increase coverage by 44.11%.
Report is 15 commits behind head on main.
The diff coverage is 9.88%.

❗ Current head e5e71be differs from pull request most recent head efeb602. Consider uploading reports for the commit efeb602 to get more accurate results

Files	Coverage Δ
crates/interfaces/src/provider.rs	`100.00% <ø> (+100.00%)`	⬆️
crates/primitives/src/transaction/mod.rs	`79.69% <ø> (+42.93%)`	⬆️
crates/stages/src/stages/tx_lookup.rs	`97.08% <100.00%> (+94.89%)`	⬆️
crates/storage/codecs/src/lib.rs	`93.91% <ø> (+57.14%)`	⬆️
crates/storage/db/src/snapshot.rs	`84.31% <100.00%> (+84.31%)`	⬆️
...torage/provider/src/providers/database/provider.rs	`77.39% <100.00%> (+49.91%)`	⬆️
crates/primitives/src/snapshot/segment.rs	`46.66% <53.84%> (+46.66%)`	⬆️
bin/reth/src/db/snapshots/headers.rs	`0.00% <0.00%> (ø)`
bin/reth/src/db/snapshots/bench.rs	`0.00% <0.00%> (ø)`
crates/storage/provider/src/traits/transactions.rs	`0.00% <0.00%> (ø)`
... and 8 more

... and 382 files with indirect coverage changes

Flag	Coverage Δ
integration-tests	`25.96% <0.00%> (-0.11%)`	⬇️
unit-tests	`61.74% <9.88%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
reth binary	`40.87% <0.00%> (+15.09%)`	⬆️
blockchain tree	`81.11% <ø> (+52.64%)`	⬆️
pipeline	`89.41% <100.00%> (+84.37%)`	⬆️
storage (db)	`74.96% <31.51%> (+44.98%)`	⬆️
trie	`95.00% <ø> (+72.46%)`	⬆️
txpool	`62.05% <ø> (+20.67%)`	⬆️
networking	`78.86% <ø> (+47.96%)`	⬆️
rpc	`58.91% <ø> (+32.43%)`	⬆️
consensus	`75.96% <ø> (+50.89%)`	⬆️
revm	`24.38% <ø> (+14.53%)`	⬆️
payload builder	`14.16% <ø> (ø)`
primitives	`85.68% <53.84%> (+56.50%)`	⬆️

crates/snapshot/src/segments/mod.rs

bin/reth/src/db/snapshots/receipts.rs

bin/reth/src/db/snapshots/transactions.rs

crates/interfaces/src/provider.rs

crates/snapshot/src/segments/mod.rs

crates/storage/db/src/tables/mod.rs

crates/storage/provider/src/traits/transactions.rs

Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>

mattsse

lgtm,

the comment re parallel hashing can be addressed separately, I wonder, do we use the same logic in the mdbx implementation, if so we also need to revise that

crates/interfaces/src/provider.rs

mattsse · 2023-10-25T17:18:42Z

crates/storage/provider/src/providers/database/provider.rs

+        let tx_range_size = tx_range.clone().count();
+        let tx_walker = tx_cursor.walk_range(tx_range)?;
+
+        let chunk_size = (tx_range_size / rayon::current_num_threads()).max(1);


I think this can be problematic, because this can negatively impact everything else.

we're usually calling this on a per-block basis, right?

so in worst case we're spawning num CPU parallel worker threads with >1 txs.

the cutoff should be way higher so that the overhead of using the threadpool is worth it.

do we have benchmarks for this?

let's say we have 200 tx in a block and 16cores results in a chunk size of 12, which I think is a bit too low.

I think we should rather define a minimum txs per thread, maybe go with 32 or something until we have benchmarks, then we also need a variant where we don't even need to spawn a single thread.

but we can do all of this in a followup if we track it

Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>

joshieDo added 12 commits October 6, 2023 16:59

move tx_lookup hash calculation to db crate

5b429fa

update doc on tx hash calc

8090431

Merge remote-tracking branch 'origin/main' into joshie/snapshot-bin-2

f5762b4

use databaseprovider instead

8eaab06

add find_transaction_range

95e0574

cargo deps

e03a3c0

add TransactionsProviderExt

16e4457

add transactions snapshot segment

a6edbc4

snapshot benches compare results

765ea2e

add TransactionNotFound error

1b41008

add additional empty provider impls to SnapshotProvider

b6789e4

add SegmentHeader to snapshot jars

a900860

joshieDo added C-enhancement New feature or request A-static-files Related to static files labels Oct 13, 2023

joshieDo added 5 commits October 13, 2023 00:18

add wip transaction_by_hash

9ae5939

fix doc test

96f77fc

fmt

16c33a7

add with_hash to walk bench as well

d14a483

fix recover_hashes range

055a818

joshieDo added 6 commits October 16, 2023 12:32

Merge remote-tracking branch 'origin/main' into joshie/snapshot-bin-2

5bef37f

fix doc test

a66b603

fix test_snap

f327d91

share dataset_for_compression implementation

5582db3

add snapshot segment

13136ed

add receipts command

d4c8b33

joshieDo changed the title ~~feat: add reth db snapshot transactions command~~ feat: add reth db snapshot transactions | receipts commands Oct 16, 2023

joshieDo marked this pull request as ready for review October 16, 2023 13:39

joshieDo requested review from onbjerg and rakita as code owners October 16, 2023 13:39

joshieDo mentioned this pull request Oct 23, 2023

feat: move ZstdDictionary inside NippyJar and create a snapshot manager #5139

Merged

mattsse reviewed Oct 23, 2023

View reviewed changes

crates/snapshot/src/segments/mod.rs Show resolved Hide resolved

shekhirin reviewed Oct 23, 2023

View reviewed changes

joshieDo and others added 10 commits October 24, 2023 11:10

Update bin/reth/src/db/snapshots/receipts.rs

b4d1d1e

Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>

Update bin/reth/src/db/snapshots/transactions.rs

79c72db

Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>

Update bin/reth/src/db/snapshots/transactions.rs

8880d6f

Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>

Update bin/reth/src/db/snapshots/receipts.rs

6224354

Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>

Update crates/interfaces/src/provider.rs

8e56d1d

Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>

add transaction_hashes_by_range to provider

4fb1871

add TxHashOrNumber type

9ab6333

Update bin/reth/src/db/snapshots/receipts.rs

c562e0c

Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>

Update bin/reth/src/db/snapshots/transactions.rs

27d6ab2

Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>

Merge remote-tracking branch 'origin/main' into joshie/snapshot-bin-2

21edb80

joshieDo requested review from mattsse and shekhirin October 24, 2023 11:05

joshieDo added 2 commits October 24, 2023 11:11

clippy

2f8d303

yes clippy

e5e71be

joshieDo requested a review from Rjected as a code owner October 24, 2023 12:01

fix doc

e920318

mattsse approved these changes Oct 25, 2023

View reviewed changes

joshieDo added 2 commits October 26, 2023 11:33

Merge remote-tracking branch 'origin/main' into joshie/snapshot-bin-2

30c4748

fmt

efeb602

joshieDo mentioned this pull request Oct 26, 2023

Investigate TransactionsProviderExt::transaction_hashes_by_range performance #5190

Closed

shekhirin approved these changes Oct 26, 2023

View reviewed changes

joshieDo added this pull request to the merge queue Oct 26, 2023

Merged via the queue into main with commit 0116b80 Oct 26, 2023

joshieDo deleted the joshie/snapshot-bin-2 branch October 26, 2023 12:11

mattsse pushed a commit that referenced this pull request Nov 8, 2023

feat: add reth db snapshot transactions | receipts commands (#5007)

351e8e6

Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add `reth db snapshot transactions | receipts` commands#5007

feat: add `reth db snapshot transactions | receipts` commands#5007
joshieDo merged 39 commits intomainfrom
joshie/snapshot-bin-2

joshieDo commented Oct 13, 2023 •

edited

Loading

Uh oh!

codecov bot commented Oct 13, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mattsse left a comment

Uh oh!

Uh oh!

mattsse Oct 25, 2023

Uh oh!

joshieDo Oct 26, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

joshieDo commented Oct 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Oct 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mattsse left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mattsse Oct 25, 2023

Choose a reason for hiding this comment

Uh oh!

joshieDo Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

joshieDo commented Oct 13, 2023 •

edited

Loading

codecov bot commented Oct 13, 2023 •

edited

Loading