perf: bench merkle stage by joshieDo · Pull Request #1497 · paradigmxyz/reth

joshieDo · 2023-02-22T07:30:12Z

Continues the work from #1324

Fixes some issues on how the trie is inserting values into the DB during its execution, and how some of the testdata is being wrongly generated.

Mainly coming from the use of tx.put::<Table>(k,v) with DUPSORT tables (such as StoragesTrie, PlainStorageState, etc...) where the value of subkey is not actually overwritten but only appended. One needs to first delete the previous entry, before upserting.

Adds reth dump-stage merkle as well.

Future improvements:

trie DB should take a cursor instead
batch storage addresses (right now its one db.tx per address)

Co-authored-by: lambdaclass-user <github@lambdaclass.com>

joshieDo · 2023-02-24T06:16:40Z

Found some problematic block ranges again, so changing it to draft

gakonst · 2023-02-24T21:57:56Z

@joshieDo where'd you land here?

joshieDo · 2023-02-27T07:13:21Z

Fixed some issue on unwind during dump-stage
Fixed one issue with StorageHashingStage execution when going for the incremental branch. (not checking return from seek_by_key_subkey)

ci clippy failure unrelated

tried some other block ranges, and seems finally fine.

onbjerg

Smol nits, otherwise LGTM

crates/stages/src/stages/merkle.rs

crates/stages/src/trie/mod.rs

onbjerg

LGTM, not sure if CI related or fixed?

gakonst · 2023-03-01T01:47:23Z

@joshieDo apologies for this, #1568 broke your Dump Execution Stage command; do you mind fixing? you'll need to instantiate the Executor similarly to how I have changed it in the PR

gakonst · 2023-03-01T06:20:05Z

Sick - ty

rakita · 2023-03-01T10:40:56Z

crates/stages/src/stages/hashing_storage.rs

-                        if let Some(value) = val {
-                            hashed_storage.upsert(address, StorageEntry { key, value })?;
-                        }
+                        hashed_storage.upsert(hashed_address, StorageEntry { key, value: val })?;


Hey @joshieDo is this statement correct: if the storage key is not found (Is zero) it shouldn't be found in hashed table?
I did some refactoring and this came up. .filter(|v| v.key == key) was a bug but this shouldn't be ( I added unwrap_or_default that is why it compares to U256::ZERO)
want to double check

MegaRedHand and others added 26 commits February 10, 2023 17:12

Refactor insert_* to remove duplicate code

8f5ae02

Co-authored-by: lambdaclass-user <github@lambdaclass.com>

Add benchmark for MerkleStage

f35fbdc

Co-authored-by: lambdaclass-user <github@lambdaclass.com>

Add insert_accounts_and_storages fn

1f07aac

Co-authored-by: lambdaclass-user <github@lambdaclass.com>

Add random_transition to merkle_stage

7d42600

Co-authored-by: lambdaclass-user <github@lambdaclass.com>

Add accs_testdata

301eaed

Co-authored-by: lambdaclass-user <github@lambdaclass.com>

Add transition generation helper function

62d92d0

Co-authored-by: lambdaclass-user <github@lambdaclass.com>

Add insert_transitions helper function

dbe7ea7

Co-authored-by: lambdaclass-user <github@lambdaclass.com>

Unify *_testdata functions

ab90047

Co-authored-by: lambdaclass-user <github@lambdaclass.com>

Add custom setup for merkle stage

5260e62

Co-authored-by: lambdaclass-user <github@lambdaclass.com>

Merge branch 'main' into bench-merkle

5409340

Generate valid state roots in testdata

e134691

Co-authored-by: lambdaclass-user <github@lambdaclass.com>

Fix compile warnings

b655cd6

Co-authored-by: lambdaclass-user <github@lambdaclass.com>

Tidy up some clones

1d768e0

Co-authored-by: lambdaclass-user <github@lambdaclass.com>

Fix: update blocks before insertion

de91a79

Co-authored-by: lambdaclass-user <github@lambdaclass.com>

Fix: add iter()

07c6315

Co-authored-by: lambdaclass-user <github@lambdaclass.com>

Merge branch 'main' into bench-merkle

fa58c3c

Merge branch 'main' into bench-merkle

6460cc4

Merge remote-tracking branch 'origin/main' into bench-merkle

35d47ee

fix removal on trie

8eb7e6c

throw error if unwinded calculated root does not match

29d9caa

fix insertion on test_db for accounts and storages

e6cd629

make sure root is committed

aa608ee

delete target root for repeated unwinds

b46da0a

fix and speedup trie db methods

a33659e

set DEFAULT_NUM_BLOCKS for criterion

121d5b7

clippy

14feb94

joshieDo added C-bug An unexpected or incorrect behavior A-staged-sync Related to staged sync (pipelines and stages) C-perf A change motivated by improving speed, memory usage or disk footprint labels Feb 22, 2023

joshieDo changed the title ~~perf: bench merkle~~ perf: bench merkle stage Feb 22, 2023

joshieDo added 2 commits February 23, 2023 05:32

zip keys and values

c20dc58

clippy

a50d7f8

joshieDo marked this pull request as ready for review February 23, 2023 05:42

joshieDo requested review from gakonst, onbjerg, rakita and rkrasiuk as code owners February 23, 2023 05:42

joshieDo mentioned this pull request Feb 24, 2023

feat(rpc): add partial eth_getProof implementation #1515

Merged

joshieDo marked this pull request as draft February 24, 2023 06:16

joshieDo added 4 commits February 27, 2023 03:49

unwind execution stage too for merkle dump

d25a3df

Merge remote-tracking branch 'origin/main' into perf/bench-merkle

19f004e

fix hashing storage incremental

1c10b65

propagate error

e4c0a0e

joshieDo force-pushed the perf/bench-merkle branch from 28f7014 to e4c0a0e Compare February 27, 2023 07:01

clippy

472bfeb

joshieDo marked this pull request as ready for review February 27, 2023 07:10

onbjerg approved these changes Feb 27, 2023

View reviewed changes

crates/stages/src/stages/merkle.rs Outdated Show resolved Hide resolved

crates/stages/src/stages/merkle.rs Outdated Show resolved Hide resolved

crates/stages/src/trie/mod.rs Show resolved Hide resolved

joshieDo added 2 commits February 28, 2023 02:45

use existing consensus error for diff root

54710c1

add stage id MERKLE_BOTH

1f768e3

onbjerg approved these changes Feb 28, 2023

View reviewed changes

Merge branch 'main' into perf/bench-merkle

426ae37

add missing Executor to stage

283292f

gakonst merged commit 2884eae into main Mar 1, 2023

gakonst deleted the perf/bench-merkle branch March 1, 2023 06:20

rakita reviewed Mar 1, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf: bench merkle stage #1497

perf: bench merkle stage #1497
gakonst merged 40 commits intomainfrom
perf/bench-merkle

joshieDo commented Feb 22, 2023 •

edited

Loading

Uh oh!

joshieDo commented Feb 24, 2023 •

edited

Loading

Uh oh!

gakonst commented Feb 24, 2023

Uh oh!

joshieDo commented Feb 27, 2023 •

edited

Loading

Uh oh!

onbjerg left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

onbjerg left a comment

Uh oh!

gakonst commented Mar 1, 2023

Uh oh!

gakonst commented Mar 1, 2023

Uh oh!

rakita Mar 1, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

joshieDo commented Feb 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joshieDo commented Feb 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gakonst commented Feb 24, 2023

Uh oh!

joshieDo commented Feb 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

onbjerg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

onbjerg left a comment

Choose a reason for hiding this comment

Uh oh!

gakonst commented Mar 1, 2023

Uh oh!

gakonst commented Mar 1, 2023

Uh oh!

rakita Mar 1, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

joshieDo commented Feb 22, 2023 •

edited

Loading

joshieDo commented Feb 24, 2023 •

edited

Loading

joshieDo commented Feb 27, 2023 •

edited

Loading