Worldstate-only resync behavior by garyschulte · Pull Request #4875 · besu-eth/besu

garyschulte · 2023-01-03T21:57:11Z

PR description

Implements a debug rpc endpoint and a behavior to resync just the worldstate.

Fixed Issue(s)

Documentation

I thought about documentation and added the doc-change-required label to this PR if
updates are required.

Changelog

I thought about the changelog and included a changelog update if required.

garyschulte · 2023-01-05T00:31:15Z

@shemnon as far as UX for a resync feature, 'touching' a file in the data dir is rather simple to implement and accessible. A debug rpc endpoint may or may not be a good idea for this kind of functionality. What do you think about file vs rpc, file and rpc, or some other command pattern to trigger this behavior?

shemnon · 2023-01-05T15:20:10Z

I'm not a devops maven, I'd think we should get the UX feedback from someone who manages fleets of nodes. Touching a file feels hacky to me but I've discovered lots of DevOps "best practices" feel hacky to me.

matkt · 2023-01-10T11:21:31Z

I'm not a devops maven, I'd think we should get the UX feedback from someone who manages fleets of nodes. Touching a file feels hacky to me but I've discovered lots of DevOps "best practices" feel hacky to me.

I'm no devops but I personally prefer an RPC call or a flag to a file that feels DIY.
I would just say that the RPC call must be admin because it seems too critical to me to simply be in debug

fab-10

LGTM, just a method that seems unused

fab-10 · 2023-01-10T18:25:19Z

+  public void reset() {
+    this.badBlocks.invalidateAll();
+    this.badHeaders.invalidateAll();
+    this.latestValidHashes.invalidateAll();
+  }
+


Where is this called?

https://github.com/hyperledger/besu/pull/4875/files#diff-cc598aa8b256f675dfcbd331db0955fd273916856708992c92364aa373405f12R49

As discussed oob, we will use debug namespace

Signed-off-by: garyschulte <garyschulte@gmail.com>

…ng on subsequent truncation at snap sync start Signed-off-by: garyschulte <garyschulte@gmail.com>

Signed-off-by: garyschulte <garyschulte@gmail.com>

…ndling during/after resync Signed-off-by: garyschulte <garyschulte@gmail.com>

Signed-off-by: garyschulte <garyschulte@gmail.com>

Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>

* use debug rpc endpoint to resync worldstate * Reset transaction pool state every time the initial sync is done Signed-off-by: garyschulte <garyschulte@gmail.com> Co-authored-by: Fabio Di Fabio <fabio.difabio@consensys.net>

* use debug rpc endpoint to resync worldstate * Reset transaction pool state every time the initial sync is done Signed-off-by: garyschulte <garyschulte@gmail.com> Signed-off-by: Simon Dudley <simon.dudley@consensys.net> Co-authored-by: Fabio Di Fabio <fabio.difabio@consensys.net>

* Fix transaction pool issue (#4964) * Cache empty slots (#4874) * clear after each block and copy during clone * fix transaction pool issue * add block replay * support in-memory snapshots * Keep Worldstate Storage open for Bonsai archive latest layer (#5039) * bonsai layered worldstate subscription * unsubscribe from worldstatestorage on close of BonsaiLayeredWorldState * minor txpool logging improvements * Avoid triggering a calculate root hash when empty slot cache is not empty. * use the updater cache to get an account during block processing (#4897) * Worldstate-only resync behavior (#4875) * use debug rpc endpoint to resync worldstate * Reset transaction pool state every time the initial sync is done * init heal code --------- Signed-off-by: Karim TAAM <karim.t2am@gmail.com> Signed-off-by: garyschulte <garyschulte@gmail.com> Signed-off-by: Justin Florentine <justin+github@florentine.us> Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> Signed-off-by: ahamlat <ameziane.hamlat@consensys.net> Signed-off-by: Simon Dudley <simon.dudley@consensys.net> Co-authored-by: matkt <karim.t2am@gmail.com> Co-authored-by: garyschulte <garyschulte@gmail.com> Co-authored-by: ahamlat <ameziane.hamlat@consensys.net> Co-authored-by: Simon Dudley <simon.dudley@consensys.net> Co-authored-by: Fabio Di Fabio <fabio.difabio@consensys.net>

* use debug rpc endpoint to resync worldstate * Reset transaction pool state every time the initial sync is done Signed-off-by: garyschulte <garyschulte@gmail.com> Co-authored-by: Fabio Di Fabio <fabio.difabio@consensys.net>

garyschulte force-pushed the feature/bonsai-resync-ws branch 2 times, most recently from 44a7cec to 7fd3377 Compare January 5, 2023 00:22

garyschulte force-pushed the feature/bonsai-resync-ws branch 4 times, most recently from 0a1ffae to 0346dad Compare January 10, 2023 00:13

garyschulte requested review from fab-10, gezero and matkt January 10, 2023 01:40

garyschulte marked this pull request as ready for review January 10, 2023 04:13

garyschulte force-pushed the feature/bonsai-resync-ws branch from 39aa3a3 to bf76408 Compare January 10, 2023 04:14

matkt previously requested changes Jan 10, 2023

View reviewed changes

Comment thread ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/jsonrpc/RpcMethod.java

fab-10 approved these changes Jan 10, 2023

View reviewed changes

garyschulte force-pushed the feature/bonsai-resync-ws branch 2 times, most recently from bf59581 to ee520db Compare January 11, 2023 21:55

garyschulte force-pushed the feature/bonsai-resync-ws branch from ee520db to 8bb393e Compare January 11, 2023 22:26

garyschulte and others added 9 commits January 11, 2023 20:16

initial stab at a light touch worldstate resync behavior

3344dd3

Signed-off-by: garyschulte <garyschulte@gmail.com>

added explicit worldstate clear to FastSyncDownloaderFactory

fa57cac

Signed-off-by: garyschulte <garyschulte@gmail.com>

explicitly clear worldstate for snap and checkpoint rather than relyi…

d300685

…ng on subsequent truncation at snap sync start Signed-off-by: garyschulte <garyschulte@gmail.com>

use debug rpc endpoint to resync worldstate

7c150f6

Signed-off-by: garyschulte <garyschulte@gmail.com>

reset sync status on worldstate resync, disable/enable transaction ha…

9f34c6b

…ndling during/after resync Signed-off-by: garyschulte <garyschulte@gmail.com>

reset bad blocks manager on resync

8d42daa

Signed-off-by: garyschulte <garyschulte@gmail.com>

TransactionPoolFactoryTest's and accessor for isEnabled()

e891c69

Signed-off-by: garyschulte <garyschulte@gmail.com>

remove todo comment

19a7145

Signed-off-by: garyschulte <garyschulte@gmail.com>

Reset transaction pool state every time the initial sync is done

d830f2a

Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>

garyschulte force-pushed the feature/bonsai-resync-ws branch from 8bb393e to d830f2a Compare January 12, 2023 04:17

garyschulte enabled auto-merge (squash) January 12, 2023 04:19

garyschulte merged commit 96972ff into besu-eth:main Jan 12, 2023

garyschulte deleted the feature/bonsai-resync-ws branch January 12, 2023 04:48

garyschulte restored the feature/bonsai-resync-ws branch January 17, 2023 16:20

siladu mentioned this pull request Feb 7, 2023

Adds to 22.10.4 #5054

Merged

garyschulte deleted the feature/bonsai-resync-ws branch January 24, 2024 21:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Worldstate-only resync behavior#4875

Worldstate-only resync behavior#4875
garyschulte merged 9 commits into
besu-eth:mainfrom
garyschulte:feature/bonsai-resync-ws

garyschulte commented Jan 3, 2023 •

edited

Loading

Uh oh!

garyschulte commented Jan 5, 2023

Uh oh!

shemnon commented Jan 5, 2023

Uh oh!

Uh oh!

matkt commented Jan 10, 2023

Uh oh!

fab-10 left a comment

Uh oh!

fab-10 Jan 10, 2023

Uh oh!

garyschulte Jan 11, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

garyschulte commented Jan 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR description

Fixed Issue(s)

Documentation

Changelog

Uh oh!

garyschulte commented Jan 5, 2023

Uh oh!

shemnon commented Jan 5, 2023

Uh oh!

Uh oh!

matkt commented Jan 10, 2023

Uh oh!

fab-10 left a comment

Choose a reason for hiding this comment

Uh oh!

fab-10 Jan 10, 2023

Choose a reason for hiding this comment

Uh oh!

garyschulte Jan 11, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

garyschulte commented Jan 3, 2023 •

edited

Loading