feat: use chain-specific data dirs by Rjected · Pull Request #2495 · paradigmxyz/reth

Rjected · 2023-05-01T10:03:53Z

WIP attempt at changing the reth directory structure to revolve around a data directory with chain-specific directories for all other node data.

The new structure will revolve around the --datadir command line argument, which will set up folders for chains by either their chain ID, or name (e.g. mainnet), depending on whether or not we have specific names for the chain being run.

Fixes #2179

TODO:

make it compile
s/--data-dir/--datadir

Rjected · 2023-05-01T21:27:36Z

tree output now:

dan@Dans-MacBook-Pro ~/p/reth (dan/separate-dirs)> tree /Users/dan/Library/Application\ Support/reth/
/Users/dan/Library/Application Support/reth/
├── mainnet
│   ├── db
│   │   ├── mdbx.dat
│   │   └── mdbx.lck
│   ├── jwtsecret
│   │   └── jwt.hex
│   ├── net
│   │   └── known_peers.json
│   └── p2p
│       └── secret
└── reth.toml

6 directories, 6 files

codecov-commenter · 2023-05-01T21:36:54Z

Codecov Report

Merging #2495 (3c70582) into main (a887557) will decrease coverage by 0.07%.
The diff coverage is 26.11%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@            Coverage Diff             @@
##             main    #2495      +/-   ##
==========================================
- Coverage   71.68%   71.61%   -0.07%     
==========================================
  Files         488      489       +1     
  Lines       60828    61100     +272     
==========================================
+ Hits        43603    43758     +155     
- Misses      17225    17342     +117

Flag	Coverage Δ
integration-tests	`17.98% <0.00%> (-0.07%)`	⬇️
unit-tests	`66.64% <26.11%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
bin/reth/src/args/network_args.rs	`3.50% <0.00%> (-1.76%)`	⬇️
bin/reth/src/args/rpc_server_args.rs	`41.15% <0.00%> (+0.16%)`	⬆️
bin/reth/src/args/secret_key.rs	`0.00% <0.00%> (ø)`
bin/reth/src/config.rs	`6.25% <0.00%> (+0.36%)`	⬆️
bin/reth/src/dump_stage/execution.rs	`0.00% <0.00%> (ø)`
bin/reth/src/dump_stage/hashing_account.rs	`0.00% <0.00%> (ø)`
bin/reth/src/dump_stage/hashing_storage.rs	`0.00% <0.00%> (ø)`
bin/reth/src/dump_stage/merkle.rs	`0.00% <0.00%> (ø)`
bin/reth/src/stage/mod.rs	`1.78% <7.14%> (-1.82%)`	⬇️
bin/reth/src/dump_stage/mod.rs	`5.35% <9.09%> (-2.06%)`	⬇️
... and 8 more

... and 56 files with indirect coverage changes

Rjected · 2023-05-01T21:39:50Z

tree output when setting the datadir:

dan@Dans-MacBook-Pro ~/p/reth (dan/separate-dirs)> cargo run -- node --debug.tip 0xdc2d938e4cd0a149681e9e04352953ef5ab399d59bcd5b0357f6c0797470a524 --datadir ~/temp-reth-datadir
..... syncing logs
dan@Dans-MacBook-Pro ~/p/reth (dan/separate-dirs)> tree ~/temp-reth-datadir/
/Users/dan/temp-reth-datadir/
├── db
│   ├── mdbx.dat
│   └── mdbx.lck
├── jwtsecret
│   └── jwt.hex
└── p2p
    └── secret

4 directories, 4 files

gakonst

some ideas to simplify further per my comment in the original issue. also we want reth.toml to also follow the datadir, vs being left hanging outside of it. NO file should be outside of the datadir.

bin/reth/src/chain/import.rs

bin/reth/src/stage/mod.rs

bin/reth/src/dirs.rs

Rjected · 2023-05-01T22:12:14Z

Ok, now:

dan@Dans-MacBook-Pro ~/p/reth (dan/separate-dirs)> cargo run -- node --debug.tip 0xdc2d938e4cd0a149681e9e04352953ef5ab399d59bcd5b0357f6c0797470a524
    Finished dev [unoptimized + debuginfo] target(s) in 0.30s
     Running `target/debug/reth node --debug.tip 0xdc2d938e4cd0a149681e9e04352953ef5ab399d59bcd5b0357f6c0797470a524`
2023-05-01T22:10:34.912767Z  INFO reth 0.1.0 starting
2023-05-01T22:10:34.918184Z  INFO Configuration loaded path="/Users/dan/Library/Application Support/reth/mainnet/reth.toml"
2023-05-01T22:10:34.918214Z  INFO Opening database path="/Users/dan/Library/Application Support/reth/mainnet/db"
...rest of syncing logs
dan@Dans-MacBook-Pro ~/p/reth (dan/separate-dirs)> tree /Users/dan/Library/Application\ Support/reth/
/Users/dan/Library/Application Support/reth/
└── mainnet
    ├── db
    │   ├── mdbx.dat
    │   └── mdbx.lck
    ├── jwtsecret
    │   └── jwt.hex
    ├── p2p
    │   └── secret
    └── reth.toml

5 directories, 5 files

With custom dir:

dan@Dans-MacBook-Pro ~/p/reth (dan/separate-dirs) [1]> cargo run -- node --debug.tip 0xdc2d938e4cd0a149681e9e04352953ef5ab399d59bcd5b0357f6c0797470a524 --datadir ~/temp-reth-datadir
   Compiling reth v0.1.0 (/Users/dan/projects/reth/bin/reth)
    Finished dev [unoptimized + debuginfo] target(s) in 10.48s
     Running `target/debug/reth node --debug.tip 0xdc2d938e4cd0a149681e9e04352953ef5ab399d59bcd5b0357f6c0797470a524 --datadir /Users/dan/temp-reth-datadir`
2023-05-01T22:06:17.296553Z  INFO reth 0.1.0 starting
2023-05-01T22:06:17.301951Z  INFO Configuration loaded path="/Users/dan/temp-reth-datadir/reth.toml"
2023-05-01T22:06:17.301979Z  INFO Opening database path="/Users/dan/temp-reth-datadir/db"
... rest of syncing logs
dan@Dans-MacBook-Pro ~/p/reth (dan/separate-dirs)> tree ~/temp-reth-datadir/
/Users/dan/temp-reth-datadir/
├── db
│   ├── mdbx.dat
│   └── mdbx.lck
├── jwtsecret
│   └── jwt.hex
├── p2p
│   └── secret
└── reth.toml

4 directories, 5 files

Rjected · 2023-05-01T22:13:50Z

removing the extra dirs next

Rjected · 2023-05-01T22:23:16Z

New structure:

dan@Dans-MacBook-Pro ~/p/reth (dan/separate-dirs)> tree /Users/dan/Library/Application\ Support/reth/
/Users/dan/Library/Application Support/reth/
└── mainnet
    ├── db
    │   ├── mdbx.dat
    │   └── mdbx.lck
    ├── discovery-secret
    ├── jwt.hex
    ├── known-peers.json
    └── reth.toml

3 directories, 6 files

gakonst · 2023-05-02T19:22:50Z

bin/reth/src/chain/init.rs

+    /// The path to the database folder. If not specified, it will be set in the data dir for the
+    /// chain being used.
+    #[arg(long, value_name = "PATH", verbatim_doc_comment)]
+    db: Option<PathBuf>,


let's remove in follow-up, open issue and we can have 3rd party close it

Tracked in #2528

gakonst · 2023-05-02T19:23:00Z

bin/reth/src/db/mod.rs

+    /// The path to the database folder. If not specified, it will be set in the data dir for the
+    /// chain being used.
+    #[arg(long, value_name = "PATH", verbatim_doc_comment)]
+    db: Option<PathBuf>,



gakonst · 2023-05-02T19:23:11Z

bin/reth/src/dirs.rs

+
+    /// Returns the path to the db directory for this chain.
+    pub fn db_path(&self) -> PathBuf {
+        self.0.join("db").into()
+    }
+
+    /// Returns the path to the reth p2p secret key for this chain.
+    pub fn p2p_secret_path(&self) -> PathBuf {
+        self.0.join("discovery-secret").into()
+    }
+
+    /// Returns the path to the known peers file for this chain.
+    pub fn known_peers_path(&self) -> PathBuf {
+        self.0.join("known-peers.json").into()
+    }
+
+    /// Returns the path to the config file for this chain.
+    pub fn config_path(&self) -> PathBuf {
+        self.0.join("reth.toml").into()
+    }
+
+    /// Returns the path to the jwtsecret file for this chain.
+    pub fn jwt_path(&self) -> PathBuf {
+        self.0.join("jwt.hex").into()
+    }
 }


Rjected added 2 commits May 1, 2023 11:58

add initial data dir

23993ad

add new helpers for data dir derived paths

d62e8f4

barnabasbusa mentioned this pull request May 1, 2023

adding reth chart ethpandaops/ethereum-helm-charts#173

Merged

Rjected added 6 commits May 1, 2023 18:16

set up dirs in reth node

0185d0c

add for stage and import

55ca005

add datadir support for the rest of the commands

f0fe547

change to --datadir

a11f3b0

add too_many_arguments lint relaxation for start_servers

5cf0958

fix dirs rustdoc

16790e5

Rjected changed the title ~~wip: use chain-specific data dirs~~ feat: use chain-specific data dirs May 1, 2023

use datadir for reth config

3b1bc40

Rjected marked this pull request as ready for review May 1, 2023 21:27

Rjected requested a review from onbjerg as a code owner May 1, 2023 21:27

Rjected requested a review from gakonst May 1, 2023 21:28

gakonst requested changes May 1, 2023

View reviewed changes

bin/reth/src/chain/import.rs Outdated Show resolved Hide resolved

bin/reth/src/stage/mod.rs Outdated Show resolved Hide resolved

bin/reth/src/stage/mod.rs Outdated Show resolved Hide resolved

bin/reth/src/dirs.rs Outdated Show resolved Hide resolved

move config into data dir

ced168c

Rjected added 2 commits May 2, 2023 00:22

remove single level dirs

f664ee6

rename discoverysecret

9da1600

Rjected added 2 commits May 2, 2023 00:25

remove db option from stage and import

dd2d403

rename to jwt.hex

3c70582

Rjected requested a review from gakonst May 2, 2023 14:44

gakonst approved these changes May 2, 2023

View reviewed changes

gakonst merged commit ecc7ae9 into main May 2, 2023

gakonst deleted the dan/separate-dirs branch May 2, 2023 19:23

This was referenced May 2, 2023

Remove remaining --db options from cli commands #2528

Closed

use datadir instead of db arg in hive paradigmxyz/hive#24

Merged

mightypenguin mentioned this pull request May 3, 2023

fix: --p2p-secret-key silently reverts to default location if file is not found (#2498) #2508

Closed

mattsse mentioned this pull request May 4, 2023

Cli paths broken, user input ignored #2551

Closed

Conversation

Rjected commented May 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Rjected commented May 1, 2023

Uh oh!

codecov-commenter commented May 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Rjected commented May 1, 2023

Uh oh!

gakonst left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Rjected commented May 1, 2023

Uh oh!

Rjected commented May 1, 2023

Uh oh!

Rjected commented May 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gakonst May 2, 2023

Choose a reason for hiding this comment

Uh oh!

Rjected May 2, 2023

Choose a reason for hiding this comment

Uh oh!

gakonst May 2, 2023

Choose a reason for hiding this comment

Uh oh!

gakonst May 2, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Rjected commented May 1, 2023 •

edited

Loading

codecov-commenter commented May 1, 2023 •

edited

Loading

Rjected commented May 1, 2023 •

edited

Loading