Implemented raw block storage and RPC endpoint for retrieving raw block by number #340
Conversation
There was a problem hiding this comment.
Looks good! Thank you! I left a couple of non-blocking comments inline.
Also:
- Let's create an issue to figure out how to do streaming.
- The number of block we'll create per year is probably going to be over 10 million and I'm not sure much of a problem it would be. Regardless of this, we may want to figure out some strategy for storing blocks in sub-directories (e.g., define an epoch and store blocks for each epoch in a separate sub-directory). Let's create an issue for this.
Yes, I agree with you, such a big number of files in a single directory might be inconvenient to manage. I introduced epochs in this PR, please take a look! The main idea here is that we get first 16 bits of block number as an epoch, it means, each epoch will store 65536 blocks, it's about 7.5 days per epoch with current block issuing frequency (one block per 10 seconds). |
| fn block_path(&self, block_num: u32) -> PathBuf { | ||
| let epoch = block_num >> 16; | ||
| let epoch_dir = self.store_dir.join(format!("{epoch:04x}")); | ||
| epoch_dir.join(format!("block_{block_num:08x}.dat")) | ||
| } |
There was a problem hiding this comment.
I think this is fine for now, but I would still create an issue to discuss what should be the length of an epoch. For example, our plan is to reduce block times to 3 - 4 seconds, in which case, the current approach would mean that an epoch is just 2 - 3 days. This may be OK but also may be too short.
There was a problem hiding this comment.
If we use epoch only for limiting file count in the directory, I think, current epoch size is good. Even if we want to use it for something else, 2-3 days is still good (for example, in Solana each epoch lasts for about 2 days).
There was a problem hiding this comment.
Sounds good! I guess we can always introduce a higher-level periods that would be something like top 14 bits.
One potential change here: should epoch be formatted as: {epoch:08x} rather than {epoch:04x}?
crates/store/src/state.rs
Outdated
| /// The underlying database. | ||
| db: Arc<Db>, |
There was a problem hiding this comment.
I would update the comment to something like: "The database which stores block headers, nullifiers, notes, and the latests states of accounts."
crates/store/src/state.rs
Outdated
| /// The underlying block store. | ||
| block_store: Arc<BlockStore>, |
There was a problem hiding this comment.
I would change the comment to something like: "The block store which stores full block contents for all blocks."
# Conflicts: # CHANGELOG.md # Cargo.lock # crates/store/src/db/mod.rs
Resolves #316