Skip to content
This repository was archived by the owner on Aug 27, 2024. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
368c7bc
Update storage.md
shawntabrizi Sep 24, 2019
ac6d78b
skeleton of storage doc
shawntabrizi Sep 24, 2019
09fb5dd
typo
shawntabrizi Sep 24, 2019
7b8b959
Update storage.md
shawntabrizi Sep 24, 2019
97e85f5
Integrate feedback
shawntabrizi Sep 26, 2019
402bba0
fixes
shawntabrizi Sep 27, 2019
89c720c
clarify kind of node
shawntabrizi Sep 27, 2019
d77a40b
Clarify
shawntabrizi Sep 27, 2019
999d39f
typo, fix line width
shawntabrizi Sep 27, 2019
68a8592
fixes
shawntabrizi Sep 27, 2019
5bbead7
replace misleading data
shawntabrizi Sep 27, 2019
ec323ac
Update docs/conceptual/core/storage.md
shawntabrizi Sep 28, 2019
5813b88
Update docs/conceptual/core/storage.md
shawntabrizi Sep 28, 2019
d6d7292
Update docs/conceptual/core/storage.md
shawntabrizi Sep 28, 2019
1c5993e
Update docs/conceptual/core/storage.md
shawntabrizi Sep 28, 2019
ab8caad
update from feedback
shawntabrizi Sep 28, 2019
e3ceff2
Merge branch 'shawntabrizi-storage-doc' of https://github.com/shawnta…
shawntabrizi Sep 28, 2019
f4570ee
Add a section on why to use child tries
shawntabrizi Sep 28, 2019
9db4f54
Update docs/conceptual/core/storage.md
shawntabrizi Sep 28, 2019
23f725b
Update docs/conceptual/core/storage.md
shawntabrizi Sep 28, 2019
59f529a
Update docs/conceptual/core/storage.md
shawntabrizi Sep 28, 2019
07026a3
Update docs/conceptual/core/storage.md
shawntabrizi Sep 28, 2019
9240c48
Update docs/conceptual/core/storage.md
shawntabrizi Sep 28, 2019
2cc0216
Update docs/conceptual/core/storage.md
shawntabrizi Sep 28, 2019
9cb7da5
Update docs/conceptual/core/storage.md
shawntabrizi Sep 28, 2019
df67541
Update docs/conceptual/core/storage.md
shawntabrizi Sep 28, 2019
96fee72
Update docs/conceptual/core/storage.md
shawntabrizi Sep 28, 2019
330d8a1
Update docs/conceptual/core/storage.md
shawntabrizi Sep 28, 2019
72550f5
Final fixes
shawntabrizi Sep 28, 2019
2018486
add todo
shawntabrizi Sep 28, 2019
0e33445
Merge branch 'source' into shawntabrizi-storage-doc
shawntabrizi Sep 29, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 82 additions & 3 deletions docs/conceptual/core/storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,87 @@
title: Storage
---

* Key Value Database, RocksDB
The Substrate uses a simple a key-value data store implemented as a
database-backed modified Merkle tree.

* Trie abstraction
## Key-Value Database

* Runtime Storage abstraction
Substrate implements its storage database with [RocksDB](https://rocksdb.org/),
a persistent key-value store for fast storage environments.

This is used for all the components of Substrate which require persistent
Comment thread
shawntabrizi marked this conversation as resolved.
Outdated
storage such as:

- Substrate clients
- Substrate light-clients
- Off-chain workers

## Trie Abstraction

One advantage of using a simple key-value store is that you are able to easily
abstract other storage structures on top.
Comment thread
shawntabrizi marked this conversation as resolved.
Outdated

Substrate uses a Base-16 Modified Merkle Patricia tree ("trie") from
[`paritytech/trie`](https://github.com/paritytech/trie) to provide a trie
structure whose contents can be modified and whose root hash is recalculated
efficiently.

Tries are important tool for blockchains because they allow for efficient
Comment thread
shawntabrizi marked this conversation as resolved.
Outdated
Comment thread
shawntabrizi marked this conversation as resolved.
Outdated
storing and sharing of the historical block state. The root hash of the trie
provides a cryptographic fingerprint of the final resulting storage after
executing all state transitions by a blockchain. Thus, two blockchain nodes can
easily verify they have the same final state by simply comparing their trie
root.
Comment thread
shawntabrizi marked this conversation as resolved.
Outdated
Comment thread
shawntabrizi marked this conversation as resolved.
Outdated

Accessing trie data is costly. Each read operation takes O(log N) time, where N
is the number of elements stored in the trie. To mitigate this, we use a key
value cache.
Comment thread
shawntabrizi marked this conversation as resolved.
Outdated

All trie nodes are stored in RocksDB and part of the trie state can get pruned,
i.e. a key-value pair can be deleted from the storage when it is out of pruning
Comment thread
joepetrowski marked this conversation as resolved.
range for non archive nodes. The trie nodes partial path prefixes the hash of
Comment thread
shawntabrizi marked this conversation as resolved.
Outdated
the encoded node for the RocksDB key to avoid key collision. We do not use
[reference counting](http://en.wikipedia.org/wiki/Reference_counting) for
performance reasons.

### State Trie

Substrate has a single main trie, called the state trie, whose changing root
Comment thread
shawntabrizi marked this conversation as resolved.
Outdated
hash is placed in each block header. This is used to easily verify the state of
the blockchain and provide a basis for light clients to verify proofs.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would be interested to know more about the light clients and what exactly is proved to them. I know they only read block headers (not full blocks) so they get to see the state root before and after each block, but how do they actually know that the transactions in the block were executed correctly and lead to the resulting state root?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is proved to the light client is validity of an operation over a state of the chain. Say the light client does not have the full state, he get the info he need with a proof that it comes from a valid full state.

simplier example should be:


Querying keys from state trie, the request will be (from cone/network/src/protocol/message.rs):

	#[derive(Debug, PartialEq, Eq, Clone, Encode, Decode)]
	/// Remote storage read request.
	pub struct RemoteReadRequest<H> {
		/// Unique request id.
		pub id: RequestId,
		/// Block at which to perform call.
		pub block: H,
		/// Storage key.
		pub keys: Vec<Vec<u8>>,
	}

then the reply will be

#[derive(Debug, PartialEq, Eq, Clone, Encode, Decode)]
/// Remote read response.
pub struct RemoteReadResponse {
	/// Id of a request this response was made for.
	pub id: RequestId,
	/// Read proof.
	pub proof: Vec<Vec<u8>>,
}

To make sense out of this reply we fetch request by id and the state trie root of the requested block hash, then we query keys over the proof.
The proof is here a subset of the memorydb the state trie is build upon.
This subset contains only the trie nodes the full client does record when running the query on its side (for every keys from the request).
Then the light client run the same keys query over a trie build from the block state root (he know it from the cht) and the trie nodes he just received (field proof of response). From this incomplete state trie he can get the resulting values of every keys in input (and since every node of trie refers to each over through a crypto hash it is proof it is in the chain state).


but this way of recording some operation on a full client and re-executing over this record on a light client can apply to many thing (if you look in message.rs there is a few query, call for instance uses executor on light blockchain).
Possible future design for substrate light client seems to be allowing evaluation of some wasm code which could be a better solution than chaining queries result like it was done in eth. (better solution to reducing the number of queries roundtrip).

Not 100% sure on the following point, but proving the transition between two consecutive blocks is not something we do (I think), we just rely on the fact that the blocks got validated and we know they are chained. Surely a full client can execute the transition by running a full block on its state, so it should be possible for him to send to a light client all accessed db keys during the full block execution and next root calculation, and the light client will be able to execute the block on those keys and produce next root. But I guess the proof will be really massive, so relying on network having validating those state may be enough in the light client usecase.

But you can run any call already (see RemoteCallRequest) or query the deltas between block through RemoteChangesRequest.

@shawntabrizi shawntabrizi Sep 27, 2019

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a slightly simpler, but hopefully still fully accurate explanation of light client proofs come from merkle proof in general.

image

from here: https://www.quora.com/Cryptography-How-does-a-Merkle-proof-actually-work

The image hopefully shows that if you want to prove TX3 is in the trie, you dont need ALL the nodes, just the full branch will leads to TX3 and all the nodes next to the nodes on that full branch. Obviously WAYWAY less data on big tries like the ones on blockchains.

Full nodes act as the provider to light clients of the proof it needs. They do so pretty selflessly, but there are thoughts to have light clients start micro-payments to full nodes for their services.


This trie only stores content for the canonical chain, not forks. There is a
separate `state_db` layer that maintain the trie state with references counted
Comment thread
shawntabrizi marked this conversation as resolved.
Outdated
in memory for all that is non canonical. See JournalDB.
Comment thread
shawntabrizi marked this conversation as resolved.
Outdated

### Child Trie

Substrate also provides an API to generate new child tries with their own root
hash that can be used in the runtime.
Comment thread
shawntabrizi marked this conversation as resolved.
Outdated

Child tries are identical to main state trie, except their root is stored and
Comment thread
shawntabrizi marked this conversation as resolved.
Outdated
updated in the main trie instead of the block header. Since their headers are a
Comment thread
shawntabrizi marked this conversation as resolved.
Outdated
part of the main state trie, it is still easy to verify of the complete node
state when it includes child tries.

## Runtime Storage API

The Substrate runtime support library provides utilities which generates unique,
Comment thread
shawntabrizi marked this conversation as resolved.
Outdated
deterministic keys for your runtime module storage items. These storage items
are placed in the state trie and are accessible by querying the trie by key.

## Next Steps

### Learn More

- Learn how to add [storage items](development/module/storage.md) into your
Substrate runtime modules.

### Examples

- View an example of creating [child tries]() in your Substrate runtime module.
Comment thread
joepetrowski marked this conversation as resolved.
Outdated

### References

- Visit the reference docs for
[`paritytech/trie`](https://substrate.dev/rustdocs/master/trie_db/trait.Trie.html).
2 changes: 1 addition & 1 deletion docs/development/module/storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ TODO

### Examples

TODO
* View this example to see how you can use a `double_map` to act as a `killable` single-map.

### References

Expand Down