-
Notifications
You must be signed in to change notification settings - Fork 209
Conceptual: Substrate Storage #255
Changes from 11 commits
368c7bc
ac6d78b
09fb5dd
7b8b959
97e85f5
402bba0
89c720c
d77a40b
999d39f
68a8592
5bbead7
ec323ac
5813b88
d6d7292
1c5993e
ab8caad
e3ceff2
f4570ee
9db4f54
23f725b
59f529a
07026a3
9240c48
2cc0216
9cb7da5
df67541
96fee72
330d8a1
72550f5
2018486
0e33445
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,8 +2,87 @@ | |
| title: Storage | ||
| --- | ||
|
|
||
| * Key Value Database, RocksDB | ||
| The Substrate uses a simple a key-value data store implemented as a | ||
| database-backed modified Merkle tree. | ||
|
|
||
| * Trie abstraction | ||
| ## Key-Value Database | ||
|
|
||
| * Runtime Storage abstraction | ||
| Substrate implements its storage database with [RocksDB](https://rocksdb.org/), | ||
| a persistent key-value store for fast storage environments. | ||
|
|
||
| This is used for all the components of Substrate which require persistent | ||
| storage such as: | ||
|
|
||
| - Substrate clients | ||
| - Substrate light-clients | ||
| - Off-chain workers | ||
|
|
||
| ## Trie Abstraction | ||
|
|
||
| One advantage of using a simple key-value store is that you are able to easily | ||
| abstract other storage structures on top. | ||
|
shawntabrizi marked this conversation as resolved.
Outdated
|
||
|
|
||
| Substrate uses a Base-16 Modified Merkle Patricia tree ("trie") from | ||
| [`paritytech/trie`](https://github.com/paritytech/trie) to provide a trie | ||
| structure whose contents can be modified and whose root hash is recalculated | ||
| efficiently. | ||
|
|
||
| Tries are important tool for blockchains because they allow for efficient | ||
|
shawntabrizi marked this conversation as resolved.
Outdated
shawntabrizi marked this conversation as resolved.
Outdated
|
||
| storing and sharing of the historical block state. The root hash of the trie | ||
| provides a cryptographic fingerprint of the final resulting storage after | ||
| executing all state transitions by a blockchain. Thus, two blockchain nodes can | ||
| easily verify they have the same final state by simply comparing their trie | ||
| root. | ||
|
shawntabrizi marked this conversation as resolved.
Outdated
shawntabrizi marked this conversation as resolved.
Outdated
|
||
|
|
||
| Accessing trie data is costly. Each read operation takes O(log N) time, where N | ||
| is the number of elements stored in the trie. To mitigate this, we use a key | ||
| value cache. | ||
|
shawntabrizi marked this conversation as resolved.
Outdated
|
||
|
|
||
| All trie nodes are stored in RocksDB and part of the trie state can get pruned, | ||
| i.e. a key-value pair can be deleted from the storage when it is out of pruning | ||
|
joepetrowski marked this conversation as resolved.
|
||
| range for non archive nodes. The trie nodes partial path prefixes the hash of | ||
|
shawntabrizi marked this conversation as resolved.
Outdated
|
||
| the encoded node for the RocksDB key to avoid key collision. We do not use | ||
| [reference counting](http://en.wikipedia.org/wiki/Reference_counting) for | ||
| performance reasons. | ||
|
|
||
| ### State Trie | ||
|
|
||
| Substrate has a single main trie, called the state trie, whose changing root | ||
|
shawntabrizi marked this conversation as resolved.
Outdated
|
||
| hash is placed in each block header. This is used to easily verify the state of | ||
| the blockchain and provide a basis for light clients to verify proofs. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would be interested to know more about the light clients and what exactly is proved to them. I know they only read block headers (not full blocks) so they get to see the state root before and after each block, but how do they actually know that the transactions in the block were executed correctly and lead to the resulting state root? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. what is proved to the light client is validity of an operation over a state of the chain. Say the light client does not have the full state, he get the info he need with a proof that it comes from a valid full state. simplier example should be: Querying keys from state trie, the request will be (from cone/network/src/protocol/message.rs): then the reply will be To make sense out of this reply we fetch request by id and the state trie root of the requested block hash, then we query keys over the proof. but this way of recording some operation on a full client and re-executing over this record on a light client can apply to many thing (if you look in message.rs there is a few query, call for instance uses executor on light blockchain). Not 100% sure on the following point, but proving the transition between two consecutive blocks is not something we do (I think), we just rely on the fact that the blocks got validated and we know they are chained. Surely a full client can execute the transition by running a full block on its state, so it should be possible for him to send to a light client all accessed db keys during the full block execution and next root calculation, and the light client will be able to execute the block on those keys and produce next root. But I guess the proof will be really massive, so relying on network having validating those state may be enough in the light client usecase. But you can run any call already (see
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think a slightly simpler, but hopefully still fully accurate explanation of light client proofs come from merkle proof in general. from here: https://www.quora.com/Cryptography-How-does-a-Merkle-proof-actually-work The image hopefully shows that if you want to prove TX3 is in the trie, you dont need ALL the nodes, just the full branch will leads to TX3 and all the nodes next to the nodes on that full branch. Obviously WAYWAY less data on big tries like the ones on blockchains. Full nodes act as the provider to light clients of the proof it needs. They do so pretty selflessly, but there are thoughts to have light clients start micro-payments to full nodes for their services. |
||
|
|
||
| This trie only stores content for the canonical chain, not forks. There is a | ||
| separate `state_db` layer that maintain the trie state with references counted | ||
|
shawntabrizi marked this conversation as resolved.
Outdated
|
||
| in memory for all that is non canonical. See JournalDB. | ||
|
shawntabrizi marked this conversation as resolved.
Outdated
|
||
|
|
||
| ### Child Trie | ||
|
|
||
| Substrate also provides an API to generate new child tries with their own root | ||
| hash that can be used in the runtime. | ||
|
shawntabrizi marked this conversation as resolved.
Outdated
|
||
|
|
||
| Child tries are identical to main state trie, except their root is stored and | ||
|
shawntabrizi marked this conversation as resolved.
Outdated
|
||
| updated in the main trie instead of the block header. Since their headers are a | ||
|
shawntabrizi marked this conversation as resolved.
Outdated
|
||
| part of the main state trie, it is still easy to verify of the complete node | ||
| state when it includes child tries. | ||
|
|
||
| ## Runtime Storage API | ||
|
|
||
| The Substrate runtime support library provides utilities which generates unique, | ||
|
shawntabrizi marked this conversation as resolved.
Outdated
|
||
| deterministic keys for your runtime module storage items. These storage items | ||
| are placed in the state trie and are accessible by querying the trie by key. | ||
|
|
||
| ## Next Steps | ||
|
|
||
| ### Learn More | ||
|
|
||
| - Learn how to add [storage items](development/module/storage.md) into your | ||
| Substrate runtime modules. | ||
|
|
||
| ### Examples | ||
|
|
||
| - View an example of creating [child tries]() in your Substrate runtime module. | ||
|
joepetrowski marked this conversation as resolved.
Outdated
|
||
|
|
||
| ### References | ||
|
|
||
| - Visit the reference docs for | ||
| [`paritytech/trie`](https://substrate.dev/rustdocs/master/trie_db/trait.Trie.html). | ||

Uh oh!
There was an error while loading. Please reload this page.