Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 40 additions & 13 deletions docs/advanced/datastructures/custom.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,17 +13,17 @@ manipulate the contract's storage. However, contract authors should know that th
also create their own custom data structures.

## Using custom types on storage
Any custom type wanting to be compatible with ink! storage must implement the
[`Storable`](https://use-ink.github.io/ink/ink_storage_traits/trait.Storable.html)
trait, so it can be SCALE
[`encoded`](https://docs.rs/parity-scale-codec/latest/parity_scale_codec/trait.Encode.html)
and
[`decoded`](https://docs.rs/parity-scale-codec/latest/parity_scale_codec/trait.Decode.html).
Additionally, the traits
[`StorageLayout`](https://use-ink.github.io/ink/ink_storage_traits//trait.StorageLayout.html)
and [`TypeInfo`](https://docs.rs/scale-info/latest/scale_info/trait.TypeInfo.html)
Any custom type wanting to be compatible with ink! storage must implement the [`Storable`][Storable] trait,
so it can be SCALE [`encoded`][scale-encode] and [`decoded`][scale-decode].
Additionally, the traits [`StorageLayout`][StorageLayout] and [`TypeInfo`][TypeInfo]
are required as well. But don't worry, usually these traits can just be derived:

[Storable]: https://use-ink.github.io/ink/ink_storage_traits/trait.Storable.html
[scale-encode]: https://docs.rs/parity-scale-codec/latest/parity_scale_codec/trait.Encode.html
[scale-decode]: https://docs.rs/parity-scale-codec/latest/parity_scale_codec/trait.Decode.html
[StorageLayout]: https://use-ink.github.io/ink/ink_storage_traits/trait.StorageLayout.html
[TypeInfo]: https://docs.rs/scale-info/latest/scale_info/trait.TypeInfo.html

```rust
/// A custom type that we can use in our contract storage
#[ink::scale_derive(Encode, Decode, TypeInfo)]
Expand All @@ -41,11 +41,12 @@ pub struct ContractStorage {
}
```

Even better: there is a macro
[`#[ink::storage_item]`](https://use-ink.github.io/ink/ink_macro/attr.storage_item.html),
Even better: there is a macro [`#[ink::storage_item]`][storage_item],
which derives all necessary traits for you. If there is no need to implement any special
behavior, the above code example can be simplified further as follows:

[storage_item]: https://use-ink.github.io/ink/ink_macro/attr.storage_item.html

```rust
/// A custom type that we can use in our contract storage
#[ink::storage_item]
Expand All @@ -65,17 +66,43 @@ the relevant trait documentations for more information.
:::note

The `#[ink::storage_item]` macro is responsible for storage key calculation of
non-[`Packed`](https://use-ink.github.io/ink/ink_storage_traits//trait.Packed.html)
non-[`Packed`](https://use-ink.github.io/ink/ink_storage_traits/trait.Packed.html)
types. Without it, the key for non-`Packed` fields will be zero. Using this macro is
necessary if you don't plan to use a
[`ManualKey`](https://use-ink.github.io/ink/ink_storage_traits//struct.ManualKey.html)
[`ManualKey`](https://use-ink.github.io/ink/ink_storage_traits/struct.ManualKey.html)
on a non-`Packed` type.

Types with custom implementations of the ink! storage traits can still use this macro only
for key calculation by disabling the derives: `#[ink::storage_item(derive = false)]`.

:::

:::note
The `#[ink::storage_item]` macro defaults to deriving a "non-Packed" layout
(i.e. the derived storage type doesn't implement the [`Packed`][Packed] trait by default).
So if the intention is to use the custom storage type in a position that expects
a packed storage type (e.g. as `V` in `Mapping<K, V: Packed>` or `T` in `Vec<T: Packed>`),
you can use the `packed` argument/flag to derive an implementation of the `Packed` trait
e.g.

```rust
/// A `Packed` custom storage type that can be used as a `T: Packed` type
/// (e.g. as `V` in `Mapping<K, V: Packed>` or `T` in `Vec<T: Packed>`).
#[ink::storage_item(packed)]
pub struct Inner {
value: bool,
}

#[ink(storage)]
pub struct ContractStorage {
inner: Mapping<AccountId, Inner>,
}

```
:::

[Packed]: https://use-ink.github.io/ink/ink_storage_traits/trait.Packed.html

## Generic storage fields

It is possible to use generic data types in your storage, as long as any generic type
Expand Down
27 changes: 19 additions & 8 deletions docs/advanced/datastructures/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,24 +9,35 @@ hide_title: true
# Overview

The `ink_storage` crate acts as the standard storage library for ink! smart contracts.
At the moment it provides two primitives for interacting with storage,
[`Mapping`](https://use-ink.github.io/ink/ink_storage/struct.Mapping.html)
and [`Lazy`](https://use-ink.github.io/ink/ink_storage/struct.Lazy.html).
At the moment it provides three primitives for interacting with storage,
[`Mapping`][mapping], [`Lazy`][lazy] and [`StorageVec`][storage-vec].

`Mapping` is a mapping of key-value pairs directly to the contract storage. It is very
similar to traditional hash tables and comparable to the `mapping` type Solidity offers.
[mapping]: https://use-ink.github.io/ink/ink_storage/struct.Mapping.html
[lazy]: https://use-ink.github.io/ink/ink_storage/struct.Lazy.html
[storage-vec]: https://use-ink.github.io/ink/ink_storage/struct.StorageVec.html

[`Mapping`][mapping] is a mapping of key-value pairs directly to the contract storage.
It is very similar to traditional hash tables and comparable to the `mapping` type Solidity offers.
As a core ingredient to the ink! language, its main advantage is being simple and
lightweight: It favors being efficient in terms of gas costs and code size
over providing a lot of high-level functionality found in other implementations
like the `ink::prelude::collections::HashMap` type.
Overall, the ink! `Mapping` will be a solid choice for most contracts. Moreover, smart
contracts developers can implement advanced features themselves.
Overall, the ink! `Mapping` will be a solid choice for most contracts.
Moreover, smart contracts developers can implement advanced features themselves.
You can learn more about this in the [dedicated `Mapping` section](./mapping.md).

`Lazy` is a wrapper type that can be used over any other storage compatible type.
[`Lazy`][lazy] is a wrapper type that can be used over any other storage compatible type.
This allows smart contract developers fine-grained manual control over the layout of
the contract storage by assigning a separate storage cell for the field. For example,
it can be used to prevent the contract from eagerly loading large storage fields
during each contract call.
Conceivably, it may be desirable to change certain aspects on how your contract deals with
its storage variables. You can find out more about this in the section about the ink!
[Storage Layout](./storage-layout.md).

[`StorageVec`][storage-vec] is a vector of values/elements directly on contract storage.
However, unlike `Vec<T>`, which stores all of its elements in a single storage cell,
`StorageVec` stores each of its elements in its own storage cell.
This layout is more gas efficient for reads and writes of single elements,
and also allows higher maximum capacity for `StorageVec` compared to `Vec<T>`.
You can learn more about this in the [dedicated `StorageVec` section](./storagevec.md).
2 changes: 1 addition & 1 deletion docs/advanced/datastructures/storage-in-metadata.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ The storage will be reflected inside the metadata as like follows:

We observe that the storage layout is represented as a tree, where tangible storage values
end up inside a `leaf`. Because of
[`Packed`](https://use-ink.github.io/ink/ink_storage_traits//trait.Packed.html)
[`Packed`](https://use-ink.github.io/ink/ink_storage_traits/trait.Packed.html)
encoding, leafs can share the same storage key, and
in order to reach them you'd need to fetch and decode the whole storage cell under this key.

Expand Down
81 changes: 44 additions & 37 deletions docs/advanced/datastructures/storage-layout.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,28 +24,30 @@ to ink! by the [`pallet-revive`](https://github.com/paritytech/polkadot-sdk/tree
<img src={useBaseUrl('/img/kv-revive.svg')} alt="Storage Organization: Layout" />
</div>

Storage data is always encoded with the
[`SCALE`](https://docs.polkadot.com/polkadot-protocol/basics/data-encoding/#scale-codec-libraries) codec.
The storage API operates by storing and loading entries into and from a single storage
cells, where each storage cell is accessed under its own dedicated storage key. To some
extent, the storage API works similar to a traditional key-value database.
Storage data is always encoded with the [`SCALE`][SCALE] codec.
The storage API operates by storing and loading entries into and from single storage cells,
where each storage cell is accessed under its own dedicated storage key.
To some extent, the storage API works similar to a traditional key-value database.

[SCALE]: https://docs.polkadot.com/polkadot-protocol/basics/data-encoding/#scale-codec-libraries

## Packed vs Non-Packed layout

Types that can be stored entirely under a single storage cell are considered
[`Packed`](https://use-ink.github.io/ink/ink_storage_traits//trait.Packed.html).
Types that can be stored entirely under a single storage cell are considered [`Packed`][Packed].
By default, ink! tries to store all storage struct fields under a single storage cell.
Consequentially, with a `Packed` storage layout, any message interacting with the contract
storage will always need to operate on the entire contract storage struct.

[Packed]: https://use-ink.github.io/ink/ink_storage_traits/trait.Packed.html

For example, if we have a somewhat small contract storage struct consisting of only a few
tiny fields, pulling everything from the storage inside every message is not
problematic. It may even be advantageous - especially if we expect most messages to
interact with most of the storage fields.

On the other hand, this can get problematic if we're storing a large `ink::prelude::vec::Vec`
in the contract storage but provide messages that do not need to read and write from this
`Vec`. In that scenario, each and every contract message bears runtime overhead by dealing
in the contract storage but provide messages that do not need to read and write from this `Vec`.
In that scenario, each and every contract message bears runtime overhead by dealing
with that `Vec`, regardless whether they access it or not. This results in extra gas costs.
To solve this problem we need to turn our storage into a non-packed layout somehow.

Expand All @@ -55,23 +57,25 @@ To solve this problem we need to turn our storage into a non-packed layout someh

:::caution

If any type exhibiting `Packed` layout gets large enough (an ever-growing `Vec` might be
a prime candidate for this), it will break your contract.
This is because for encoding and decoding storage items, there is a buffer with only limited
capacity (around 16KB in the default configuration) available. This means any contract
trying to decode more than that will trap! If you are unsure about the potential size a
data structure might get, consider using an ink! `Mapping`, which can store an arbitrary
number of elements, instead.
If any type exhibiting `Packed` layout gets large enough (an ever-growing `Vec`
might be a prime candidate for this), it will break your contract.
This is because for encoding and decoding storage items, there is a buffer
with only limited capacity (around 16KB in the default configuration) available.
This means any contract trying to decode more than that will trap!
If you are unsure about the potential size a data structure might get, consider using
an ink! `Mapping` or `StorageVec`, which can store an arbitrary number of elements, instead.

:::

## Eager Loading vs. Lazy Loading

ink! provides means of breaking the storage up into smaller pieces, which can be loaded
on demand, with the
[`Lazy`](https://use-ink.github.io/ink/ink/storage/struct.Lazy.html) primitive.
Wrapping any storage field inside a `Lazy` struct makes the storage
struct in which that field appears also
non-`Packed`, preventing it from being eagerly loaded during arbitrary storage operations:
on demand, with the [`Lazy`][Lazy] primitive.
Wrapping any storage field inside a `Lazy` struct makes the storage struct
in which that field appears also non-`Packed`, preventing it from being eagerly loaded
during arbitrary storage operations:

[Lazy]: https://use-ink.github.io/ink/ink/storage/struct.Lazy.html

<div class="schema">
<img src={useBaseUrl('/img/storage-layout-revive.svg')} alt="Storage Organization: Layout with a Lazy field" />
Expand Down Expand Up @@ -127,22 +131,20 @@ mod mycontract {

:::caution

`ink::prelude::vec::Vec`'s are always loaded in their entirety. This is because all elements
of the `ink::prelude::vec::Vec` live under a single storage key. Wrapping the
`ink::prelude::vec::Vec` inside `Lazy`, like the
provided example above does, has no influence on its inner layout. If you are dealing with
large or sparse arrays on contract storage, consider using a `Mapping` instead.
`ink::prelude::vec::Vec`'s are always loaded in their entirety.
This is because all elements of the `ink::prelude::vec::Vec` live under a single storage key.
Wrapping the `ink::prelude::vec::Vec` inside `Lazy`, like the provided example above does,
has no influence on its inner layout. If you are dealing with large or sparse arrays
on contract storage, consider using a `Mapping` instead.

:::

## Manual vs. Automatic Key Generation

By default, keys are calculated automatically for you, thanks to the
[`AutoKey`](https://use-ink.github.io/ink/ink_storage_traits//struct.AutoKey.html)
primitive. They'll be generated at compile time and ruled out for conflicts.
However, for non-`Packed` types like `Lazy` or the `Mapping`, the
[`ManualKey`](https://use-ink.github.io/ink/ink_storage_traits//struct.ManualKey.html)
primitive allows manual control over the storage key of a field like so:
By default, keys are calculated automatically for you, thanks to the [`AutoKey`][AutoKey] primitive.
They'll be generated at compile time and ruled out for conflicts.
However, for non-`Packed` types like `Lazy` or the `Mapping`, the [`ManualKey`][ManualKey] primitive
allows manual control over the storage key of a field like so:

```rust
#[ink(storage)]
Expand All @@ -156,6 +158,9 @@ This may be advantageous: Your storage key will always stay the same, regardless
the version of your contract or ink! itself (note that the key calculation algorithm may
change with future ink! versions).

[AutoKey]: https://use-ink.github.io/ink/ink_storage_traits/struct.AutoKey.html
[ManualKey]: https://use-ink.github.io/ink/ink_storage_traits/struct.ManualKey.html

:::tip

Using `ManualKey` instead of `AutoKey` might be especially desirable for upgradable
Expand All @@ -178,13 +183,15 @@ pub struct MyContract<KEY: StorageKey = ManualKey<0xcafebabe>> {

## Considerations

It might be worthwhile to think about the desired storage layout of your contract. While
using a `Packed` layout will keep your contracts overall code size smaller, it can cause
unnecessarily high gas costs. Thus, we consider it a good practice to break up large
or complex storage layouts into reasonably sized distinct storage cells.
It might be worthwhile to think about the desired storage layout of your contract.
While using a `Packed` layout will keep your contracts overall code size smaller,
it can cause unnecessarily high gas costs.
Thus, we consider it a good practice to break up large or complex storage layouts
into reasonably sized distinct storage cells.

:::note

ink! `Mapping`s are always non-`Packed` and loaded lazily, one key-value pair at the time.
ink! `Mapping`s and `StorageVec`s are always non-`Packed` and loaded lazily,
one key-value pair at the time.

:::
27 changes: 17 additions & 10 deletions docs/advanced/datastructures/storagevec.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,29 +74,36 @@ mod mycontract {

## Difference between `StorageVec` and Rusts `Vec` type

Any Rust `Vec<T>` will exhibit `Packed` storage layout; where
`StorageVec` stores each value under it's own storage key.
Any Rust `Vec<T>` will exhibit `Packed` storage layout
(i.e. all values are stored in a single storage cell);
whereas `StorageVec` stores each value under its own storage key.

Hence, any read or write from or to a `Vec` on storage will load
or store _all_ of its elements.

This can be undesirable:
The cost of reading or writing a _single_ element grows linearly
corresponding to the number of elements in the vector (its length).
Additionally, the maximum capacity of the _whole_ vector is limited by
the size of [ink!'s static buffer](https://github.com/use-ink/ink/blob/master/ARCHITECTURE.md#communication-with-the-pallet)
used during ABI encoding and decoding (default 16 KiB).
This can be undesirable because:
- The cost of reading or writing a _single_ element grows linearly
corresponding to the number of elements in the vector (i.e. its length).
- Additionally, the maximum capacity of the _whole_ vector is limited by
[`pallet-revive's` storage limit for a single cell][pallet-revive-limits]
(currently `416 bytes`).
- Lastly, the maximum capacity of the _whole_ vector is also limited by
the size of [ink!'s static buffer][static-buffer] used during ABI encoding
and decoding (current default is `16 KiB`).

`StorageVec` on the other hand allows to access each element individually.
Thus, it can theoretically grow to infinite size.
However, we currently limit the length at 2 ^ 32 elements. In practice,
even if the vector elements are single bytes, it'll allow to store
more than 4 GB data in blockchain storage.

[pallet-revive-limits]: https://docs.polkadot.com/polkadot-protocol/smart-contract-basics/evm-vs-polkavm/#current-memory-limits
[static-buffer]: https://github.com/use-ink/ink/blob/master/ARCHITECTURE.md#communication-with-the-pallet

### Caveats

Iterators are not provided. `StorageVec` is expected to be used to
store a lot elements, where iterating through the elements would be
store a lot of elements, where iterating through the elements would be
rather inefficient. Manually iterating over the elements using a loop
is possible but considered an anti-pattern for most cases.

Expand All @@ -118,7 +125,7 @@ storage read.

### Storage Layout

At given `StorageKey` `K`, the length of the `StorageVec` is hold.
At given `StorageKey` `K`, the length of the `StorageVec` is held.
Each element `E` is then stored under a combination of the `StorageVec`
key `K` and the elements index.

Expand Down
2 changes: 1 addition & 1 deletion sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,8 @@ module.exports = {
label: "Data Structures",
items: [
"advanced/datastructures/overview",
"advanced/datastructures/storagevec",
"advanced/datastructures/mapping",
"advanced/datastructures/storagevec",
"advanced/datastructures/storage-layout",
"advanced/datastructures/custom",
"advanced/datastructures/storage-in-metadata"
Expand Down
2 changes: 1 addition & 1 deletion src/data/projects.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ export const projects: Project[] = [
{
link: 'https://analyze.ink/',
logo: 'img/projects/ink-analyzer-logo.svg',
title: 'A collection of modular and reusable libraries and tools for improving of ink! language support in IDEs/editors.',
title: 'A collection of modular and reusable libraries and tools for improving ink! language support in IDEs/editors.',
about: (
<>
<p>
Expand Down