Implementer's guide: downward messages and HRMP

roadmap/implementers-guide/src/glossary.md

@@ -7,8 +7,10 @@ Here you can find definitions of a bunch of jargon, usually specific to the Polk
 - Backed Candidate: A Backable Candidate noted in a relay-chain block
 - Backing: A set of statements proving that a Parachain Candidate is backable.
 - Collator: A node who generates Proofs-of-Validity (PoV) for blocks of a specific parachain.
+- DMP: Downward Message Passing. Message passing from the relay-chain to a parachain.
 - Extrinsic: An element of a relay-chain block which triggers a specific entry-point of a runtime module with given arguments.
 - GRANDPA: (Ghost-based Recursive ANcestor Deriving Prefix Agreement). The algorithm validators use to guarantee finality of the Relay Chain.
+- HRMP: (Horizontally Relay-routed Message Passing). A mechanism for message passing between parachains (hence horizontal) that leverages the relay-chain storage. Predates XCMP.
 - Inclusion Pipeline: The set of steps taken to carry a Parachain Candidate from authoring, to backing, to availability and full inclusion in an active fork of its parachain.
 - Module: A component of the Runtime logic, encapsulating storage, routines, and entry-points.
 - Module Entry Point: A recipient of new information presented to the Runtime. This may trigger routines.
@@ -26,8 +28,12 @@ Here you can find definitions of a bunch of jargon, usually specific to the Polk
 - Runtime API: A means for the node-side behavior to access structured information based on the state of a fork of the blockchain.
 - Secondary Checker: A validator who has been randomly selected to perform secondary approval checks on a parablock which is pending approval.
 - Subsystem: A long-running task which is responsible for carrying out a particular category of work.
+- UMP: (Upward Message Passing) A vertical message passing mechanism from a parachain to the relay chain.
+- DMP: (Downward Message Passing) A vertical message passing mechanism from the relay chain to a parachain.
 - Validator: Specially-selected node in the network who is responsible for validating parachain blocks and issuing attestations about their validity.
 - Validation Function: A piece of Wasm code that describes the state-transition function of a parachain.
+- VMP: (Vertical Message Passing) A family of mechanisms that are responsible for message exchange between the relay chain and parachains.
+- XCMP (Cross-Chain Message Passing) A type of horizontal message passing (i.e. between parachains) that allows secure message passing directly between parachains and has minimal resource requirements from the relay chain, thus highly scalable.
 
 Also of use is the [Substrate Glossary](https://substrate.dev/docs/en/knowledgebase/getting-started/glossary).
 

roadmap/implementers-guide/src/runtime/inclusion.md

@@ -68,13 +68,18 @@ All failed checks should lead to an unrecoverable error making the block invalid
   1. Transform each [`CommittedCandidateReceipt`](../types/candidate.md#committed-candidate-receipt) into the corresponding [`CandidateReceipt`](../types/candidate.md#candidate-receipt), setting the commitments aside.
   1. check the backing of the candidate using the signatures and the bitfields, comparing against the validators assigned to the groups, fetched with the `group_validators` lookup.
   1. check that the upward messages, when combined with the existing queue size, are not exceeding `config.max_upward_queue_count` and `config.watermark_upward_queue_size` parameters.
+  1. call `Router::ensure_processed_downward_messages(para, commitments.processed_downward_messages)` for each candidate to check rules of processing the downward message queue.
+  1. check that in the commitments of each candidate the horizontal messages are sorted by ascending recipient ParaId and there is no two horizontal messages have the same recipient.
+  1. using `Router::ensure_horizontal_messages_fit(sender, commitments.horizontal_messages)` ensure that the each candidate's para doesn't overfill any downward queue.
   1. create an entry in the `PendingAvailability` map for each backed candidate with a blank `availability_votes` bitfield.
   1. create a corresponding entry in the `PendingAvailabilityCommitments` with the commitments.
   1. Return a `Vec<CoreIndex>` of all scheduled cores of the list of passed assignments that a candidate was successfully backed for, sorted ascending by CoreIndex.
 * `enact_candidate(relay_parent_number: BlockNumber, CommittedCandidateReceipt)`:
   1. If the receipt contains a code upgrade, Call `Paras::schedule_code_upgrade(para_id, code, relay_parent_number + config.validationl_upgrade_delay)`.
     > TODO: Note that this is safe as long as we never enact candidates where the relay parent is across a session boundary. In that case, which we should be careful to avoid with contextual execution, the configuration might have changed and the para may de-sync from the host's understanding of it.
   1. call `Router::queue_upward_messages` for each backed candidate, using the [`UpwardMessage`s](../types/messages.md#upward-message) from the [`CandidateCommitments`](../types/candidate.md#candidate-commitments).
+  1. call `Router::drain_downward_messages` with the para id of the candidate and `processed_downward_messages` taken from the commitment,
+  1. call `Router::queue_horizontal_messages` with the para id of the candidate and the list of horizontal messages taken from the commitment,
   1. Call `Paras::note_new_head` using the `HeadData` from the receipt and `relay_parent_number`.
 * `collect_pending`:
 

roadmap/implementers-guide/src/runtime/router.md

@@ -1,8 +1,6 @@
 # Router Module
 
-The Router module is responsible for storing and dispatching Upward and Downward messages from and to parachains respectively. It is intended to later handle the XCMP logic as well.
-
-For each enacted block the `queue_upward_messages` entry-point is called.
+The Router module is responsible for all messaging mechanisms supported between paras and the relay chain, specifically: UMP, DMP, HRMP and later XCMP.
 
 ## Storage
 
@@ -19,6 +17,17 @@ RelayDispatchQueues: map ParaId => Vec<UpwardMessage>;
 RelayDispatchQueueSize: map ParaId => (u32, u32);
 /// The ordered list of `ParaId`s that have a `RelayDispatchQueue` entry.
 NeedsDispatch: Vec<ParaId>;
+/// The mapping that tracks how many bytes / messages are sent by a certain sender - recipient pair.
+///
+/// First item in the tuple is the count of messages for the (sender, recipient) pair and the second
+/// item is the total length (in bytes) of the message payloads.
+HorizontalMessagesResourceUsage: map (ParaId, ParaId) => (u32, u32);
+/// The downward messages addressed for a certain para. These vectors are not bounded directly, but
+/// rather each possible sender can put only a limited amount of messages in the downward queue.
+DownwardMessageQueues: map ParaId => Vec<DownwardMessage>;
+/// The number of downward messages originated from the relay chain to a certain para. This is subject
+/// to the `max_relay_chain_downward_messages` limit found in `HostConfiguration`.
+RelayChainDownwardMessages: map ParaId => u32;
 ```
 
 ## Initialization
@@ -27,6 +36,61 @@ No initialization routine runs for this module.
 
 ## Routines
 
+There are situations when actions that took place within the relay chain could lead to a downward message
+sent to a para. For example, if an entry-point to transfer some funds to a para was called.
+
+For these cases, there are two routines, `has_dmq_capacity_for_relay_chain` and `send_downward_messages`,
+intended for use by the relay chain.
+
+`send_downward_messages` is used for enqueuing one or more downward messages for a certain recipient. Since downward
+message queues can hold only so many messages per one sender (and the relay chain is not an exception),
+`send_downward_messages` can fail refusing enqueuing a message that would have exceeded the limit. In those cases
+`has_dmq_capacity_for_relay_chain` can be used for checking in advance if there is enough space for a given
+number of messages.
+
+Note that an HRMP message can only be sent by para candidates.
+
+* `has_dmq_capacity_for_relay_chain(recipient: ParaId, n: u32)`.
+  1. Checks that the sum of the number `RelayChainDownwardMessages` for `recipient` and `n` is less
+  than or equal to `config.max_relay_chain_downward_messages`.
+* `send_downward_messages(recipient: ParaId, Vec<DownwardMessage>)`.
+  1. Checks that there is enough capacity in the receipient's downward queue using `has_dmq_capacity_for_relay_chain`.
+  1. For each downward message `DM`:
+    1. Checks that `DM` is not of type `HorizontalMessage`.
+    1. Appends `DM` into the `DownwardMessageQueues` corresponding to `recipient`.
+  1. Increments `RelayChainDownwardMessages` for the `recipient` according to the number of messages sent.
+
+The following routines are intended for use during the course of inclusion or enactment of para candidates.
+For checking the validity of message passing within a candidate the `ensure_processed_downward_messages`
+and `ensure_horizontal_messages_fit` routines are called. When a candidate is enacted the
+`drain_downward_messages`, `queue_horizontal_messages` and `queue_upward_messages` are called.
+
+* `ensure_processed_downward_messages(recipient: ParaId, processed_downward_messages: u32)`:
+  1. Checks that `DownwardMessageQueues` for `recipient` is at least `processed_downward_messages` long.
+  1. Checks that `processed_downward_messages` is at least 1 if `DownwardMessageQueues` for `recipient` is not empty.
+* `ensure_horizontal_messages_fit(sender, Vec<HorizontalMessage>)`:
+  1. For each horizontal message `HM`, with recipient `R`:
+    1. Fetches the current usage level for the pair `(sender, R)`. The usage level is defined by
+    a tuple of `(msg_count, total_byte_size)`.
+    1. Checks that `msg_count + 1` is less or equal than `config.max_hrmp_queue_count_per_sender`.
+    1. Checks that the sum of the payload size occupied by `HM` and `total_byte_size` is less than or
+    equal to `config.max_hrmp_queue_size_per_sender`.
+* `drain_downward_messages(recipient: ParaId, processed_downward_messages)`:
+  1. Prunes `processed_downward_messages` from the beginning of the downward message queue. For each pruned message `DM`:
+    1. If `DM` is a horizontal message sent from a sender `S`,
+      1. With the mapping from `HorizontalMessagesResourceUsage` that corresponds to `(S, recipient)` represented by
+      `(msg_count, total_byte_size)`.
+          1. Decrements `msg_count` by 1.
+          1. Decrements `total_byte_size` according to the payload size of `DM`.
+    1. Otherwise, decrements `RelayChainDownwardMessages` for the `recipient`.
+* `queue_horizontal_messages(sender: ParaId, Vec<HorizontalMessage>)`:
+  1. For each horizontal message `HM`, with recipient `R`:
+    1. Using the payload from `HM` and the `sender` creates a downward message `DM`.
+    1. Appends `DM` into the `DownwardMessageQueues` corresponding to `R`.
+    1. With the mapping from `HorizontalMessagesResourceUsage` that corresponds to `(sender, R)` represented by
+      `(msg_count, total_byte_size)`.
+        1. Increment `msg_count` by 1.
+        1. Increment `total_byte_size` according to the payload size of `DM`.
 * `queue_upward_messages(ParaId, Vec<UpwardMessage>)`:
   1. Updates `NeedsDispatch`, and enqueues upward messages into `RelayDispatchQueue` and modifies the respective entry in `RelayDispatchQueueSize`.
 

roadmap/implementers-guide/src/types/candidate.md

@@ -157,6 +157,8 @@ The execution and validation of parachain or parathread candidates produces a nu
 struct CandidateCommitments {
 	/// Fees paid from the chain to the relay chain validators.
 	fees: Balance,
+	/// Messages directed to other paras routed via the relay chain.
+	horizontal_messages: Vec<HorizontalMessage>,
 	/// Messages destined to be interpreted by the Relay chain itself.
 	upward_messages: Vec<UpwardMessage>,
 	/// The root of a block's erasure encoding Merkle tree.
@@ -165,6 +167,8 @@ struct CandidateCommitments {
 	new_validation_code: Option<ValidationCode>,
 	/// The head-data produced as a result of execution.
 	head_data: HeadData,
+	/// The number of processed downward messages by the para.
+	processed_downward_messages: u32,
 }
 ```
 
@@ -193,11 +197,15 @@ struct ValidationOutputs {
 	global_validation_schedule: GlobalValidationSchedule,
 	/// The local validation data.
 	local_validation_data: LocalValidationData,
+	/// Messages directed to other paras routed via the relay chain.
+	horizontal_messages: Vec<HorizontalMessage>,
 	/// Upwards messages to the relay chain.
 	upwards_messages: Vec<UpwardsMessage>,
 	/// Fees paid to the validators of the relay-chain.
 	fees: Balance,
 	/// The new validation code submitted by the execution, if any.
 	new_validation_code: Option<ValidationCode>,
+	/// The number of processed downward messages by the para.
+	processed_downward_messages: u32,
 }
 ```

roadmap/implementers-guide/src/types/messages.md

@@ -2,6 +2,9 @@
 
 Types of messages that are passed between parachains and the relay chain: UMP, DMP, XCMP.
 
+There is also HRMP (Horizontally Relay-routed Message Passing) which provides the same functionality
+although with smaller scalability potential.
+
 ## Upward Message
 
 A type of messages dispatched from a parachain to the relay chain.
@@ -26,3 +29,38 @@ struct UpwardMessage {
 	pub data: Vec<u8>,
 }
 ```
+
+## Horizontal Message
+
+This is a message sent from a parachain to another parachain that travels through the relay chain.
+This message ends up in the recipient's mailbox. A size of a horizontal message is defined by its
+`data` payload.
+
+```rust,ignore
+struct HorizontalMessage {
+	/// The para that will get this message in its downward message queue.
+	pub recipient: ParaId,
+	/// The message payload.
+	pub data: Vec<u8>,
+}
+```
+
+## Downward Message
+
+A message that go down from the relay chain to a parachain. Such a message could be initiated either
+as a result of an operation took place on the relay chain or sent using a horizontal message.
+
+```rust,ignore
+enum DownwardMessage {
+	/// Some funds were transferred into the parachain's account. The hash is the identifier that
+	/// was given with the transfer.
+	TransferInto(AccountId, Balance, Remark),
+	/// This downward message is a result of a horizontal message represented as opaque bytes sent
+	/// by the specified sender.
+	HorizontalMessage(ParaId, Vec<u8>),
+	/// An opaque message which interpretation is up to the recipient para. This variant ought
+	/// to be used as a basis for special protocols between the relay chain and, typically system,
+	/// paras.
+	ParachainSpecific(Vec<u8>),
+}
+```

roadmap/implementers-guide/src/types/runtime.md

@@ -40,5 +40,14 @@ struct HostConfiguration {
 	/// no further messages may be added to it. If it exceeds this then the queue may contain only
 	/// a single message.
 	pub watermark_upward_queue_size: u32,
+	/// The maximum number of downward messages originated from the relay chain in a downward message
+	/// queue.
+	pub max_relay_chain_downward_messages: u32,
+	/// The maximum number of horizontal messages allowed in a downward message queue per one sender
+	/// para at the same time.
+	pub max_hrmp_queue_count_per_sender: u32,
+	/// The maximum total size of messages in bytes allowed in a downward message queue per one
+	/// sender para at the same time.
+	pub max_hrmp_queue_size_per_sender: u32,
 }
 ```