Implementer's guide: downward messages and HRMP

roadmap/implementers-guide/src/runtime/inclusion.md

@@ -68,13 +68,18 @@ All failed checks should lead to an unrecoverable error making the block invalid
   1. Transform each [`CommittedCandidateReceipt`](../types/candidate.md#committed-candidate-receipt) into the corresponding [`CandidateReceipt`](../types/candidate.md#candidate-receipt), setting the commitments aside.
   1. check the backing of the candidate using the signatures and the bitfields, comparing against the validators assigned to the groups, fetched with the `group_validators` lookup.
   1. check that the upward messages, when combined with the existing queue size, are not exceeding `config.max_upward_queue_count` and `config.watermark_upward_queue_size` parameters.
+  1. call `Router::ensure_processed_downward_messages(para, processed_downward_messages)` to check rules of processing the downward message queue.
+  1. check that the horizontal messages are sorted by ascending recipient ParaId and there is no two horizontal messages have the same recipient.
+  1. using `Router::ensure_horizontal_messages_fit(sender, horizontal_messages)` ensure that the sender para doesn't overfill any downward queue.
   1. create an entry in the `PendingAvailability` map for each backed candidate with a blank `availability_votes` bitfield.
   1. create a corresponding entry in the `PendingAvailabilityCommitments` with the commitments.
   1. Return a `Vec<CoreIndex>` of all scheduled cores of the list of passed assignments that a candidate was successfully backed for, sorted ascending by CoreIndex.
 * `enact_candidate(relay_parent_number: BlockNumber, CommittedCandidateReceipt)`:
   1. If the receipt contains a code upgrade, Call `Paras::schedule_code_upgrade(para_id, code, relay_parent_number + config.validationl_upgrade_delay)`.
     > TODO: Note that this is safe as long as we never enact candidates where the relay parent is across a session boundary. In that case, which we should be careful to avoid with contextual execution, the configuration might have changed and the para may de-sync from the host's understanding of it.
   1. call `Router::queue_upward_messages` for each backed candidate, using the [`UpwardMessage`s](../types/messages.md#upward-message) from the [`CandidateCommitments`](../types/candidate.md#candidate-commitments).
+  1. call `Router::drain_downward_messages` with the para id of the candidate and `processed_downward_messages` taken from the commitment,
+  1. call `Router::queue_horizontal_messages` with the para id of the candidate and the list of horizontal messages taken from the commitment,
   1. Call `Paras::note_new_head` using the `HeadData` from the receipt and `relay_parent_number`.
 * `collect_pending`:
 

roadmap/implementers-guide/src/runtime/router.md

@@ -19,6 +19,17 @@ RelayDispatchQueues: map ParaId => Vec<UpwardMessage>;
 RelayDispatchQueueSize: map ParaId => (u32, u32);
 /// The ordered list of `ParaId`s that have a `RelayDispatchQueue` entry.
 NeedsDispatch: Vec<ParaId>;
+/// The mapping that tracks how many bytes / messages are sent by a certain sender - recipient pair.
+///
+/// First item in the tuple is the count of messages for the (sender, recipient) pair and the second
+/// item is the total length (in bytes) of the message payloads.
+HorizontalMessagesResourceUsage: map (ParaId, ParaId) => (u32, u32);
+/// The downward messages addressed for a certain para. These vectors are not bounded directly, but
+/// rather each possible sender can put only a limited amount of messages in the downward queue.
+DownwardMessageQueues: map ParaId => Vec<DownwardMessage>;
+/// The number of downward messages originated from the relay chain to a certain para. This is subject
+/// to the `max_relay_chain_downward_messages` limit found in `HostConfiguration`.
+RelayChainDownwardMessages: map ParaId => u32;
 ```
 
 ## Initialization
@@ -27,6 +38,52 @@ No initialization routine runs for this module.
 
 ## Routines
 
+There are two routines intended for use by the relay chain extrinsics: `ensure_downward_messages_fits`
+and `queue_downward_messages`. The former function is used before performing relay chain operations
+that results in downward messages sent to a given `recipient` to check if sending those messages will
+exceed the limits on the number of messages that the relay chain can send to a single recipient para.
+The latter routine is intended to perform the send of the downward messages.
+
+Note that the HRMP message can only be sent by para candidates.
+
+* `ensure_downward_messages_fits(recipient: ParaId, n: u32)`.
+  1. Checks that the sum of the number `RelayChainDownwardMessages` for `recipient` and `n` is less
+  than or equal to `config.max_relay_chain_downward_messages`.
+* `queue_downward_messages(recipient: ParaId, Vec<DownwardMessage>)`.
+  1. Checks that there is enough capacity in the receipient's downward queue using `ensure_downward_messages_fits`.
+  1. For each downward message `DM`:
+    1. Checks that `DM` is not of type `HorizontalMessage`.
+    1. Appends `DM` into the `DownwardMessageQueues` corresponding to `recipient`.
+  1. Increments `RelayChainDownwardMessages` for the `recipient` according to the number of messages sent.
+
+The following routines are intended for use during the course of inclusion or enactment of para candidates.
+
+* `ensure_processed_downward_messages(recipient: ParaId, processed_downward_messages: u32)`:
+  1. Checks that `processed_downward_messages` is at least 1,
+  1. Checks that `DownwardMessageQueues` for `recipient` is at least `processed_downward_messages` long.
+* `ensure_horizontal_messages_fit(sender, Vec<HorizontalMessage>)`:
+  1. For each horizontal message `HM`, with recipient `R`:
+    1. Fetches the current usage level for the pair `(sender, R)`. The usage level is defined by
+    a tuple of `(msg_count, total_byte_size)`.
+    1. Checks that `msg_count + 1` is less or equal than `config.max_hrmp_queue_count_per_sender`.
+    1. Checks that the sum of the payload size occupied by `HM` and `total_byte_size` is less than or
+    equal to `config.max_hrmp_queue_size_per_sender`.
+* `drain_downward_messages(recipient: ParaId, processed_downward_messages)`:
+  1. Prunes `processed_downward_messages` from the beginning of the downward message queue. For each pruned message `DM`:
+    1. If `DM` is a horizontal message sent from a sender `S`,
+      1. With the mapping from `HorizontalMessagesResourceUsage` that corresponds to `(S, recipient)` represented by
+      `(msg_count, total_byte_size)`.
+          1. Decrements `msg_count` by 1.
+          1. Decrements `total_byte_size` according to the payload size of `DM`.
+    1. Otherwise, decrements `RelayChainDownwardMessages` for the `recipient`.
+* `queue_horizontal_messages(sender: ParaId, Vec<HorizontalMessage>)`:
+  1. For each horizontal message `HM`, with recipient `R`:
+    1. Using the payload from `HM` and the `sender` creates a downward message `DM`.
+    1. Appends `DM` into the `DownwardMessageQueues` corresponding to `R`.
+    1. With the mapping from `HorizontalMessagesResourceUsage` that corresponds to `(sender, R)` represented by
+      `(msg_count, total_byte_size)`.
+        1. Increment `msg_count` by 1.
+        1. Increment `total_byte_size` according to the payload size of `DM`.
 * `queue_upward_messages(ParaId, Vec<UpwardMessage>)`:
   1. Updates `NeedsDispatch`, and enqueues upward messages into `RelayDispatchQueue` and modifies the respective entry in `RelayDispatchQueueSize`.
 

roadmap/implementers-guide/src/types/candidate.md

@@ -157,6 +157,8 @@ The execution and validation of parachain or parathread candidates produces a nu
 struct CandidateCommitments {
 	/// Fees paid from the chain to the relay chain validators.
 	fees: Balance,
+	/// Messages directed to other paras routed via the relay chain.
+	horizontal_messages: Vec<HorizontalMessage>,
 	/// Messages destined to be interpreted by the Relay chain itself.
 	upward_messages: Vec<UpwardMessage>,
 	/// The root of a block's erasure encoding Merkle tree.
@@ -165,6 +167,8 @@ struct CandidateCommitments {
 	new_validation_code: Option<ValidationCode>,
 	/// The head-data produced as a result of execution.
 	head_data: HeadData,
+	/// The number of processed downward messages by the para.
+	processed_downward_messages: u32,
 }
 ```
 
@@ -193,6 +197,8 @@ struct ValidationOutputs {
 	global_validation_schedule: GlobalValidationSchedule,
 	/// The local validation data.
 	local_validation_data: LocalValidationData,
+	/// Messages directed to other paras routed via the relay chain.
+	horizontal_messages: Vec<HorizontalMessage>,
 	/// Upwards messages to the relay chain.
 	upwards_messages: Vec<UpwardsMessage>,
 	/// Fees paid to the validators of the relay-chain.

roadmap/implementers-guide/src/types/messages.md

@@ -2,6 +2,9 @@
 
 Types of messages that are passed between parachains and the relay chain: UMP, DMP, XCMP.
 
+There is also HRMP (Horizontally Relay-routed Message Passing) which provides the same functionality
+although with smaller scalability potential.
+
 ## Upward Message
 
 A type of messages dispatched from a parachain to the relay chain.
@@ -26,3 +29,34 @@ struct UpwardMessage {
 	pub data: Vec<u8>,
 }
 ```
+
+## Horizontal Message
+
+This is a message sent from a parachain to another parachain that travels through the relay chain.
+This message ends up in the recipient's mailbox. A size of a horizontal message is defined by its
+`data` payload.
+
+```rust,ignore
+struct HorizontalMessage {
+	/// The para that will get this message in its downward message queue.
+	pub recipient: ParaId,
+	/// The message payload.
+	pub data: Vec<u8>,
+}
+```
+
+## Downward Message
+
+A message that go down from the relay chain to a parachain. Such a message could be initiated either
+as a result of an operation took place on the relay chain or sent using a horizontal message.
+
+```rust,ignore
+enum DownwardMessage {
+	/// Some funds were transferred into the parachain's account. The hash is the identifier that
+	/// was given with the transfer.
+	TransferInto(AccountId, Balance, Remark),
+	/// This downward message is a result of a horizontal message represented as opaque bytes sent
+	/// by the specified sender.
+	HorizontalMessage(AccountId, Vec<u8>),
+}
+```

roadmap/implementers-guide/src/types/runtime.md

@@ -40,5 +40,14 @@ struct HostConfiguration {
 	/// no further messages may be added to it. If it exceeds this then the queue may contain only
 	/// a single message.
 	pub watermark_upward_queue_size: u32,
+	/// The maximum number of downward messages originated from the relay chain in a downward message
+	/// queue.
+	pub max_relay_chain_downward_messages: u32,
+	/// The maximum number of horizontal messages allowed in a downward message queue per one sender
+	/// para at the same time.
+	pub max_hrmp_queue_count_per_sender: u32,
+	/// The maximum total size of messages in bytes allowed in a downward message queue per one
+	/// sender para at the same time.
+	pub max_hrmp_queue_size_per_sender: u32,
 }
 ```