Add Cell Dissemination via Partial Message Specification

specs/fulu/p2p-interface.md

@@ -8,11 +8,16 @@
   - [Configuration](#configuration)
   - [Containers](#containers)
     - [`DataColumnsByRootIdentifier`](#datacolumnsbyrootidentifier)
+    - [`PartialDataColumnSidecar`](#partialdatacolumnsidecar)
+    - [`PartialDataColumnPartsMetadata`](#partialdatacolumnpartsmetadata)
+    - [`PartialDataColumnHeader`](#partialdatacolumnheader)
   - [Helpers](#helpers)
     - [Modified `compute_fork_version`](#modified-compute_fork_version)
     - [`verify_data_column_sidecar`](#verify_data_column_sidecar)
     - [`verify_data_column_sidecar_kzg_proofs`](#verify_data_column_sidecar_kzg_proofs)
     - [`verify_data_column_sidecar_inclusion_proof`](#verify_data_column_sidecar_inclusion_proof)
+    - [`verify_partial_data_column_header_inclusion_proof`](#verify_partial_data_column_header_inclusion_proof)
+    - [`verify_partial_data_column_sidecar_kzg_proofs`](#verify_partial_data_column_sidecar_kzg_proofs)
     - [`compute_subnet_for_data_column_sidecar`](#compute_subnet_for_data_column_sidecar)
   - [MetaData](#metadata)
   - [The gossip domain: gossipsub](#the-gossip-domain-gossipsub)
@@ -23,6 +28,18 @@
         - [Deprecated `blob_sidecar_{subnet_id}`](#deprecated-blob_sidecar_subnet_id)
         - [`data_column_sidecar_{subnet_id}`](#data_column_sidecar_subnet_id)
         - [Distributed blob publishing using blobs retrieved from local execution-layer client](#distributed-blob-publishing-using-blobs-retrieved-from-local-execution-layer-client)
+        - [Partial Messages on `data_column_sidecar_{subnet_id}`](#partial-messages-on-data_column_sidecar_subnet_id)
+    - [Partial columns for Cell Dissemination](#partial-columns-for-cell-dissemination)
+      - [Partial message group ID](#partial-message-group-id)
+      - [Parts metadata](#parts-metadata)
+      - [Encoding and decoding responses](#encoding-and-decoding-responses)
+      - [Eager pushing](#eager-pushing)
+      - [Interaction with standard gossipsub](#interaction-with-standard-gossipsub)
+        - [Requesting partial messages](#requesting-partial-messages)
+        - [Mesh](#mesh)
+        - [Fanout](#fanout)
+        - [Scoring](#scoring)
+        - [Forwarding](#forwarding)
   - [The Req/Resp domain](#the-reqresp-domain)
     - [Messages](#messages)
       - [Status v2](#status-v2)
@@ -76,6 +93,67 @@ class DataColumnsByRootIdentifier(Container):
     columns: List[ColumnIndex, NUMBER_OF_COLUMNS]
 ```
 
+#### `PartialDataColumnSidecar`
+
+The `PartialDataColumnSidecar` is similar to the `DataColumnSidecar` container,
+except that only the cells and proofs identified by the bitmap are present.
+
+*Note*: The column index is inferred from the gossipsub topic subnet.
+
+```python
+class PartialDataColumnSidecar(Container):
+    cells_present_bitmap: Bitlist[MAX_BLOB_COMMITMENTS_PER_BLOCK]
+    partial_column: List[Cell, MAX_BLOB_COMMITMENTS_PER_BLOCK]
+    kzg_proofs: List[KZGProof, MAX_BLOB_COMMITMENTS_PER_BLOCK]
+    # Optional header, only sent on eager pushes
+    header: List[PartialDataColumnHeader, 1]
+```
+
+#### `PartialDataColumnPartsMetadata`
+
+Peers communicate the cells available with a bitmap. A set bit (`1`) at index
+`i` means that the peer has the cell at index `i`. The bitmap is encoded as a
+`Bitlist`. Peers explicitly request cells with a second request bitmap of the
+same length that is set to `1` if the peer would like to receive or provide this
+cell.
+
+This is encoded as the following SSZ container:
+
+```python
+class PartialDataColumnPartsMetadata(Container):
+    available: Bitlist[MAX_BLOB_COMMITMENTS_PER_BLOCK]
+    requests: Bitlist[MAX_BLOB_COMMITMENTS_PER_BLOCK]
+```
+
+This means that for each cell there are two bits of state:
+
+| Bits | Description                                          |
+| :--: | ---------------------------------------------------- |
+|  00  | The peer does not have the cell and does not want it |
+|  01  | The peer does not have the cell and does want it     |
+|  10  | Unused, ignore                                       |
+|  11  | The peer has the cell and is willing to provide it   |
+
+Having a cell but not willing to provide it is functionally the same as not
+having the cell and not wanting it, so it does not need a separate state.
+
+Clients MUST only provide or request a cell if the second bit is set to `1`.
+
+#### `PartialDataColumnHeader`
+
+The `PartialDataColumnHeader` is the header that is common to all columns for a
+given block. It lets a peer identify which blobs are included in a block, as
+well as validating cells and proofs. This header is only sent on eager pushes
+because a peer can only make a request after having the data in this header.
+This header can be derived from a beacon block or a `DataColumnSidecar`.
+
+```python
+class PartialDataColumnHeader(Container):
+    kzg_commitments: List[KZGCommitment, MAX_BLOB_COMMITMENTS_PER_BLOCK]
+    signed_block_header: SignedBeaconBlockHeader
+    kzg_commitments_inclusion_proof: Vector[Bytes32, KZG_COMMITMENTS_INCLUSION_PROOF_DEPTH]
+```
+
 ### Helpers
 
 #### Modified `compute_fork_version`
@@ -164,6 +242,48 @@ def verify_data_column_sidecar_inclusion_proof(sidecar: DataColumnSidecar) -> bo
     )
 ```
 
+#### `verify_partial_data_column_header_inclusion_proof`
+
+```python
+def verify_partial_data_column_header_inclusion_proof(header: PartialDataColumnHeader) -> bool:
+    """
+    Verify if the given KZG commitments are included in the given beacon block.
+    """
+    return is_valid_merkle_branch(
+        leaf=hash_tree_root(header.kzg_commitments),
+        branch=header.kzg_commitments_inclusion_proof,
+        depth=KZG_COMMITMENTS_INCLUSION_PROOF_DEPTH,
+        index=get_subtree_index(get_generalized_index(BeaconBlockBody, "blob_kzg_commitments")),
+        root=header.signed_block_header.message.body_root,
+    )
+```
+
+#### `verify_partial_data_column_sidecar_kzg_proofs`
+
+```python
+def verify_partial_data_column_sidecar_kzg_proofs(
+    sidecar: PartialDataColumnSidecar,
+    all_commitments: List[KZGCommitment, MAX_BLOB_COMMITMENTS_PER_BLOCK],
+    column_index: ColumnIndex,
+) -> bool:
+    """
+    Verify the KZG proofs.
+    """
+    # Get the blob indices from the bitmap
+    blob_indices = [i for i, b in enumerate(sidecar.cells_present_bitmap) if b]
+
+    # The cell index is the column index for all cells in this column
+    cell_indices = [CellIndex(column_index)] * len(blob_indices)
+
+    # Batch verify that the cells match the corresponding commitments and proofs
+    return verify_cell_kzg_proof_batch(
+        commitments_bytes=[all_commitments[i] for i in blob_indices],
+        cell_indices=cell_indices,
+        cells=sidecar.partial_column,
+        proofs_bytes=sidecar.kzg_proofs,
+    )
+```
+
 #### `compute_subnet_for_data_column_sidecar`
 
 ```python
@@ -292,6 +412,149 @@ gossip. In particular, clients MUST:
 - Update gossip rule related data structures (i.e. update the anti-equivocation
   cache).
 
+###### Partial Messages on `data_column_sidecar_{subnet_id}`
+
+Validating partial messages happens in two parts. First, the
+`PartialDataColumnHeader` needs to be validated, then the cell and proof data.
+
+Once a `PartialDataColumnHeader` is validated for a corresponding block on any
+subnet (gossipsub topic), it can be used for all subnets.
+
+Due to the nature of partial messages, it is possible to get the
+`PartialDataColumnHeader` with no cells, and get cells in a future response.
+
+For all partial messages:
+
+- _[IGNORE]_ If the received partial message contains only cell data, the node
+  has seen the corresponding `PartialDataColumnHeader`.
+
+For verifying the `PartialDataColumnHeader` in a partial message:
+
+- _[IGNORE]_ The header is the first valid header for the given block root.
+- _[REJECT]_ The hash of the block header in `signed_block_header` MUST be the
+  same as the partial message's group id.
+- _[REJECT]_ The header's `kzg_commitments` list is non-empty.
+- _[IGNORE]_ The header is not from a future slot (with a
+  `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) -- i.e. validate that
+  `block_header.slot <= current_slot` (a client MAY queue future headers for
+  processing at the appropriate slot).
+- _[IGNORE]_ The header is from a slot greater than the latest finalized slot --
+  i.e. validate that
+  `block_header.slot > compute_start_slot_at_epoch(state.finalized_checkpoint.epoch)`
+- _[REJECT]_ The proposer signature of `signed_block_header` is valid with
+  respect to the `block_header.proposer_index` pubkey.
+- _[IGNORE]_ The header's block's parent (defined by `block_header.parent_root`)
+  has been seen (via gossip or non-gossip sources) (a client MAY queue header
+  for processing once the parent block is retrieved).
+- _[REJECT]_ The header's block's parent (defined by `block_header.parent_root`)
+  passes validation.
+- _[REJECT]_ The header is from a higher slot than the header's block's parent
+  (defined by `block_header.parent_root`).
+- _[REJECT]_ The current `finalized_checkpoint` is an ancestor of the header's
+  block -- i.e.
+  `get_checkpoint_block(store, block_header.parent_root, store.finalized_checkpoint.epoch) == store.finalized_checkpoint.root`.
+- _[REJECT]_ The header's `kzg_commitments` field inclusion proof is valid as
+  verified by `verify_partial_data_column_header_inclusion_proof`.
+- _[REJECT]_ The header is proposed by the expected `proposer_index` for the
+  block's slot in the context of the current shuffling (defined by
+  `block_header.parent_root`/`block_header.slot`). If the `proposer_index`
+  cannot immediately be verified against the expected shuffling, the header MAY
+  be queued for later processing while proposers for the block's branch are
+  calculated -- in such a case _do not_ `REJECT`, instead `IGNORE` this message.
+
+For verifying the cells in a partial message:
+
+- _[REJECT]_ The cells present bitmap length is equal to the number of KZG
+  commitments in the `PartialDataColumnHeader`.
+- _[REJECT]_ The sidecar's cell and proof data is valid as verified by
+  `verify_partial_data_column_sidecar_kzg_proofs(sidecar, header.kzg_commitments, column_index)`.
+
+#### Partial columns for Cell Dissemination
+
+Gossipsub's
+[Partial Message Extension](https://github.com/libp2p/specs/pull/685) enables
+exchanging selective parts of a message rather than the whole. The specification
+here describes how consensus-layer clients use Partial Messages to disseminate
+cells.
+
+##### Partial message group ID
+
+When sending a partial message, the gossipsub group ID MUST be the block root
+prefixed by a single byte used for versioning. The version byte MUST be zero.
+Other versions may be defined later.
+
+##### Parts metadata
+
+The parts metadata is encoded with the `PartialDataColumnPartsMetadata`
+container.
+
+##### Encoding and decoding responses
+
+All responses MUST be encoded and decoded with the `PartialDataColumnSidecar`
+container.
+
+##### Eager pushing
+
+In contrast to standard gossipsub, a client explicitly requests missing parts
+from a peer. A client can send its request before receiving a peer's parts
+metadata. This registers interest in certain parts, even if the peer does not
+have these parts yet.
+
+This request can introduce extra latency compared to a peer unconditionally
+pushing messages, especially in the first hop of dissemination.
+
+To address this tradeoff, a client MAY choose to eagerly push some (or all) of
+the cells it has. Clients SHOULD only do this when they are reasonably confident
+that a peer does not have the provided cells. For example, a proposer including
+private blobs SHOULD eagerly push the cells corresponding to the private blobs.
+
+Clients SHOULD eagerly push the `PartialDataColumnHeader` to inform peers as to
+which blobs are included in this block, and therefore which cells they are
+missing. Clients SHOULD NOT send a `PartialDataColumnHeader` non-eagerly, as
+this is wasted bandwidth.
+
+Clients MAY choose to not eagerly push the `PartialDataColumnHeader` if it has
+previously sent the header to the peer on another topic.
+
+Clients SHOULD request cell data from peers after validating a
+`PartialDataColumnHeader`, even if the corresponding block has not been seen
+yet.
+
+##### Interaction with standard gossipsub
+
+###### Requesting partial messages
+
+A peer requests partial messages for a topic by setting the `partial` field in
+gossipsub's `SubOpts` RPC message to `true`.
+
+###### Mesh
+
+The Partial Message Extension uses the same mesh peers for a given topic as the
+standard gossipsub topics for `DataColumnSidecar`s.
+
+###### Fanout
+
+The Partial Message Extension uses the same fanout peers for a given topic as
+the standard gossipsub topics for `DataColumnSidecar`s.
+
+###### Scoring
+
+On receiving useful novel data from a peer, the client should report to
+gossipsub a positive first message delivery.
+
+On receiving invalid data, the client should report to gossipsub an invalid
+message delivery.
+
+###### Forwarding
+
+Once clients can construct the full `DataColumnSidecar` after receiving missing
+cells, they should forward the full `DataColumnSidecar` over standard gossipsub
+to peers that do not support partial messages. This provides backwards
+compatibility with nodes that do not yet support partial messages.
+
+Avoid forwarding the full `DataColumnSidecar` message to peers that requested
+partial messages for that given topic. It is purely redundant information.
+
 ### The Req/Resp domain
 
 #### Messages

specs/gloas/p2p-interface.md

@@ -9,6 +9,7 @@
   - [Configuration](#configuration)
   - [Containers](#containers)
     - [Modified `DataColumnSidecar`](#modified-datacolumnsidecar)
+    - [Modified `PartialDataColumnHeader`](#modified-partialdatacolumnheader)
     - [New `ProposerPreferences`](#new-proposerpreferences)
     - [New `SignedProposerPreferences`](#new-signedproposerpreferences)
   - [Helpers](#helpers)
@@ -25,6 +26,7 @@
         - [`proposer_preferences`](#proposer_preferences)
       - [Blob subnets](#blob-subnets)
         - [`data_column_sidecar_{subnet_id}`](#data_column_sidecar_subnet_id)
+        - [Partial Messages on `data_column_sidecar_{subnet_id}`](#partial-messages-on-data_column_sidecar_subnet_id)
       - [Attestation subnets](#attestation-subnets)
         - [`beacon_attestation_{subnet_id}`](#beacon_attestation_subnet_id)
   - [The Req/Resp domain](#the-reqresp-domain)
@@ -77,6 +79,23 @@ class DataColumnSidecar(Container):
     beacon_block_root: Root
 ```
 
+#### Modified `PartialDataColumnHeader`
+
+*Note*: These are the same changes as the changes for `DataColumnSidecar` above.
+
+```python
+class PartialDataColumnHeader(Container):
+    kzg_commitments: List[KZGCommitment, MAX_BLOB_COMMITMENTS_PER_BLOCK]
+    # [Modified in Gloas:EIP7732]
+    # Removed `signed_block_header`
+    # [Modified in Gloas:EIP7732]
+    # Removed `kzg_commitments_inclusion_proof`
+    # [New in Gloas:EIP7732]
+    slot: Slot
+    # [New in Gloas:EIP7732]
+    beacon_block_root: Root
+```
+
 #### New `ProposerPreferences`
 
 *[New in Gloas:EIP7732]*
@@ -417,6 +436,54 @@ The following validations MUST pass before forwarding the
   be queued for later processing while proposers for the block's branch are
   calculated -- in such a case _do not_ `REJECT`, instead `IGNORE` this message.
 
+###### Partial Messages on `data_column_sidecar_{subnet_id}`
+
+*[Modified in Gloas:EIP7732]*
+
+*Note*: These are the same changes as the changes in validation rules for full
+messages on `data_column_sidecar_{subnet_id}` as defined above.
+
+**Added in Gloas:**
+
+- _[IGNORE]_ The header's `beacon_block_root` has been seen via a valid signed
+  execution payload bid. A client MAY queue the sidecar for processing once the
+  block is retrieved.
+- _[REJECT]_ The header's `slot` matches the slot of the block with root
+  `beacon_block_root`.
+- _[REJECT]_ The hash of the header's `kzg_commitments` matches the
+  `blob_kzg_commitments_root` in the corresponding builder's bid for
+  `header.beacon_block_root`.
+
+**Removed from Fulu:**
+
+- _[IGNORE]_ The header is not from a future slot (with a
+  `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) -- i.e. validate that
+  `block_header.slot <= current_slot` (a client MAY queue future headers for
+  processing at the appropriate slot).
+- _[IGNORE]_ The header is from a slot greater than the latest finalized slot --
+  i.e. validate that
+  `block_header.slot > compute_start_slot_at_epoch(state.finalized_checkpoint.epoch)`
+- _[REJECT]_ The proposer signature of `signed_block_header` is valid with
+  respect to the `block_header.proposer_index` pubkey.
+- _[IGNORE]_ The header's block's parent (defined by `block_header.parent_root`)
+  has been seen (via gossip or non-gossip sources) (a client MAY queue header
+  for processing once the parent block is retrieved).
+- _[REJECT]_ The header's block's parent (defined by `block_header.parent_root`)
+  passes validation.
+- _[REJECT]_ The header is from a higher slot than the header's block's parent
+  (defined by `block_header.parent_root`).
+- _[REJECT]_ The current `finalized_checkpoint` is an ancestor of the header's
+  block -- i.e.
+  `get_checkpoint_block(store, block_header.parent_root, store.finalized_checkpoint.epoch) == store.finalized_checkpoint.root`.
+- _[REJECT]_ The header's `kzg_commitments` field inclusion proof is valid as
+  verified by `verify_partial_data_column_header_inclusion_proof`.
+- _[REJECT]_ The header is proposed by the expected `proposer_index` for the
+  block's slot in the context of the current shuffling (defined by
+  `block_header.parent_root`/`block_header.slot`). If the `proposer_index`
+  cannot immediately be verified against the expected shuffling, the header MAY
+  be queued for later processing while proposers for the block's branch are
+  calculated -- in such a case _do not_ `REJECT`, instead `IGNORE` this message.
+
 ##### Attestation subnets
 
 ###### `beacon_attestation_{subnet_id}`