Builder spec changes for Capella & EIP-4844 by jimmygchen · Pull Request #58 · ethereum/builder-specs

jimmygchen · 2022-12-01T14:31:17Z

Description

Updated ExecutionPayload & BeaconBlockBody types to align with EIP-4844 specs. (Depends on types from this Beacon API PR: Update types to support EIP-4844 beacon-APIs#271)
Add blob_kzg_commitments to getExecutionPayloadHeader response (SignedBuilderBid), so that validator can include it in the block and create a signature over the blinded block.
Update submitBlindedBlock endpoint for 4844, that includes blobs_sidecar in the response (ExecutionPayloadAndBlobsSidecar).

Changes for EIP-4844

sequenceDiagram
    participant validator
    participant consensus
    participant mev_boost
    Title: Block Proposal with Builder API (EIP-4844)
    validator->>consensus: GET v2/validator/blinded_blocks/{slot}
    consensus->>mev_boost: GET v1/builder/header/{slot}/{parent_hash}/{pubkey}
    mev_boost-->>consensus: GET v1/builder/header/{slot}/{parent_hash}/{pubkey} response
    Note left of mev_boost: add `blob_kzg_commitments` to response (`SignedBuilderBid`)
    consensus-->>validator: GET v2/validator/blinded_blocks/{slot} response
    Note left of consensus: add `blob_kzg_commitments` to response
    Note over validator: include `blob_kzg_commitments` in block and sign the header
    validator->>consensus: POST beacon/blinded_blocks
    consensus->>mev_boost: POST /eth/v2/builder/blinded_blocks
    mev_boost-->>consensus: POST /eth/v2/builder/blinded_blocks response
    Note left of mev_boost: revealed payload will have to include `blobs_sidecar` and kzg proof
    Note over consensus: construct block and broadcast block

Add capella & 4844 fork version support

apis/builder/blinded_blocks.yaml

0xGabi

Great work on adding the blobs_sidecar, I believe we are getting close to mark the PR ready for review.

I had some general thoughts about writing the spec:

While looking through the execution-apis spec I start wondering if we want to further clarify the changes between submitBlindedBlockV1 and submitBlindedBlockV2.
If we follow the same versioning criteria of the execution API. We may need to version the structures (i.e. SignedBuilderBidV2) and as a consequence have a getHeaderV2. I'm not sure we want to follow that path but wanted to bring it for discussion.

.gitmodules

apis/builder/blinded_blocks.yaml

types/eip4844/block_and_blobs_sidecar.yaml

apis/builder/blinded_blocks.yaml

jimmygchen · 2022-12-02T07:31:50Z

While looking through the execution-apis spec I start wondering if we want to further clarify the changes between submitBlindedBlockV1 and submitBlindedBlockV2.

If we follow the same versioning criteria of the execution API. We may need to version the structures (i.e. SignedBuilderBidV2) and as a consequence have a getHeaderV2. I'm not sure we want to follow that path but wanted to bring it for discussion.

Great point, I'm not sure tbh, and I'd like to hear some more discussions on this too. There were some discussions on discord earlier, and it looks like we don't need a new version just for new fields, however in this case we're changing the endpoint to return a SignedBeaconBlockAndSidecar instead of ExecutionPayload, so it's a breaking change.
https://discord.com/channels/595666850260713488/1031999860997619843/1038023245900820531

apis/builder/header.yaml

metachris · 2022-12-02T10:23:10Z

apis/builder/blinded_blocks_v2.yaml

@@ -0,0 +1,58 @@
+post:


from the discussions i've had with EF/CL peers, there seems to be a preference for reusing the existing API call, rather than introducing a new V2 one. cc/ @jtraglia @lightclient @ralexstokes

Hey thanks for the insight @metachris. I've seen some discussion on discord earlier too. My understanding was that the version header is used for determining the payload structure, so clients should be able to parse the same payload type from different forks. It make sense to me to keep the same API if the response type is the same, i.e. if we return Eip4844.ExeuctionPayload.

However I think we'd also want to include blobs and the kzg proof, so it would be more convenient to change the response type to SignedBeaconBlockAndBlobsSidecar (with the revealed ExecutionPayload wrapped within the block), and the clients can just broadcast the block after receiving it, rather than having to retrieve the sidecar and construct the beacon block. Does this make sense? I haven't seen an API allowing a completely different response type, so I thought we may want to increment the version, but keen to hear your thoughts!

(cc: @ajsutton)

I haven't seen an API allowing a completely different response type

this should be fine -- we should be able to just add the right type to the v1 endpoint

let's drop v2 here and extend v1 to also support a new type that contains the ExecutionPayload and the BlobsSidecar, calling it something like ExecutionPayloadWithBlobsSidecar

Done, thanks!

metachris · 2022-12-02T15:35:23Z

Just wanted to say thanks for the contribution! 🚀
Would love to understand more about where this is coming from.

pinging @lightclient @jtraglia @lightclient @terencechain @tbenr @ajsutton (and others) for review and feedback

tbenr · 2022-12-12T15:02:16Z

Regarding submitBlindedBlock, It makes sense to me to have a V2 returning the entireSignedBeaconBlockAndBlobsSidecar. It reduces complexity and the overhead is minimal since ExecutionPayload and BlobsSidecar are the biggest objects we need to get anyway.

ralexstokes

excellent work!

I left some questions and feedback -- this is a great start :)

This PR is fine as is but something to keep in mind for the future is that it is much easier to review smaller PRs, so this one could easily be a PR just for capella changes and then a separate one for 4844 changes

.gitmodules

ralexstokes · 2022-12-13T21:01:07Z

types/capella/bid.yaml

+      value:
+        $ref: "../../beacon-apis/types/primitive.yaml#/Uint256"
+        description: "Payment in wei that will be paid to the `fee_recipient` account."
+      pubkey:
+        $ref: "../../beacon-apis/types/primitive.yaml#/Pubkey"
+        description: "BLS public key of builder."


I think we can define a BuilderBidCommon schema to reduce the repetition here

Done, thanks

ralexstokes · 2022-12-13T21:04:33Z

examples/capella/execution_payload.json

+      "withdrawals": [
+        "0x02f878831469668303f51d843b9ac9f9843b9aca0082520894c93269b73096998db66be0441e836d873535cb9c8894a19041886f000080c001a031cc29234036afbf9a1fb9476b463367cb1f957ac0b919b69bbc798436e604aaa018c4e9c3914eb27aadd0b91e10b18655739fcf8c1fc398763a9f1beecb8ddc86"
+      ]


ideally we have an actual hex-encoded withdrawal here

ah nice catch, updated now - just realised withdrawals is a list of objects.. wonder why validation didn't pick up 🤔

ralexstokes · 2022-12-13T21:05:15Z

examples/eip4844/execution_payload.json

+        "0x02f878831469668303f51d843b9ac9f9843b9aca0082520894c93269b73096998db66be0441e836d873535cb9c8894a19041886f000080c001a031cc29234036afbf9a1fb9476b463367cb1f957ac0b919b69bbc798436e604aaa018c4e9c3914eb27aadd0b91e10b18655739fcf8c1fc398763a9f1beecb8ddc86"
+      ],
+      "withdrawals": [
+        "0x02f878831469668303f51d843b9ac9f9843b9aca0082520894c93269b73096998db66be0441e836d873535cb9c8894a19041886f000080c001a031cc29234036afbf9a1fb9476b463367cb1f957ac0b919b69bbc798436e604aaa018c4e9c3914eb27aadd0b91e10b18655739fcf8c1fc398763a9f1beecb8ddc86"


same on withdrawal example here

ralexstokes · 2022-12-13T21:14:26Z

types/eip4844/block_and_blobs_sidecar.yaml

+  SignedBeaconBlockAndBlobsSidecar:
+    type: object
+    description: "The `SignedBeaconBlockAndBlobsSidecar` object from the EIP-4844 CL spec."
+    properties:
+      beacon_block:
+        $ref: "../../beacon-apis/types/eip4844/block.yaml#/EIP4844/SignedBeaconBlock"
+      blobs_sidecar:
+        $ref: "#/EIP4844/BlobsSidecar"
+
+  BlobsSidecar:
+    type: object
+    description: "The `BlobsSidecar` object from the EIP-4844 CL spec."
+    properties:
+      beacon_block_root:
+        $ref: "../../beacon-apis/types/primitive.yaml#/Root"
+      beacon_block_slot:
+        $ref: "../../beacon-apis/types/primitive.yaml#/Uint64"
+      blobs:
+        type: array
+        maxItems: 16
+        minItems: 0
+        items:
+          $ref: "#/EIP4844/Blob"
+      kzg_aggregated_proof: 
+        $ref: "#/EIP4844/KZGProof"
+
+  KZGProof:
+    type: string
+    format: hex
+    pattern: "^0x[a-fA-F0-9]{96}$"
+    description: "An aggregated KZG proof. Same check as `KZGCommitment`"
+
+  Blob:
+    type: string
+    format: hex
+    pattern: "^0x[a-fA-F0-9]{262144}$"
+    description: "A blob is `FIELD_ELEMENTS_PER_BLOB * size_of(BLSFieldElement) = 4096 * 32 = 131072` bytes (`DATA`) representing a SSZ-encoded Blob as defined in EIP-4844"


what do you think about moving SignedBeaconBlockAndBlobsSidecar to the beacon-apis and then just referencing them here?

@0xGabi made a comment about this here as well - I wasn't sure about this as it's not used in beacon-apis (and may lead to validation warnings if there's one), but happy to move this since we've got more arguments for moving over to beacon-apis

ralexstokes · 2022-12-13T21:16:28Z

apis/builder/header.yaml

                example: "bellatrix"
              data:
-                oneOf:
+                anyOf:


not sure on the semantics of oneOf vs anyOf but it sounds like we want oneOf here

Yes, it should be oneOf, I had a bit of issues getting this to work earlier (noted in this comment) - perhaps we'll need to make all new properties in schema versions required in order for the examples pass validation::

I'm having some issues with lint here complaining about these examples matching multiple schemas, and not sure what's the best way to resolve this.
I think the json examples are matching multiple schemas because the schema allows for additional properties. I think we could set additionalProperties to false so that it doesn't match all, but I haven't seen this approach used anywhere else in the API repos.

~/builder-specs/examples/bellatrix/signed_builder_bid.json 4:13 error oas3-valid-media-example "data" property must match exactly one schema in oneOf value.data ~/builder-specs/examples/capella/signed_builder_bid.json 4:13 error oas3-valid-media-example "data" property must match exactly one schema in oneOf value.data ~/builder-specs/examples/eip4844/signed_builder_bid.json 4:13 error oas3-valid-media-example "data" property must match exactly one schema in oneOf value.data

I don't see any easy way around without adding required for new properties everywhere, which is a bit ugly, so I went with using anyOf instead - if anyone has a better solution for this please let me know!

Changed to oneOf. Got this working by adding additioanlProperties: false to ExecutionPayloadHeader and BuilderBid types.

ralexstokes · 2022-12-13T21:19:43Z

apis/builder/blinded_blocks_v2.yaml

@@ -0,0 +1,58 @@
+post:


I haven't seen an API allowing a completely different response type

this should be fine -- we should be able to just add the right type to the v1 endpoint

let's drop v2 here and extend v1 to also support a new type that contains the ExecutionPayload and the BlobsSidecar, calling it something like ExecutionPayloadWithBlobsSidecar

0xGabi

Well done. I like ExecutionPayloadAndBlobSidecar.

.gitmodules

Remove comment.

jimmygchen · 2022-12-16T03:42:24Z

Thanks for the review @ralexstokes @0xGabi! 🙏 I've addressed your comments, please review again.

0xGabi

LGTM

0xGabi · 2022-12-16T06:14:54Z

examples/capella/execution_payload.json

      ],
      "withdrawals": [
-        "0x02f878831469668303f51d843b9ac9f9843b9aca0082520894c93269b73096998db66be0441e836d873535cb9c8894a19041886f000080c001a031cc29234036afbf9a1fb9476b463367cb1f957ac0b919b69bbc798436e604aaa018c4e9c3914eb27aadd0b91e10b18655739fcf8c1fc398763a9f1beecb8ddc86"
+        {


metachris · 2022-12-23T09:35:29Z

I think it would make sense to split this PR into two -- one for Capella changes, and another one for EIP4844 changes? (The 4844 PR could point to the Capella PR to reuse anything that's already in there)

g11tech · 2023-01-02T15:44:11Z

examples/eip4844/signed_beacon_block_and_blobs_sidecar.json

+              "0x02f878831469668303f51d843b9ac9f9843b9aca0082520894c93269b73096998db66be0441e836d873535cb9c8894a19041886f000080c001a031cc29234036afbf9a1fb9476b463367cb1f957ac0b919b69bbc798436e604aaa018c4e9c3914eb27aadd0b91e10b18655739fcf8c1fc398763a9f1beecb8ddc86"
+            ],
+            "withdrawals": [
+              "0x02f878831469668303f51d843b9ac9f9843b9aca0082520894c93269b73096998db66be0441e836d873535cb9c8894a19041886f000080c001a031cc29234036afbf9a1fb9476b463367cb1f957ac0b919b69bbc798436e604aaa018c4e9c3914eb27aadd0b91e10b18655739fcf8c1fc398763a9f1beecb8ddc86"


withdrawals json here looks incorrect

Thanks @g11tech that's fixed now!

jimmygchen · 2023-01-03T00:42:18Z

I think it would make sense to split this PR into two -- one for Capella changes, and another one for EIP4844 changes? (The 4844 PR could point to the Capella PR to reuse anything that's already in there)

Hi @metachris @ralexstokes I've addressed all comments and created separated PRs below:

Capella Add and update types for Capella #60
EIP-4844 Update getHeader and submitBlindedBlock to support blobs (EIP-4844) #61

Closing this PR.

jimmygchen and others added 13 commits November 24, 2022 18:28

Add initial files for EIP-4844 fork

8bd1d61

Add capella & 4844 fork version support

104a1f7

Update submodule ref

da1fc21

Update builder API schema and examples

cd21096

chore: generate doctoc

bc49eb1

Update submodule ref

e9ff861

Add blob_kzg_commitments to BuilderBid

5227b6e

Update examples/eip4844/signed_builder_bid.json

2ebb085

Merge pull request #1 from 0xGabi/eip4844

03c57d7

Add capella & 4844 fork version support

Add blobs and proof to revealed payload

dc4cd6e

Update beacon API remote

250b6f8

Fix incorrect ref syntax

4430897

Add submitBlindedBlockV2 endpoint

834834c

jimmygchen commented Dec 1, 2022

View reviewed changes

apis/builder/blinded_blocks.yaml Outdated Show resolved Hide resolved

jimmygchen added 2 commits December 2, 2022 01:55

Add missing field in BuilderBidd schema

7a9ed25

Fix typo

75403ff

jimmygchen mentioned this pull request Dec 1, 2022

Changes for Capella and EIP-4844 flashbots/mev-boost#392

Closed

0xGabi suggested changes Dec 2, 2022

View reviewed changes

.gitmodules Outdated Show resolved Hide resolved

apis/builder/blinded_blocks.yaml Outdated Show resolved Hide resolved

types/eip4844/block_and_blobs_sidecar.yaml Outdated Show resolved Hide resolved

apis/builder/blinded_blocks.yaml Outdated Show resolved Hide resolved

Add TODO to .gitmodules

ea928bf

jimmygchen commented Dec 2, 2022

View reviewed changes

apis/builder/header.yaml Outdated Show resolved Hide resolved

metachris reviewed Dec 2, 2022

View reviewed changes

Fix spellcheck

113787a

Fix lint error

6728e3b

jimmygchen marked this pull request as ready for review December 2, 2022 15:58

jimmygchen changed the title ~~[wip] Builder spec changes for Capella & EIP-4844~~ Builder spec changes for Capella & EIP-4844 Dec 2, 2022

jimmygchen added 3 commits December 12, 2022 22:52

fix spell check

859443c

fix spellcheck

8fe8a7c

fix spell check

0f7a4b4

remove extra whitespace

2f552a8

ralexstokes requested changes Dec 13, 2022

View reviewed changes

revert submodule fork

4b29292

This was referenced Dec 14, 2022

Update types to support EIP-4844 ethereum/beacon-APIs#271

Merged

Builder updates for Blobs (EIP-4844) sigp/lighthouse#3808

Closed

jimmygchen added 3 commits December 15, 2022 19:36

Remove blinded_blocks_v2 endpoint

96a32e6

Change getHeader response schema to oneOf

29a560a

Update Beacon API version

9b60d69

0xGabi approved these changes Dec 15, 2022

View reviewed changes

.gitmodules Outdated Show resolved Hide resolved

0xGabi and others added 3 commits December 15, 2022 18:46

Update .gitmodules

65df997

Remove comment.

Define BuilderBidCommon to reduce repetition

2072690

add withdrawals to execution payload example

69af188

jimmygchen requested review from 0xGabi and ralexstokes and removed request for 0xGabi and ralexstokes December 16, 2022 03:42

0xGabi approved these changes Dec 16, 2022

View reviewed changes

tbenr mentioned this pull request Dec 16, 2022

Support Block and Blobs production via builder Consensys/teku#6621

Closed

avalonche mentioned this pull request Dec 20, 2022

Add builder types for capella attestantio/go-builder-client#2

Closed

g11tech reviewed Jan 2, 2023

View reviewed changes

Fix withdrawals json example format

fcbc901

This was referenced Jan 3, 2023

Add and update types for Capella #60

Merged

Update getHeader and submitBlindedBlock to support blobs (EIP-4844) #61

Merged

jimmygchen closed this Jan 3, 2023

g11tech mentioned this pull request Jan 3, 2023

Ugrade builder flow for capella and eip4844 forks ChainSafe/lodestar#4964

Merged

Conversation

jimmygchen commented Dec 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes for EIP-4844

Uh oh!

Uh oh!

0xGabi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jimmygchen commented Dec 2, 2022

Uh oh!

Uh oh!

metachris Dec 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

metachris commented Dec 2, 2022

Uh oh!

tbenr commented Dec 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ralexstokes left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

0xGabi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jimmygchen commented Dec 16, 2022

Uh oh!

0xGabi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

metachris commented Dec 23, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jimmygchen commented Jan 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

jimmygchen commented Dec 1, 2022 •

edited

Loading

metachris Dec 2, 2022 •

edited

Loading

tbenr commented Dec 12, 2022 •

edited

Loading

jimmygchen commented Jan 3, 2023 •

edited

Loading