engine/blobs: move partial return support to mandatory engine_getBlobsV3

[raulk]

Context

The current specification is too lax: it does not explicitly require EL clients to support partial blob returns. Consequently, most EL clients (if not all) are omitting this support in Osaka. This gap will require additional coordination work in the future when the CL actually wants to use this feature. To prevent this, we propose biting the bullet now so that no future effort needs to be invested.

In addition, the current approach does not allow the CL (or another party) to determine which return mode the EL supports. This is suboptimal, given that the Engine API already supports capability exchange to signal feature support.

Proposed changes

After posting concerns on Discord and discussing separately with @MariusVanDerWijden, we propose:

• Making partial blob return support mandatory via the ⁠engine_getBlobsV3 method.

• Restoring the all-or-nothing return behavior for ⁠engine_getBlobsV2.

This approach seems to hit the optimal tradeoff:

• Mandatorily adds functional support for partial blob returns.
• Avoids the serde penalty while the CL consumes V2.
• Enables the CL to adopt V3 when optimizations are implemented, without further ACD coordination.
• Leverages built-in capability exchange mechanisms.

Implementation

This should be straightforward to implement. For reference, it's already done in Geth: ethereum/go-ethereum#32170.

Misc

Also fixed a bug in the OpenRPC spec that did not account for literal null returns from engine_getBlobsV2.

[MarcoPolo]

This allows an EL client to still return null as a response. From the CL's perspective V3 is identical to main's V2.

[raulk]

V3 guarantees that partial responses will be served (unless no responses can be served at all, and this is not due to an internal error, which is what the null literal case covers here). Main's V2 does not make any such behavioural guarantee.

[MarcoPolo]

V3 changes the EL behavior relative to V2, no argument there.

From the CL's perspective V3 is identical to main's V2.

This is my point. The CL has to handle receiving null or a list of partial responses, just as it does with V2. This is not a point against the PR, just a note.

[raulk]

This was a deliberate choice. V1 would return a null filled array, but it’s clearer semantics to return a single null literal that translates to an Option or pointer type in Rust and Go, given this outcome affects the whole request anyway. IMO it’s even cleaner to return an error, but I assumed there was a reason V1 didn’t from the get-go.

[mkalinin]

In general, looks good to me 👍
Let’s get some approvals from EL and CL devs and then merge!

[kasey]

In addition, the current approach does not allow the CL (or another party) to determine which return mode the EL supports.

I can think of 2 advantages to the changes this implies to v2:

1. Save resources if the CL isn't able to do anything meaningful with partial responses.

• As stated in discord, EL clients can already decide to save this overhead by just returning a nil value. So this seems like we aren't gaining anything.

2. Mitigate risk of a bug in a dormant and insufficiently tested CL code path that gets triggered when an EL starts sending partial responses after an upgrade.

• The extra version mitigates this risk and the need for extra testing at the fusaka fork.

TL;DR the choice is between extra testing at fusaka or a small bit of tech debt. If we are confident the partial response code in CL clients is safe, we wind up with simpler specs and code. If we want to avoid risk, the extra engine api version just adds a bit of boilerplate and complicates the getblobs code path.

[raulk]

@kasey thanks for chiming in. The risk argument is compelling -- I hadn't considered that advantage. Yes, in the case something goes wrong with partial responses (V3), CL clients can roll back to V2 (and the associated unoptimized CL code) without requiring any further action in the EL (e.g. changing a config flag). That's a benefit of stricter and more deterministic/explicit behaviour definitions for each method.

[raulk]

Not sure what happened in this commit, but some old wording crept in and it went unnoticed. Restored the correct text.

[raulk]

Tracking implementation status.

• Execution layer.
  • Geth implementation: eth/catalyst: implement getBlobsV3 go-ethereum#32170
  • Besu implementation: getBlobsV2 prohibits partial responses hyperledger/besu#8967
• Consensus layer.
  • Nothing for CLs to do if they're still expecting V2 to be all-or-nothing, i.e. if they had not implemented what's on master right now.
  • No code required for V3.
• Testing.
  • V2 is exercised in devnets and testnets during normal testing.
  • V3 can be exercised through simple Hive tests.

[raulk]

Closing as per #676.