Add REST endpoint to retrieve historical_summaries

[kdeme]

In draft currently as it is not in spec. Needs to be accompanied still with a proposal for beacon REST API specifications.

The idea is that historical_summaries is very useful for verifying if beacon blocks (with their accompanied proofs) are part of the canonical chain, however there is no way to actually retrieve the historical_summaries except by using the /eth/v2/debug/beacon/states/{state_id} endpoint, which is quite the download in terms of size (We are currently using this endpoint in Portal/fluffy for our portal_bridge, but it takes long).

This PR adds an endpoint that provides the historical_summaries with its proof against the state root. This is done so that the historical_summaries can then be verified for example with the finalized state root that a (light) node might hold from its finality update.
Again, this is specifically useful in Portal (as a portal_bridge can inject it in the p2p network as is, and receiving nodes can verify it if they are light client synced) . But you could make an argument that it could also be useful on a consensus light client (assuming blocks with proofs would be somehow available).

edit: moved to /nimbus/v1/debug/ namespace and removed draft

[github-actions]

Unit Test Results

       15 files  ±  0    2 614 suites  ±0   1h 16m 26s ⏱️ + 1m 5s
  6 412 tests +  1    5 891 ✔️ +  1  521 💤 ±0  0 ❌ ±0 
44 616 runs  +10  43 898 ✔️ +10  718 💤 ±0  0 ❌ ±0 

Results for commit 6cebf61. ± Comparison against base commit 3a91604.

♻️ This comment has been updated with latest results.

[tersec]

One consideration for me is that if this does get standardized, we don't end up with both a pre-standardized and standardized version which needs to be supported.

[kdeme]

One consideration for me is that if this does get standardized, we don't end up with both a pre-standardized and standardized version which needs to be supported.

One approach could be that I move it under the /nimbus/v1/debug/ prefix for now, assuming that /nimbus/v1/debug/ would mean custom and subject to removal without notice (like for the debug cli flags).

[cheatfate]

One consideration for me is that if this does get standardized, we don't end up with both a pre-standardized and standardized version which needs to be supported.

One approach could be that I move it under the /nimbus/v1/debug/ prefix for now, assuming that /nimbus/v1/debug/ would mean custom and subject to removal without notice (like for the debug cli flags).

Why you need to remove it? You can establish redirect() call from one endpoint to another when it will be part of specification, we was doing this many times.

[tersec]

One consideration for me is that if this does get standardized, we don't end up with both a pre-standardized and standardized version which needs to be supported.

One approach could be that I move it under the /nimbus/v1/debug/ prefix for now, assuming that /nimbus/v1/debug/ would mean custom and subject to removal without notice (like for the debug cli flags).

Why you need to remove it? You can establish redirect() call from one endpoint to another when it will be part of specification, we was doing this many times.

Well, part of the point is not to end up with extra, redundant endpoints. The point of this subject to removal without notice would be to exactly allow experimentation without obligation such directions later.

[cheatfate]

One consideration for me is that if this does get standardized, we don't end up with both a pre-standardized and standardized version which needs to be supported.

One approach could be that I move it under the /nimbus/v1/debug/ prefix for now, assuming that /nimbus/v1/debug/ would mean custom and subject to removal without notice (like for the debug cli flags).

Why you need to remove it? You can establish redirect() call from one endpoint to another when it will be part of specification, we was doing this many times.

Well, part of the point is not to end up with extra, redundant endpoints. The point of this subject to removal without notice would be to exactly allow experimentation without obligation such directions later.

There could be exactly 2 situations:

1. Change will be proposed, but declined.
2. Change will be proposed and accepted.

In both this cases it could be better to start with /nimbus/ endpoint. Because /nimbus/ endpoint could not be regulated by EF, so we could add/remove any functions. Otherwise in case 1 you will be forced to remove it from /debug/.

[kdeme]

Went with different arrays for the different proofs as this is for example also what is done for FinalityBranch, CurrentSyncCommitteeBranch and NextSyncCommitteeBranch.

But could in theory also just use a seq[Eth2Digest] which I think would make the whole ForkedHistoricalSummariesWithProof helpers not required.

[etan-status]

I think having it consistent with other endpoints is better than seq, even though it adds some boilerplate here. There is for every fork only a single correct proof length, and using array guarantees that incorrect proof lengths are rejected early at deserialization time.

The real solution would be to adopt EIP-7688... then there are no more random proof length changes for different consensusFork. It's a design flaw that every client has to keep maintaining gindices whenever ethereum decides to release some random functionality unrelated to the client's interests.

[kdeme]

The real solution would be to adopt EIP-7688

Yes, I agree. But that is not in my sole hands here :)

[kdeme]

@etan-status and/or @cheatfate Would you mind having a look at this again? It is something crucial for our development on Portal network.

As mentioned in the comments, fork management is added, which I think was the only issue (at least for getting it under nimbus/debug namespace)?

Some remarks:

• Those HistoricalSummariesProof and ForkedHistoricalSummariesProof are added in the rest types code as I did not want to pollute the specs and forks code with types that are not actually part of the consensus specifications. But if you believe they now pollute too much the rest types, I can also move them into another new file.
• The Forked type is required as we use arrays for the proofs (which differ in size between forks). We could also opt for going with a sequence instead, but that is not consistent with what is done for example for FinalityBranch, CurrentSyncCommitteeBranch and NextSyncCommitteeBranch.

[etan-status]

There's also an effort from @Inspector-Butters and @wemeetagain to look into a general proof API. But I think as part of Nimbus namespace it's alright to have specific endpoints to unblock portal progress.

Code looks correct to me, I have put some remarks to make it less maintenance-heavy and reduce duplication. Once comments are addressed, good to go.

[etan-status]

Hmm. Do we really need to retain the exact consensusFork?

Or would a HistoricalSummariesFork make sense, similar to LightClientDataFork, EpochInfoFork or (experimental BlobFork in #6451)? That way, this code no longer has to be maintained with every fork, by simply having a capellaData and electraData member, and no denebData / fuluData etc

case x.kind
of HistoricalSummariesFork.Electra:
  const historicalFork {.inject, used.} = HistoricalSummariesFork.Electra
  template forkySummaries: untyped {.inject, used.} = x.electraData
  body
of ConsensusFork.Capella:
  const historicalFork {.inject, used.} = HistoricalSummariesFork.Capella
  template forkySummaries: untyped {.inject, used.} = x.capellaData
  body

One can still recover exact consensusFork if needed from Eth-Consensus-Version HTTP header, if needed.

[kdeme]

No, I don't think we need it.

In the past, I mostly found it confusing to work with these different fork types and always had to look back at which one contained which forks, hence why I did not go for that.

But I think I can understand how it makes maintenance probably quite a bit easier when you are dealing with that all the time. Though for an external/new reader of the code, having all these different fork types without a reference in the spec, might be confusing.

[kdeme]

So in the case we would add a HistoricalSummariesFork, I assume the code would work in such way that in the metrics API where there is now the statement:

    node.withStateForBlockSlotId(bslot):
      return withState(state):
        when consensusFork >= ConsensusFork.Capella:

we would set the HistoricalSummariesFork based on the ConsensusFork?

And then from then onward work with the HistoricalSummariesFork?

I can see the benefit of not having to update the fork templates for that, but it also allows for not failing compilation when a fork gets added which might require a change again?

I am currently undecided what I like the most, but if there is a strong pull towards adding HistoricalSummariesFork then I don't mind doing so (especially since it is already the habit in the code base).

[etan-status]

I think having it consistent with other endpoints is better than seq, even though it adds some boilerplate here. There is for every fork only a single correct proof length, and using array guarantees that incorrect proof lengths are rejected early at deserialization time.

The real solution would be to adopt EIP-7688... then there are no more random proof length changes for different consensusFork. It's a design flaw that every client has to keep maintaining gindices whenever ethereum decides to release some random functionality unrelated to the client's interests.

[kdeme]

@etan-status I addressed latest feedback, aside from the specific fork addition, which I commented on (but am open to add still).