Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[v2] Retriever server #1048

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[v2] Retriever server #1048

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions api/clients/v2/mock/retrieval_client.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
package mock

import (
"context"

"github.com/Layr-Labs/eigenda/api/clients/v2"
"github.com/Layr-Labs/eigenda/core"
corev2 "github.com/Layr-Labs/eigenda/core/v2"
"github.com/stretchr/testify/mock"
)

type MockRetrievalClient struct {
mock.Mock
}

var _ clients.RetrievalClient = (*MockRetrievalClient)(nil)

func NewRetrievalClient() *MockRetrievalClient {
return &MockRetrievalClient{}
}

func (c *MockRetrievalClient) GetBlob(ctx context.Context, blobHeader *corev2.BlobHeader, referenceBlockNumber uint64, quorumID core.QuorumID) ([]byte, error) {
args := c.Called()

result := args.Get(0)
return result.([]byte), args.Error(1)
}
120 changes: 120 additions & 0 deletions api/docs/eigenda-protos.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -624,6 +624,29 @@ <h2>Table of Contents</h2>
</ul>
</li>


<li>
<a href="#retriever%2fv2%2fretriever.proto">retriever/v2/retriever.proto</a>
<ul>

<li>
<a href="#retriever.v2.BlobReply"><span class="badge">M</span>BlobReply</a>
</li>

<li>
<a href="#retriever.v2.BlobRequest"><span class="badge">M</span>BlobRequest</a>
</li>




<li>
<a href="#retriever.v2.Retriever"><span class="badge">S</span>Retriever</a>
</li>

</ul>
</li>

<li><a href="#scalar-value-types">Scalar Value Types</a></li>
</ul>
</div>
Expand Down Expand Up @@ -3955,6 +3978,103 @@ <h3 id="retriever.Retriever">Retriever</h3>




<div class="file-heading">
<h2 id="retriever/v2/retriever.proto">retriever/v2/retriever.proto</h2><a href="#title">Top</a>
</div>
<p></p>


<h3 id="retriever.v2.BlobReply">BlobReply</h3>
<p></p>


<table class="field-table">
<thead>
<tr><td>Field</td><td>Type</td><td>Label</td><td>Description</td></tr>
</thead>
<tbody>

<tr>
<td>data</td>
<td><a href="#bytes">bytes</a></td>
<td></td>
<td><p>The blob retrieved and reconstructed from the EigenDA Nodes per BlobRequest. </p></td>
</tr>

</tbody>
</table>





<h3 id="retriever.v2.BlobRequest">BlobRequest</h3>
<p></p>


<table class="field-table">
<thead>
<tr><td>Field</td><td>Type</td><td>Label</td><td>Description</td></tr>
</thead>
<tbody>

<tr>
<td>blob_header</td>
<td><a href="#common.v2.BlobHeader">common.v2.BlobHeader</a></td>
<td></td>
<td><p>header of the blob to be retrieved </p></td>
</tr>

<tr>
<td>reference_block_number</td>
<td><a href="#uint32">uint32</a></td>
<td></td>
<td><p>The Ethereum block number at which the batch for this blob was constructed. </p></td>
</tr>

<tr>
<td>quorum_id</td>
<td><a href="#uint32">uint32</a></td>
<td></td>
<td><p>Which quorum of the blob this is requesting for (note a blob can participate in
multiple quorums). </p></td>
</tr>

</tbody>
</table>











<h3 id="retriever.v2.Retriever">Retriever</h3>
<p>The Retriever is a service for retrieving chunks corresponding to a blob from</p><p>the EigenDA operator nodes and reconstructing the original blob from the chunks.</p><p>This is a client-side library that the users are supposed to operationalize.</p><p>Note: Users generally have two ways to retrieve a blob from EigenDA V2:</p><p>1) Retrieve from the relay that the blob is assigned to: the API</p><p>is Relay.GetBlob() as defined in api/proto/relay/relay.proto</p><p>2) Retrieve directly from the EigenDA Nodes, which is supported by this Retriever.</p><p>The Relay.GetBlob() (the 1st approach) is generally faster and cheaper as the</p><p>relay manages the blobs that it has processed, whereas the Retriever.RetrieveBlob()</p><p>(the 2nd approach here) removes the need to trust the relay, with the downside of</p><p>worse cost and performance.</p>
<table class="enum-table">
<thead>
<tr><td>Method Name</td><td>Request Type</td><td>Response Type</td><td>Description</td></tr>
</thead>
<tbody>

<tr>
<td>RetrieveBlob</td>
<td><a href="#retriever.v2.BlobRequest">BlobRequest</a></td>
<td><a href="#retriever.v2.BlobReply">BlobReply</a></td>
<td><p>This fans out request to EigenDA Nodes to retrieve the chunks and returns the
reconstructed original blob in response.</p></td>
</tr>

</tbody>
</table>




<h2 id="scalar-value-types">Scalar Value Types</h2>
<table class="scalar-value-types-table">
Expand Down
76 changes: 76 additions & 0 deletions api/docs/eigenda-protos.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,12 @@

- [Retriever](#retriever-Retriever)

- [retriever/v2/retriever.proto](#retriever_v2_retriever-proto)
- [BlobReply](#retriever-v2-BlobReply)
- [BlobRequest](#retriever-v2-BlobRequest)

- [Retriever](#retriever-v2-Retriever)

- [Scalar Value Types](#scalar-value-types)


Expand Down Expand Up @@ -1737,6 +1743,76 @@ worse cost and performance.



<a name="retriever_v2_retriever-proto"></a>
<p align="right"><a href="#top">Top</a></p>

## retriever/v2/retriever.proto



<a name="retriever-v2-BlobReply"></a>

### BlobReply



| Field | Type | Label | Description |
| ----- | ---- | ----- | ----------- |
| data | [bytes](#bytes) | | The blob retrieved and reconstructed from the EigenDA Nodes per BlobRequest. |






<a name="retriever-v2-BlobRequest"></a>

### BlobRequest



| Field | Type | Label | Description |
| ----- | ---- | ----- | ----------- |
| blob_header | [common.v2.BlobHeader](#common-v2-BlobHeader) | | header of the blob to be retrieved |
| reference_block_number | [uint32](#uint32) | | The Ethereum block number at which the batch for this blob was constructed. |
| quorum_id | [uint32](#uint32) | | Which quorum of the blob this is requesting for (note a blob can participate in multiple quorums). |












<a name="retriever-v2-Retriever"></a>

### Retriever
The Retriever is a service for retrieving chunks corresponding to a blob from
the EigenDA operator nodes and reconstructing the original blob from the chunks.
This is a client-side library that the users are supposed to operationalize.

Note: Users generally have two ways to retrieve a blob from EigenDA V2:
1) Retrieve from the relay that the blob is assigned to: the API
is Relay.GetBlob() as defined in api/proto/relay/relay.proto
2) Retrieve directly from the EigenDA Nodes, which is supported by this Retriever.

The Relay.GetBlob() (the 1st approach) is generally faster and cheaper as the
relay manages the blobs that it has processed, whereas the Retriever.RetrieveBlob()
(the 2nd approach here) removes the need to trust the relay, with the downside of
worse cost and performance.

| Method Name | Request Type | Response Type | Description |
| ----------- | ------------ | ------------- | ------------|
| RetrieveBlob | [BlobRequest](#retriever-v2-BlobRequest) | [BlobReply](#retriever-v2-BlobReply) | This fans out request to EigenDA Nodes to retrieve the chunks and returns the reconstructed original blob in response. |





## Scalar Value Types

| .proto Type | Notes | C++ | Java | Python | Go | C# | PHP | Ruby |
Expand Down
38 changes: 14 additions & 24 deletions api/docs/retriever.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,22 +175,22 @@ <h2>Table of Contents</h2>


<li>
<a href="#retriever%2fretriever.proto">retriever/retriever.proto</a>
<a href="#retriever%2fv2%2fretriever.proto">retriever/v2/retriever.proto</a>
<ul>

<li>
<a href="#retriever.BlobReply"><span class="badge">M</span>BlobReply</a>
<a href="#retriever.v2.BlobReply"><span class="badge">M</span>BlobReply</a>
</li>

<li>
<a href="#retriever.BlobRequest"><span class="badge">M</span>BlobRequest</a>
<a href="#retriever.v2.BlobRequest"><span class="badge">M</span>BlobRequest</a>
</li>




<li>
<a href="#retriever.Retriever"><span class="badge">S</span>Retriever</a>
<a href="#retriever.v2.Retriever"><span class="badge">S</span>Retriever</a>
</li>

</ul>
Expand All @@ -203,12 +203,12 @@ <h2>Table of Contents</h2>


<div class="file-heading">
<h2 id="retriever/retriever.proto">retriever/retriever.proto</h2><a href="#title">Top</a>
<h2 id="retriever/v2/retriever.proto">retriever/v2/retriever.proto</h2><a href="#title">Top</a>
</div>
<p></p>


<h3 id="retriever.BlobReply">BlobReply</h3>
<h3 id="retriever.v2.BlobReply">BlobReply</h3>
<p></p>


Expand All @@ -232,7 +232,7 @@ <h3 id="retriever.BlobReply">BlobReply</h3>



<h3 id="retriever.BlobRequest">BlobRequest</h3>
<h3 id="retriever.v2.BlobRequest">BlobRequest</h3>
<p></p>


Expand All @@ -243,20 +243,10 @@ <h3 id="retriever.BlobRequest">BlobRequest</h3>
<tbody>

<tr>
<td>batch_header_hash</td>
<td><a href="#bytes">bytes</a></td>
<td></td>
<td><p>The hash of the ReducedBatchHeader defined onchain, see:
https://github.com/Layr-Labs/eigenda/blob/master/contracts/src/interfaces/IEigenDAServiceManager.sol#L43
This identifies the batch that this blob belongs to. </p></td>
</tr>

<tr>
<td>blob_index</td>
<td><a href="#uint32">uint32</a></td>
<td>blob_header</td>
<td><a href="#common.v2.BlobHeader">common.v2.BlobHeader</a></td>
<td></td>
<td><p>Which blob in the batch this is requesting for (note: a batch is logically an
ordered list of blobs). </p></td>
<td><p>header of the blob to be retrieved </p></td>
</tr>

<tr>
Expand Down Expand Up @@ -287,8 +277,8 @@ <h3 id="retriever.BlobRequest">BlobRequest</h3>



<h3 id="retriever.Retriever">Retriever</h3>
<p>The Retriever is a service for retrieving chunks corresponding to a blob from</p><p>the EigenDA operator nodes and reconstructing the original blob from the chunks.</p><p>This is a client-side library that the users are supposed to operationalize.</p><p>Note: Users generally have two ways to retrieve a blob from EigenDA:</p><p>1) Retrieve from the Disperser that the user initially used for dispersal: the API</p><p>is Disperser.RetrieveBlob() as defined in api/proto/disperser/disperser.proto</p><p>2) Retrieve directly from the EigenDA Nodes, which is supported by this Retriever.</p><p>The Disperser.RetrieveBlob() (the 1st approach) is generally faster and cheaper as the</p><p>Disperser manages the blobs that it has processed, whereas the Retriever.RetrieveBlob()</p><p>(the 2nd approach here) removes the need to trust the Disperser, with the downside of</p><p>worse cost and performance.</p>
<h3 id="retriever.v2.Retriever">Retriever</h3>
<p>The Retriever is a service for retrieving chunks corresponding to a blob from</p><p>the EigenDA operator nodes and reconstructing the original blob from the chunks.</p><p>This is a client-side library that the users are supposed to operationalize.</p><p>Note: Users generally have two ways to retrieve a blob from EigenDA V2:</p><p>1) Retrieve from the relay that the blob is assigned to: the API</p><p>is Relay.GetBlob() as defined in api/proto/relay/relay.proto</p><p>2) Retrieve directly from the EigenDA Nodes, which is supported by this Retriever.</p><p>The Relay.GetBlob() (the 1st approach) is generally faster and cheaper as the</p><p>relay manages the blobs that it has processed, whereas the Retriever.RetrieveBlob()</p><p>(the 2nd approach here) removes the need to trust the relay, with the downside of</p><p>worse cost and performance.</p>
<table class="enum-table">
<thead>
<tr><td>Method Name</td><td>Request Type</td><td>Response Type</td><td>Description</td></tr>
Expand All @@ -297,8 +287,8 @@ <h3 id="retriever.Retriever">Retriever</h3>

<tr>
<td>RetrieveBlob</td>
<td><a href="#retriever.BlobRequest">BlobRequest</a></td>
<td><a href="#retriever.BlobReply">BlobReply</a></td>
<td><a href="#retriever.v2.BlobRequest">BlobRequest</a></td>
<td><a href="#retriever.v2.BlobReply">BlobReply</a></td>
<td><p>This fans out request to EigenDA Nodes to retrieve the chunks and returns the
reconstructed original blob in response.</p></td>
</tr>
Expand Down
Loading
Loading