Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IPIP-417: Delegated Peer Routing HTTP API #417

Merged
merged 10 commits into from
Sep 25, 2023
Merged

IPIP-417: Delegated Peer Routing HTTP API #417

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
1467e48
docs: start IPIP 0417
hacdias May 29, 2023
e1b335f
add paragraph about additional fields
hacdias Aug 1, 2023
b77728b
remove known transfer protocols section
hacdias Aug 1, 2023
b1ed79e
cleanup provider records story
hacdias Aug 1, 2023
828378d
add notes to the compatibility section
hacdias Aug 1, 2023
2dbdb54
editorial touches
hacdias Aug 1, 2023
4adf8b7
finish sections in IPIP
hacdias Aug 1, 2023
86ecfa0
editorial fixes
hacdias Aug 1, 2023
f3fe9d5
ipip-0417: final editorials, list legacy schemas
lidel Aug 31, 2023
95999ef
chore: bump date
lidel Aug 31, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions src/ipips/ipip-0417.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
---
title: "IPIP-0417: Delegated Peer Routing HTTP API"
date: 2023-05-29
ipip: proposal
editors:
- name: Henrique Dias
github: hacdias
url: https://hacdias.com/
affiliation:
name: Protocol Labs
url: https://protocol.ai/
relatedIssues:
- https://github.com/ipfs/specs/pull/410
- https://github.com/ipfs/kubo/pull/9877
order: 417
tags: ['ipips']
---

## Summary

This IPIP specifies `/routing/v1/peers/{peer-id}` HTTP API to offload peer routing onto another server.

## Motivation

The motivation of this IPIP extends the one of :cite[ipip-0337] and :cite[ipip-0379],
which introduced delegated content routing and delegated naming, respectively. Now,
we expand upon those basis to introduce peer routing, reducing the barrier for interaction
across different systems.

## Detailed design

Add `/routing/v1/peers/{peer-id}` to the existing :cite[http-routing-v1] specification,
as well as the new Peer schema, that replaces the existing "known transfer protocols".

## Design rationale

In line with the remaining Routing V1 API, this IPIP introduces a new HTTP GET
endpoint that is used to retrieve peer records for a certain peer.

### User benefit

The user benefit brought by this PR is similar to the one in :cite[ipip-0379],
but instead of offloading the naming process, we offload the peer discovery and
routing.

### Compatibility

The section "Known Transfer Protocols" has been removed and replaced by a "Known Schemas"
section. Before, we used to have protocol specific schemas, such as `bitswap` and `graphsync-filecoinv1`.

The usage of these schemas is no longer encouraged. Instead, clients and
servers SHOULD be updated to use the new, more generic, `peer` schema, which
avoids returning the same peer multiple times, making results more efficient
when a peer supports more than one protocol.

See more in the "Compatibility" section of :cite[ipip-0337].

### Security

See the "Security" section of :cite[ipip-0337].

### Alternatives

See the "Alternatives" section of :cite[ipip-0337].

### Copyright

Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
176 changes: 121 additions & 55 deletions src/routing/http-routing-v1.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,16 +4,31 @@ description: >
Delegated routing is a mechanism for IPFS implementations to use for offloading
content routing and naming to another process/server. This specification describes
an HTTP API for delegated content routing.
date: 2023-03-22
date: 2023-08-31
maturity: reliable
editors:
- name: Gus Eggert
github: guseggert
affiliation:
name: Protocol Labs
url: https://protocol.ai/
- name: Masih H. Derkani
github: masih
affiliation:
name: Protocol Labs
url: https://protocol.ai/
- name: Henrique Dias
url: https://hacdias.com/
github: hacdias
affiliation:
name: Protocol Labs
url: https://protocol.ai/
- name: Marcin Rataj
github: lidel
url: https://lidel.org/
affiliation:
name: Protocol Labs
url: https://protocol.ai/
xref:
- ipns-record
order: 0
Expand All @@ -32,75 +47,89 @@ As such, human-readable encodings of types are preferred. This specification may

- CIDs are always string-encoded using a [multibase]-encoded [CIDv1].
- Multiaddrs are string-encoded according to the [human-readable multiaddr specification][multiaddr].
- Peer IDs are string-encoded according [PeerID string representation specification][peer-id-representation].
- Peer IDs are string-encoded according [PeerID string representation specification][peer-id-representation]: either a Multihash in Base58btc, or a CIDv1 with libp2p-key (`0x72`) codec.
- Multibase bytes are string-encoded according to [the Multibase spec][multibase], and SHOULD use base64.
- Timestamps are Unix millisecond epoch timestamps.

Until required for business logic, servers should treat these types as opaque strings, and should preserve unknown JSON fields.

### Versioning
## Versioning

This API uses a standard version prefix in the path, such as `/v1/...`. If a backwards-incompatible change must be made, then the version number should be increased.

### Provider Records
## Content Routing API

A provider record contains information about a content provider, including the transfer protocol and any protocol-specific information useful for fetching the content from the provider.
### `GET /routing/v1/providers/{cid}`

The information required to write a record to a router (*"write" provider records*) may be different than the information contained when reading provider records (*"read" provider records*).
#### Path Parameters

For example, indexers may require a signature in `bitswap` write records for authentication of the peer contained in the record, but the read records may not include this authentication information.
- `cid` is the [CID](https://github.com/multiformats/cid) to fetch provider records for.

Both read and write provider records have a minimal required schema as follows:
#### Response Status Codes

- `200` (OK): the response body contains 0 or more records.
- `404` (Not Found): must be returned if no matching records are found.
- `422` (Unprocessable Entity): request does not conform to schema or semantic constraints.

#### Response Body

```json
{
"Protocol": "<transfer_protocol_name>",
"Schema": "<transfer_protocol_schema>",
"Providers": [
{
"Schema": "<schema>",
"ID": "bafz...",
"Addrs": ["/ip4/..."],
...
},
...
]
}
```

Where:
The `application/json` responses SHOULD be limited to 100 providers.

- `Protocol` is the multicodec name of the transfer protocol or an opaque string (for experimenting with novel protocols without a multicodec)
- `Schema` denotes the schema to use for encoding/decoding the record
- This is separate from the `Protocol` to allow this HTTP API to evolve independently of the transfer protocol
- Implementations should switch on this when parsing records, not on `Protocol`
- `...` denotes opaque JSON, which may contain information specific to the transfer protocol
The client SHOULD be able to make a request with `Accept: application/x-ndjson` and get a [stream](#streaming) with more results.

Specifications for some transfer protocols are provided in the "Transfer Protocols" section.
Each object in the `Providers` list is a record conforming to a schema, usually the [Peer Schema](#peer-schema).

## Content Providers API
## Peer Routing API

### `GET /routing/v1/providers/{cid}`
### `GET /routing/v1/peers/{peer-id}`

#### Path Parameters

- `cid` is the [CID](https://github.com/multiformats/cid) to fetch provider records for.
- `peer-id` is the [Peer ID](https://github.com/libp2p/specs/blob/master/peer-ids/peer-ids.md) to fetch peer records for,
represented as a CIDv1 encoded with `libp2p-key` codec.

#### Response Status Codes

- `200` (OK): the response body contains 0 or more records.
- `200` (OK): the response body contains the peer record.
- `404` (Not Found): must be returned if no matching records are found.
- `422` (Unprocessable Entity): request does not conform to schema or semantic constraints.

#### Response Body

```json
{
"Providers": [
"Peers": [
{
"Protocol": "<protocol_name>",
"Schema": "<schema>",
"Protocols": ["<protocol-a>", "<protocol-b>", ...],
"ID": "bafz...",
"Addrs": ["/ip4/..."],
...
}
},
...
]
}
```

Response limit: 100 providers
The `application/json` responses SHOULD be limited to 100 peers.

The client SHOULD be able to make a request with `Accept: application/x-ndjson` and get a [stream](#streaming) with more results.

Each object in the `Providers` list is a *read provider record*.
Each object in the `Peers` list is a record conforming to the [Peer Schema](#peer-schema).

## IPNS API

Expand All @@ -123,7 +152,7 @@ Each object in the `Providers` list is a *read provider record*.

#### Response Body

The response body contains a :ref[IPNS Record] serialized using the verifiable [`application/vnd.ipfs.ipns-record`](https://www.iana.org/assignments/media-types/application/vnd.ipfs.ipns-record) protobuf format.
The response body contains a :ref[IPNS Record] serialized using the verifiable [`application/vnd.ipfs.ipns-record`](https://www.iana.org/assignments/media-types/application/vnd.ipfs.ipns-record) protobuf format.

### `PUT /routing/v1/ipns/{name}`

Expand Down Expand Up @@ -163,7 +192,7 @@ with one result per line:

:::note

Streaming is opt-in and backwards-compatibile with clients and servers that do
Streaming is opt-in and backwards-compatible with clients and servers that do
not support streaming:

- Requests without the `Accept: application/x-ndjson` header MUST default to
Expand Down Expand Up @@ -199,59 +228,96 @@ Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, OPTIONS
```

## Known Transfer Protocols
## Known Schemas

This section contains a non-exhaustive list of known transfer protocols (by name) that may be supported by clients and servers.
This section contains a non-exhaustive list of known schemas that MAY be supported by clients and servers.

### Bitswap
### Peer Schema

- Multicodec name: `transport-bitswap`
- Schema: `bitswap`
- Specification: [ipfs/specs/BITSWAP.md](https://github.com/ipfs/specs/blob/main/BITSWAP.md)
The `peer` schema represents an arbitrary peer.

#### Bitswap Read Provider Records
```json
{
"Schema": "peer",
"ID": "bafz...",
"Addrs": ["/ip4/..."],
"Protocols": ["transport-bitswap", ...]
...
}
```

- `ID`: the [Peer ID][peer-id] as Multihash in Base58btc or CIDv1 with libp2p-key codec.
- `Addrs`: an optional list of known [multiaddrs][multiaddr] for this peer.
- If missing or empty, it means the router server is missing that information, and the client should use `ID` to lookup updated peer information.
- `Protocols`: an optional list of protocols known to be supported by this peer.
- If missing or empty, it means the router server is missing that information, and the client should use `ID` and `Addrs` to lookup connect to the peer and use the [libp2p identify protocol](https://github.com/libp2p/specs/tree/master/identify) to learn about supported ones.

:::note

To allow for protocol-specific fields and future-proofing, the parser MUST
allow for unknown fields, and the clients MUST ignore unknown ones.

Below is an example on how one could include `protocol-a` and `protocol-b`
protocols that includes an additional fields `protocol-a` and `protocol-b`.

If the client knows the protocol, they are free to use the extra binary
(base64) or JSON information contained in the additional field. If that is not
the case, the field MUST be ignored.

```json
{
"Protocol": "transport-bitswap",
"Schema": "bitswap",
"ID": "12D3K...",
"Addrs": ["/ip4/..."]
"Schema": "peer",
"ID": "bafz...",
"Addrs": ["/ip4/..."],
"Protocols": ["transport-bitswap", "protocol-a", "protocol-b", ...],
"protocol-a": "[base64-blob]",
"protocol-b": { "foo": "bar" }
}
```

- `ID`: the [Peer ID][peer-id] to contact
- `Addrs`: a list of known multiaddrs for the peer
- This list may be incomplete or incorrect and should only be treated as *hints* to improve performance by avoiding extra peer lookups
:::

### Legacy Schemas

Legacy schemas include `ID` and optional `Addrs` list just like
the [`peer` schema](#peer-schema) does.

These schemas are deprecated and SHOULD be replaced with `peer` over time, but
MAY be returned by some legacy endpoints. In such case, a client MAY parse
them the same way as the `peer` schema.

The server should respect a passed `transport` query parameter by filtering against the `Addrs` list.
#### Bitswap Schema

### Filecoin Graphsync
A legacy schema used by some routers to indicate a peer supports retrieval over
the `/ipfs/bitswap[/*]` libp2p protocol.

```json
{
"Protocol": "transport-bitswap",
"Schema": "bitswap",
"ID": "bafz...",
"Addrs": ["/ip4/..."]
}
```

- Multicodec name: `transport-graphsync-filecoinv1`
- Schema: `graphsync-filecoinv1`
- Specification: [ipfs/go-graphsync/blob/main/docs/architecture.md](https://github.com/ipfs/go-graphsync/blob/main/docs/architecture.md)
#### Graphsync Schema

#### Filecoin Graphsync Read Provider Records
A legacy schema used by some routers to indicate a peer supports retrieval over
the [graphsync](https://github.com/ipfs/go-graphsync/blob/main/docs/architecture.md)
libp2p protocol.

```json
{
"Protocol": "transport-graphsync-filecoinv1",
"Schema": "graphsync-filecoinv1",
"ID": "12D3K...",
"ID": "bafz...",
"Addrs": ["/ip4/..."],
"PieceCID": "<cid>",
"VerifiedDeal": true,
"FastRetrieval": true
}
```

- `ID`: the [Peer ID][peer-id] of the provider
- `Addrs`: a list of known multiaddrs for the provider
- `PieceCID`: the CID of the [piece](https://spec.filecoin.io/systems/filecoin_files/piece/#section-systems.filecoin_files.piece) within which the data is stored
- `VerifiedDeal`: whether the deal corresponding to the data is verified
- `FastRetrieval`: whether the provider claims there is an unsealed copy of the data available for fast retrieval

[multibase]: https://github.com/multiformats/multibase
[CIDv1]: https://github.com/multiformats/cid#cidv1
[multiaddr]: https://github.com/multiformats/multiaddr#specification
Expand Down