You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* Add detailed signature procedure for PUT /v1/providers
* Add error codes
* Remove pagination and limit field
* Many other small changes based on feedback
Copy file name to clipboardExpand all lines: IPIP/0000-delegated-routing-http-api.md
+29-13
Original file line number
Diff line number
Diff line change
@@ -20,60 +20,76 @@ but it has resulted in a very unidiomatic HTTP API which is difficult to impleme
20
20
The cost of a proper redesign, implementation, and maintenance of Reframe and its implementation is too high relative to the urgency of having a delegated routing HTTP API.
21
21
22
22
Note that this does not supplant nor deprecate Reframe. Ideally in the future, Reframe and its implementation would receive the resources needed to map the IDL to idiomatic HTTP,
23
-
and this spec could then be rewritten in the IDL, maintaining backwards compatibility.
23
+
and implementations of this spec could then be rewritten in the IDL, maintaining backwards compatibility.
24
24
25
25
## Detailed design
26
26
27
-
See the [API design](../routing/DELEGATED_ROUTING_HTTP.md) included with this IPIP.
27
+
See the [Delegated Routing HTTP API design](../routing/DELEGATED_ROUTING_HTTP.md) included with this IPIP.
28
28
29
29
## Design rationale
30
30
To understand the design rationale, it is important to consider the concrete Reframe limitations that we know about:
31
31
32
-
- Reframe methods are encoded inside messages
33
-
- This prevents URL-based pattern matching on methods
32
+
- Reframe [method types](../reframe/REFRAME_KNOWN_METHODS.md) are encoded inside messages
33
+
- This prevents URL-based pattern matching on methods, which makes it hard and expensive to do basic HTTP scaling and optimizations:
34
34
- Configuring different caching strategies for different methods
35
35
- Configuring reverse proxies on a per-method basis
36
36
- Routing methods to specific backends
37
37
- Method-specific reverse proxy config such as timeouts
38
38
- Developer UX is poor as a result, e.g. for CDN caching you must encode the entire request message and pass it as a query parameter
39
39
- This was initially done by URL-escaping the raw bytes
40
-
- Not possible to consume correctly using standard JavaScript
40
+
- Not possible to consume correctly using standard JavaScript (see [edelweiss#61](https://github.com/ipld/edelweiss/issues/61))
41
41
- Shipped in Kubo 0.16
42
42
- Packing a CID into a struct, encoding it with DAG-CBOR, multibase-encoding that, percent-encoding that, and then passing it in a URL, rather than merely passing the CID in the URL, is needlessly complex from a user's perspective
43
43
- Added complexity of "Cacheable" methods supporting both POSTs and GETs
44
-
- The required streaming support and message groups add a lot of implementation complexity but isn’t very useful
44
+
- The required streaming support and message groups add a lot of implementation complexity, but streaming does not work for cachable methods sent over HTTP
45
45
- Ex for FindProviders, the response is buffered anyway for ETag calculation
46
46
- There are no limits on response sizes nor ways to impose limits and paginate
47
47
- This is useful for routers that have highly variable resolution time, to send results as soon as possible, but this is not a use case we are focusing on right now and we can add it later
48
48
- The Identify method is not implemented because it is not currently useful
49
+
- This is because Reframe's ambition is to be generic catch-all bag of methods across protocols, while delegated routing use case only requires a subset of its methods.
49
50
- Client and server implementations are difficult to write correctly, because of the non-standard wire formats and conventions
50
-
- The Go implementation is [complex](https://github.com/ipfs/go-delegated-routing/blob/main/gen/proto/proto_edelweiss.go) and [brittle](https://github.com/ipfs/go-delegated-routing/blame/main/client/provide.go#L51), and is currently maintained by IPFS Stewards who are already over-committed with other priorities
51
+
- Example: [bug reported by implementer](https://github.com/ipld/edelweiss/issues/62), and [another one](https://github.com/ipld/edelweiss/issues/61)
52
+
- The Go implementation is [complex](https://github.com/ipfs/go-delegated-routing/blob/main/gen/proto/proto_edelweiss.go) and [brittle](https://github.com/ipfs/go-delegated-routing/blame/main/client/provide.go#L51-L100), and is currently maintained by IPFS Stewards who are already over-committed with other priorities
51
53
- Only the HTTP transport has been designed and implemented, so it's unclear if the existing design will work for other transports, and what their use cases and requirements are
54
+
- This means Reframe can't be trusted to be transport-agnostic until there is at least second transport implemented (e.g. as a reframe-over-libp2p protocol).
52
55
53
56
So this API proposal makes the following changes:
54
57
55
-
- The API is defined in HTTP directly
56
-
- "Methods" and cache-relevant parameters are pushed into the URL path
57
-
- Streaming support is removed, and optional pagination is added, which limits the response size and provides a scalable mechanism for iterating over arbitrarily-large collections
58
+
- The Delegated Routing API is defined using HTTP semantics, and can be implemented without introducing Reframe concepts
59
+
- "Method names" and cache-relevant parameters are pushed into the URL path
60
+
- Streaming support is removed, and default response size limits are added along with an optional `limit` parameter for clients to specify response sizes
58
61
- We might add streaming support w/ chunked-encoded responses in the future, but it's currently not an important feature for the use cases that an HTTP API will be used for
62
+
- Pagination could be added to this in the future, if needed
59
63
- Bodies are encoded using standard JSON or CBOR, instead of using IPLD codecs
64
+
- JSON uses human-friendly string encodings of common data types
65
+
- CIDs are encoded as CIDv1 strings with a multibase prefix (e.g. base32), for consistency with CLIs, browsers, and [gateway URLs](https://docs.ipfs.io/how-to/address-ipfs-on-web/)
66
+
- Multiaddrs use the [human-readable format](https://github.com/multiformats/multiaddr#specification) that is used in existing tools and Kubo CLI commands such as `ipfs id` or `ipfs swarm peers`
67
+
- Byte array values, such as signatures, are multibase-encoded strings (with an `m` prefix indicating Base64)
60
68
- The "Identify" method and "message groups" are removed
61
69
62
70
### User benefit
63
71
64
72
The cost of building and operating content routing services will be much lower, as developers will be able to reuse existing industry-standard tooling.
73
+
They no longer need to learn Reframe-specific concepts to consume or expose the API.
65
74
This will result in more content routing providers, each providing a better experience for users, driving down content routing latency across the IPFS netowrk
66
75
and increasing data availability.
67
76
68
77
### Compatibility
69
78
70
79
#### Backwards Compatibility
71
80
IPFS Stewards will implement this API in [go-delegated-routing](https://github.com/ipfs/go-delegated-routing), using breaking changes in a new minor version.
72
-
Because the existing Reframe spec can't be safely used in JavaScript, the experimental support for Reframe in Kubo will be removed in the next release,
73
-
and delegated routing will subsequently use this HTTP API. We may decide to re-add Reframe support in the future once these issues have been resolved.
81
+
Because the existing Reframe spec can't be safely used in JavaScript and we won't be investing time and resources into changing the wire format implemented in edelweiss to fix it,
82
+
the experimental support for Reframe in Kubo will be removed in the next release and delegated routing will subsequently use this HTTP API.
83
+
We may decide to re-add Reframe support in the future once these issues have been resolved.
74
84
75
85
#### Forwards Compatibility
76
-
Standard HTTP mechanisms for forward compatibility are used--the API is versioned using a version number in the path. The `Accept` and `Content-Type` headers are used for content type negotiation. new methods will result in new paths, and parameters can be added using either new query parameters or new fields in the request/response body. Certain parts of bodies are labeled as "opaque bytes", which are passed through by the implementation, with no schema enforcement.
86
+
Standard HTTP mechanisms for forward compatibility are used:
87
+
- The API is versioned using a version number in the path
88
+
- The `Accept` and `Content-Type` headers are used for content type negotiation
89
+
- New methods will result in new paths
90
+
- Parameters can be added using either new query parameters or new fields in the request/response body.
91
+
92
+
Certain parts of bodies are labeled as "{ ... }", which are opaque JSON values passed through by the implementation, with no schema enforcement.
Copy file name to clipboardExpand all lines: routing/DELEGATED_ROUTING_HTTP.md
+60-35
Original file line number
Diff line number
Diff line change
@@ -24,78 +24,103 @@
24
24
-[Implementations](#implementations)
25
25
26
26
# API Specification
27
-
By default, the Delegated Routing HTTP API uses the `application/json` content type. Clients and servers may optionally negotiate other content types such as `application/cbor`, `application/vnd.ipfs.rpc+dag-json`, etc. using the standard `Accept` and `Content-Type` headers.
27
+
The Delegated Routing HTTP API uses the `application/json` content type by default. Clients and servers *should* support `application/cbor`, which can be negotiated using the standard `Accept` and `Content-Type` headers.
28
28
29
+
## Common Data Types:
30
+
31
+
- CIDs are always encoded using a [multibase](https://github.com/multiformats/multibase)-encoded [CIDv1](https://github.com/multiformats/cid#cidv1).
32
+
- Multiaddrs are encoded according to the [human-readable multiaddr specification](https://github.com/multiformats/multiaddr#specification)
33
+
- Peer IDs are encoded according [PeerID string representation specification](https://github.com/libp2p/specs/blob/master/peer-ids/peer-ids.md#string-representation)
34
+
- Multibase bytes are encoded according to [the Multibase spec](https://github.com/multiformats/multibase), and *should* use Base64.
- `transfer` only return providers who support the passed transfer protocols, expressed as a comma-separated list of multicodec IDs such as `2304,2320`,
54
-
- `transport` only return providers whose published multiaddrs explicitly support the passed transport protocols, such as `/quic` or `/tls/ws`.
55
-
- `GET /v1/providers/hash/{multihash}`
56
-
- This is the same as `GET /v1/providers/{CID}`, but takes a hashed CID encoded as a multihash
60
+
- `transfer` only return providers who support the passed transfer protocols, expressed as a comma-separated list of [multicodec codes](https://github.com/multiformats/multicodec/blob/master/table.csv) in decimal form such as `2304,2320`
61
+
- `transport` only return providers whose published multiaddrs explicitly support the passed transport protocols, such as `460,478` (`/quic` and `/tls/ws`)
62
+
- Servers should treat the multicodec codes used in the `transfer` and `transport` parameters as opaque, and not validate them, for forwards compatibility
63
+
- `GET /v1/providers/hashed/{multihash}`
64
+
- This is the same as `GET /v1/providers/{CID}`, but takes a hashed CID encoded as a [multihash](https://github.com/multiformats/multihash/)
57
65
- `GET /v1/ipns/{ID}`
58
66
- Reframe equivalent: GetIPNS
67
+
- `ID`: multibase-encoded bytes
59
68
- Response
60
69
- record bytes
61
-
- `POST /v1/ipns/{ID}`
70
+
- `POST /v1/ipns`
62
71
- Reframe equivalent: PutIPNS
63
72
- Body
64
-
- record bytes
65
-
- No need for idempotency
66
-
- `PUT /v1/providers/{CID}`
73
+
```json
74
+
{
75
+
"Records": [
76
+
{
77
+
"ID": "multibase bytes",
78
+
"Record": "multibase bytes"
79
+
}
80
+
]
81
+
}
82
+
```
83
+
- Not idempotent (this doesn't really make sense for IPNS)
84
+
- Default limit of 100 records per request
85
+
- `PUT /v1/providers`
67
86
- Reframe equivalent: Provide
68
87
- Body
69
88
89
+
```json
90
+
{
91
+
"Signature": "multibase bytes",
92
+
"Payload": "multibase bytes"
93
+
}
94
+
```
95
+
- `Payload` is a multibase-encoded JSON object with the following schema:
- `Signature` is a multibase-encoded signature of the bytes of the `Payload` field, signed using the private key of the Peer ID specified in the `Payload`. See the [Peer ID](https://github.com/libp2p/specs/blob/master/peer-ids/peer-ids.md#keys) specification for the encoding of Peer IDs. Servers must verify the payload by deserializing it, extracting the public key from the Peer ID, and verifying the signature against the `Payload` bytes. If the verification fails, the server must return a 403 status code.
91
115
- Idempotent
116
+
- Default limit of 100 keys per request
92
117
- `GET /v1/ping`
93
-
- This is absent from Reframe but is necessary for supporting e.g. the accelerated DHT client which can take many minutes to bootstrap
118
+
- This is absent from Reframe but is necessary for supporting e.g. the accelerated DHT client which can take many minutes to bootstrap, and light clients who want to probe multiple HTTP endpoints and use the fastest one
94
119
- Returns 200 once the server is ready to accept requests
95
120
- An alternate approach is w/ an orchestration dance in the server by not listening on the socket until the dependencies are ready, but this makes the “dance” easier to implement
96
-
- Pagination
121
+
- Limits
97
122
- Responses with collections of results must have a default limit on the number of results that will be returned in a single response
98
-
- Servers may optionally implement pagination by responding with an opaque page token which, when provided as a subsequent query parameter, will fetch the next page of results.
99
-
- Clients may continue paginating until no `NextPageToken` is returned.
100
-
- Clients making calls that return collections may limit the number of per-page results returned with the `limit` query parameter, i.e. `GET /v1/providers/{CID}?limit=10`
101
-
- Additional filtering/sorting operations may be defined on a per-path basis, as needed
123
+
- Pagination and/or dynamic limit configuration may be added to this spec in the future, once there is a concrete requirement
124
+
- Error Codes
125
+
- A 404 must be returned if a resource was not found
126
+
- A 501 must be returned if a method is not supported
0 commit comments