Skip to content

feat: HTTP retrieval enabled by default #10836

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lidel merged 2 commits into from
Jun 16, 2025
Merged

feat: HTTP retrieval enabled by default #10836

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8dec810
feat: HTTP retrieval enabled by default
lidel Jun 12, 2025
48d24c7
test: expect httpnet metrics by default
lidel Jun 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion config/http_retrieval.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ type HTTPRetrieval struct {
}

const (
DefaultHTTPRetrievalEnabled = false // opt-in for now, until we figure out https://github.com/ipfs/specs/issues/496
DefaultHTTPRetrievalEnabled = true
DefaultHTTPRetrievalNumWorkers = 16
DefaultHTTPRetrievalTLSInsecureSkipVerify = false // only for testing with self-signed HTTPS certs
DefaultHTTPRetrievalMaxBlockSize = "2MiB" // matching bitswap: https://specs.ipfs.tech/bitswap-protocol/#block-sizes
Expand Down
7 changes: 7 additions & 0 deletions docs/changelogs/v0.36.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ This release was brought to you by the [Shipyard](http://ipshipyard.com/) team.

- [Overview](#overview)
- [🔦 Highlights](#-highlights)
- [HTTP Retrieval client enabled by default](#http-retrieval-client-enabled-by-default)
- [Update go-log to v2](#update-go-log-to-v2)
- [AutoNATv2 Client](#autonatv2-client)
- [Smarter AutoTLS registration](#smarter-autotls-registration)
Expand All @@ -24,6 +25,12 @@ This release was brought to you by the [Shipyard](http://ipshipyard.com/) team.

### 🔦 Highlights

#### HTTP Retrieval client enabled by default

This release promotes the HTTP Retrieval client from an experimental feature to a standard feature that is enabled by default. When possible, Kubo will be retrieving blocks over plain HTTPS (HTTP/2) without any extra user configuration.

See [`HTTPRetrieval`](https://github.com/ipfs/kubo/blob/master/docs/config.md#httpretrieval) for more details.

#### Update go-log to v2

go-log v2 has been out for quite a while now and it is time to deprecate v1.
Expand Down
19 changes: 10 additions & 9 deletions docs/config.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2686,25 +2686,26 @@ Type: `object`

### `HTTPRetrieval.Enabled`

> [!CAUTION]
> This feature is **EXPERIMENTAL** and may change in future release. Enable with caution, and provide feedback via GitHub issues.

Controls whether HTTP-based block retrieval is enabled.

When enabled, Kubo will be able to act on `/tls/http` (HTTP/2) providers ([Trustless HTTP Gateways](https://specs.ipfs.tech/http-gateways/trustless-gateway/)) returned by the [`Routing.DelegatedRouters`](#routingdelegatedrouters)
When enabled, Kubo will act on `/tls/http` (HTTP/2) providers ([Trustless HTTP Gateways](https://specs.ipfs.tech/http-gateways/trustless-gateway/)) returned by the [`Routing.DelegatedRouters`](#routingdelegatedrouters)
to perform pure HTTP [block retrievals](https://specs.ipfs.tech/http-gateways/trustless-gateway/#block-responses-application-vnd-ipld-raw)
in addition to [Bitswap over Libp2p](#bitswap).
(`/ipfs/cid?format=raw`, `Accept: application/vnd.ipld.raw`)
alongside [Bitswap over Libp2p](#bitswap).

HTTP requests for `application/vnd.ipld.raw` will be issued instead of Bitswap if a peer has a `/tls/http` multiaddr
HTTP requests for `application/vnd.ipld.raw` will be made instead of Bitswap when a peer has a `/tls/http` multiaddr
and the HTTPS server returns HTTP 200 for the [probe path](https://specs.ipfs.tech/http-gateways/trustless-gateway/#dedicated-probe-paths).

> [!IMPORTANT]
> - Requires TLS and HTTP/2.
> This feature is relatively new. Please report any issues via [Github](https://github.com/ipfs/kubo/issues/new).
>
> Important notes:
> - TLS and HTTP/2 are required. For privacy reasons, and to maintain feature-parity with browsers, unencrypted `http://` providers are ignored and not used.
> - This feature works in the same way as Bitswap: connected HTTP-peers receive optimistic block requests even for content that they are not announcing.
> - HTTP client does not follow redirects. Providers should keep announcements up to date.
> - For performance reasons, and to avoid loops, the HTTP client does not follow redirects. Providers should keep announcements up to date.
> - IPFS ecosystem is working towards [supporting HTTP providers on Amino DHT](https://github.com/ipfs/specs/issues/496). Currently, HTTP providers are mostly limited to results from [`Routing.DelegatedRouters`](#routingdelegatedrouters) endpoints and requires `Routing.Type=auto|autoclient`.

Default: `false`
Default: `true`

Type: `flag`

Expand Down
16 changes: 16 additions & 0 deletions test/sharness/t0119-prometheus-data/prometheus_metrics
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,22 @@ exchange_bitswap_wantlists_seconds_bucket
exchange_bitswap_wantlists_seconds_count
exchange_bitswap_wantlists_seconds_sum
exchange_bitswap_wantlists_total
exchange_httpnet_request_duration_seconds_bucket
exchange_httpnet_request_duration_seconds_count
exchange_httpnet_request_duration_seconds_sum
exchange_httpnet_request_sent_bytes
exchange_httpnet_requests_body_failure
exchange_httpnet_requests_failure
exchange_httpnet_requests_in_flight
exchange_httpnet_requests_total
exchange_httpnet_response_bytes_bucket
exchange_httpnet_response_bytes_count
exchange_httpnet_response_bytes_sum
exchange_httpnet_wantlists_items_total
exchange_httpnet_wantlists_seconds_bucket
exchange_httpnet_wantlists_seconds_count
exchange_httpnet_wantlists_seconds_sum
exchange_httpnet_wantlists_total
go_gc_duration_seconds
go_gc_duration_seconds_count
go_gc_duration_seconds_sum
Expand Down
Loading