Use ipnisync with libp2phttp

[gammazero]

Overview

IPNI has a new capability to perform advertisement sync using HTTP over libp2p. This replaces the current data-transfer/graphsync based mechanism, and does not require any change in configuration.

For boost providers serving advertisements using plain HTTP, their current configurations will continue serving plain HTTP as well as HTTP over libp2p. Serving advertisements over libp2p can be disabled if needed.

Technical design

New capabilities within the ipni publisher, contained in the go-libipni/dagsync and index-provider/engine packages, serve advertisements using the boost libp2p stream host. A new protocol ID for HTTP is registered and libp2p stream host directs requests for this protocol to use the ipni publisher as a handler for incoming requests. The requests are handled as HTTP requests and the ipni publisher handles these requests the same way that it handles HTTP requests delivered by a plain HTTP server.

This allows an existing boost provider to serve the new http-based ipni protocol over libp2p without making any changes to its existing configuration, which previously used data-transfer/graphsync to publish advertisements. Since communication is done using the existing libp2p host, there is no need to make any infrastructure changes such as exposing HTTP endpoints or configuring a proxy. The same addresses that were sent in announcement messages to announce new advertisements will continue to work with the new libp2phttp ipni protocol.

UI changes required

No UI changes are needed, unless necessary to give access to new configuration options. The new configuration options are documented in the documentation update.

Implementation

The only new implementation is a dependency update and a modification to fix an existing unit test.

Testing plan

The current tests are run using the default configuration, which serves advertisements using HTTP over libp2p.

Remaining Tasks

• Update to release versions of dependencies
• Test in production

[LexLuthr]

@gammazero A few questions about dependencies:

1. Do we need to upgrade the Go version? Lotus latest is still dependent on v1.19 and I would like to Boost on the same version as Lotus.
2. Can we update Lotus dependency to point to v1.23.4-rc1 instead of master? I am not sure how reliable is Lotus master dependency.
3. Any specific reason to bump Ubuntu to 22? I am asking as there was a thread in Slack about things not working smoothly on 22. I can follow up with the TSE team on that side to confirm if there are any known issues.
4. We will need to upgrade the FFI based on point 2.

[gammazero]

@LexLuthr

1. Various dependencies such as libp2p require that the go version is upgraded, at least to v1.20. I do not think it is reasonable to ask that all these dependencies downgrade their go version, so it will be necessary for boost to upgrade.
2. Yes, [email protected] appears to be compatible. I will make that change to this PR.
3. It needed to at least be upgraded to a more current version of Ubuntu 20 to use the current tagging in CircleCI. CircleCI has made Ubuntu 22 their "standard" linux image so that seemed like the best choice.
4. I have upgraded the FFI in this PR (see submodule link) as was needed to work with the newer lotus.

[LexLuthr]

@gammazero Let me know once changes are done to downgrade Lotus dependency. After that, we can test in filcollins.

[gammazero]

@LexLuthr Is [email protected] the correct version? See commit 28644a0 above my previous comment.

[LexLuthr]

@gammazero Apologies! I completely missed that commit. I will start the deployment and testing on filcollins.

[LexLuthr]

I am merging this PR in a temporary branch. The reason for this approach:

1. Allows anyone to push commits to this change
2. Would allow me to test with multiple version of Lotus in parallel