Reprovider to reannounce blocks to the network

[vasco-santos]

This issue is a child of Connectivity Magic #1459, and is intended to discuss the implementation of the reprovider in js-ipfs.

The current implementation of reprovider in go-ipfs consists of iterating all blocks in blockstore and provide them every X hours (being X configurable through the config file). Other than that, the reprovider may have different strategies for deciding which blocks it should reprovide ipfs/go-ipfs/docs/config.md#reprovider.

However, go-ipfs team is currently working on improving their reprovider. Consequently, it would be great to have an idea of what is the plan, in order to decide if we should start with this simpler implementation or with the newest one.

Another aspect to take into consideration in the JS implementation is if we should create the concept of Provider. In this case, js-ipfs-bitswap would receive its reference and the Provider could abstract the providing logic from there. I don't know the current logic of js-ipfs-bitswap yet to have strong conclusions in this regard.

cc @michaelavila @Kubuxu @dirkmc @vmx

[michaelavila]

@vasco-santos thanks for reaching out!

However, go-ipfs team is currently working on improving their reprovider. Consequently, it would be great to have an idea of what is the plan, in order to decide if we should start with this simpler implementation or with the newest one.

The immediate next iteration of the reprovider gets simpler:

• We introduce a Tracker that is used by both the provider and the reprovider
• The Provider adds entries into the tracker
• All the Reprovider does is announce every block in the Tracker

Most of our effort is going into reworking the provider itself. Ultimately, the goal is to provide a subset of all the blocks we are needing to provide now, while retaining overall discoverability of blocks. This will necessitate things like backlinks or some kind of meta to determine which blocks we can call FindProviders on.

One additional small thing we've done, and intend to retain, is prioritizing announcing root blocks first.

Another aspect to take into consideration in the JS implementation is if we should create the concept of Provider. In this case, js-ipfs-bitswap would receive its reference and the Provider could abstract the providing logic from there. I don't know the current logic of js-ipfs-bitswap yet to have strong conclusions in this regard.

I am currently in the process of ripping all Provide specific behavior out of go-bitswap. Instead we are making a new Provider subsystem in the go-ipfs repository. This is because we want way more control over how providing happens and how it can be configured. I think this is in line with what you were saying "... we should create the concept of Provider."

I'm happy to provide (tsk tsk) whatever info I can.

[dirkmc]

The PR for a Garbage Collection implementation in js-ipfs has some refactoring that exposes a method that can be used to retrieve blocks that are used internally by the pinner. We will presumably want to filter out these blocks when reproviding.

[michaelavila]

I just wanted to point out some recent PRs in go-ipfs that give some indication of where we are going:

This PR introduces a rudimentary provider and the corresponding calls for doing the provide during add and pin add:

ipfs/kubo#6068

This PR moves all the existing provider functionality behind one system so that we can plug in our new, more robust system as needed (and also removes providing from bitswap):

ipfs/kubo#6292

Full disclosure, the new more robust system is currently under development.

[vasco-santos]

Thanks for your feedback @michaelavila and @dirkmc ! I will hopefully start working on this tomorrow and I will ping you if needed.

[michaelavila]

@vasco-santos happy to help! I'll be at the team week, so if you're there we can meet up and chat about this stuff in person. Just let me know.

[SgtPooki]

js-ipfs is being deprecated in favor of Helia. You can learn more about this deprecation and read the migration guide.

Please feel to reopen with any comments by 2023-06-02. We will do a final pass on reopened issues afterwards (see #4336).