Application/ipfs utilities 2

[dominikdem]

Project Abstract

This is a follow-up project for Substrate IPFS Utilities MS1 and MS2. During the previous project some issues occurred that are related to already existing IPFS APIs and patterns used. Goal of this project is to solve those issues and make the IPFS integration ready for real use cases.

Grant level

• Level 1: Up to $10,000, 2 approvals
• Level 2: Up to $30,000, 3 approvals
• Level 3: Unlimited, 5 approvals (for >$100k: Web3 Foundation Council approval)

Application Checklist

• The application template has been copied and aptly renamed (project_name.md).
• I have read the application guidelines.
• Payment details have been provided (bank details via email or BTC, Ethereum (USDC/DAI) or Polkadot/Kusama (USDT) address in the application).
• The software delivered for this grant will be released under an open-source license specified in the application.
• The initial PR contains only one commit (squash and force-push if needed).
• The grant will only be announced once the first milestone has been accepted (see the announcement guidelines).
• I prefer the discussion of this application to take place in a private Element/Matrix channel. My username is: @_______:matrix.org (change the homeserver if you use a different one)

[CLAassistant]

All committers have signed the CLA.

[dominikdem]

Hey @Noc2, we have developed follow-up ideas for the IPFS integration project. We look forward to your feedback.

[Noc2]

Sorry for the delay here, and thanks for the application. I have a few initial questions/comments:

• Not sure checking the availability of the file is super helpful. Theoretically, it could be deleted immediately after the file is marked as available. Unless you basically implemented something like filecoin, availability won’t be guaranteed. What are your plans in this regard?
• Could you expand a little bit on why the research on InputStreams is needed and what you are planning to do?

[dobschal]

Hey @Noc2, thanks for your reply and the questions.

Not sure checking the availability of the file is super helpful. Theoretically, it could be deleted immediately after the file is marked as available. Unless you basically implemented something like filecoin, availability won’t be guaranteed. What are your plans in this regard?

We designed the IPFS API to be an integrated closed system within Substrate and only offer, extrinsic that adds or returns stored data. Deletion method do exists within the Pallets, but are not attached. With running that embedded IPFS node we can guarantee, that the files added aren't deleted from the IPFS network.

Could you expand a little bit on why the research on InputStreams is needed and what you are planning to do?

A file upload would be beneficial to the system, since there might be many use cases, in which people need to upload data to a server and the result and or request data needs to be stored onchain. For example, if it must to be determined, that a specific user did actually upload the file by her-/himself or that it should be traced, that a file was successfully received by a server. We believe, that this could offer unique opportunities for the Substate system.

Right now it is only possible to use a RPC with JSON content. As a result, RPC APIs on both server and client sides need to be implemented and there are limitations such as amount of accepted bytes. Those can be configured within the Substrate runtime, but still would be way more handy and intuitive using streaming functionality. Using streams of large data and handling those chunk by chunk is much more performant, then handling all the incoming data at once. The rest of the chains runtime would be also less affected in terms of performance and I/O.

@Noc2 Does that answer your questions?

[dobschal]

Hey, good morning @Noc2,
what are the next steps? Is there anything we can help with?

Thanks!

[Noc2]

Thanks for the reply. And again, I’m sorry for the delay. I got covid. Regarding availability, I’m not sure I follow you. In my opinion, you can only be 100% certain that the file isn’t deleted if you store it yourself in your case. Even if the software doesn’t provide an easy (on-chain) way to delete a file, you can not be sure that other nodes are actually storing the data correctly unless it’s completely on-chain. Any off-chain storage system can be changed/updated without other nodes being informed about it.

Independent of it, I mark your application as ready for review so that others can take a look at it.

[dominikdem]

Hey @Noc2, no worries, health comes first, and we hope you're doing better now. Regarding your questions about the file deletion - according to the IPFS documentation, there is no official API for removing a file from the network. https://docs.ipfs.tech/concepts/persistence/#long-term-storage

The only way to create an inconsistency between the on-chain data and the IPFS node is if files get deleted automatically because they were not pinned and no node is holding the file anymore. But our embedded IPFS node ensures that the file stays in the network.

So the chance is quite low and does not counterbalance the possibilities. Storing and accessing the information if a file was uploaded could be essential information for Substrate usage. For example, if people want to build up a system such as File Coin, in combination with streaming functionality, it could be implemented quickly and almost out of the box using Substrate and IPFS.

[keeganquigley]

Hi @dominikdem thanks for the follow-up application.

• Even though it's a follow-up, you can still re-label the milestones as 1 & 2 instead of 3 & 4. That way it follows the template convention.
• Can you specify USDC or DAI next to the payment address?

[dominikdem]

Hi @keeganquigley, I hope you're doing well. Do you have any open questions about our proposal?

[semuelle]

The only way to create an inconsistency between the on-chain data and the IPFS node is if files get deleted automatically because they were not pinned and no node is holding the file anymore. But our embedded IPFS node ensures that the file stays in the network.

I think what @Noc2 is referring to is this: what happens when I upload a file locally, wait for to be "Done", then shut down my node? There is no way to know the file is available anywhere, much less pinned. Or am I missing something?

[Noc2]

Thanks for the updates. Just to follow up here: I’m aware that you can not delete the file from the entire network (because you don’t control other ipfs nodes). But you can permanently delete it locally on your node (https://docs.ipfs.tech/concepts/lifecycle/#_4-deleting), and there is a huge incentive for other nodes not to store your data. Additionally, as Sebastian pointed out, you can not ensure that all the nodes are always online or that the data is propagated to other nodes. So if I understand it correctly, unless you run your own IPFS node and store the data yourself, you can not be sure that the data is still available.

[semuelle]

pinging @dominikdem & @dobschal

[Noc2]

I'm closing this application due to inactivity. Let me know if I should reopen it.

[dominikdem]

Hi @Noc2 , thanks for reaching out - we are still on it. Unfortunately, we're blocked by other projects at the moment. We'll come back to you as soon as we can.

[Noc2]

Feel free to ping me here once you want to update the application.