Files API - How can we make it better for users

[daviddias]

This is a thread to get a single discussion around ipfs/specs#98 and #1360 (comment) for the JavaScript land of IPFS.

[achingbrain]

I think we need to give some of the commands on ipfs/specs#98 a bit more thought. For example ipfs files cat and ipfs files read (from this comment - which is where I think that thread concludes) look really similar but currently work very differently in terms of inputs in that read takes mfs paths and cat takes ipfs paths.

ipfs files cp takes both which can lead to oddness if you have an mfs directory in your repo called /ipfs.

What would be great is to support all three (as I see them) types of paths in all file commands (where it makes sense):

1. IPFS-at-large (e.g. /ipfs/QmPbFXUdgtyrCjExNAjugJTro6Lx4GG4MuvXTiHcZQkiif)
2. MFS (e.g. /a-file-in-mfs)
3. Local files (e.g. /home/me/some-file.txt)

[olizilla]

The spec on the existing behaviour of ipfs files cp is vague about the arguments. What is supported currently?

I can guess that the argument in the from position is supposed to be an mfs path, but what is supported in the to position right now?

Is there a reason why it needs to take an array of two items and not an arguments list or named object properties?

[alanshaw]

re: files cp, this is due to get much better soon when MFS lands in js-ipfs! ipfs-inactive/interface-js-ipfs-core#277

[olizilla]

We can't perfectly disambiguate an local fs path from an mfs path from an IPFS path as you could have mounted ipfs in your local file system under /ipfs, and you might choose to be a chaos monkey and create a /ipfs dir at the root of your mfs.

We could do a best effort:

• stat the path in MFS and use it if exists
• stat the path in the file system and use it if exists
• if it starts with /ipfs or /ipns or it parses as a valid cid then try and get it from ipfs.
• else, fail...

[olizilla]

I'm running into issues where I want the files api to work on any unixfs node, not just ones in my local MFS. I can't ipfs.files.stat a CID. I can ipfs.object.get it and read the metadata, like the cumulative size, but I can't use the links array off of the response, as I'd have to do extra work to figure out if the links point to files or shards of large files.

[mikeal]

Just bumped into this today. I've been using js-ipfs-api and when I went to do some stuff w/ js-ipfs the file methods I was used to were missing (read, write, mkdir, rm).

Yes, when I was first learning these methods it was confusing since some took cid, some took ipfs path, and some took mfs path, but right now there isn't even feature parity, which would be better than nothing :)

[achingbrain]

js-mfs is coming. Soon. Really soon.

[ntninja]

I have a different idea for how the paths could be disambigued: How about having all MFS paths start with /ipfs/local (similar to how the current node key is at /ipns/local)?

This way to would always be clear which file path is being referred to:

1. MFS: /ipfs/local/*
2. IPFS/IPNS: /ip[fn]s/<CID>/*
3. Local FS: /*

Since local is not a valid CID there would not be any conflict. The downsides would be of course:

1. Not backwards-compatible WRT MFS – can probably be worked around by exposing new API calls on the HTTP API layer, but will likely cause issues for some clients
2. Longer paths for MFS

But the unambiguousness of this solution IMHO beats the above two issues. We're also in Alpha still so backwards-incompatible changes can be made if really necessary.

[alanshaw]

As per #1394 - progress information for files.cat/files.get

[achingbrain]

How about having all MFS paths start with /ipfs/local

Does this handle the case where the user has /ipfs/local on their filesystem and in MFS?

e.g. where would the files come from in this case:

ipfs files cp /ipfs/local/foo.txt /ipfs/local/bar.txt

[ntninja]

No it does not, but we assume that /ipns and /ipfs are available on the system already – at least when mounting. And if they already exist and contain meaningful data then lots of IPFS operations will feel weird, not just copying files.

[daviddias]

@olizilla @alanshaw can you update this thread with the new top level Files API proposal?

[alanshaw]

Pong, I will get to this asap...

[olizilla]

@alanshaw does the files api proposal include changes for having items added via ipfs add show up some how in the mfs view of the repo? Right now the Web UI can't visualise files you ipfs added which is a suprising UX. Could you link to an issue or add a quick summary here?

[daviddias]

Distilling the convo from here ipfs-inactive/interface-js-ipfs-core#378, we essentially set on upgrading the Files API in 3 milestones:

• M1 - Move add, get, cat to top level and distinguish them from the Files(MFS) commands. This is 2/3 done, only missing getting realeased on js-ipfs
• M2 - Move to async await everywhere (this is half of what ipfsx is about)
• M3 - Revisit how the all the commands that touch files are designed. This milestone is original intention of this issue and the other half of what ipfsx is about.

@olizilla I am in favor of adding that property to an ipfs add and also getting closer to an "ipfs drive" notion which is very familiar to the users.

For reference, another great thread to acquire more context is: ipfs/specs#98

[Gozala]

M1 - Move add, get, cat to top level and distinguish them from the Files(MFS) commands. This is 2/3 done, only missing getting realeased on js-ipfs

Is the Files API open for feedback ? If so how can I provide it ? Or is this ship already has sailed ?

[Gozala]

I've just posted my feedback here https://via.hypothes.is/https://github.com/ipfs/interface-ipfs-core/blob/master/SPEC/FILES.md

[daviddias]

@Gozala the ship hasn't sailed yet, M3 aka https://github.com/ipfs-shipyard/ipfsx is the Future™

Btw, love how you provided the feedback over hypothesis :)

[lidel]

Somehow related to M3 (Revisit how the all the commands that touch files are designed), this discussion hints that our current Files API does not pass the red face test when we start explaining it to people (add vs write etc).

I feel it is a good time to revisit the idea of cleaning up the MFS/Files API mess by starting fresh with better abstractions under something like ipfs drive and incentivizing people to switch to it by making ti much easier to understand and giving it superpowers such as automatic publishing to IPNS.

Is this issue the best place to track this?
Would opening a PR with proposed ipfs drive API be a good next step?

[achingbrain]

js-ipfs is being deprecated in favor of Helia. You can #4336 and read the migration guide.

Helia has no strong opinion about what the files API should look like (mostly due to these sorts of issues TBH), @helia/unixfs is one approach, people are free to implement their own ever-so-slightly-rounder version as they see fit.

I look forward to everyone's creativity being unleashed!