track: Files API + MFS

[daviddias]

The files API offers an interface to work with unixfs, a Merklelized Data Structure that represents a typical FileSystem used in a Unix OS. with files, dirs and symlinks. This API is what enables IPFS to add and retrieve files without having to ask the users to convert their data into a graph data structure.

To learn more about the Files API, you can

• watch https://www.youtube.com/watch?v=FX_AXNDsZ9k
• check files (mfs) implementation in go https://github.com/ipfs/go-ipfs/tree/master/mfs
• check unixfs implementation in go https://github.com/ipfs/go-ipfs/tree/master/unixfs

Implementation roadmap

• Files API spec Files API spec specs#88
• Unixfs Spec
• jsipfs files add (current ipfs add)
  • feature/files add #67
• jsipfs files cat (current ipfs cat)
  • cat subfiles
• jsipfs files get (current ipfs get)
• jsipfs files mv - Move files.
• jsipfs files cp - Copy files into mfs.
• jsipfs files ls - List directories.
• jsipfs files mkdir - Make directories.
• jsipfs files stat - Display file status.
• jsipfs files rm ... - Remove a file.
• jsipfs files read - Read a file in a given mfs.
• jsipfs files write - Write to a mutable file in a given filesystem.

[daviddias]

Updated this issue

[nginnever]

jsipfs files get

Taking a note from @noffle and a stab at jsipfs files get

Relevant go-ipfs links:
https://github.com/ipfs/go-ipfs/blob/master/core/commands/get.go

ARGUMENTS:

<ipfs-path> - The path to the IPFS object(s) to be outputted

OPTIONS:

-o, --output            string - The path where output should be stored
-a, --archive           bool   - Output a TAR archive
-C, --compress          bool   - Compress the output with GZIP compression
-l, --compression-level int    - The level of compression (1-9)

STEPS

1. Get the gzip compression options, bail if err (TODO: Look into js gzip modules) - return NoCompression if compress bool is false - return DefaultCompression if compress bool is true but compress level is undefined - return NoCompression and error if bool is true and compress level is not between 1-9 - return compress level if none of the above conditions are met 2. Get the node from the command request context. 3. Get the path from the argument . - TODO: List what happens here 4. Resolve the path, bail if err 5. Set archive to the option --archive provided. 6. DagArchive (this is equivalent to `ipfs getdag $hash | maybe_tar | maybe_gzip`) - TODO: list what happens here 7. create a readable stream of the dagnode 8. Check the stream bail if error 9. Check the path option. If it does not exist, defaut path to the multihash 10. Get the compression level from options 11. Get the archive boolean from options 12. Pass the readable stream and path to a write() function - If archive or compression options != gzip.NoCompress then pass to writeArchive() function - else pass the stream and path to writeExtracted() function - **If writeArchive()** - check archive is set to true - if it is, check if the ending of the path is `path + '.tar'` or `path + '.tar.gz'` - if it does not then add .tar to path name - adjust path name to .gz if compression level != gzip.NoCompress - create the file on disk - print to console `Saving archive to $path` - show a progress bar - **if writeExtracted()** - print to console `Saving file(s) to $path` - show progress bar - TODO: figure out what extractor is

[nginnever]

jsipfs files cat

A look at jsipfs files cat

Relevant go-ipfs links:
https://github.com/ipfs/go-ipfs/blob/master/core/commands/cat.go
https://github.com/ipfs/go-ipfs/blob/master/core/coreunix/cat.go

ARGUMENTS:

<ipfs-path>... - The path to the IPFS object(s) to be outputted

DESCRIPTION:

Retrieves the object named by <ipfs-or-ipns-path> and outputs the data
it contains.

STEPS

1. Get the node from the command request context. 2. If node.onlineMode = false, setupOfflineRouting() - TODO: List setupOfflineRouting steps 3. call cat() function passing in the context, the node, and the path(s) from the arguments - create read streams array for each path provided - for each path, call coreunix.Cat(ctx, node, fpath) - resolve the dagnode - return a readable stream - append the stream from resolve to the read streams array - collect the total length by adding each buffer length returned from resolve - return the readers array, length, and null or error 4. Do a conditional GC 5. set the length in the response 6. combine the readers streams into one read stream 7. check if the length < the smallest progress bar size 8. create the progress bar if not 9. set the res to the read stream.

[daviddias]

merged #67

[daviddias]

@nginnever I know you are working on this. Can you make a PR tagged with WIP from a branch under this project?

[daviddias]

-> #156

[daviddias]

The remaining items (specific to mfs) go into the Milestone 6 - Feature parity

[daviddias]

//cc @victorbjelkholm

[daviddias]

Now Files Add work as expected with the option to do trickle-dag too! #707

[daviddias]

cat/get subtrees is now there too :)

[daviddias]

Now tracked here https://github.com/ipfs/ipfs/blob/master/IMPLEMENTATION_STATUS.md

tl;dr; Files is done, MFS isn't