[WIP] Filestore Implementation

[kevina]

Closes Issue #875, avoid duplicating files added to ipfs

NOT READY FOR MERGE

Rebased #2600 on master.

Quicklinks: Code, README,

TODO to get this merged:

• Rebase on master turning the chain of commits into a reasonable ChangeSet
• Agree on major infrastructure changes
  • Multi-blockstore (Discussion: Support for multiple blockstores. #3119)
• Separate out non-filestore bits of infrastructure change into there own pull request
  • Multi-blockstore: WIP: Add support for multiple blockstores #3257
  • DAGService.GetLinks: Add DAGService.GetLinks() method and use it in the GC and elsewhere. #3255
  • Others: see milestone Filestore implementation
• Code review
• Merge

Note The filestore is very basic right now, but it is functional. I will likely continue to improve and submit new pull requests for the enhanced functionally but right now I fell it is important a basic implementation gets in so that it will get used, it can be labeled as an experimental feature and disabled by default, but available for those that want to use it. I consider the code production ready.

Resolves #875

[whyrusleeping]

aww, renaming this from RawBlock to BasicBlock makes the diff so noisy... I was hoping to avoid that by getting that change merged separately

[kevina]

Sorry, I missed that change as it was in a separate commit. Let me know if you know if you want me to do anything about it, like submitting this change separately.

[kevina]

@whyrusleeping, this rename is only in one file, it is exported because I might eventually refactor my Filestore specific changes to blocks/blocks.go into another file.

[kevina]

I imagine that this is going to take a while before it gets merged. So for now I think it will be best to maintain this as a separate fork while the goal of eventually merging everything in.

@whyrusleeping I will still like to work with you on API issues and work to get major API changes merged first to avoid stagnation.

I have created a README for the new filestore available here https://github.com/ipfs-filestore/go-ipfs/blob/kevina/filestore/filestore/README.md some notes on my fork are available here https://github.com/ipfs-filestore/go-ipfs/wiki.

@jefft0 if you are interested I could use some serious testing now. Others all of course welcome to test it out.

[whyrusleeping]

this can probably be changed so that it only accepts the absolute path instead of a filename, path and absolute path.

[whyrusleeping]

(and a similar change can be made throughout this part of the codebase)

[kevina]

@whyrusleeping I added absolute path, because I was unsure how FullPath was used in the code base. Is it acceptable for full path to be an absolute path? I am unsure how FullPath and "add -r" interact.

[whyrusleeping]

To be honest, i'm not entirely sure either. But that said, our tests in that area are quite thorough, If you change something, go into test/sharness, run make deps and then run ./t0040-add.sh, and maybe some of the other t004* tests too to make sure nothing weird happened

[kevina]

All right I will start by trying to change FullPath to an absolute path add see what happens when used with "add -r". I will make sure the contents of the directory objects are correct.

[kevina]

It turns our that when adding directories the directory is part of the filename. For example after adding some debug output here is the result of one of the test:

expecting success: 
        mkdir mountdir/planets &&
        echo "Hello Mars!" >mountdir/planets/mars.txt &&
        echo "Hello Venus!" >mountdir/planets/venus.txt &&
        ipfs add -r mountdir/planets >actual

Filename: planets/mars.txt
Path: mountdir/planets/mars.txt
Filename: planets/venus.txt
Path: mountdir/planets/venus.txt
ok 42 - 'ipfs add -r' succeeds

Thus we still need to pass in both a filename and a path. It seams okay to just make the path absolute, so I will do that I remove the change that adds an absolute path to the various parts in the files package.

[whyrusleeping]

We could extract the filename from the passed in path though

[kevina]

I tried that and it is not going to work. This test will fail. The filename needs to be "planets/mars.txt" if you try to extract it it will be just "mars.txt". You will get this:

expecting success: 
        mkdir mountdir/planets &&
        echo "Hello Mars!" >mountdir/planets/mars.txt &&
        echo "Hello Venus!" >mountdir/planets/venus.txt &&
        ipfs add -r mountdir/planets >actual

ok 42 - 'ipfs add -r' succeeds

expecting success: 
        PLANETS="QmWSgS32xQEcXMeqd3YPJLrNBLSdsfYCep2U7CFkyrjXwY" &&
        MARS="QmPrrHqJzto9m7SyiRzarwkqPcCSsKR2EB1AyqJfe8L8tN" &&
        VENUS="QmU5kp3BH3B8tnWUU2Pikdb2maksBNkb92FHRr56hyghh4" &&
        echo "added $MARS planets/mars.txt" >expected &&
        echo "added $VENUS planets/venus.txt" >>expected &&
        echo "added $PLANETS planets" >>expected &&
        test_cmp expected actual

> diff -u expected actual
--- expected    2016-05-11 03:20:12.742743658 +0000
+++ actual      2016-05-11 03:20:12.466738156 +0000
@@ -1,3 +1,2 @@
-added QmPrrHqJzto9m7SyiRzarwkqPcCSsKR2EB1AyqJfe8L8tN planets/mars.txt
-added QmU5kp3BH3B8tnWUU2Pikdb2maksBNkb92FHRr56hyghh4 planets/venus.txt
-added QmWSgS32xQEcXMeqd3YPJLrNBLSdsfYCep2U7CFkyrjXwY planets
+added QmPrrHqJzto9m7SyiRzarwkqPcCSsKR2EB1AyqJfe8L8tN mars.txt
+added QmU5kp3BH3B8tnWUU2Pikdb2maksBNkb92FHRr56hyghh4 venus.txt

not ok 43 - 'ipfs add -r' output looks good
#
#               PLANETS="QmWSgS32xQEcXMeqd3YPJLrNBLSdsfYCep2U7CFkyrjXwY" &&
#               MARS="QmPrrHqJzto9m7SyiRzarwkqPcCSsKR2EB1AyqJfe8L8tN" &&
#               VENUS="QmU5kp3BH3B8tnWUU2Pikdb2maksBNkb92FHRr56hyghh4" &&
#               echo "added $MARS planets/mars.txt" >expected &&
#               echo "added $VENUS planets/venus.txt" >>expected &&
#               echo "added $PLANETS planets" >>expected &&
#               test_cmp expected actual
#

Here is a quick patch to try, it might not apply cleanly for you though.

diff --git a/commands/files/readerfile.go b/commands/files/readerfile.go
index 2508fe1..d7f5443 100644
--- a/commands/files/readerfile.go
+++ b/commands/files/readerfile.go
@@ -4,6 +4,7 @@ import (
        "errors"
        "io"
        "os"
+       gopath "path"
 )

 // ReaderFile is a implementation of File created from an `io.Reader`.
@@ -17,7 +18,11 @@ type ReaderFile struct {
        baseInfo ExtraInfo
 }

-func NewReaderFile(filename, path string, reader io.ReadCloser, stat os.FileInfo) *ReaderFile {
+func NewReaderFile(_, path string, reader io.ReadCloser, stat os.FileInfo) *ReaderFile {
+       filename := path 
+       if path != "" {
+               filename = gopath.Base(path)
+       }
        return &ReaderFile{filename, path, reader, stat, 0, PosInfo{0, path}}
 }

[whyrusleeping]

@kevina Yeah, theres a LOT here, so its going to take some time. I don't disagree with you maintaining a fork in the meantime. That said, as I find time my plan is to pick out parts of the code that can be merged up easily. I think the next candidate for that is the commands/files code refactor to use absolute paths.

[kevina]

@whyrusleeping, One idea I had which might avoid some special casing is to replicate what is done with add --only-hash

                if hash {
                        nilnode, err := core.NewNode(n.Context(), &core.BuildCfg{
                                //TODO: need this to be true or all files
                                // hashed will be stored in memory!
                                NilRepo: true,
                        })
                        ...
                        n = nilnode
                }

That is create a special "node" for adding files to the filestore. This could help avoid special case code in at least a few places. It will at minimal avoid the special case code in the blockstore package as I can use my own implementation.

Do you think this is an idea worth pursuing? I image the special node will share many of the data structures from the real node to avoid any data race or related issues.

[whyrusleeping]

@kevina I think that might be a good idea, my only concern is that in order to read the blocks out of the filestore would you still need that custom created node?

[kevina]

@whyrusleeping, no the custom node will be for adding only. We will still need some sort of multi-datastore for reading blocks. See my comments for #2747.

[kevina]

@whyrusleeping there is a lot going on in an Ipfs Node. What I need is not a mock node like the code that --only-hash uses but rather create a new Node that acts like a view into the original node except that add's are handled differently. I was not confident I could pull this if with a full Ipfs Node.

With that in mind, I refactored the adder code a little to bundle all the needed services into the DataServices struct see commit c41a47e. A proper interface that exports just methods needed by the adder would be nice, but I think this is a good start. This new interface could be used to simply a lot of code elsewhere, including the code for just computing the hashes, but I don't fully understand what is going on there so I left the code for add --hash-only alone.

I use this new interface to to create a DataServices view in commit e6bfbfa that has uses an alternative Blockstore and leaves the existing Blockstore and caching Blockstore wrapper alone. In commit b54f2f4 I factor out the filestore specific code from the DagServices and instead install small hooks.

In commit f9a51b1 I completely eliminated the AdvReader code you didn't like and instead computed the offset's in the DAG builder code (the Balanced builder to be specific, the Trickle builder doesn't support the filestore just yet).

All in all I cleaned up a lot of the special case code.

I you are okay with the changes in c41a47e I can separate that out into it's own pull request.

[whyrusleeping]

@kevina from irc, we can move towards making the NewAdder signature be: NewAddr(context.Context, bs bstore.Blockstore, pin pinning.Pinner, ds dag.DAGService) which eliminates the need for the partial initialization stuff

[kevina]

@whyrusleeping f5fb3c9 basically does what you ask for.

[rht]

This sure is a feat, pulled singlehandedly on filestore, 👏 👏, @kevina! If this PR is carefully splitted into 10 PR's consisting of 10 commits each (if 50% have already been merged), this should be doable within a week (or two the longest).

[whyrusleeping]

🙌 🎆 #3629 has been merged, basic filestore functionality is in master. Go try it out :)