Add: recursion fix

[jbenet]

Currently subblocks are not added with addNode directly, but as part of their file dags. Currently, dir-file recursion happens via calls to addNode, and subblock recursion happens when addNode (on a file) calls DAG.AddRecursive(nd).

The fact that addNode calls DAG.AddRecursive(nd) means it will try to add nodes again and again as addNode is called with parent dirs. The two alternatives are to either do everything with recursive backtracking, or generate the whole dag first, and then call DAG.AddRecursively(root). The former approach is less memory intensive (as the whole DAG does not need to be in memory) and parallelizes better, as can work on siblings simultaneously (without relying on the parallelism being inside DAG.AddRecursively).

Two related changes needed:

1. The file-sublock recursion should be the same as dir-file recursion (i.e. calling addNode on the subblock).
2. addNode should call DAG.Add(nd)

[jbenet]

cc @whyrusleeping

[whyrusleeping]

SGTM

[jbenet]

I think this is a non-issue now that ipfs add writes differently.

cc @whyrusleeping

[whyrusleeping]

Eh, now that i think about it, this isnt entirely fixed since the directories will be recursively added after the files under them are.

[jbenet]

@whyrusleeping is this fixed now?

[whyrusleeping]

I dont think so...

right here: https://github.com/jbenet/go-ipfs/blob/master/core/commands/add.go#L129

we recursively add the given node, when all of its children should have already been added to the DAG during importer.BuildDagFromReader. The simple fix to this i think is changing that call to not be recursive.

[jbenet]

@whyrusleeping i think you're right. if this is not correct, will the tests we have catch it? if so, let's try it.

[whyrusleeping]

Yeah, the tests will catch that one. lets give it a try