Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add: recursion fix #158

Closed
jbenet opened this issue Oct 13, 2014 · 8 comments
Closed

Add: recursion fix #158

jbenet opened this issue Oct 13, 2014 · 8 comments
Labels
help wanted Seeking public contribution on this issue

Comments

@jbenet
Copy link
Member

jbenet commented Oct 13, 2014

Currently subblocks are not added with addNode directly, but as part of their file dags. Currently, dir-file recursion happens via calls to addNode, and subblock recursion happens when addNode (on a file) calls DAG.AddRecursive(nd).

The fact that addNode calls DAG.AddRecursive(nd) means it will try to add nodes again and again as addNode is called with parent dirs. The two alternatives are to either do everything with recursive backtracking, or generate the whole dag first, and then call DAG.AddRecursively(root). The former approach is less memory intensive (as the whole DAG does not need to be in memory) and parallelizes better, as can work on siblings simultaneously (without relying on the parallelism being inside DAG.AddRecursively).

Two related changes needed:

  1. The file-sublock recursion should be the same as dir-file recursion (i.e. calling addNode on the subblock).
  2. addNode should call DAG.Add(nd)
@jbenet
Copy link
Member Author

jbenet commented Oct 13, 2014

cc @whyrusleeping

jbenet added a commit that referenced this issue Oct 13, 2014
@whyrusleeping
Copy link
Member

SGTM

@jbenet jbenet added the help wanted Seeking public contribution on this issue label Oct 13, 2014
@jbenet
Copy link
Member Author

jbenet commented Nov 4, 2014

I think this is a non-issue now that ipfs add writes differently.

cc @whyrusleeping

@jbenet jbenet closed this as completed Nov 4, 2014
@whyrusleeping
Copy link
Member

Eh, now that i think about it, this isnt entirely fixed since the directories will be recursively added after the files under them are.

@jbenet jbenet reopened this Nov 4, 2014
@jbenet
Copy link
Member Author

jbenet commented Jan 15, 2015

@whyrusleeping is this fixed now?

@whyrusleeping
Copy link
Member

I dont think so...

right here: https://github.com/jbenet/go-ipfs/blob/master/core/commands/add.go#L129

we recursively add the given node, when all of its children should have already been added to the DAG during importer.BuildDagFromReader. The simple fix to this i think is changing that call to not be recursive.

@jbenet
Copy link
Member Author

jbenet commented Jan 17, 2015

@whyrusleeping i think you're right. if this is not correct, will the tests we have catch it? if so, let's try it.

@whyrusleeping
Copy link
Member

Yeah, the tests will catch that one. lets give it a try

@Stebalien Stebalien mentioned this issue May 26, 2020
77 tasks
ariescodescream pushed a commit to ariescodescream/go-ipfs that referenced this issue Oct 23, 2021
@aschmahmann aschmahmann mentioned this issue Dec 1, 2021
80 tasks
laurentsenta pushed a commit to laurentsenta/kubo that referenced this issue Feb 25, 2022
laurentsenta pushed a commit to laurentsenta/kubo that referenced this issue Feb 25, 2022
laurentsenta pushed a commit to laurentsenta/kubo that referenced this issue Mar 4, 2022
laurentsenta pushed a commit to laurentsenta/kubo that referenced this issue Mar 4, 2022
ariescodescream pushed a commit to ariescodescream/go-ipfs that referenced this issue Apr 7, 2022
@ajnavarro ajnavarro mentioned this issue Aug 24, 2022
72 tasks
Jorropo pushed a commit that referenced this issue May 30, 2023
* chore: interop with go-ipfs 0.13

Applies necessary changes to ensure 'block/put' works
and is backward-compatible.

Context:
#8568

* chore: 0.3.1

bumping as patch because we bumped to 0.3.0 recently, 
as part of other (unreleased) go-ipfs 0.13 work

This commit was moved from ipfs/go-ipfs-http-client@ecf364c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Seeking public contribution on this issue
Projects
None yet
Development

No branches or pull requests

2 participants