an attempt at making the editor more efficient #1567

whyrusleeping · 2015-08-12T20:46:44Z

Using this, we would be able to pass an 'offline' dagstore to the editor, and then use the WriteOutputTo method on the editor to write the changed nodes to our actual dagservice.

@jbenet thoughts?

License: MIT
Signed-off-by: Jeromy [email protected]

jbenet · 2015-08-13T03:57:17Z

@whyrusleeping SGTM. the right solution, i think

whyrusleeping · 2015-08-13T18:49:44Z

@whyrusleeping SGTM. the right solution, i think

Cool, I'll finish working it into the add command if you'd like.

jbenet · 2015-08-17T02:43:22Z

merkledag/utils/utils.go

+
+func (e *Editor) WriteOutputTo(ds dag.DAGService) error {
+	return copyDag(e.GetNode(), e.ds, ds)
+}


this doesn't seem to get called? shouldn't it be called by the add command at the end?

it's called now.

whyrusleeping · 2015-08-17T22:49:34Z

I broke some test here... I'll need to take a look.

whyrusleeping · 2015-08-19T00:18:57Z

@jbenet if you get a chance, could you take a look at the test failure here in t0080-repo.sh? Something is a little off in the pinning, and i'm not sure what...

jbenet · 2015-08-19T00:45:18Z

Spelunking i also found this:

> cd test/sharness
> make bin/ipfs
> ./t0080-repo.sh
...
> cd trash\ directory.t0080-repo.sh
> ../bin/ipfs add -q multiblock
QmaPMMQBVm4mir7ZuL8b4MEFpEhoH2HA4oFdFAgSvsGCkG
Error: blockstore: block not found

the Error: blockstore: block not found shouldn't happen...

jbenet · 2015-08-19T00:51:05Z

merkledag/utils/utils.go

@@ -153,3 +148,32 @@ func rmLink(ctx context.Context, ds dag.DAGService, root *dag.Node, path []strin

 	return root, nil
 }
+
+func (e *Editor) WriteOutputTo(ds dag.DAGService) error {
+	return copyDag(e.GetNode(), e.ds, ds)


i think this needs what i wrote in 978c9fa#diff-bd0ca9f1e2d5c7ba128af01ec4c1131cR314

jbenet · 2015-08-19T00:51:22Z

In my run (the hashes are different because we use random without a seed), i get:

> diff -u refsout_sorted indirectpins_sorted
--- refsout_sorted  2015-08-19 00:41:36.000000000 +0000
+++ indirectpins_sorted 2015-08-19 00:41:36.000000000 +0000
@@ -8,5 +8,6 @@
 QmY8EMo9ykRSkJAc9QTZbLUfusViyonfcbCb5PyKjYsQn2 indirect
 QmYCvbfNbCwFR45HiNP45rwJgvatpiW38D961L5qAhUM5Y indirect
 QmZTR5bcpQD7cFgTorqxZDYaew1Wqgfbd2ud9QqGPAkK2V indirect
+QmaPMMQBVm4mir7ZuL8b4MEFpEhoH2HA4oFdFAgSvsGCkG indirect
 Qmc8afjX8uzstEbK4ziHcW52TSHuL8UjSFp2NnAFF84DSR indirect
 QmdQs4tAY9Me7iUEv8n4ZD7jwEJQ4M8YCHRePRuh7CmBqr indirect

not ok 25 - 'ipfs pin ls --type=indirect' is correct

and QmaPMMQBVm4mir7ZuL8b4MEFpEhoH2HA4oFdFAgSvsGCkG is the root hash of running

> ipfs add -q multiblock
QmaPMMQBVm4mir7ZuL8b4MEFpEhoH2HA4oFdFAgSvsGCkG
Error: blockstore: block not found (<--- yeah, this shouldnt be there)

QmaPMMQBVm4mir7ZuL8b4MEFpEhoH2HA4oFdFAgSvsGCkG is listed as indirect. It should be recursive, because we're not wrapping.

This may be related to something i ran into when i wrote 978c9fa:

when not wrapping, if the add root (/) only has one link out, discard the root, and treat the value of the link as the root. See 978c9fa#diff-bd0ca9f1e2d5c7ba128af01ec4c1131cR314
looking at your diff, it seems to be missing the equivalent: https://github.com/ipfs/go-ipfs/pull/1567/files#r37370653

jbenet · 2015-08-23T02:53:09Z

@whyrusleeping I think I CRed this already. see #1567 (comment) -- otherwise LGTM

whyrusleeping · 2015-09-04T18:57:22Z

@jbenet fixed this! should be good to go.

jbenet · 2015-09-05T01:29:03Z

@whyrusleeping: 991cd4c committed sadhack.

ipfs add -p

.... err, i meant

git add -p

whyrusleeping · 2015-09-05T01:34:33Z

@jbenet nuuuuu. the more i have to do this, the more resolve i get to fix the problem in the first place

License: MIT Signed-off-by: Jeromy <[email protected]>

…eOutputTo License: MIT Signed-off-by: Jeromy <[email protected]>

jbenet · 2015-09-05T02:58:57Z

Rebased on top of #1655

an attempt at making the editor more efficient

jbenet added the status/in-progress In progress label Aug 12, 2015

jbenet reviewed Aug 17, 2015
View reviewed changes

whyrusleeping force-pushed the editor-staging branch 2 times, most recently from 3333b81 to 48b115f Compare August 17, 2015 17:35

jbenet reviewed Aug 19, 2015
View reviewed changes

jbenet mentioned this pull request Aug 22, 2015

Sprint Aug 17 ipfs/team-mgmt#28

Closed

16 tasks

daviddias mentioned this pull request Sep 3, 2015

Sprint Aug 31 ipfs/team-mgmt#30

Closed

42 tasks

whyrusleeping force-pushed the editor-staging branch from 48b115f to 991cd4c Compare September 4, 2015 18:52

whyrusleeping force-pushed the editor-staging branch from 991cd4c to 920f557 Compare September 5, 2015 02:01

whyrusleeping added 3 commits September 5, 2015 04:56

an attempt at making the editor more efficient

b522107

License: MIT Signed-off-by: Jeromy <[email protected]>

add-only-hash no longer stores entirety of everything in memory

30d10ce

License: MIT Signed-off-by: Jeromy <[email protected]>

move mem-dag construction to its own function, and actually call Writ…

b3aee28

…eOutputTo License: MIT Signed-off-by: Jeromy <[email protected]>

jbenet force-pushed the editor-staging branch from 920f557 to b3aee28 Compare September 5, 2015 02:57

jbenet added a commit that referenced this pull request Sep 5, 2015

Merge pull request #1567 from ipfs/editor-staging

637e221

an attempt at making the editor more efficient

jbenet merged commit 637e221 into master Sep 5, 2015

jbenet removed the status/in-progress In progress label Sep 5, 2015

jbenet deleted the editor-staging branch September 5, 2015 03:03

ajnavarro mentioned this pull request Aug 24, 2022

Release 0.15 #9152

Closed

72 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

an attempt at making the editor more efficient #1567

an attempt at making the editor more efficient #1567

whyrusleeping commented Aug 12, 2015

jbenet commented Aug 13, 2015

whyrusleeping commented Aug 13, 2015

jbenet Aug 17, 2015

jbenet Aug 19, 2015

whyrusleeping commented Aug 17, 2015

whyrusleeping commented Aug 19, 2015

jbenet commented Aug 19, 2015

jbenet Aug 19, 2015

jbenet commented Aug 19, 2015

jbenet commented Aug 23, 2015

whyrusleeping commented Sep 4, 2015

jbenet commented Sep 5, 2015

whyrusleeping commented Sep 5, 2015

jbenet commented Sep 5, 2015

an attempt at making the editor more efficient #1567

an attempt at making the editor more efficient #1567

Conversation

whyrusleeping commented Aug 12, 2015

jbenet commented Aug 13, 2015

whyrusleeping commented Aug 13, 2015

jbenet Aug 17, 2015

Choose a reason for hiding this comment

jbenet Aug 19, 2015

Choose a reason for hiding this comment

whyrusleeping commented Aug 17, 2015

whyrusleeping commented Aug 19, 2015

jbenet commented Aug 19, 2015

jbenet Aug 19, 2015

Choose a reason for hiding this comment

jbenet commented Aug 19, 2015

jbenet commented Aug 23, 2015

whyrusleeping commented Sep 4, 2015

jbenet commented Sep 5, 2015

whyrusleeping commented Sep 5, 2015

jbenet commented Sep 5, 2015