replace nodebuilder with a nicer interface #1572

whyrusleeping · 2015-08-13T18:51:31Z

a less extreme version of #1557 that uses a BuildCfg to set all the same options.

License: MIT
Signed-off-by: Jeromy [email protected]

whyrusleeping · 2015-08-13T23:27:13Z

@jbenet let me know what you think of this interface. If its what we want, i'll go ahead and switch over a bunch of the codebase to use it instead of the old way.

jbenet · 2015-08-14T00:48:32Z

i like it :)

whyrusleeping · 2015-08-14T03:52:45Z

alright! Choo Choo! time for the refactor train :D

rht · 2015-08-14T05:44:32Z

Also, have been wondering, why the https://github.com/ipfs/go-ipfs/pull/1572/files#diff-54f2c4a5481de30b8ccb59ad5b6588b3L242? Sounds redundant with something that sounds "NodeBuilder".

Can't the node be asserted to exist at the creation of the commands.Context object?

rht · 2015-08-14T05:48:58Z

https://github.com/ipfs/go-ipfs/blob/master/util/util.go#L99 and https://github.com/ipfs/go-ipfs/blob/master/thirdparty/multierr/multierr.go#L8 -- only one can exist.

whyrusleeping · 2015-08-14T20:34:35Z

@rht on the invocContext.ConstructNode call, I think its so that the http client doesnt have to waste time constructing the node if it doesnt need to. Although that can likely be refactored... @jbenet thoughts?

whyrusleeping · 2015-08-14T20:35:17Z

interesting failure here: https://travis-ci.org/ipfs/go-ipfs/jobs/75662605

whyrusleeping · 2015-08-14T21:41:11Z

core/mock/mock.go

+}
+
+// TODO: shrink this function as much as possible, moving logic into NewNode
+func NewMockNodeOnNet(ctx context.Context, mn mocknet.Mocknet) (*core.IpfsNode, error) {


Since it proved difficult to combine mocknet and the node constructor i was working on, i have this method here as a bit of a shim until i figure out the right way to do it.

whyrusleeping · 2015-08-17T22:54:17Z

if anyone has a good idea on how to make the node construction process work nicely with the MockNet, i would love to hear your thoughts. cc @jbenet @Heems @rht

whyrusleeping · 2015-08-18T06:13:35Z

much better :) RFCR

whyrusleeping · 2015-08-18T06:15:22Z

The simplest case of getting an offline node for testing something is now:

node, err := core.NewNode(ctx, nil)

rht · 2015-08-18T09:17:30Z

core/builder.go

+	}
+	n.Resolver = &path.Resolver{DAG: n.DAG}
+
+	success = true


should be just n.teardown() (no defer) since it is only used for this path.

no, there are three different places we could return early, triggering the need for a teardown.

i c, I got this reversed. teardown is not called only for this path.

This can be used for res.SetError

defer func(){ if err != nil { res.SetError(err, cmds.ErrNormal) } }()

this function is a scary still, and the success/defer stuff is not easy to modify safely (error prone).

how about this (it should work exactly):

func cfgDefaults(cfg *BuildCfg) (*BuildCfg, error) { if cfg == nil { cfg = new(BuildCfg) } err := cfg.fillDefaults() if err != nil { return nil, err } } func NewNode(ctx context.Context, cfg *BuildCfg) (*IpfsNode, error) { cfg, err := cfgDefaults(cfg) if err != nil { return nil, err } n := &IpfsNode{ mode: offlineMode, Repo: cfg.Repo, ctx: ctx, Peerstore: peer.NewPeerstore(), } if cfg.Online { n.mode = onlineMode } // TODO: this is a weird circular-ish dependency, rework it n.proc = goprocessctx.WithContextAndTeardown(ctx, n.teardown) if err := tryNodeSetup(ctx, cfg, n); err != nil { n.Close() return nil, err } return n, nil } func tryNodeSetup(ctx context.Context, cfg *BuildCfg, n *IpfsNode) error { // setup local peer ID (private key is loaded in online setup) if err := n.loadID(); err != nil { return nil, err } bs := bstore.NewBlockstore(n.Repo.Datastore()) n.Blockstore, err = bstore.WriteCached(bs, kSizeBlockstoreWriteCache) if err != nil { return nil, err } if cfg.Online { do := setupDiscoveryOption(n.Repo.Config().Discovery) if err := n.startOnlineServices(ctx, cfg.Routing, cfg.Host, do); err != nil { return nil, err } } else { n.Exchange = offline.Exchange(n.Blockstore) } n.Blocks = bserv.New(n.Blockstore, n.Exchange) n.DAG = dag.NewDAGService(n.Blocks) n.Pinning, err = pin.LoadPinner(n.Repo.Datastore(), n.DAG) if err != nil { // TODO: we should move towards only running 'NewPinner' explicity on // node init instead of implicitly here as a result of the pinner keys // not being found in the datastore. // this is kinda sketchy and could cause data loss n.Pinning = pin.NewPinner(n.Repo.Datastore(), n.DAG) } n.Resolver = &path.Resolver{DAG: n.DAG} return nil }

Note: call n.Close(), or even n.proc.Close(). do not call n.teardown() directly!! or it may be called twice and break semantics (maybe panic)

ah, i like that better.

rht · 2015-08-20T06:41:16Z

@whyrusleeping there is still one use left of mocknet.FullMeshLinked in p2p/net/mock/mock_notif_test.go. If this is replaced with mocknet.New and mn.Linkall() then mocknet.FullMeshLinked can be removed.

jbenet · 2015-08-21T23:41:36Z

@whyrusleeping should i take another look here?

whyrusleeping · 2015-08-22T00:27:23Z

@jbenet yeah, i think i got it right now.

jbenet · 2015-08-23T00:27:49Z

core/builder.go

-		online:   false,
-		routing:  DHTOption,
-		peerhost: DefaultHostOption,
+func (cfg *BuildCfg) fillDefaults() error {


this should error when:

cfg.Repo != nil && cfg.NilRepo

jbenet · 2015-08-23T00:32:59Z

test/integration/bitswap_wo_routing_test.go

-	peers := mn.Peers()
-	if len(peers) < numPeers {
-		t.Fatal(errors.New("test initialization error"))
-	}


why removing this? it can catch serious errors that might otherwise drive people mad.

(maybe it just belongs elsewhere)

Because no peers are created by the mocknet constructor, i'm creating them manually in a loop and adding them.

jbenet · 2015-08-23T00:35:17Z

👍 i think this is way cleaner. comments above

whyrusleeping · 2015-08-24T06:29:51Z

@rht @jbenet do you think FullMeshLinked should be removed? if so, in this PR? or a subsequent one?

jbenet · 2015-08-24T12:38:31Z

@rht @jbenet do you think FullMeshLinked should be removed? if so, in this PR? or a subsequent one?

No, i don't. i meant just New() so that people use FullMeshLinked and friends

rht · 2015-08-24T14:33:17Z

FullMeshLinked should be removed only if it can't be modified to use the new NewNode manual construction.

This would mean FullMeshConntected is to be replaced with mn.ConnectAll() at the end of its construction.

It is part of this PR, I think, because the NewNode change makes it necessary (not optional) to use mn.LinkAll()/mn.ConnectAll() at the end instead of the beginning.

whyrusleeping · 2015-08-25T16:40:51Z

@jbenet yeah, i think we should do away with the FullMeshLinked and FullMeshConnected and also WithNPeers constructors in favor of explicity calling LinkAll and ConnectAll because it makes it difficult to construct ipfs nodes from the resulting networks

jbenet · 2015-08-26T15:26:17Z

Ok.

@whyrusleeping this needs rebasing i think

jbenet · 2015-08-26T15:27:26Z

i think that's the last thing, right? this should be good to go

whyrusleeping · 2015-08-26T19:47:42Z

rebased!

jbenet · 2015-08-26T20:39:58Z

@whyrusleeping do all commits pass? either git push-each or squash.

whyrusleeping · 2015-08-26T21:04:20Z

squashed.

jbenet · 2015-08-27T00:58:47Z

@whyrusleeping conflicts?

License: MIT Signed-off-by: Jeromy <[email protected]> use NewNode instead of NewIPFSNode in most of the codebase License: MIT Signed-off-by: Jeromy <[email protected]> make mocknet work with node constructor better License: MIT Signed-off-by: Jeromy <[email protected]> finish cleanup of old construction method License: MIT Signed-off-by: Jeromy <[email protected]> blockservice.New doesnt return an error anymore License: MIT Signed-off-by: Jeromy <[email protected]> break up node construction into separate function License: MIT Signed-off-by: Jeromy <[email protected]> add error case to default filling on node constructor License: MIT Signed-off-by: Jeromy <[email protected]>

whyrusleeping · 2015-08-27T15:45:45Z

@jbenet conflicts resolved.

replace nodebuilder with a nicer interface

jbenet added the status/in-progress In progress label Aug 13, 2015

whyrusleeping mentioned this pull request Aug 13, 2015

improve nodebuilder code to not depend on NewIPFSNode #1557

Closed

whyrusleeping force-pushed the node-construct-v2 branch from cd3d6df to 686f5a3 Compare August 13, 2015 23:26

whyrusleeping force-pushed the node-construct-v2 branch from bcbdeca to ab486fa Compare August 14, 2015 20:26

whyrusleeping reviewed Aug 14, 2015
View reviewed changes

whyrusleeping force-pushed the node-construct-v2 branch from ab486fa to 63fc7bb Compare August 17, 2015 22:51

whyrusleeping assigned jbenet Aug 18, 2015

whyrusleeping added the codereview label Aug 18, 2015

rht reviewed Aug 18, 2015
View reviewed changes

jbenet mentioned this pull request Aug 22, 2015

Sprint Aug 17 ipfs/team-mgmt#28

Closed

16 tasks

jbenet reviewed Aug 23, 2015
View reviewed changes

whyrusleeping force-pushed the node-construct-v2 branch from 373d334 to 3e2b7ae Compare August 24, 2015 06:24

whyrusleeping force-pushed the node-construct-v2 branch from 3e2b7ae to 969b8b8 Compare August 26, 2015 19:44

whyrusleeping force-pushed the node-construct-v2 branch from 969b8b8 to a12661b Compare August 26, 2015 21:04

whyrusleeping force-pushed the node-construct-v2 branch from a12661b to 94000e6 Compare August 27, 2015 15:43

jbenet added a commit that referenced this pull request Aug 27, 2015

Merge pull request #1572 from ipfs/node-construct-v2

b55cf12

replace nodebuilder with a nicer interface

jbenet merged commit b55cf12 into master Aug 27, 2015

jbenet removed codereview status/in-progress In progress labels Aug 27, 2015

jbenet deleted the node-construct-v2 branch August 27, 2015 16:01

replace nodebuilder with a nicer interface #1572

replace nodebuilder with a nicer interface #1572

Conversation

whyrusleeping commented Aug 13, 2015

whyrusleeping commented Aug 13, 2015

jbenet commented Aug 14, 2015

whyrusleeping commented Aug 14, 2015

rht commented Aug 14, 2015

rht commented Aug 14, 2015

whyrusleeping commented Aug 14, 2015

whyrusleeping commented Aug 14, 2015

Choose a reason for hiding this comment

whyrusleeping commented Aug 17, 2015

whyrusleeping commented Aug 18, 2015

whyrusleeping commented Aug 18, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rht commented Aug 20, 2015

jbenet commented Aug 21, 2015

whyrusleeping commented Aug 22, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbenet commented Aug 23, 2015

whyrusleeping commented Aug 24, 2015

jbenet commented Aug 24, 2015

rht commented Aug 24, 2015

whyrusleeping commented Aug 25, 2015

jbenet commented Aug 26, 2015

jbenet commented Aug 26, 2015

whyrusleeping commented Aug 26, 2015

jbenet commented Aug 26, 2015

whyrusleeping commented Aug 26, 2015

jbenet commented Aug 27, 2015

whyrusleeping commented Aug 27, 2015