plugin: create plugin API and loader, add ipld-git plugin #4033

whyrusleeping · 2017-07-04T18:57:07Z

This is very much a work in progress, but it works and is pretty fun to play around with.

If you want to try it, pull this branch down and:

make install
mkdir $IPFS_PATH/plugins
go build -buildmode=plugin -o="$IPFS_PATH/plugins/git.so" ./plugins/git

Then start ipfs, it will now be loaded with git support. You can install https://github.com/magik6k/git-remote-ipld and then try out git push ipld:: to push a git repo into ipfs and then git clone ipld::GITSHA to clone it from ipfs.

whyrusleeping · 2017-07-04T18:59:39Z

cc @kumavis and @hermanjunge, this will be how we load in blockchain support too

victorb · 2017-07-04T20:17:00Z

Really cool to have plugin support finally :) However, the following disclaimer exists in the documentation: Currently plugins only work on Linux. but I'm unsure if this is for building with plugins, building the plugins or including the plugins...

whyrusleeping · 2017-07-04T20:29:28Z

@victorbjelkholm its for all of the above. The plugin stuff only works on linux. If people want this on platforms other than linux, pester the go team.

Kubuxu · 2017-07-05T13:17:17Z

cmd/ipfs/plugin_linux.go

+	LoadPlugins = LoadPluginsLinux
+}
+
+func LoadPluginsLinux(cfgroot string) error {


this can be private.

Kubuxu · 2017-07-05T13:20:58Z

plugins/git/git.go

+	"github.com/ipfs/go-cid"
+	"github.com/ipfs/go-ipfs/core/coredag"
+	git "github.com/ipfs/go-ipld-git"
+	"gx/ipfs/QmUBtPvHKFAX43XMsyxsYpMi3U5VwZ4jYFTo4kFhvAR33G/go-ipld-format"


Spacing and go-ipld-git is not imported.

Kubuxu · 2017-07-05T13:23:10Z

core/coredag/dagtransl.go

+var DefaultInputEncParsers = InputEncParsers{
+	"json": DefaultJsonParsers,
+	"raw":  DefaultRawParsers,
+}


I am not sure if making this global is the right decision.
This will make testing it harder.

If we dont use a global, registration becomes a much more complicated thing to do. Any ideas on a good way to do that?

Can't we do the same thing we're doing with BlockDecoders (pass it to register)?

Yeah, I am planning that.

BlockDecoder is doing the same (it is using DefaultBlockDecoder from go-ipld-format).

Good point. The problem isn't really registration, it's using the the decoders (e.g. in ParseInputs).

In the Plugin API for IPLD I've sketched out I've made the registration map an argument so at least the plugins can be tested.

We have to figure out how to use the decoders well with non-global states.

Kubuxu · 2017-07-05T13:25:44Z

cmd/ipfs/plugin_linux.go

+		regfunci, err := plugin.Lookup("Register")
+		if err != nil {
+			return fmt.Errorf("ipld plugin had no Register function: %s", err)
+		}


Regarding API for plugins themselves, I think it might be better to have a plugin expose a structure of callback functions, implemented interface or overridden structure. This way the API is cleanly defined in Go code instead of having to be written somewhere separately.

Hrm... so we could maybe do something like:

plug, _ := plugin.Lookup("Plugin") switch plug := plug.(type) { case IpldPlugin: case Libp2pPlugin: case DatastorePlugin: default: // unsupported plugin type! }

That would work.

Kubuxu · 2017-07-05T13:27:18Z

Once we decide on structure more I will add Makefile support for it.

Stebalien · 2017-07-07T21:28:22Z

This may be beyond the scope of this PR but it would be really nice to make encoders/decoders composable and separate. That is, when we re-encode, always decode to a common (in-memory) format/interface and then re-encode to the target format. Unfortunately, the only way I think of doing this in go is to decode to a dynamically typed interface{} of maps, arrays, integers, strings, etc... So, how about something like ipfs/go-ipld-format#18.

Kubuxu · 2017-07-09T18:48:46Z

This should be ready for first CR, I will try adding some tests and more UX regarding plugins on other systems (display warnings if there is a plugin in directory but system is not supported).

Kubuxu · 2017-07-09T19:11:34Z

Hah, I can't request review from @whyrusleeping's own PR.

whyrusleeping · 2017-07-10T19:16:48Z

would be pretty cool if github had a 'take over PR' button

whyrusleeping · 2017-07-10T19:20:20Z

plugin/loader/load_linux.go

+	typePls, ok := pls.([]iplugin.Plugin)
+	if !ok {
+		return nil, errors.New("filed 'Plugins' didn't contain correct type")
+	} else {


no need for the else

whyrusleeping · 2017-07-10T19:21:36Z

would be pretty cool if github had a 'take over PR' button.

You also probably don't want to commit the plugin git.so binary.

Kubuxu · 2017-07-10T19:23:25Z

Oops.

License: MIT Signed-off-by: Jeromy <[email protected]>

License: MIT Signed-off-by: Jakub Sztandera <[email protected]>

Kubuxu · 2017-07-13T21:09:41Z

Whoa, tests are green. Git IPLD plugin is being loaded dynamically and works (passes sharness tests written by @magik6k).

Should be ready for CR: @whyrusleeping @Stebalien

This PR contains quite a bit of Makefile magic to glue plugins and tests together, provide option for preloading plugins (plugin/loader/preload_list) and allow for dynamic loading of plugins and preloading them (to load plugin dynamically it has to be defined in a main package, to preload it can't).

Kubuxu · 2017-07-13T21:10:07Z

I just saw that coverage collection failed, so please do not merge I have to fix it first.

Stebalien

What I've looked at so far looks good. There are a few things I'd like to see improved eventually but they're not reasons to hold this up.

I'm a bit concerned about loading plugins from a directory owned by the user IPFS is running under but this is very convenient (and probably not a problem). However, in the future, we'll probably want to allow loading plugins out of /usr/lib/plugins/... (for distros and package managers).

Now, it would be awesome if we could load plugins directly from IPFS but we can discuss that later.

I really don't like having one stream parser per input-output pair. However, doing this the correct way would require quit a bit of work and thought so I'm going to say that this is good enough for now.

Stebalien · 2017-07-13T22:00:30Z

plugin/loader/load_linux.go

+)
+
+func init() {
+	loadPluginsFunc = linxuLoadFunc


s/linxu/linux/g

Stebalien · 2017-07-13T22:20:38Z

plugin/loader/initializer.go

+	format "gx/ipfs/QmYNyRZJBUYPNrLszFmrBrPJbsBh2vMsefz5gnDpB5M1P6/go-ipld-format"
+)
+
+func initalize(plugins []plugin.Plugin) error {


s/initalize/initialize/g

License: MIT Signed-off-by: Jakub Sztandera <[email protected]>

Kubuxu · 2017-07-14T10:17:58Z

I don't think it is possible for me to build plugins with coverage collection. This means I will either have to preload plugins for coverage collection or disable plugins during coverage collection.

I think for now I will disable plugins and plugin tests during coverage collection.

License: MIT Signed-off-by: Jakub Sztandera <[email protected]>

Kubuxu · 2017-07-14T15:24:57Z

Coverage of the patch is low as the sharness won't work well with the plugins.
In future I might work on running go-ipfs with plugins preloaded for testing.

I can't test the dynamic plugin loading in the sharness and this is the part that misses coverage the most.

whyrusleeping · 2017-07-15T22:01:18Z

plugin/loader/initializer.go

+	return nil
+}
+
+func runIPLDPlugin(pl plugin.Plugin) error {


do a type switch up in the 'run' function, and have this function accept the proper type. Then we can error out if the types don't match correctly instead of silently ignoring the problem (which could easily happen with a mis-compiled plugin)

I was thinking about it, but a Plugin could in theory implement multiple interfaces, then typeswich isn't really helping us as it will choose just one interface (the first in the switch statement).

Miss-compiled plugin will error out or log if: the plugin lib can't be loaded, there is no Plugins variable, it has wrong type.

whyrusleeping · 2017-07-15T22:01:39Z

plugin/loader/initializer.go

+	}
+
+	var err error
+	err = ipldpl.RegisterBlockDecoders(format.DefaultBlockDecoder)


why not just err := .... ?

whyrusleeping · 2017-07-15T22:01:58Z

plugin/loader/initializer.go

+		return err
+	}
+
+	err = ipldpl.RegisterInputEncParsers(coredag.DefaultInputEncParsers)


return ipldpl.Register....

whyrusleeping · 2017-07-15T22:05:17Z

plugin/plugins/.gitignore

@@ -0,0 +1,2 @@
+*.so
+*/main


what is this main ignore for?

This is autogenerated package in directory of every plugin. This is what I have to do so the plugin can be either pre-loaded (statically compiled with go-ipfs, for non-linux builds) or be compiled for dynamic linking as plugin.

Go requires plugins to be defined in package named main, and at the same time it doesn't allow for static linking of main package. So I have main plugin inside its own package and I generate main package that rebinds Plugins variable.

whyrusleeping · 2017-07-15T22:07:52Z

test/sharness/t0280-plugin-git.sh

@@ -0,0 +1,57 @@
+#!/bin/sh
+#
+# Copyright (c) 2016 Jakub Sztandera


its 2017 btw

whyrusleeping · 2017-07-15T22:09:12Z

This LGTM, great work @Kubuxu :)

I'll wait for your confirmation before merging, but I think this is ready to go.

License: MIT Signed-off-by: Jakub Sztandera <[email protected]>

whyrusleeping added the status/in-progress In progress label Jul 4, 2017

Kubuxu reviewed Jul 5, 2017

View reviewed changes

cmd/ipfs/plugin_linux.go Outdated

LoadPlugins = LoadPluginsLinux

}

func LoadPluginsLinux(cfgroot string) error {

Copy link

Member

Kubuxu Jul 5, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this can be private.

Kubuxu reviewed Jul 5, 2017

View reviewed changes

whyrusleeping mentioned this pull request Jul 5, 2017

Use block decoding framework #4040

Closed

Kubuxu force-pushed the feat/git-plugin branch 2 times, most recently from 510f5c1 to 8c84c76 Compare July 9, 2017 18:36

Kubuxu force-pushed the feat/git-plugin branch from 8c84c76 to 76b2a96 Compare July 9, 2017 19:17

whyrusleeping commented Jul 10, 2017

View reviewed changes

Kubuxu force-pushed the feat/git-plugin branch from 76b2a96 to 39e4854 Compare July 10, 2017 19:52

Kubuxu changed the title ~~Feat/git plugin~~ plugin: create plugin API and loader, add ipld-git plugin Jul 10, 2017

Kubuxu force-pushed the feat/git-plugin branch 6 times, most recently from a91be3c to 9b2cfae Compare July 12, 2017 18:50

whyrusleeping and others added 2 commits July 12, 2017 21:53

plugin: create plugin API and loader, add ipld-git plugin

7203c43

License: MIT Signed-off-by: Jeromy <[email protected]>

test: add TEST_NO_PLUGIN and PLUGIN prereq

285a4d8

License: MIT Signed-off-by: Jakub Sztandera <[email protected]>

Kubuxu added 2 commits July 13, 2017 08:49

make codeclimate more happy

b067972

License: MIT Signed-off-by: Jakub Sztandera <[email protected]>

plugin: make the plugin preload work again

c569b0b

License: MIT Signed-off-by: Jakub Sztandera <[email protected]>

Kubuxu force-pushed the feat/git-plugin branch from 20f500f to c569b0b Compare July 13, 2017 19:16

Kubuxu added 5 commits July 13, 2017 21:57

plugin: fix error when plugins dir does not exist

e293de1

License: MIT Signed-off-by: Jakub Sztandera <[email protected]>

misc: make codecov really happy

314d2c3

License: MIT Signed-off-by: Jakub Sztandera <[email protected]>

Disable ipldgit preload

be5ee0b

License: MIT Signed-off-by: Jakub Sztandera <[email protected]>

misc: add go fmt to main packages of plugins

b135a1f

License: MIT Signed-off-by: Jakub Sztandera <[email protected]>

plugin: fix plugin loading

6da6e77

License: MIT Signed-off-by: Jakub Sztandera <[email protected]>

Stebalien reviewed Jul 13, 2017

View reviewed changes

Kubuxu added 2 commits July 14, 2017 10:05

plugin: fix typos

fb618a2

License: MIT Signed-off-by: Jakub Sztandera <[email protected]>

coverage: add dependency on DEPS_GO for coverage

36cbfa4

License: MIT Signed-off-by: Jakub Sztandera <[email protected]>

Kubuxu added 3 commits July 14, 2017 12:18

coverage: disable plugin tests during sharness coverage collection

2df6186

License: MIT Signed-off-by: Jakub Sztandera <[email protected]>

sharness: add debug output for TEST_NO_PLUGIN

b5ea00e

License: MIT Signed-off-by: Jakub Sztandera <[email protected]>

fix typo

3e8e039

License: MIT Signed-off-by: Jakub Sztandera <[email protected]>

Kubuxu mentioned this pull request Jul 14, 2017

gx: update ipldcbor #4067

Merged

whyrusleeping commented Jul 15, 2017

View reviewed changes

misc: small code style edit

c0ee7d6

License: MIT Signed-off-by: Jakub Sztandera <[email protected]>

whyrusleeping merged commit 71bb417 into master Jul 16, 2017

whyrusleeping deleted the feat/git-plugin branch July 16, 2017 09:02

whyrusleeping removed the status/in-progress In progress label Jul 16, 2017

magik6k added this to the Ipfs 0.4.11 milestone Aug 13, 2017

		@@ -0,0 +1,2 @@
		*.so
		*/main

plugin: create plugin API and loader, add ipld-git plugin #4033

plugin: create plugin API and loader, add ipld-git plugin #4033

Conversation

whyrusleeping commented Jul 4, 2017

whyrusleeping commented Jul 4, 2017

victorb commented Jul 4, 2017

whyrusleeping commented Jul 4, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kubuxu commented Jul 5, 2017

Stebalien commented Jul 7, 2017

Kubuxu commented Jul 9, 2017

Kubuxu commented Jul 9, 2017

whyrusleeping commented Jul 10, 2017

Choose a reason for hiding this comment

whyrusleeping commented Jul 10, 2017

Kubuxu commented Jul 10, 2017

Kubuxu commented Jul 13, 2017

Kubuxu commented Jul 13, 2017

Stebalien left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kubuxu commented Jul 14, 2017

Kubuxu commented Jul 14, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

whyrusleeping commented Jul 15, 2017

Kubuxu commented Jul 14, 2017 •

edited

Loading