Proposal: Driver Plugins

[ehazlett]

This proposal is to start a discussion (finally) around Machine driver plugins. We have discussed this in passing before, but due to the influx in driver PRs we have pushed this up and have this slated for 0.5.

Background

Docker Machine offers an interface for creating drivers. This allowed for relatively easy driver creation which caused great feedback and response with drivers working with varying hypervisors and cloud providers. However, at this time it is proving to be extremely difficult for us to keep up with reviewing and testing each of these drivers for inclusion in the Machine core. We really want to switch to a more pluggable model, as well as polish up a few things about the driver model which need to be changed to ensure a smooth and sustainable future.

Implementation

To re-iterate, this is meant to be the start of a discussion. I'm just putting ideas here as starting blocks.

Plugin Directory

Machine Driver plugins will be stored in a known directory (i.e. ~/.docker/machine/plugins). For the first iteration, Machine will simply list the available files in the directory and use this list as the available plugins.

Binaries

Each plugin will be a binary. This makes it easy to distribute and execute.

Communication

Plugin communication would happen via stdin and stdout as a JSON stream. We will need to design the spec but something like this:

{
  "action": "start",
  "machine": "dev",
  "args": null
}

This would be sent as stdin to the plugin. The plugin would respond with something like:

{
  "exit_code": 0,
  "errors": [],
  "output": [
    "Machine created successfully"
  ]
}

Distribution

For the first iteration we would not include a distribution mechanism. We would maintain a list on the wiki of available drivers and their location, maintainer, etc.

[timfallmk]

To play devil's advocate for a moment, would a full fledged API (and spec) be better than distributing binaries and relying on people to get the stdin/out communication implemented properly?

[nathanleclaire]

To play devil's advocate for a moment, would a full fledged API (and spec) be better than distributing binaries and relying on people to get the stdin/out communication implemented properly?

Could you elaborate a bit more on how you envision this working?

[nathanleclaire]

Something I thought of about driver plugins today: I think that we (the Docker Machine team) should maintain some kind of "template" / "boilerplate" repo for developing a driver plugin. It would include a Makefile, Dockerfile, etc. and instructions on how to develop and integrate your plugin. That way, we could have problems like cross-compilation (and possibly Github release automation etc. which might be a pain for others to manage manually) solved out of the box for plugin authors.

[hairyhenderson]

Plugin communication would happen via stdin and stdout as a JSON stream.

My gut reaction to this was "why not use unix domain sockets?" but I think I do like stdin/stdout as one of the simplest possible implementations. The key is going to be making sure we handle garbage input gracefully.

For the first iteration we would not include a distribution mechanism.

Hmmm... While I agree with the sentiment, this is going to make it significantly more difficult for new users of Machine to get started than it is now. I'm not sure it'd be that difficult to implement some kind of docker-machine plugin get github.com/exoscale/docker-machine-driver type of experience using the GitHub releases API and some a few conventions around binary name, etc...

But, I haven't thought deeply about it yet - there may be some significant roadblocks that I haven't considered ;)

Also - I'm assuming this design means driver plugins can be written in any language, or is there a reason we should restrict support to drivers written in Go?

[nathanleclaire]

My gut reaction to this was "why not use unix domain sockets?"

The issue here is mainly Windows. I like Docker's "auto-detect if there's a socket in a certain directory" model, but I don't think UNIX sockets work on Windows, and using TCP sockets introduces all kinds of fun new complications ("How do I get an open port and communicate what that is" etc.).

I'm not sure it'd be that difficult to implement some kind of docker-machine plugin get github.com/exoscale/docker-machine-driver type of experience using the GitHub releases API and some a few conventions around binary name, etc...

I have a feeling this will be trickier than it looks, for a variety of reasons. It's a lot of moving parts and we want to get the local experience relatively solid before introducing distribution to the model. Distribution will be much easier to add in than to change or take out, IMO.

Also - I'm assuming this design means driver plugins can be written in any language, or is there a reason we should restrict support to drivers written in Go?

Depending on the model we end up going with, maybe. If we support a pure STDIN/STDOUT model, it seems theoretically feasible. However, all of the wrappers and UX niceness around development, will be Go-oriented (e.g. "Just fulfill this interface and the libmachine dependency will handle boilerplate demarshalling etc. for you!").

[hairyhenderson]

but I don't think UNIX sockets work on Windows

Right... Pesky old Windows 😉

using TCP sockets introduces all kinds of fun new complications ("How do I get an open port and communicate what that is" etc.).

Communicating open ports isn't so hard - either the parent process can open the port before execing the child process, and then communicate that as a command line flag, or the child process can report its port on stdout.

However, I do agree that stdout/stdin is the simplest. 😄

we want to get the local experience relatively solid before introducing distribution to the model

Ok, that's fair.

[thaJeztah]

perhaps @jhowardmsft and @ahmetalpbalkan can chime in for additional ideas on the Windows front?

[rickard-von-essen]

I would like to propose that you look into the architecture of Packer's plugins system.

See for some docs see docs - plugins and docs - developing plugins.

To summarise the code/architecture a bit:

• A plugin has to implement a plugin interface, e. g. packer/plugin/builder.go
• There is a plugin server component which handles all plugins, it uses TCP for Windows and UNIX socket for *NIX.
• The RPC implementation is implemented in package packer/rpc, e.g. packer/rpc/builder.go

The main benefit of designing it this way is that it gives a Go interface to implement agains with static type checking etc and also probably makes it much easier to handle messages and debug logging in a uniform and good way.

PS. I think it is extremely beneficial to quickly get a simple distribution system up and running. Something as simple as:

docker-machine plugin install parallells/machine-parallels

as soon as there is a usable plugin system.

[ahmetb]

So far @rickard-von-essen's idea sounds good and can support Windows too (where Unix sockets are missing). I think implementing a wire-protocol RPC would be an extra thing to worry about whereas we can implement a HTTP server that has a JSON API very easily.

In such a setup, what will go wrong in the long run is versioning, as Driver interface changes, the API/RPC interface will need to keep up.

To address that, we can provide a “stub/base implementation” of a Machine Driver API Server in Go and by keeping the driver interface closely tied to the Go API Server interface, and we can even ensure stuff like type safety in those (e.g. missing methods) ––and even possibly API versioning like /v1/....

+1 for a need of distribution system. My comment on that is, a path like ~/.docker/machine/plugins is rarely a good one to store binaries (e.g. if you switch to sudo from your user, you'll need to install the plugin again).

Are there any proposals about how the arguments of each driver will be discovered and exposed on machine create?

[tianon]

https https://golang.org/pkg/net/rpc/:// https://golang.org/pkg/net/rpc/
golang.org https://golang.org/pkg/net/rpc//pkg/net/
https://golang.org/pkg/net/rpc/rpc https://golang.org/pkg/net/rpc//
https://golang.org/pkg/net/rpc/ is probably worth taking a look at too
for possible rpc glue

[ghost]

I also think stdin/stdout is simplest.
Then, for large-scale docker machine deployment, could docker/libchan be an option?

[crunchywelch]

I'd like to echo @timfallmk here, if the issue is a lack of time and resources to do code reviews and managing the volume of contributions, why not ask for volunteers to focus on this part of the project? I'd be happy to review and get PRs into good shape before handing off to the maintainers to merge, I'm sure there are other folks who would also help out.

This seems like adding complexity to the project in an effort to save effort, but may end up just being as much work (or more) in the end. Right now there are a lot of activity around provider drivers, but I suspect that would tail off relatively quickly. libcloud, which is arguably the most comprehensive cloud driver library, only has 102 compute drivers by my count, which is a lot, but not unmanageable. Trying to coordinate compiled binaries and/or some other plugin system seems unwieldy and more difficult to manage, not less.

From a UX perspective, one of the advantages to docker machine currently is you can just drop the binary on your box and go from there, would this mean you'd now have to always enable a plugin for the providers you are using and then compile every time? Seems confusing and cumbersome for the end user. I'd love to hear from someone using docker downstream like @ibuildthecloud at Rancher to get their perspective on it, maybe they don't care.

I guess I don't see an argument that the current model is broken from an architecture standpoint, so why not just continue to refine and define the driver interface?

In any case, happy to contribute to whatever system ultimately is decided on, it's not that much code or time from my perspective, but this seems like not a great solution for the stated problem. If a different system is a requirement, then I'd say the one suggestion that seems the best so far (but probably also the most work) is @rickard-von-essen's but even then, seems like a lot of complexity for not a huge amount of upside IMO.

[FrenchBen]

Just throwing a small stone into the machine, as I came across Heka
They use GO and LUA as plugin languages, which may help add more plugin support :)

[ehazlett]

Here is the first (of hopefully monthly) Docker Machine community hangouts: https://plus.google.com/events/cs0iir01kd9ac2df7kv1k2uslts (Aug. 20 @ 3p EDT)

We will be discussing this as a main topic.

[miqui]

@ehazlett thanks!! +1

[ehazlett]

It appears we will have more people joining than hangouts supports (awesome!!). We will use BlueJeans instead (this supports up to 100 participants). Thanks!

https://bluejeans.com/399850491

[miqui]

@ehazlett awesome!! +1

[hairyhenderson]

As a possible starting point, we could steal terraform's RPC-based plugin model...

[crunchywelch]

@hairyhenderson +1 on starting there. FWIW, I'm waiting on this PR for Packet support in terraform, it was pretty easy to port our machine driver to that:

hashicorp/terraform#2260

[nathanleclaire]

For anyone interested in this, my progress on driver plugins can be seen here: nathanleclaire#3

I'd love to start getting some people tinkering with it, so if you're interested in writing a plugin, let me know ([email protected] is best) and additionally I'll start putting together a little guide.

It is based on the libmachine work so getting that merged in the next week or so is high priority for me.

• N

[allingeek]

Has there been any talks about Docker/Docker Machine plugin hosting on DockerHub? Or anything like a centralized list of plugins?

[nathanleclaire]

Has there been any talks about Docker/Docker Machine plugin hosting on DockerHub? Or anything like a centralized list of plugins?

We've talked about a few things in this regard. In general, what we are doing is:

1. For the immediate future, the Machine repo (in a Markdown file to be determined) will include a list of available plugins that anyone can fork and add theirs too if desired.
2. Longer-term, investigating the possibility of running some plugins in containers (which would allow us to use Docker Hub for distribution).

[hairyhenderson]

the possibility of running some plugins in containers (which would allow us to use Docker Hub for distribution).

...with the obvious caveat that this could present a catch-22 scenario... ;)

[nathanleclaire]

...with the obvious caveat that this could present a catch-22 scenario... ;)

Indeed -- it doesn't make much sense to do it for local virtualization providers for a variety of reasons. They will likely always be binaries or similar.

[hairyhenderson]

Now that #1902 has landed, should this be closed?

[dmp42]

\o/
Close!

[thaJeztah]

🎉