Skip to content
This repository has been archived by the owner on Sep 26, 2021. It is now read-only.

Proposal: Driver Plugins #1626

Closed
ehazlett opened this issue Jul 31, 2015 · 44 comments
Closed

Proposal: Driver Plugins #1626

ehazlett opened this issue Jul 31, 2015 · 44 comments
Assignees
Milestone

Comments

@ehazlett
Copy link
Contributor

This proposal is to start a discussion (finally) around Machine driver plugins. We have discussed this in passing before, but due to the influx in driver PRs we have pushed this up and have this slated for 0.5.

Background

Docker Machine offers an interface for creating drivers. This allowed for relatively easy driver creation which caused great feedback and response with drivers working with varying hypervisors and cloud providers. However, at this time it is proving to be extremely difficult for us to keep up with reviewing and testing each of these drivers for inclusion in the Machine core. We really want to switch to a more pluggable model, as well as polish up a few things about the driver model which need to be changed to ensure a smooth and sustainable future.

Implementation

To re-iterate, this is meant to be the start of a discussion. I'm just putting ideas here as starting blocks.

Plugin Directory

Machine Driver plugins will be stored in a known directory (i.e. ~/.docker/machine/plugins). For the first iteration, Machine will simply list the available files in the directory and use this list as the available plugins.

Binaries

Each plugin will be a binary. This makes it easy to distribute and execute.

Communication

Plugin communication would happen via stdin and stdout as a JSON stream. We will need to design the spec but something like this:

{
  "action": "start",
  "machine": "dev",
  "args": null
}

This would be sent as stdin to the plugin. The plugin would respond with something like:

{
  "exit_code": 0,
  "errors": [],
  "output": [
    "Machine created successfully"
  ]
}

Distribution

For the first iteration we would not include a distribution mechanism. We would maintain a list on the wiki of available drivers and their location, maintainer, etc.

@timfallmk
Copy link

To play devil's advocate for a moment, would a full fledged API (and spec) be better than distributing binaries and relying on people to get the stdin/out communication implemented properly?

@nathanleclaire
Copy link
Contributor

To play devil's advocate for a moment, would a full fledged API (and spec) be better than distributing binaries and relying on people to get the stdin/out communication implemented properly?

Could you elaborate a bit more on how you envision this working?

@nathanleclaire
Copy link
Contributor

Something I thought of about driver plugins today: I think that we (the Docker Machine team) should maintain some kind of "template" / "boilerplate" repo for developing a driver plugin. It would include a Makefile, Dockerfile, etc. and instructions on how to develop and integrate your plugin. That way, we could have problems like cross-compilation (and possibly Github release automation etc. which might be a pain for others to manage manually) solved out of the box for plugin authors.

@hairyhenderson
Copy link
Contributor

Plugin communication would happen via stdin and stdout as a JSON stream.

My gut reaction to this was "why not use unix domain sockets?" but I think I do like stdin/stdout as one of the simplest possible implementations. The key is going to be making sure we handle garbage input gracefully.

For the first iteration we would not include a distribution mechanism.

Hmmm... While I agree with the sentiment, this is going to make it significantly more difficult for new users of Machine to get started than it is now. I'm not sure it'd be that difficult to implement some kind of docker-machine plugin get github.com/exoscale/docker-machine-driver type of experience using the GitHub releases API and some a few conventions around binary name, etc...

But, I haven't thought deeply about it yet - there may be some significant roadblocks that I haven't considered ;)

Also - I'm assuming this design means driver plugins can be written in any language, or is there a reason we should restrict support to drivers written in Go?

@nathanleclaire
Copy link
Contributor

My gut reaction to this was "why not use unix domain sockets?"

The issue here is mainly Windows. I like Docker's "auto-detect if there's a socket in a certain directory" model, but I don't think UNIX sockets work on Windows, and using TCP sockets introduces all kinds of fun new complications ("How do I get an open port and communicate what that is" etc.).

I'm not sure it'd be that difficult to implement some kind of docker-machine plugin get github.com/exoscale/docker-machine-driver type of experience using the GitHub releases API and some a few conventions around binary name, etc...

I have a feeling this will be trickier than it looks, for a variety of reasons. It's a lot of moving parts and we want to get the local experience relatively solid before introducing distribution to the model. Distribution will be much easier to add in than to change or take out, IMO.

Also - I'm assuming this design means driver plugins can be written in any language, or is there a reason we should restrict support to drivers written in Go?

Depending on the model we end up going with, maybe. If we support a pure STDIN/STDOUT model, it seems theoretically feasible. However, all of the wrappers and UX niceness around development, will be Go-oriented (e.g. "Just fulfill this interface and the libmachine dependency will handle boilerplate demarshalling etc. for you!").

@hairyhenderson
Copy link
Contributor

but I don't think UNIX sockets work on Windows

Right... Pesky old Windows 😉

using TCP sockets introduces all kinds of fun new complications ("How do I get an open port and communicate what that is" etc.).

Communicating open ports isn't so hard - either the parent process can open the port before execing the child process, and then communicate that as a command line flag, or the child process can report its port on stdout.

However, I do agree that stdout/stdin is the simplest. 😄

we want to get the local experience relatively solid before introducing distribution to the model

Ok, that's fair.

@thaJeztah
Copy link
Member

perhaps @jhowardmsft and @ahmetalpbalkan can chime in for additional ideas on the Windows front?

@rickard-von-essen
Copy link

I would like to propose that you look into the architecture of Packer's plugins system.

See for some docs see docs - plugins and docs - developing plugins.

To summarise the code/architecture a bit:

The main benefit of designing it this way is that it gives a Go interface to implement agains with static type checking etc and also probably makes it much easier to handle messages and debug logging in a uniform and good way.

PS. I think it is extremely beneficial to quickly get a simple distribution system up and running. Something as simple as:

docker-machine plugin install parallells/machine-parallels

as soon as there is a usable plugin system.

@ahmetb
Copy link
Contributor

ahmetb commented Aug 3, 2015

So far @rickard-von-essen's idea sounds good and can support Windows too (where Unix sockets are missing). I think implementing a wire-protocol RPC would be an extra thing to worry about whereas we can implement a HTTP server that has a JSON API very easily.

In such a setup, what will go wrong in the long run is versioning, as Driver interface changes, the API/RPC interface will need to keep up.

To address that, we can provide a “stub/base implementation” of a Machine Driver API Server in Go and by keeping the driver interface closely tied to the Go API Server interface, and we can even ensure stuff like type safety in those (e.g. missing methods) ––and even possibly API versioning like /v1/....

+1 for a need of distribution system. My comment on that is, a path like ~/.docker/machine/plugins is rarely a good one to store binaries (e.g. if you switch to sudo from your user, you'll need to install the plugin again).

Are there any proposals about how the arguments of each driver will be discovered and exposed on machine create?

@tianon
Copy link
Contributor

tianon commented Aug 3, 2015

@ghost
Copy link

ghost commented Aug 5, 2015

I also think stdin/stdout is simplest.
Then, for large-scale docker machine deployment, could docker/libchan be an option?

@crunchywelch
Copy link
Contributor

I'd like to echo @timfallmk here, if the issue is a lack of time and resources to do code reviews and managing the volume of contributions, why not ask for volunteers to focus on this part of the project? I'd be happy to review and get PRs into good shape before handing off to the maintainers to merge, I'm sure there are other folks who would also help out.

This seems like adding complexity to the project in an effort to save effort, but may end up just being as much work (or more) in the end. Right now there are a lot of activity around provider drivers, but I suspect that would tail off relatively quickly. libcloud, which is arguably the most comprehensive cloud driver library, only has 102 compute drivers by my count, which is a lot, but not unmanageable. Trying to coordinate compiled binaries and/or some other plugin system seems unwieldy and more difficult to manage, not less.

From a UX perspective, one of the advantages to docker machine currently is you can just drop the binary on your box and go from there, would this mean you'd now have to always enable a plugin for the providers you are using and then compile every time? Seems confusing and cumbersome for the end user. I'd love to hear from someone using docker downstream like @ibuildthecloud at Rancher to get their perspective on it, maybe they don't care.

I guess I don't see an argument that the current model is broken from an architecture standpoint, so why not just continue to refine and define the driver interface?

In any case, happy to contribute to whatever system ultimately is decided on, it's not that much code or time from my perspective, but this seems like not a great solution for the stated problem. If a different system is a requirement, then I'd say the one suggestion that seems the best so far (but probably also the most work) is @rickard-von-essen's but even then, seems like a lot of complexity for not a huge amount of upside IMO.

@FrenchBen
Copy link
Contributor

Just throwing a small stone into the machine, as I came across Heka
They use GO and LUA as plugin languages, which may help add more plugin support :)

@ehazlett
Copy link
Contributor Author

Here is the first (of hopefully monthly) Docker Machine community hangouts: https://plus.google.com/events/cs0iir01kd9ac2df7kv1k2uslts (Aug. 20 @ 3p EDT)

We will be discussing this as a main topic.

@miqui
Copy link
Contributor

miqui commented Aug 14, 2015

@ehazlett thanks!! +1

@ehazlett
Copy link
Contributor Author

It appears we will have more people joining than hangouts supports (awesome!!). We will use BlueJeans instead (this supports up to 100 participants). Thanks!

https://bluejeans.com/399850491

@miqui
Copy link
Contributor

miqui commented Aug 17, 2015

@ehazlett awesome!! +1

@hairyhenderson
Copy link
Contributor

As a possible starting point, we could steal terraform's RPC-based plugin model...

@crunchywelch
Copy link
Contributor

@hairyhenderson +1 on starting there. FWIW, I'm waiting on this PR for Packet support in terraform, it was pretty easy to port our machine driver to that:

hashicorp/terraform#2260

@nathanleclaire
Copy link
Contributor

For anyone interested in this, my progress on driver plugins can be seen here: nathanleclaire#3

I'd love to start getting some people tinkering with it, so if you're interested in writing a plugin, let me know ([email protected] is best) and additionally I'll start putting together a little guide.

It is based on the libmachine work so getting that merged in the next week or so is high priority for me.

  • N

@allingeek
Copy link
Contributor

Has there been any talks about Docker/Docker Machine plugin hosting on DockerHub? Or anything like a centralized list of plugins?

@nathanleclaire
Copy link
Contributor

Has there been any talks about Docker/Docker Machine plugin hosting on DockerHub? Or anything like a centralized list of plugins?

We've talked about a few things in this regard. In general, what we are doing is:

  1. For the immediate future, the Machine repo (in a Markdown file to be determined) will include a list of available plugins that anyone can fork and add theirs too if desired.
  2. Longer-term, investigating the possibility of running some plugins in containers (which would allow us to use Docker Hub for distribution).

@hairyhenderson
Copy link
Contributor

the possibility of running some plugins in containers (which would allow us to use Docker Hub for distribution).

...with the obvious caveat that this could present a catch-22 scenario... ;)

@nathanleclaire
Copy link
Contributor

...with the obvious caveat that this could present a catch-22 scenario... ;)

Indeed -- it doesn't make much sense to do it for local virtualization providers for a variety of reasons. They will likely always be binaries or similar.

@hairyhenderson
Copy link
Contributor

Now that #1902 has landed, should this be closed?

@dmp42
Copy link
Contributor

dmp42 commented Oct 19, 2015

\o/
Close!

@dmp42 dmp42 closed this as completed Oct 19, 2015
@thaJeztah
Copy link
Member

🎉

openstack-gerrit pushed a commit to openstack/openstack that referenced this issue Mar 21, 2017
Project: openstack-infra/project-config  c0e25e8bc5619bc1efba26c90b50ec80cdf12a77

Standalone Docker Machine driver for OpenStack

docker-machine folks have designed a driver API to be able
to plugin external drivers:
docker/machine#1626

There is existing code in that repo, however driver is outdated,
we should take over that code, update to latest dependencies
(gophercloud/gophercloud), make it work with minikube etc.

Change-Id: I3fb035a39243ad135ff5513f8a8b651a5fd13970
openstack-gerrit pushed a commit to openstack/project-config that referenced this issue Mar 21, 2017
docker-machine folks have designed a driver API to be able
to plugin external drivers:
docker/machine#1626

There is existing code in that repo, however driver is outdated,
we should take over that code, update to latest dependencies
(gophercloud/gophercloud), make it work with minikube etc.

Change-Id: I3fb035a39243ad135ff5513f8a8b651a5fd13970
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests