Quantify suitability of ESX agent to perform low-level tasks

[corrieb]

There are a number of architectural decisions that pivot on our decision whether or not to use an ESX agent to provide very low-level services to multiple tenants.

Possible functions/benefits of an ESX agent:

• vSocket support for tether interaction. This would eliminate our need to use serial-over-LAN for communications between tether and VCH. Serial-over-LAN currently inhibits vMotion and won't run on free ESX due to license restrictions.
• Authentication proxy. Not only would we do authentication out-of-band, but it would mean that the a VCH could be untrusted (not run in the VC management network). We would need to consider how the authenticating proxy might present in the container guestOS and a VMOMI gateway may well be the appropriate mechanism (guest sends SOAP requests to an endpoint which provides validation and authentication before forwarding).
• Out-of-band VMDK preparation. Currently VMDK prep is a bottleneck in the docker pull path, given the need to attach and detach disks to/from VMs in order to be able to write to them. When we have a viable solution for out-of-band VMDK prep, we will need an endpoint to delegate to.

The investigation work that needs to be done is:

• Write and deploy a HelloWorld agent to an ESX host. How involved is the toolchain / build process?
• Investigate mechanisms for installing/uninstalling/upgrading as part of the VIC product. What if a host is added to a cluster? Can the agent be pushed to the host automatically?
• In addition to this, we need to have some decisions around all of the above functions/benefits (timeframe, basic designs) so that we can decide how critical an agent might be in the short term.
• Are there other optimizations that the agent may be good for?
• What implications would the ESX agent have on container vMotion? How would the vMotioned guest re-attach to a new agent on a new host?
• A significant consideration is how we build a VIB without access to internal tool chains. Are there precedents for this from VMware partners? Are there libraries we can link to? How would we make an SDK available to OSS contributors to build against?

Using very simple passthrough API for the agents lets us be very API version agnostic w.r.t the contents of the stream.

[msterin]

FYI, some of this is already resolved in https://github.com/vmware/docker-vmdk-plugin (Go module for vSocket communication, ESX-side python code, vib package) and more is coming as the plugin is currently WIP.

[dougm]

Moving this to the backlog as the parent/epic issue with focus split into the referenced issues above.

[mdubya66]

still relevant @dougm