Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added hyperkit options for enterprise VPN support #9

Closed
wants to merge 1 commit into from

Conversation

seborama
Copy link

@seborama seborama commented May 28, 2018

The purpose of these changes is to enhance Hyperkit support from the
minikube command line for better integration with enterprise networks
behind a VPN.

uuid: Provide VM UUID to restore MAC address (only supported with
Hyperkit driver).
vpnkitSock: Location of the VPNKit socket used for networking. If empty,
disables Hyperkit VPNKitSock, if 'auto' uses Docker for Mac
VPNKit connection, otherwise uses the specified VSock."
vsockPorts: List of guest VSock ports that should be exposed as sockets
on the host (Only supported on with hyperkit now).

The purpose of these changes is to enhance Hyperkit support from the
minikube command line for better integration with enterprise networks
behind a VPN.

uuid: Provide VM UUID to restore MAC address (only supported with
      Hyperkit driver).
vpnkitSock: Location of the VPNKit socket used for networking. If empty,
            disables Hyperkit VPNKitSock, if 'auto' uses Docker for Mac
            VPNKit connection, otherwise uses the specified VSock."
vsockPorts: List of guest VSock ports that should be exposed as sockets
            on the host (Only supported on with hyperkit now).

Note:
tests pass but file:
`vendor/github.com/google/certificate-transparency/go/x509/root_darwin.go`
has to be edited to correct an issue - not committed since this is in
the vendor directory.
Copy link
Collaborator

@praveenkumar praveenkumar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good to me, we do need to test this piece and merge it.

@@ -103,7 +107,7 @@ func (d *Driver) Create() error {

isoPath := d.ResolveStorePath(isoFilename)
if err := d.extractKernel(isoPath); err != nil {
return err
return errors.Wrap(err, "extracting kernel")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@@ -175,25 +179,32 @@ func (d *Driver) Restart() error {

// Start a host
func (d *Driver) Start() error {
h, err := hyperkit.New("", "", filepath.Join(d.StorePath, "machines", d.MachineName))
h, err := hyperkit.New("", d.VpnKitSock, filepath.Join(d.StorePath, "machines", d.MachineName))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@seborama
Copy link
Author

seborama commented Jun 1, 2018

Hi,

With regards to testing, below are examples of ~/.minikube/machines/minikube/hyperkit.json with different minikube start options.

EXAMPLE 1

./out/minikube-darwin-amd64 start --vm-driver=hyperkit --cache-images --memory=6144 --logtostderr --loglevel=0 -v=5 --uuid=8de16914-60d3-11e8-b5f4-784f438fc458 --hyperkit-vpnkit-sock=auto --hyperkit-vsock-ports="2376,8443,30000,8080" && ./out/minikube-darwin-amd64 ip

~/.minikube/machines/minikube/hyperkit.json:
Notice the values of the uuid, vsock, vsock_ports, vpnkit_sock. Also, the VM was given the same IP address as previously given for this uuid.
{"hyperkit":"/usr/local/bin/hyperkit","argv0":"","state_dir":"/Users/someone/.minikube/machines/minikube","vpnkit_sock":"/Users/someone/Library/Containers/com.docker.docker/Data/s50","vpnkit_uuid":"","vpnkit_preferred_ipv4":"","uuid":"8de16914-60d3-11e8-b5f4-784f438fc458","disks":[{"path":"/Users/someone/.minikube/machines/minikube/minikube.rawdisk","size":20000,"format":"","driver":"virtio-blk"}],"iso":["/Users/someone/.minikube/machines/minikube/boot2docker.iso"],"vsock":true,"vsock_ports":[2376,8443,30000,8080],"vsock_guest_cid":3,"vmnet":true,"9p_sockets":null,"kernel":"/Users/someone/.minikube/machines/minikube/bzImage","initrd":"/Users/someone/.minikube/machines/minikube/initrd","bootrom":"","cpus":2,"memory":6144,"console":1,"extra_files":null,"pid":6305,"arguments":["-A","-u","-F","/Users/someone/.minikube/machines/minikube/hyperkit.pid","-c","2","-m","6144M","-s","0:0,hostbridge","-s","31,lpc","-s","1:0,virtio-vpnkit,path=/Users/someone/Library/Containers/com.docker.docker/Data/s50","-s","2:0,virtio-net","-U","8de16914-60d3-11e8-b5f4-784f438fc458","-s","3:0,virtio-blk,/Users/someone/.minikube/machines/minikube/minikube.rawdisk","-s","4,virtio-sock,guest_cid=3,path=/Users/someone/.minikube/machines/minikube,guest_forwards=2376;8443;30000;8080","-s","5,ahci-cd,/Users/someone/.minikube/machines/minikube/boot2docker.iso","-s","6,virtio-rnd","-l","com1,autopty=/Users/someone/.minikube/machines/minikube/tty,log=/Users/someone/.minikube/machines/minikube/console-ring","-f","kexec,/Users/someone/.minikube/machines/minikube/bzImage,/Users/someone/.minikube/machines/minikube/initrd,earlyprintk=serial loglevel=3 user=docker console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 systemd.legacy_systemd_cgroup_controller=yes base host=minikube"],"cmdline":"/usr/local/bin/hyperkit -A -u -F /Users/someone/.minikube/machines/minikube/hyperkit.pid -c 2 -m 6144M -s 0:0,hostbridge -s 31,lpc -s 1:0,virtio-vpnkit,path=/Users/someone/Library/Containers/com.docker.docker/Data/s50 -s 2:0,virtio-net -U 8de16914-60d3-11e8-b5f4-784f438fc458 -s 3:0,virtio-blk,/Users/someone/.minikube/machines/minikube/minikube.rawdisk -s 4,virtio-sock,guest_cid=3,path=/Users/someone/.minikube/machines/minikube,guest_forwards=2376;8443;30000;8080 -s 5,ahci-cd,/Users/someone/.minikube/machines/minikube/boot2docker.iso -s 6,virtio-rnd -l com1,autopty=/Users/someone/.minikube/machines/minikube/tty,log=/Users/someone/.minikube/machines/minikube/console-ring -f kexec,/Users/someone/.minikube/machines/minikube/bzImage,/Users/someone/.minikube/machines/minikube/initrd,earlyprintk=serial loglevel=3 user=docker console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 systemd.legacy_systemd_cgroup_controller=yes base host=minikube"}

EXAMPLE 2

./out/minikube-darwin-amd64 start --vm-driver=hyperkit --cache-images --memory=6144 --logtostderr --loglevel=0 -v=5 && ./out/minikube-darwin-amd64 ip

~/.minikube/machines/minikube/hyperkit.json:
A new IP was provided, no more vsock activation
{"hyperkit":"/usr/local/bin/hyperkit","argv0":"","state_dir":"/Users/someone/.minikube/machines/minikube","vpnkit_sock":"","vpnkit_uuid":"","vpnkit_preferred_ipv4":"","uuid":"4b8107f3-656a-11e8-9069-784f438fc457","disks":[{"path":"/Users/someone/.minikube/machines/minikube/minikube.rawdisk","size":20000,"format":"","driver":"virtio-blk"}],"iso":["/Users/someone/.minikube/machines/minikube/boot2docker.iso"],"vsock":false,"vsock_ports":null,"vsock_guest_cid":3,"vmnet":true,"9p_sockets":null,"kernel":"/Users/someone/.minikube/machines/minikube/bzImage","initrd":"/Users/someone/.minikube/machines/minikube/initrd","bootrom":"","cpus":2,"memory":6144,"console":1,"extra_files":null,"pid":8607,"arguments":["-A","-u","-F","/Users/someone/.minikube/machines/minikube/hyperkit.pid","-c","2","-m","6144M","-s","0:0,hostbridge","-s","31,lpc","-s","1:0,virtio-net","-U","4b8107f3-656a-11e8-9069-784f438fc457","-s","2:0,virtio-blk,/Users/someone/.minikube/machines/minikube/minikube.rawdisk","-s","3,ahci-cd,/Users/someone/.minikube/machines/minikube/boot2docker.iso","-s","4,virtio-rnd","-l","com1,autopty=/Users/someone/.minikube/machines/minikube/tty,log=/Users/someone/.minikube/machines/minikube/console-ring","-f","kexec,/Users/someone/.minikube/machines/minikube/bzImage,/Users/someone/.minikube/machines/minikube/initrd,earlyprintk=serial loglevel=3 user=docker console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 systemd.legacy_systemd_cgroup_controller=yes base host=minikube"],"cmdline":"/usr/local/bin/hyperkit -A -u -F /Users/someone/.minikube/machines/minikube/hyperkit.pid -c 2 -m 6144M -s 0:0,hostbridge -s 31,lpc -s 1:0,virtio-net -U 4b8107f3-656a-11e8-9069-784f438fc457 -s 2:0,virtio-blk,/Users/someone/.minikube/machines/minikube/minikube.rawdisk -s 3,ahci-cd,/Users/someone/.minikube/machines/minikube/boot2docker.iso -s 4,virtio-rnd -l com1,autopty=/Users/someone/.minikube/machines/minikube/tty,log=/Users/someone/.minikube/machines/minikube/console-ring -f kexec,/Users/someone/.minikube/machines/minikube/bzImage,/Users/someone/.minikube/machines/minikube/initrd,earlyprintk=serial loglevel=3 user=docker console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 systemd.legacy_systemd_cgroup_controller=yes base host=minikube"}

@seborama
Copy link
Author

seborama commented Jun 2, 2018

Sorry, I have found one problem with the parsing of ports. They're defined as int rather than uint16 by Hyperkit and the conversion sometimes fails.
Investigating options...

@seborama
Copy link
Author

seborama commented Jun 2, 2018

OK, it seems the problem may have been caused by me. I have 3 versions of the driver and 2 of minikube owing to the forked repos and one from homebrew.
I've re-sync'ed both minikube and the hyperkit driver from a fresh compilation and the problem has gone away. I can only assume that different versions caused a clash.

@LalatenduMohanty
Copy link

@seborama Thanks for the PR, it will be helpful for other projects (including Minishift) using hyperkit.

@seborama
Copy link
Author

seborama commented Jun 4, 2018

Hi @LalatenduMohanty / @dlorenc / @praveenkumar ,
More work would be necessary in a future PR to improve the experience. I'm trying to understand how the minikube VM IP is generated. I seem to gather that the docker hyperkit process requests the IP from the DHCP server based on the generated MAC address of the minikube VM. Is this correct?
Could you confirm and enrich my understanding?
Several scenarios exist currently which lead to failure when starting minikube as the IP is not generated for the minikube - Hyperkit configuration.

@praveenkumar
Copy link
Collaborator

I seem to gather that the docker hyperkit process requests the IP from the DHCP server based on the generated MAC address of the minikube VM. Is this correct?

Yes that's correct https://github.com/machine-drivers/docker-machine-driver-hyperkit/blob/master/pkg/hyperkit/network.go#L48-L65

@praveenkumar
Copy link
Collaborator

I can also see this PR is merged from minikube side so @seborama do you think we can merge it here also and then gradually built it by resolving the issues.

@seborama
Copy link
Author

seborama commented Jun 7, 2018

Hi Praveen,

Thanks for your reply.

I'd say so yes. The code itself works fine as far as I can tell.

The intricacies (or my understanding thereof 😛) come from how the Minikube VM's IP address is allocated by Hyperkit. Starting a new machine with vsock for the first time won't allocate an IP. The machine needs creating once then deleted and finally re-created with vsock and the uuid of the defunct machine. Awkward. I need to look into the Hyperkit code to gain a better understanding.

@nanzhong
Copy link

Is this PR ready to be merged over on this side? I currently make heavy use of this and and building the driver from source, but it would be nice to see these changes merged.

@seborama
Copy link
Author

It is for this iteration.
I think that further enhancements can be part of separate PR's. For instance, we found that routing in the Minikube K8s cluster to external addresses is not as functional as with say VirtualBox for instance.

@seborama
Copy link
Author

Hi @praveenkumar,

Is there anything outstanding on this PR?

Cheers

@seborama
Copy link
Author

seborama commented Jul 24, 2018

Hey @nanzhong ,

Currently the code from this PR exists both in the minikube repo (already merged by @dlorenc ) and this repo (this PR, pending).

This means that you can compile the drivers from the Minikube repo (use make driver and optionally make install install-hyperkit-driver or copy the file file manually from the out directory and apply the correct permissions).

HOWEVER... As documented at https://github.com/kubernetes/minikube/blob/master/docs/drivers.md#hyperkit-driver, I've had a look at the binary distribution at https://storage.googleapis.com/minikube/releases/latest/docker-machine-driver-hyperkit.

The binary has been updated from the minikube repo. So you don't need to compile it yourself if you don't want to, just follow the steps in the doc from the minikube project (above link).

Best of luck.

PS: details of the driver update as reported at https://storage.googleapis.com/minikube/:
(in short: yesterday)

<Contents>
<Key>releases/latest/docker-machine-driver-hyperkit</Key>
<Generation>1532362270225121</Generation>
<MetaGeneration>1</MetaGeneration>
<LastModified>2018-07-23T16:11:10.224Z</LastModified>
<ETag>"6c90be83e9882c5bd838e2bb8461dc59"</ETag>
<Size>26811748</Size>
</Contents>

@nanzhong
Copy link

A perfect, thanks!

@seborama
Copy link
Author

Hi,
This PR is stale. I'm closing it.
Best regards.

@seborama seborama closed this Aug 23, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants