hyper-v: docs and feature to make easy NAT networking setup #1627

philips · 2017-06-21T21:56:55Z

Environment:

OS (e.g. from /etc/os-release): Windows 10
VM Driver hyperv
ISO version: -v0.19.1.i
Install tools:
Others:

What happened:

Setup a NAT on windows and booted minikube. Minikube gets no IP!!

What you expected to happen:

IP

How to reproduce it (as minimally and precisely as possible):

Create NAT: https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/user-guide/setup-nat-network

Use that switch:

minikube-windows-amd64.exe start --vm-driver=hyperv --hyperv-virtual-switch=NATSwitch --iso-url=https://storage.googleapis.com/minikube/iso/minikube-v0.19.1.iso  -v 9

If you run the demo DHCP server on the NAT everything works! https://github.com/krolaw/dhcp4

Anything else do we need to know:

I intend to turn this sketch into real docs and feature.

The text was updated successfully, but these errors were encountered:

gbraad · 2017-08-21T10:18:15Z

We have the same issue with Minishift. We have recommended people before to use a simple DHCP server on the virtual switch:

However, as I remember, this shortcoming of the Hyper-V Virtual Networking stack will be addressed in coming versions (just when is unknown).

I would rather find a solution in which the IPv6 address that actually gets assigned is usable. Or you didn't get one? Might be that the MobyLinuxVM (DFW) and CentOS image we use might react differently.

aaron-prindle · 2017-08-25T05:43:14Z

When I create a NAT using https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/user-guide/setup-nat-network

PS C:\users\admin\downloads> .\minikube-windows-amd64.exe start --vm-driver=hyperv --hyperv-virtual-switch="SwitchName"
Starting local Kubernetes v1.7.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Starting cluster components...
Connecting to cluster...
Setting up kubeconfig...
Kubectl is now configured to use the cluster.
PS C:\users\admin\downloads> .\minikube-windows-amd64.exe ssh

᭕$
$ ^C

$ ls
$ cd /
$ ls
bin   dev  home  lib    linuxrc  mnt  proc  run   srv  tmp  var
data  etc  init  lib64  media    opt  root  sbin  sys  usr
$ exit
logout
PS C:\users\admin\downloads> .\minikube-windows-amd64.exe ip
fe80::215:5dff:fe38:101

I get an ipv6 address for minikube but kubectl is unable to interact with the cluster.

PS C:\users\admin\downloads> .\kubectl.exe get po
Unable to connect to the server: dial tcp [fe80::215:5dff:fe38:101]:8443: connectex: No connection could be made because
 the target machine actively refused it.

It seems the VM can't reach the internet so the switch isn't configured properly

PS C:\users\admin\downloads> .\minikube-windows-amd64.exe ssh -- curl google.com
curl: (6) Couldn't resolve host 'google.com'

EDIT: it's the same issue as minishift/minishift#418

gbraad · 2017-08-25T05:47:55Z

@aaron-prindle minishift/minishift#418 We have the same issue. The IPv6 address does not allow connectivity either. Haven't been able to spend a lot of time on it yet, but when assigning a static IP address to the VM (or have DHCP hand a IPv4 address, www.dhcpserver.de) it will work as expected.

aaron-prindle · 2017-08-25T06:44:11Z

I configured the go dhcp server as you suggested and it appears to be working now.

PS C:\users\admin\downloads> .\minikube-windows-amd64.exe ip
172.30.0.39

It seems that the pods are still not resolving thought, need to investigate.

PS C:\users\admin\downloads> .\kubectl.exe get po -n kube-system
NAME                          READY     STATUS              RESTARTS   AGE
kube-addon-manager-minikube   0/1       ContainerCreating   0          5m

gbraad · 2017-08-29T01:15:24Z

There are some known limitations with the NAT implementation on Windows: https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/user-guide/setup-nat-network

Note: Currently, you are limited to one NAT network per host. For additional details on the Windows NAT (WinNAT) implementation, capabilities, and limitations, please reference the WinNAT capabilities and limitations blog

This means that when someone has installed Docker for Windows we might run into some problems when re-using the DockerNAT switch. We might be claiming IP addresses that are in use by the containers. We can of course use a smaller range starting from .128 and upwards... but this does not really solve the possible conflicts.

This is however the easiest solution...

gbraad · 2017-08-29T01:21:43Z

For Minishift I have been looking to use the Data Exchange service that Hyper-V adds to the VM. When the Hyper-V agents are installed in the guest, it is possible to exchange simple data structures using a key-value pair. Part of this spike can be found at: minishift/minishift#1293 This way it is possible to assign an IP address on start, with for instance a flag --ip-address. The VM will boot and read from /var/lib/hyperv/.kvp_pool_0 what keys are available. If no key is available, use DHCP. When a key 'IpAddress' is available, assign the static address. I think with minimal changes to the boot process of the LiveCD/ISO we are able to implement this without breaking current DHCP. The caveat... this is ONLY possible for Hyper-V.

Note; a solution for the other hypervisors will likely have to be a two stage boot when a static IP address is needed (pretty much like the IP change scripts we have seen as workarounds :-( )

gbraad · 2017-09-01T04:52:57Z

Note: Implementation task for a static IP address on HyperV and Minishift: minishift/minishift#1316

gbraad · 2017-09-27T10:26:33Z

PR with a possible solution is available at Minishift's repo for testing. Instructions are here: minishift/minishift#1316 (comment)
If results are usable, we will also integrate this in minikube and the iso.

Command looks like:

C:\> minishift.exe start ^
   --network-ipaddress 10.0.75.124 ^
   --network-gateway 10.0.75.1 ^
   --network-nameserver 8.8.8.8

briansmith · 2017-10-30T01:18:49Z

Upgrade to Windows 10 version 1703 (Creator's Update). This will add a "Default Switch" in Hyper-V that is NAT and which has a built-in DHCP (NAT didn't work in earlier versions of Hyper-V because it had no DHCP server).
minikube start --kubernetes-version="v1.8.0" --vm-driver="hyperv" --memory=4096 --cpus=4 --hyperv-virtual-switch="Default Switch" --v=7 --alsologtostderr

Note that step 2 might fail because of #754. Either disable IPv6 in the "Default Switch" adapter or try step 2 again (and again and again) until it selects an IPv4 address.

briansmith · 2017-12-04T10:55:00Z

Either disable IPv6 in the "Default Switch" adapter

Sadly that doesn't work. It seems I just got lucky the first few times I tried it.

gbraad · 2017-12-04T11:24:49Z

@briansmith I have setup many hosts (through installing 1703 as update) and I have never seen the "Default Switch" with DHCP. Also, I find no reference that the Creator's Update offers DHCP. I guess you mean to say 1709 (Fall Creator's update): https://blogs.technet.microsoft.com/virtualization/2017/11/13/whats-new-in-hyper-v-for-windows-10-fall-creators-update/. It is identified by ID: c08cb7b8-9b3c-408e-8e30-5e16a3aeb444

briansmith · 2017-12-04T11:42:41Z

I guess you mean to say 1709 (Fall Creator's update)

Yes, that's right.

gbraad · 2017-12-04T14:27:42Z

this means the documentation might have to include two situations.

pre Fall Creator's update (eg. 1703), to have the network specified
Fall Creator's update (1709), with use of the "Default Switch".

Note: the roll out of the Fall Creators update is still on-going, and not always applies successfully.

Will have to try using the "Default Switch" and see how IPv6 is handled.

gbraad · 2017-12-06T17:28:45Z

The same restrictions apply: https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/user-guide/setup-nat-network#troubleshooting Onlu one NAT network is available per host. You can verify if you have one with the command:

PS> Get-NetNAT

If you have a NAT network, it will render the "Default Switch" interoperable. It will assign an IP address, such as 172.26.219.111. I collect this information here: minishift/minishift#1792 (comment)

gbraad · 2017-12-22T08:00:16Z

@briansmith

Can you check what ends up in your /etc/resolve.conf when using the "Default Switch"?

djensen47 · 2018-01-03T23:34:56Z

I've read through this thread half a dozen times. I'm still not sure what I should try. I have Docker for Windows installed but if I try to use "DockerNAT" it seems to assign a IPv6 IP and just fail.

Any help would be appreciated.

gbraad · 2018-01-09T08:12:59Z

@briansmith Using

Get-NetIPInterface -InterfaceAlias "vEthernet (Default Switch)" -AddressFamily IPv4 | Get-NetIPAddress | ForEach-Object { $_.IPAddress }

seems to provide me with the correct address for use as DNS (and Gateway) for the VMs on the "Default Switch".

gbraad · 2018-01-09T08:13:46Z

@djensen47 make sure you have used New-NetNAT to setup a local NAT network

djensen47 · 2018-01-09T08:26:05Z

make sure you have used New-NetNAT to setup a local NAT network

Isn't there an issue if DockerNAT is already on the system?

gbraad · 2018-01-09T08:27:28Z

DockerNAT is not a real NAT-enabled network. It uses VPNKit to provide this behaviour instead

briansmith · 2018-01-10T18:44:17Z

Can you check what ends up in your /etc/resolve.conf when using the "Default Switch"?

When it works doing minikube ssh and then cat /etc/resolv.conf shows:

nameserver 172.21.111.65

When it doesn't work...I'm not sure, because I haven't been able to reproduce the problem where an IPv6 address is chosen by the C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikube ).networkadapters[0]).ipaddresses[0] step. It's always returning an IPv4 address the last few times I've tried. I'm not sure if I fixed something or if Microsoft fixed something or if it's just a coincidence.

mcolomerc · 2018-01-17T16:32:39Z

By now, I was able to start minikube with Internal Switch using IPV4

My environment:
Windows 10 Enterprise
Version: 1703
OS Build: 15063.786

Running Docker for Windows with Hyper-V

$ docker --version Docker version 17.09.0-ce, build afdb6d4

$ minikube version minikube version: v0.24.1

I needed to create the internal switch:

  New-VMSwitch -SwitchName "NATSwitch" -SwitchType Internal
  New-NetIPAddress –IPAddress 192.168.1.1 -PrefixLength 24 -InterfaceAlias "vEthernet (NATSwitch)"

And create the NAT:
New-NetNat –Name MinikubeNetwork -InternalIPInterfaceAddressPrefix 192.168.0.0/24

Start the minikube using the Switch:
minikube start --vm-driver="hyperv" --hyperv-virtual-switch="NATSwitch" --v=7 --alsologtostderr

Then:

$ minikube ip
192.168.0.10

$ kubectl cluster-info
Kubernetes master is running at https://192.168.0.10:8443

fejta-bot · 2018-04-17T17:05:52Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

gbraad · 2018-04-17T17:14:42Z

Minishift is currently considering the use of the "Default Switch" as the network to use wen nothing is configured. This will greatly simplify the setup

fejta-bot · 2018-05-17T17:53:14Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

briansmith · 2018-05-17T17:59:27Z

Minishift is currently considering the use of the "Default Switch" as the
network to use wen nothing is configured. This will greatly simplify the
setup

My experience using the "Default Switch" with minikube 0.25 is that it sometimes works and sometimes doesn't. In particular I still run into the "I got an IPv6 address that doesn't work" problem that is reported in various issues here. (I suspect the IPv6 address is some link-local address or another kind with limited connectivity.) I've tried all the workarounds regarding disabling IPv6 in various ways to no avail. The only workaround I know of is to "minikube stop && minikube start" until it works.

IMO getting the "Default Switch" working would make minikube much more usable on Windows.

fejta-bot · 2018-06-16T18:39:57Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

mahendra-012 · 2018-07-20T09:02:21Z

Environment:

OS: Windows 10 Enterprise
VM Driver: hyperv
Version: 1703
OS Build: 16299.125
minikube version: v0.27.0
kubernetes version: v1.10.0

When I create a NAT using https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/user-guide/setup-nat-network

PS C:\WINDOWS\system32> minikube start Starting local Kubernetes v1.10.0 cluster... Starting VM... Getting VM IP address... Moving files into cluster... Setting up certs... Connecting to cluster... Setting up kubeconfig... Starting cluster components... E0720 14:26:12.140736 9672 start.go:281] Error restarting cluster: restarting kube-proxy: waiting for kube-proxy to be up for configmap update: timed out waiting for the condition

r2d4 added co/hyperv HyperV related issues kind/feature Categorizes issue or PR as related to a new feature. os/windows labels Jun 21, 2017

gbraad mentioned this issue Aug 24, 2017

Support for predictable IP's across restarts #951

Closed

gbraad mentioned this issue Dec 4, 2017

Enable using Hyper-V Default Switch for Fall Creator's update (Windows 10) minishift/minishift#1792

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 17, 2018

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels May 17, 2018

k8s-ci-robot closed this as completed Jun 16, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

hyper-v: docs and feature to make easy NAT networking setup #1627

hyper-v: docs and feature to make easy NAT networking setup #1627

philips commented Jun 21, 2017

gbraad commented Aug 21, 2017

aaron-prindle commented Aug 25, 2017 •

edited

Loading

gbraad commented Aug 25, 2017 •

edited

Loading

aaron-prindle commented Aug 25, 2017 •

edited

Loading

gbraad commented Aug 29, 2017 •

edited

Loading

gbraad commented Aug 29, 2017

gbraad commented Sep 1, 2017

gbraad commented Sep 27, 2017 •

edited

Loading

briansmith commented Oct 30, 2017

briansmith commented Dec 4, 2017

gbraad commented Dec 4, 2017

briansmith commented Dec 4, 2017

gbraad commented Dec 4, 2017 •

edited

Loading

gbraad commented Dec 6, 2017

gbraad commented Dec 22, 2017

djensen47 commented Jan 3, 2018

gbraad commented Jan 9, 2018

gbraad commented Jan 9, 2018

djensen47 commented Jan 9, 2018

gbraad commented Jan 9, 2018

briansmith commented Jan 10, 2018

mcolomerc commented Jan 17, 2018

fejta-bot commented Apr 17, 2018

gbraad commented Apr 17, 2018 via email

fejta-bot commented May 17, 2018

briansmith commented May 17, 2018 •

edited

Loading

fejta-bot commented Jun 16, 2018

mahendra-012 commented Jul 20, 2018

hyper-v: docs and feature to make easy NAT networking setup #1627

hyper-v: docs and feature to make easy NAT networking setup #1627

Comments

philips commented Jun 21, 2017

gbraad commented Aug 21, 2017

aaron-prindle commented Aug 25, 2017 • edited Loading

gbraad commented Aug 25, 2017 • edited Loading

aaron-prindle commented Aug 25, 2017 • edited Loading

gbraad commented Aug 29, 2017 • edited Loading

gbraad commented Aug 29, 2017

gbraad commented Sep 1, 2017

gbraad commented Sep 27, 2017 • edited Loading

briansmith commented Oct 30, 2017

briansmith commented Dec 4, 2017

gbraad commented Dec 4, 2017

briansmith commented Dec 4, 2017

gbraad commented Dec 4, 2017 • edited Loading

gbraad commented Dec 6, 2017

gbraad commented Dec 22, 2017

djensen47 commented Jan 3, 2018

gbraad commented Jan 9, 2018

gbraad commented Jan 9, 2018

djensen47 commented Jan 9, 2018

gbraad commented Jan 9, 2018

briansmith commented Jan 10, 2018

mcolomerc commented Jan 17, 2018

fejta-bot commented Apr 17, 2018

gbraad commented Apr 17, 2018 via email

fejta-bot commented May 17, 2018

briansmith commented May 17, 2018 • edited Loading

fejta-bot commented Jun 16, 2018

mahendra-012 commented Jul 20, 2018

aaron-prindle commented Aug 25, 2017 •

edited

Loading

gbraad commented Aug 25, 2017 •

edited

Loading

aaron-prindle commented Aug 25, 2017 •

edited

Loading

gbraad commented Aug 29, 2017 •

edited

Loading

gbraad commented Sep 27, 2017 •

edited

Loading

gbraad commented Dec 4, 2017 •

edited

Loading

briansmith commented May 17, 2018 •

edited

Loading