minikube start with hyper-v won't resolve guest ip-address

[a3nick]

BUG REPORT

Executing minikube start --vm-driver="hyperv" --hyperv-virtual-switch="ext1", where ext1 is my external virtual switch on hyper-v, the system hangs on "Starting VM" message.
Checking the log it seems that the powershell command used to retireve the ip address of the guest machine don't returns any value (i tested running the command on a separate powershell with the same result).
I obtained the same result on two different machines.

Environment:

Minikube version: v0.22.0

• OS: Windows 10x64 - 1703
• VM Driver: hyper-v
• ISO version: -v0.23.3
• Install tools: kubectl & minikube installed using chocolatey packages

Output of minikube logs:

[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] :
[stderr =====>] :

[mikemcgo]

It looks like hv_kvp_daemon isn't running by default in the vm. This is the daemon that allows the guest os to communicate with the host. If you start it manually (/usr/sbin/hv_kvp_daemon), the ip shows up in hyperv manager. I think its a bug that it isn't starting by default.

[richardjharding]

I can confirm this with 0.22.1 and hyperv on windows 10

Tried starting hv_kvp_daemon as suggested but the vm still doesn't seem to get an ip

any suggestions?

[a3nick]

I obtained the same result running the daemon, the ip address isn't returned

[richardjharding]

Does anyone know if this is a new bug - is there an earlier version that works?

[vcx]

Same issue here. minikube vm starts, external virtual switch configured OK, and I have an IP address that I can verify from inside the VM. From outside, I got the powershell command that minikube is trying to execute via procmon, and it matches the same posted in the first message of this issue.

Also, kvp service failed to start (as verified with systemctl status hv_kvp_daemon.service). The output says that KVP service started and one second later process exited, status=1/FAILURE. Unit entered failed state.

I've studied how systemctl works, found the command line, ran it and it worked. Tried to start the hv-kvp manually using systemctl and it worked too. After some seconds, I could see the IP address in the powershell command... but minikube still hangs on the "Waiting for SSH to be available".

with Procmon, I found that minikube is looping trying to connect to port 22 on my own address. As a test, tried to run again using a new, different switch. Added an Internal switch (in hyper-v world, this means no internet, vm and physical machine can communicate) and set static IPs from my physical machine and the VM (192.168.20.1 an 2, respectively). Then, ran the minikube start command again. Crossed that point, but now got looping while configuring the VM... looks like docker service doesn't start.

Now it looks like I have a different problem, and will open a different issue later. But I hope this troubleshooting info helps your case too.

[richardjharding]

thanks I get something similar after starting the hv_kvp daemon using systemctl as you suggest

This is the output I see in powershell admin session

[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Get-VM minikube
 ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikub
e ).networkadapters[0]).ipaddresses[0]
[stdout =====>] : 192.168.1.244

[stderr =====>] :
SSH binary not found, using native Go implementation
&{{{<nil> 0 [] [] []} docker [0x8526f0] <nil>  [] 0s} 192.168.1.244 22 <nil> <nil>}
About to run SSH command:
sudo systemctl -f restart docker
SSH cmd err, output: Process exited with status 1: Job for docker.service failed because the control process exited with
 error code.
See "systemctl status docker.service" and "journalctl -xe" for details.

[a3nick]

It works with minikube version v0.22.2 and ISO v0.23.4

[vcx]

Confirmed working. Thank you very much, guys!

[ryegrass2003]

seems I have the same issue with v0.24.1
windows 10 pro hyperv

Error starting host: Error starting stopped host: exit status 1

[ghost]

I have the same problem

[dnldaniel]

The same. Works with minikube 0.22.2 but doesn't for 0.24.1 (hangs at "starting vm").

[stackcats]

The same problem with v0.24.1. Win10

[cforce]

I have the issue with 0.24.1,0.22.2 and 0.20.0 on windows 10 md or psheel in admin - none of thme works for me

minikube start --vm-driver hyperv --hyperv-virtual-switch "Primary Virtual Switch"
There is a newer version of minikube available (v0.24.1).  Download it here:
https://github.com/kubernetes/minikube/releases/tag/v0.24.1

To disable this notification, run the following:
minikube config set WantUpdateNotification false
Starting local Kubernetes v1.7.5 cluster...
Starting VM...
E0113 09:00:25.864532   18144 start.go:147] Error starting host: Error starting stopped host: exit status 1.

 Retrying.
E0113 09:00:25.872533   18144 start.go:153] Error starting host:  Error starting stopped host: exit status 1

or PowerShell

Windows PowerShell
Copyright (C) Microsoft Corporation. Alle Rechte vorbehalten.

PS C:\WINDOWS\system32> minikube start --vm-driver hyperv --hyperv-virtual-switch "Primary Virtual Switch"v0.23.4
Starting local Kubernetes v1.7.5 cluster...
Starting VM...
E0113 16:01:46.090360   14388 start.go:143] Error starting host: Error starting stopped host: exit status 1.

 Retrying.
E0113 16:01:46.099362   14388 start.go:149] Error starting host:  Error starting stopped host: exit status 1
PS C:\WINDOWS\system32>

[maxcapraro]

I am facing the error too on a Windows 10 Pro machine. I have the same log output like @cforce. I tested with 0.24.1, 0.22.2, and 0.19.1. In addition, I checked whether /usr/sbin/hv_kvp_daemon is running (as suggested by @mikemcgo). It is running.

The first mention of this bug/behaviour re-appearing was 2018-01-05 - over a month after 0.24.1 was released. This could be an indicator that a Windows Update or similar triggered this.

[dbovenzi]

I have the same issue described here.
minikube version: v0.25.0

`Starting local Kubernetes v1.9.0 cluster...
Starting VM...
E0206 12:42:59.777557 7560 start.go:159] Error starting host: Error starting stopped host: exit status 1.

Retrying.
E0206 12:42:59.782057 7560 start.go:165] Error starting host: Error starting stopped host: exit status 1
`

[HB-2012]

@dbovenzi Make sure you run PowerShell as administrator. Also add --alsologtostderr to see some more detailed information.

[j-rewerts]

Also running into this issue.

Minikube version v0.25.0

I get these logs by running this command:

minikube start --vm-driver="hyperv" --memory=4096 --cpus=4 --hyperv-virtual-switch="New Virtual
Switch" --v=7 --alsologtostderr

W0317 23:56:24.407104 12708 root.go:148] Error reading config file at C:\Program Files\Docker\Minikube.minikube\config\config.json: open C:\Program Files\Docker\Minikube.minikube\config\config.json: The system cannot find the file specified.
I0317 23:56:24.409605 12708 notify.go:109] Checking for updates...
I0317 23:56:24.676586 12708 cache_images.go:292] Attempting to cache image: gcr.io/k8s-minikube/storage-provisioner:v1.8.0 at C:\Program Files\Docker\Minikube.minikube\cache\images\gcr.io\k8s-minikube\storage-provisioner_v1.8.0
Starting local Kubernetes v1.9.0 cluster...
Starting VM...
I0317 23:56:24.676586 12708 cache_images.go:292] Attempting to cache image: k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.14.5 at C:\Program Files\Docker\Minikube.minikube\cache\images\k8s.gcr.io\k8s-dns-dnsmasq-nanny-amd64_1.14.5
I0317 23:56:24.677587 12708 cluster.go:70] Machine does not exist... provisioning new machine
I0317 23:56:24.676586 12708 cache_images.go:292] Attempting to cache image: gcr.io/google-containers/kube-addon-manager:v6.5 at C:\Program Files\Docker\Minikube.minikube\cache\images\gcr.io\google-containers\kube-addon-manager_v6.5
I0317 23:56:24.676586 12708 cache_images.go:292] Attempting to cache image: k8s.gcr.io/pause-amd64:3.0 at C:\Program Files\Docker\Minikube.minikube\cache\images\k8s.gcr.io\pause-amd64_3.0
I0317 23:56:24.676586 12708 cache_images.go:292] Attempting to cache image: k8s.gcr.io/kubernetes-dashboard-amd64:v1.8.1 at C:\Program Files\Docker\Minikube.minikube\cache\images\k8s.gcr.io\kubernetes-dashboard-amd64_v1.8.1
I0317 23:56:24.676586 12708 cache_images.go:292] Attempting to cache image: k8s.gcr.io/k8s-dns-kube-dns-amd64:1.14.5 at C:\Program Files\Docker\Minikube.minikube\cache\images\k8s.gcr.io\k8s-dns-kube-dns-amd64_1.14.5
I0317 23:56:24.676586 12708 cache_images.go:292] Attempting to cache image: k8s.gcr.io/k8s-dns-sidecar-amd64:1.14.5 at C:\Program Files\Docker\Minikube.minikube\cache\images\k8s.gcr.io\k8s-dns-sidecar-amd64_1.14.5
I0317 23:56:24.677587 12708 cluster.go:71] Provisioning machine with config: {MinikubeISO:https://storage.googleapis.com/minikube/iso/minikube-v0.25.1.iso Memory:4096 CPUs:4 DiskSize:20000 VMDriver:hyperv XhyveDiskDriver:ahci-hd DockerEnv:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch:New Virtual Switch KvmNetwork:default Downloader:{} DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID:}
I0317 23:56:24.678586 12708 cache_images.go:79] Successfully cached all images.
I0317 23:56:24.678586 12708 downloader.go:56] Not caching ISO, using https://storage.googleapis.com/minikube/iso/minik
ube-v0.25.1.iso
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive @(Get-Command Get
-VM).ModuleName
[stdout =====>] : Hyper-V

[stderr =====>] :
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive @([Security.Principal.WindowsPrincipal][Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole("S-1-5-32-578")
[stdout =====>] : False

[stderr =====>] :
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive @([Security.Principal.WindowsPrincipal][Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole([Security.Principal.WindowsBuiltInRole] "Administrator")
[stdout =====>] : True

[stderr =====>] :
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (Get-VMSwitch).Name
[stdout =====>] : DockerNAT
Default Switch
New Virtual Switch

[stderr =====>] :
Downloading C:\Program Files\Docker\Minikube.minikube\cache\boot2docker.iso from file://C:/Program Files/Docker/Minikube/.minikube/cache/iso/minikube-v0.25.1.iso...
Creating SSH key...
Creating VM...
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (Get-VMSwitch).Name
[stdout =====>] : DockerNAT
Default Switch
New Virtual Switch

[stderr =====>] :
Using switch "New Virtual Switch"
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive @([Security.Princ
ipal.WindowsPrincipal][Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole([Security.Principal.WindowsBuiltInRol
e] "Administrator")
[stdout =====>] : True

[stderr =====>] :
Creating VHD
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive New-VHD -Path 'C:\Program Files\Docker\Minikube.minikube\machines\minikube\fixed.vhd' -SizeBytes 10MB -Fixed
[stdout =====>] :

ComputerName : DESKTOP-9SMC76M
Path : C:\Program Files\Docker\Minikube.minikube\machines\minikube\fixed.vhd
VhdFormat : VHD
VhdType : Fixed
FileSize : 10486272
Size : 10485760
MinimumSize :
LogicalSectorSize : 512
PhysicalSectorSize : 512
BlockSize : 0
ParentPath :
DiskIdentifier : C7A2C5E7-513F-4E09-B2C3-94EDE15E8413
FragmentationPercentage : 0
Alignment : 1
Attached : False
DiskNumber :
IsPMEMCompatible : False
AddressAbstractionType : None
Number :

[stderr =====>] :
Writing magic tar header
Writing SSH key tar header
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive Convert-VHD -Path 'C:\Program Files\Docker\Minikube.minikube\machines\minikube\fixed.vhd' -DestinationPath 'C:\Program Files\Docker\Minikube.minikube\machines\minikube\disk.vhd' -VHDType Dynamic -DeleteSource
[stdout =====>] :
[stderr =====>] :
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive Resize-VHD -Path 'C:\Program Files\Docker\Minikube.minikube\machines\minikube\disk.vhd' -SizeBytes 20000MB
[stdout =====>] :
[stderr =====>] :
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive New-VM minikube -Path 'C:\Program Files\Docker\Minikube.minikube\machines\minikube' -SwitchName 'New Virtual Switch' -MemoryStartupBytes 4096MB
[stdout =====>] :
Name State CPUUsage(%) MemoryAssigned(M) Uptime Status Version

---

minikube Off 0 0 00:00:00 Operating normally 8.2

[stderr =====>] :
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive Set-VMProcessor minikube -Count 4
[stdout =====>] :
[stderr =====>] :
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive Set-VMDvdDrive -VMName minikube -Path 'C:\Program Files\Docker\Minikube.minikube\machines\minikube\boot2docker.iso'
[stdout =====>] :
[stderr =====>] :
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive Add-VMHardDiskDrive -VMName minikube -Path 'C:\Program Files\Docker\Minikube.minikube\machines\minikube\disk.vhd'
[stdout =====>] :
[stderr =====>] :
Starting VM...
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive Start-VM minikube
[stdout =====>] :
[stderr =====>] :
Waiting for host to start...
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] :
[stderr =====>] :
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] :
[stderr =====>] :

[j-rewerts]

Then it just seems to repeatedly check for an ip address, but gets nothing.

Could this be something to do with my DHCP on my router not giving up an ip for some reason?

[krumware]

I had this same trouble on my machine. After some troubleshooting and head banging, it became apparent that the external network adapter had an ethernet adapter selected that was not connected to a network. (ethernet selected, instead of the wireless adapter). Of course an IP wont be allocated on an external network if none is connected.

Once the connected wireless adapter was selected (an adapter actually connected to a DHCP network), all was good.
While rather silly on the end, I hope this helps someone with the same "IP loop" issue

[bretcj7]

Same issue on:

Windows 10 Pro
minikub v0.32.0

[suvasishm]

Same issue on:

Windows 10 Pro
Minikube v0.30.0

[hecsalazar]

Same issue on:

Windows 10 Pro
minikube v0.32.0

[suvasishm]

I would suggest enable K8s from Windows Docker (desktop app) and use 'docker-for-desktop' as Context.

[bretcj7]

@suvasishm I have tried that as well but the kubernetes option in docker never starts up and runs

[suvasishm]

@bretcj7 Is virtualization enabled for your system?

[ghost]

Same issue on:

Windows 10 Pro
Minikube v0.35.0

forever stopping after this...

PS C:\Windows\system32> minikube start --vm-driver=hyperv --hyperv-virtual-switch="vSwitch" --v=3
o   minikube v0.35.0 on windows (amd64)
>   Creating hyperv VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
@   Downloading Minikube ISO ...
 184.42 MB / 184.42 MB [============================================] 100.00% 0s
Creating CA: C:\Users\xxx\.minikube\certs\ca.pem
Creating client certificate: C:\Users\xxx\.minikube\certs\cert.pem
Downloading C:\Users\xxx\.minikube\cache\boot2docker.iso from file://C:/Users/xxx/.minikube/cache/iso/minikube-v0.35.0.iso...
Creating SSH key...
Creating VM...
Using switch "vSwitch"
Creating VHD
Starting VM...
Waiting for host to start...

[legatusAlex]

Hung on:
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] :
[stderr =====>] :

Works for me when changed:
minikube start --vm-driver hyperv --hyperv-virtual-switch "Primary Virtual Switch"
to:
minikube start --vm-driver hyperv --hyperv-virtual-switch "Default Switch"

[cyb3rfist]

I was able to resolve this issue by creating an external virtual switch instead of using an internal one in hyper-V. It seemed somehow the minikube VM was not getting the IP address from the in built DHCP in my WiFi router. So creating an external switch created a direct bridge between the VM and router's DHCP. If you get into the same IP loop you would see on your machine DNS cache (ipconfig /displaydns) for minikube is not changing. It will remain the same no matter how many times you try 'minikube delete' and 'minikube start' which tell you that IP is not getting assigned to new VM from DHCP and hence not getting updated in DNS cache of host OS either.

I took a network capture when I ran it with external switch pointed to Wifi.

625 6:49:22 PM 4/11/2020 10.5034268 svchost.exe 0.0.0.0 255.255.255.255 DHCP DHCP:Request, MsgType = DISCOVER, TransactionID = 0xDEF6CA67 {DHCP:112, UDP:114, IPv4:113}
626 6:49:22 PM 4/11/2020 10.5154494 svchost.exe 192.168.17.129 192.168.17.130 DHCP DHCP:Reply, MsgType = OFFER, TransactionID = 0xDEF6CA67 {DHCP:112, UDP:114, IPv4:113}
627 6:49:22 PM 4/11/2020 10.5167556 svchost.exe 0.0.0.0 255.255.255.255 DHCP DHCP:Request, MsgType = REQUEST, TransactionID = 0xDEF6CA67 {DHCP:112, UDP:114, IPv4:113}
628 6:49:22 PM 4/11/2020 10.5279982 svchost.exe 192.168.17.129 192.168.17.130 DHCP DHCP:Reply, MsgType = ACK, TransactionID = 0xDEF6CA67 {DHCP:112, UDP:114, IPv4:113}

As you can see here minikube VM got offered 192.168.17.130 from WiFi's DHCP which later on reflected in DNS cache as well.

Taking a network capture and filtering the traffic with DHCP at the time of issue will help you narrow down the problem