hyperv: remote command exited without exit status or exit signal #3856

lluck42 · 2019-03-12T01:52:51Z

PS C:\Users\Administrator> minikube start --registry-mirror="https://registry.docker-cn.com" --vm-driver="hyperv" --memory=2048 --hyperv-virtual-switch="nat" --mount --mount-string="/f::/f"
o minikube v0.35.0 on windows (amd64)

Creating hyperv VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...

"minikube" IP address is fe80::215:5dff:fe01:8acd
Configuring Docker as the container runtime ...
Preparing Kubernetes environment ...
@ Downloading kubelet v1.13.4
@ Downloading kubeadm v1.13.4
! Failed to update cluster: downloading binaries: copy: wait: remote command exited without exit status or exit signal

Sorry that minikube crashed. If this was unexpected, we would love to hear from you:

https://github.com/kubernetes/minikube/issues/new

xarses · 2019-03-13T03:57:06Z

Literally running into the same issue

./minikube-windows-amd64.exe start --v 5 --vm-driver="hyperv" --cpus=2 --memory=2048 --disk-size=20g --service-cluster-ip-range="10.0.0.1/24" --kubernetes-version v1.13.3
o   minikube v0.35.0 on windows (amd64)
>   Creating hyperv VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
@   Downloading Minikube ISO ...
 184.42 MB / 184.42 MB [============================================] 100.00% 0s
Creating CA: C:\Users\awoodward\.minikube\certs\ca.pem
Creating client certificate: C:\Users\awoodward\.minikube\certs\cert.pem
Downloading C:\Users\awoodward\.minikube\cache\boot2docker.iso from file://C:/Users/awoodward/.minikube/cache/iso/minikube-v0.35.0.iso...
Creating SSH key...
Creating VM...
Using switch "myCluster"
Creating VHD
Starting VM...
Waiting for host to start...
Setting Docker configuration on the remote daemon...
-   "minikube" IP address is 172.18.67.193
-   Configuring Docker as the container runtime ...
-   Preparing Kubernetes environment ...
@   Downloading kubeadm v1.13.3
@   Downloading kubelet v1.13.3
!   Failed to update cluster: downloading binaries: copy: wait: remote command exited without exit status or exit signal

*   Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
-   https://github.com/kubernetes/minikube/issues/new

however switching to 0.34.1 it works (although, "downloading" took forever)

PS C:\Users\awoodward\Downloads> .\minikube-windows-amd64-0.34.1.exe start --v 5 --vm-driver="hyperv" --cpus=2 --memo
ry=2048 --disk-size=20g --service-cluster-ip-range="10.0.0.1/24" --kubernetes-version v1.13.3
There is a newer version of minikube available (v0.35.0).  Download it here:
https://github.com/kubernetes/minikube/releases/tag/v0.35.0

To disable this notification, run the following:
minikube config set WantUpdateNotification false
o   minikube v0.34.1 on windows (amd64)
>   Creating hyperv VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
@   Downloading Minikube ISO ...
 184.28 MB / 184.30 MB [============================================]  99.99% 0s
 184.30 MB / 184.30 MB [============================================] 100.00% 0sCreating CA: C:\Users\awoodward\.minikube\certs\ca.pem
Creating client certificate: C:\Users\awoodward\.minikube\certs\cert.pem
Downloading C:\Users\awoodward\.minikube\cache\boot2docker.iso from file://C:/Users/awoodward/.minikube/cache/iso/minikube-v0.34.0.iso...
Creating SSH key...
Creating VM...
Using switch "myCluster"
Creating VHD
Starting VM...
Waiting for host to start...
Setting Docker configuration on the remote daemon...
-   "minikube" IP address is 172.18.66.64
-   Configuring Docker as the container runtime ...
-   Preparing Kubernetes environment ...
@   Downloading kubeadm v1.13.3
@   Downloading kubelet v1.13.3
-   Pulling images required by Kubernetes v1.13.3 ...
-   Launching Kubernetes v1.13.3 using kubeadm ...
-   Configuring cluster permissions ...
-   Verifying component health .....
+   kubectl is now configured to use "minikube"
=   Done! Thank you for using minikube!
PS C:\Users\awoodward\Downloads>
PS C:\Users\awoodward\Downloads> .\minikube-windows-amd64-0.34.1.exe delete
-   Powering off "minikube" via SSH ...
x   Deleting "minikube" from hyperv ...
-   The "minikube" cluster has been deleted.

nothing fun in minikube logs

PS C:\Users\awoodward\Downloads> .\minikube-windows-amd64-0.35.0.exe logs
E0312 13:51:24.901826   19944 logs.go:141] Failed to list containers for "kube-apiserver": Process exited with status 137 from signal KILL
E0312 13:51:26.731933   19944 logs.go:141] Failed to list containers for "coredns": Process exited with status 137 from signal KILL
E0312 13:51:28.934086   19944 logs.go:141] Failed to list containers for "kube-scheduler": Process exited with status 137 from signal KILL
==> kubelet <==
-- Logs begin at Tue 2019-03-12 13:31:48 UTC, end at Tue 2019-03-12 20:51:28 UTC. --
-- No entries --

the last few lines of dmesg are quite telling

[ 1217.114676] active_anon:7 inactive_anon:23 isolated_anon:0
                active_file:17 inactive_file:18 isolated_file:0
                unevictable:141617 dirty:0 writeback:21 unstable:0
                slab_reclaimable:4955 slab_unreclaimable:6699
                mapped:20072 shmem:0 pagetables:387 bounce:0
                free:3419 free_pcp:281 free_cma:0
[ 1217.114679] Node 0 active_anon:28kB inactive_anon:92kB active_file:68kB inactive_file:72kB unevictable:566468kB isolated(anon):0kB isolated(file):0kB mapped:80288kB dirty:0kB writeback:84kB shmem:0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ 1217.114680] Node 0 DMA free:7180kB min:44kB low:56kB high:68kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 1217.114683] lowmem_reserve[]: 0 1785 1785 1785
[ 1217.114685] Node 0 DMA32 free:6496kB min:5380kB low:7208kB high:9036kB active_anon:112kB inactive_anon:520kB active_file:68kB inactive_file:72kB unevictable:566468kB writepending:0kB present:2080704kB managed:2011528kB mlocked:0kB kernel_stack:2816kB pagetables:1548kB bounce:0kB free_pcp:1124kB local_pcp:484kB free_cma:0kB
[ 1217.114689] lowmem_reserve[]: 0 0 0 0
[ 1217.114691] Node 0 DMA: 1*4kB (U) 1*8kB (U) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 1*4096kB (M) = 7180kB
[ 1217.114699] Node 0 DMA32: 126*4kB (UME) 69*8kB (UMEH) 26*16kB (UMEH) 21*32kB (UME) 12*64kB (UMEH) 4*128kB (ME) 4*256kB (ME) 2*512kB (M) 1*1024kB (E) 0*2048kB 0*4096kB = 6496kB
[ 1217.114710] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 1217.114711] 141694 total pagecache pages
[ 1217.114712] 41 pages in swap cache
[ 1217.114713] Swap cache stats: add 292687, delete 292673, find 202449/325207
[ 1217.114713] Free swap  = 984316kB
[ 1217.114714] Total swap = 1023996kB
[ 1217.114714] 524174 pages RAM
[ 1217.114715] 0 pages HighMem/MovableOnly
[ 1217.114715] 17315 pages reserved
[ 1217.114715] [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[ 1217.114720] [ 1239]     0  1239     9126      941   110592      132             0 systemd-journal
[ 1217.114722] [ 1931]     0  1931    11137      999   126976      390         -1000 systemd-udevd
[ 1217.114723] [ 1984]     0  1984     3298        0    69632      131             0 rpc.mountd
[ 1217.114726] [ 2076]     0  2076     4946      510    81920       78             0 rpcbind
[ 1217.114727] [ 2077]     0  2077     7088      898   102400      131             0 systemd-logind
[ 1217.114729] [ 2081]     0  2081     1756      436    57344       47             0 hv_kvp_daemon
[ 1217.114731] [ 2104]     0  2104     2310       85    65536       29             0 getty
[ 1217.114732] [ 2105]     0  2105     2310      106    61440       29             0 getty
[ 1217.114734] [ 2107]  1002  2107     5656      744    77824      115          -900 dbus-daemon
[ 1217.114736] [ 2147]     0  2147     5529      973    94208      130         -1000 sshd
[ 1217.114738] [ 2693]     0  2693    85046     8246   229376     3154          -500 dockerd
[ 1217.114740] [ 2700]     0  2700    80691     4811   188416     1548          -500 docker-containe
[ 1217.114742] [ 2857]  1008  2857     9723     1086   102400      150             0 systemd-network
[ 1217.114744] [ 2859]  1009  2859     8534     1097   110592      113             0 systemd-resolve
[ 1217.114746] [ 2880]     0  2880     5529     1029    90112      143             0 sshd
[ 1217.114747] [ 2882]  1001  2882     5529      636    90112      127             0 sshd
[ 1217.114749] [ 2899]     0  2899     5529     1002    77824      112             0 sshd
[ 1217.114751] [ 2901]  1001  2901     5529      679    77824      112             0 sshd
[ 1217.114753] [ 2912]  1001  2912     9837     4114   131072     1095             0 docker
[ 1217.114754] Out of memory: Kill process 2912 (docker) score 6 or sacrifice child
[ 1217.114784] Killed process 2912 (docker) total-vm:39348kB, anon-rss:0kB, file-rss:16456kB, shmem-rss:0kB
[ 1217.116826] oom_reaper: reaped process 2912 (docker), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

increasing the 0.35.0 minikube to 4096 of ram does not solve the problem, it simply spins around 80% usage and commands don't progress

tstromberg · 2019-03-13T05:21:32Z

Any chance that dynamic memory allocation in HyperV is involved? I wonder if this may be due to #1766

If someone has the time, could you try disabling the dynamic memory feature in HyperV to see if it fixes the issue?

xarses · 2019-03-13T16:33:20Z

After bashing in the command to get the dynamic memory feature off, I was able to get 0.35.0 to start a 4096 mb instance

PS C:\Users\awoodward\Downloads> .\minikube-windows-amd64-0.35.0.exe start --v 5 --vm-driver="hyperv" --cpus=2 --memory=
4096 --disk-size=20g --service-cluster-ip-range="10.0.0.1/24" --kubernetes-version v1.13.3
o   minikube v0.35.0 on windows (amd64)
>   Creating hyperv VM (CPUs=2, Memory=4096MB, Disk=20000MB) ...
Downloading C:\Users\awoodward\.minikube\cache\boot2docker.iso from file://C:/Users/awoodward/.minikube/cache/iso/minikube-v0.35.0.iso...
Creating SSH key...
Creating VM...
Using switch "myCluster"
Creating VHD
Starting VM...
Waiting for host to start...
Setting Docker configuration on the remote daemon...
-   "minikube" IP address is 172.18.68.188
-   Configuring Docker as the container runtime ...
-   Preparing Kubernetes environment ...
@   Downloading kubeadm v1.13.3
@   Downloading kubelet v1.13.3
-   Pulling images required by Kubernetes v1.13.3 ...
-   Launching Kubernetes v1.13.3 using kubeadm ...
:   Waiting for pods: apiserver proxy etcd scheduler controller addon-manager dns
-   Configuring cluster permissions ...
-   Verifying component health .....
+   kubectl is now configured to use "minikube"
=   Done! Thank you for using minikube!

simple things, like Powering off "minikube" via SSH ... actually work now

and hey 2048 mb works now too

PS C:\Users\awoodward\Downloads> .\minikube-windows-amd64-0.35.0.exe start --v 5 --vm-driver="hyperv" --cpus=2 --memory=
2048 --disk-size=20g --service-cluster-ip-range="10.0.0.1/24" --kubernetes-version v1.13.3
o   minikube v0.35.0 on windows (amd64)
>   Creating hyperv VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
Downloading C:\Users\awoodward\.minikube\cache\boot2docker.iso from file://C:/Users/awoodward/.minikube/cache/iso/minikube-v0.35.0.iso...
Creating SSH key...
Creating VM...
Using switch "myCluster"
Creating VHD
Starting VM...
Waiting for host to start...
Setting Docker configuration on the remote daemon...
-   "minikube" IP address is 172.18.66.2
-   Configuring Docker as the container runtime ...
-   Preparing Kubernetes environment ...
-   Pulling images required by Kubernetes v1.13.3 ...
-   Launching Kubernetes v1.13.3 using kubeadm ...
:   Waiting for pods: apiserver proxy etcd scheduler controller addon-manager dns
-   Configuring cluster permissions ...
-   Verifying component health .....
+   kubectl is now configured to use "minikube"
=   Done! Thank you for using minikube!
PS C:\Users\awoodward\Downloads>

xarses · 2019-03-13T16:37:44Z

I'm going to declare this a duplicate of #1766 since it totally fixed this

xarses · 2019-03-14T02:13:39Z

specifically hammering Set-VMMemory -VMName 'minikube' -DynamicMemoryEnabled $false is what is needed while the minikube vm is creating, if you manage it before it starts, I get about 5 seconds that It can be run, it seems to go great

zaf187 · 2019-04-04T10:51:59Z

I was facing this problem but instead of setting the flag i upped the memory on the minikube start command, instead of the default (2048mb) i set it to (4096mb) and it came up without that error

tstromberg changed the title ~~remote command exited without exit status or exit signal~~ hyperv: remote command exited without exit status or exit signal Mar 13, 2019

tstromberg closed this as completed Mar 14, 2019

tstromberg added the triage/duplicate Indicates an issue is a duplicate of other open issue. label Mar 14, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

hyperv: remote command exited without exit status or exit signal #3856

hyperv: remote command exited without exit status or exit signal #3856

lluck42 commented Mar 12, 2019

xarses commented Mar 13, 2019

tstromberg commented Mar 13, 2019 •

edited

Loading

xarses commented Mar 13, 2019

xarses commented Mar 13, 2019

xarses commented Mar 14, 2019

zaf187 commented Apr 4, 2019

hyperv: remote command exited without exit status or exit signal #3856

hyperv: remote command exited without exit status or exit signal #3856

Comments

lluck42 commented Mar 12, 2019

xarses commented Mar 13, 2019

tstromberg commented Mar 13, 2019 • edited Loading

xarses commented Mar 13, 2019

xarses commented Mar 13, 2019

xarses commented Mar 14, 2019

zaf187 commented Apr 4, 2019

tstromberg commented Mar 13, 2019 •

edited

Loading