hyperkit: no internet connectivity, can't pull kube-addon-manager #5951

douglascamata · 2019-11-19T18:08:53Z

The exact command to reproduce the issue:

minikube start

The full output of the command that failed:

😄  minikube v1.5.2 on Darwin 10.14.3
✨  Automatically selected the 'hyperkit' driver (alternates: [virtualbox])
💾  Downloading driver docker-machine-driver-hyperkit:
    > docker-machine-driver-hyperkit.sha256: 65 B / 65 B [---] 100.00% ? p/s 0s
    > docker-machine-driver-hyperkit: 10.79 MiB / 10.79 MiB  100.00% 636.16 KiB
🔑  The 'hyperkit' driver requires elevated permissions. The following commands will be executed:

    $ sudo chown root:wheel /Users/dcamata/.minikube/bin/docker-machine-driver-hyperkit
    $ sudo chmod u+s /Users/dcamata/.minikube/bin/docker-machine-driver-hyperkit

🔥  Creating hyperkit VM (CPUs=2, Memory=2000MB, Disk=20000MB) ...
⚠️  VM may be unable to resolve external DNS records
⚠️  VM is unable to access k8s.gcr.io, you may need to configure a proxy or set --image-repository
🐳  Preparing Kubernetes v1.16.2 on Docker '18.09.9' ...
🚜  Pulling images ...
❌  Unable to pull images, which may be OK: running cmd: "/bin/bash -c \"sudo env PATH=/var/lib/minikube/binaries/v1.16.2:$PATH kubeadm config images pull --config /var/tmp/minikube/kubeadm.yaml\"": Process exited with status 1
🚀  Launching Kubernetes ...
⌛  Waiting for: apiserverj
🏄  Done! kubectl is now configured to use "minikube"

The output of the minikube logs command:

Huge spam of this:

Nov 19 18:03:53 minikube kubelet[2721]: E1119 18:03:53.519954    2721 pod_workers.go:191] Error syncing pod c3e29047da86ce6690916750ab69c40b ("kube-addon-manager-minikube_kube-system(c3e29047da86ce6690916750ab69c40b)"), skipping: failed to "StartContainer" for "kube-addon-manager" with ImagePullBackOff: "Back-off pulling image \"k8s.gcr.io/kube-addon-manager:v9.0.2\""
Nov 19 18:04:07 minikube kubelet[2721]: E1119 18:04:07.531993    2721 remote_image.go:113] PullImage "k8s.gcr.io/kube-addon-manager:v9.0.2" from image service failed: rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.2:60592->192.168.64.1:53: read: connection refused
Nov 19 18:04:07 minikube kubelet[2721]: E1119 18:04:07.532050    2721 kuberuntime_image.go:50] Pull image "k8s.gcr.io/kube-addon-manager:v9.0.2" failed: rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.2:60592->192.168.64.1:53: read: connection refused
Nov 19 18:04:07 minikube kubelet[2721]: E1119 18:04:07.532087    2721 kuberuntime_manager.go:783] container start failed: ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.2:60592->192.168.64.1:53: read: connection refused
Nov 19 18:04:07 minikube kubelet[2721]: E1119 18:04:07.532111    2721 pod_workers.go:191] Error syncing pod c3e29047da86ce6690916750ab69c40b ("kube-addon-manager-minikube_kube-system(c3e29047da86ce6690916750ab69c40b)"), skipping: failed to "StartContainer" for "kube-addon-manager" with ErrImagePull: "rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.2:60592->192.168.64.1:53: read: connection refused"
Nov 19 18:04:18 minikube kubelet[2721]: E1119 18:04:18.518574    2721 pod_workers.go:191] Error syncing pod c3e29047da86ce6690916750ab69c40b ("kube-addon-manager-minikube_kube-system(c3e29047da86ce6690916750ab69c40b)"), skipping: failed to "StartContainer" for "kube-addon-manager" with ImagePullBackOff: "Back-off pulling image \"k8s.gcr.io/kube-addon-manager:v9.0.2\""

The operating system version: macOS 10.14.3

The text was updated successfully, but these errors were encountered:

douglascamata · 2019-11-19T18:16:41Z

Quick note: changing the driver from hyperkit to virtualox makes it work.

douglascamata · 2019-11-19T18:28:35Z

I tried SSHing into the machine and effectively it cannot even ping google.com.

tstromberg · 2019-11-20T15:01:18Z

#5935 will fix the addon-manager, but as you mentioned, the root cause is that your hyperkit installation apparently does not have any network connectivity. It is very interesting that switching to virtualbox makes it work however.

Do you have any special VPN or Docker for Desktop configuration on this host by any chance?

Thanks!

douglascamata · 2019-11-20T15:26:13Z

Hi @tstromberg!

Do you have any special VPN or Docker for Desktop configuration on this host by any chance?

I have a standard Docker for Mac installation, nothing special. Sometimes I use corporate VPNs (even a VPN over another VPN).

I always disconnect all the VPNs when using minikube, as I know they don't place nice together.

If there are any commands you would like me to run to give you more information I'm happy to help.

@michaelklishin

Only enabled on Darwin (a.k.a. macOS, a.k.a. OS X) because it's the only platform that we have tested. Contributions for other platforms are welcome. We should uninstall VirtualBox, Minikube & kubectl and test that the target works from a clean slate. We are using VirtualBox as the VM driver for Minikube because kubernetes/minikube#5951 Pair @michaelklishin

smoya · 2019-12-03T18:02:55Z

Same issue here.

➜ minikube start
😄  minikube v1.5.2 on Darwin 10.15.1
💾  Downloading driver docker-machine-driver-hyperkit:
    > docker-machine-driver-hyperkit.sha256: 65 B / 65 B [---] 100.00% ? p/s 0s
    > docker-machine-driver-hyperkit: 10.79 MiB / 10.79 MiB  100.00% 4.88 MiB p
🔑  The 'hyperkit' driver requires elevated permissions. The following commands will be executed:

    $ sudo chown root:wheel /Users/smoya/.minikube/bin/docker-machine-driver-hyperkit
    $ sudo chmod u+s /Users/smoya/.minikube/bin/docker-machine-driver-hyperkit


Password:
💿  Downloading VM boot image ...
    > minikube-v1.5.1.iso.sha256: 65 B / 65 B [--------------] 100.00% ? p/s 0s
    > minikube-v1.5.1.iso: 143.76 MiB / 143.76 MiB [-] 100.00% 6.37 MiB p/s 23s
🔥  Creating hyperkit VM (CPUs=2, Memory=2000MB, Disk=20000MB) ...
⚠️  VM may be unable to resolve external DNS records
⚠️  VM is unable to access k8s.gcr.io, you may need to configure a proxy or set --image-repository
🐳  Preparing Kubernetes v1.16.2 on Docker '18.09.9' ...
💾  Downloading kubeadm v1.16.2
💾  Downloading kubelet v1.16.2
🚜  Pulling images ...
❌  Unable to pull images, which may be OK: running cmd: "/bin/bash -c \"sudo env PATH=/var/lib/minikube/binaries/v1.16.2:$PATH kubeadm config images pull --config /var/tmp/minikube/kubeadm.yaml\"": Process exited with status 1
🚀  Launching Kubernetes ...
⌛  Waiting for: apiserver
🏄  Done! kubectl is now configured to use "minikube"

➜ kubectl get pods --all-namespaces
NAMESPACE     NAME                          READY   STATUS             RESTARTS   AGE
kube-system   coredns-5644d7b6d9-2p2s9      1/1     Running            0          67s
kube-system   coredns-5644d7b6d9-4c6k7      1/1     Running            0          67s
kube-system   etcd-minikube                 1/1     Running            0          18s
kube-system   kube-addon-manager-minikube   0/1     ImagePullBackOff   0          68s
kube-system   kube-apiserver-minikube       1/1     Running            0          9s
kube-system   kube-proxy-8fsj6              1/1     Running            0          67s
kube-system   kube-scheduler-minikube       1/1     Running            0          10s

The error:

Failed to pull image "k8s.gcr.io/kube-addon-manager:v9.0.2": rpc error: code = Unknown desc = Error response from daemon
: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.2:58839->192.168.64.1:53: read: connection refused

jorijn · 2019-12-09T07:42:53Z

I found this issue because I was experiencing the same problem. I also found the culprit:

When SSH'ing into the machine I found out it's using the host machine's IP as resolving IP. When using nslookup I first got connection refused, then connection timeout. This makes no sense, but it triggered me for something something else:

For local development I've been using dnsmasq, so in fact some other DNS server was running as opposed to the one hyperkit needs. When stopping dnsmasq, setting up a new minikube instance worked as intended. ✅

douglascamata · 2019-12-09T13:44:59Z

@jorijn you think maybe then this could be related to internal DNS configuration/servers used by VPN software? I have both Cisco AnyConnect and Tunnelblick installed.

jorijn · 2019-12-12T06:43:22Z

@douglascamata it might! you could try disconnecting VPN connections, quitting those programs and trying again.

douglascamata · 2019-12-12T11:14:50Z

@jorijn I tried this already. I always disconnect and close all VPN software when using minikube (even before this problem), as I know they don't place nice together.

bzub · 2019-12-13T23:20:39Z

I see this issue as well. I believe it has something to do with the Blue Coat Unified Agent" installed on my corporate laptop. Do others having this issue see a bcua-service or symbol on their taskbar? I'm on MacOS.

For now as mentioned above I use minikube start --vm-driver=virtualbox to work around the problem.

tycooon · 2020-01-15T20:47:22Z

I have the same problem when connected to a corporate VPN via Tunnelblick. In case I turn it off the network inside minikube ssh is working just fine.

fhopfensperger · 2020-01-28T13:47:59Z

I have the same problem when connected to a corporate VPN via Cisco AnyConnect. Is there a way to configure hyperkit to use a different network connection?

priyawadhwa · 2020-01-28T20:18:31Z

I'm not sure how to fix the network connection issue, but the latest version of minikube no longer pulls kube-addon-manager. If you upgrade to our latest release then that should resolve a piece of this issue.

medyagh · 2020-03-04T19:52:13Z

@douglascamata I haven't heard from you I will close this issue, and I am sorry that you faced this isssue, please feel free to reopen this issue
I also suggest to try our new driver "docker" with latest version of minikbue which might solve your issue.

minikube start --vm-driver=docker

nomansadiq11 · 2020-07-18T13:51:36Z

restart network adapter and its works for me

kokizzu · 2022-01-28T05:57:08Z

@nomansadiq11 from host or from minikube ssh?

tstromberg added co/hyperkit Hyperkit related issues kind/support Categorizes issue or PR as a support question. labels Nov 20, 2019

tstromberg changed the title ~~v1.5.2 cannot pull kube-addon-manager-minikube's image~~ hyperkit: no internet connectivity, can't pull kube-addon-manager Nov 20, 2019

medyagh closed this as completed Mar 4, 2020

douglascamata mentioned this issue Aug 19, 2020

hyperkit: trouble accessing https://k8s.gcr.io #8902

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

hyperkit: no internet connectivity, can't pull kube-addon-manager #5951

hyperkit: no internet connectivity, can't pull kube-addon-manager #5951

douglascamata commented Nov 19, 2019 •

edited

Loading

douglascamata commented Nov 19, 2019

douglascamata commented Nov 19, 2019

tstromberg commented Nov 20, 2019

douglascamata commented Nov 20, 2019 •

edited

Loading

smoya commented Dec 3, 2019

jorijn commented Dec 9, 2019

douglascamata commented Dec 9, 2019

jorijn commented Dec 12, 2019

douglascamata commented Dec 12, 2019

bzub commented Dec 13, 2019

tycooon commented Jan 15, 2020

fhopfensperger commented Jan 28, 2020

priyawadhwa commented Jan 28, 2020

medyagh commented Mar 4, 2020

nomansadiq11 commented Jul 18, 2020

kokizzu commented Jan 28, 2022

hyperkit: no internet connectivity, can't pull kube-addon-manager #5951

hyperkit: no internet connectivity, can't pull kube-addon-manager #5951

Comments

douglascamata commented Nov 19, 2019 • edited Loading

douglascamata commented Nov 19, 2019

douglascamata commented Nov 19, 2019

tstromberg commented Nov 20, 2019

douglascamata commented Nov 20, 2019 • edited Loading

smoya commented Dec 3, 2019

jorijn commented Dec 9, 2019

douglascamata commented Dec 9, 2019

jorijn commented Dec 12, 2019

douglascamata commented Dec 12, 2019

bzub commented Dec 13, 2019

tycooon commented Jan 15, 2020

fhopfensperger commented Jan 28, 2020

priyawadhwa commented Jan 28, 2020

medyagh commented Mar 4, 2020

nomansadiq11 commented Jul 18, 2020

kokizzu commented Jan 28, 2022

douglascamata commented Nov 19, 2019 •

edited

Loading

douglascamata commented Nov 20, 2019 •

edited

Loading