Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mystery hang during /usr/bin/kubeadm init (with no --extra-config options) #2939

Closed
hsebastian opened this issue Jun 27, 2018 · 23 comments
Closed

Mystery hang during /usr/bin/kubeadm init (with no --extra-config options) #2939

hsebastian opened this issue Jun 27, 2018 · 23 comments
Assignees
@tstromberg
Labels
ev/hung-start kind/bug Categorizes issue or PR as related to a bug. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. os/linux priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.

Comments

@hsebastian
Copy link

BUG REPORT

Environment:

$ minikube version
minikube version: v0.28.0

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.4 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.4 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

$ grep DriverName ~/.minikube/machines/minikube/config.json
    "DriverName": "virtualbox",

$ grep -i ISO ~/.minikube/machines/minikube/config.json
        "Boot2DockerURL": "file:///home/hans/.minikube/cache/iso/minikube-v0.28.0.iso",

$ cat /home/hans/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority: /home/hans/.minikube/ca.crt
    server: https://192.168.99.103:8443
  name: minikube
contexts:
- context:
    cluster: minikube
    user: minikube
  name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
  user:
    client-certificate: /home/hans/.minikube/client.crt
    client-key: /home/hans/.minikube/client.key

What happened: When running minikube start it hangs at Starting cluster components.... In the console of the minikube VM, I get prompted for minikube login: . I can login with root.

What you expected to happen: minikube is started and configured.

How to reproduce it (as minimally and precisely as possible):

minikube delete
minikube start --logtostderr --loglevel 0

Output of minikube logs (if applicable):

Connecting to cluster...
Setting up kubeconfig...
I0627 11:33:56.307740   24241 config.go:101] Using kubeconfig:  /home/hans/.kube/config
Starting cluster components...
I0627 11:33:56.308563   24241 ssh_runner.go:80] Run with output: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI

Anything else do we need to know:
Installed minikube with

curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/

Installed virtualbox with

sudo apt-get remove --purge virtualbox-5.2
sudo apt-get remove --purge virtualbox
rm -rf ~/.config/VirtualBox
sudo dpkg -i ~/Downloads/virtualbox-5.2_5.2.12-122591\~Ubuntu\~xenial_amd64.deb
@imroc
Copy link

imroc commented Jun 27, 2018

I got the same issue

@Arnavion
Copy link

Are you deploying a 1.11.0 cluster? If yes, it's because of #2942

@hsebastian
Copy link
Author

@Arnavion No, I remember it was 1.10.0 and I also tried with 1.9.4 but still fails. I didn't know there is 1.11.0.

@IBMRob
Copy link

IBMRob commented Jul 4, 2018

We are also hitting this, it looks the same as we were hitting previously as well but the issue was closed - #2765

For now some of us are using 0.25.2 OK.
Others have problem with that version which need a newer docker version to pick up a fix so are completely blocked as all newer versions hit this problem

@Karreg
Copy link

Karreg commented Aug 7, 2018 •
edited
Loading

Same here for Minikube 0.28.2.
It happens with both VirtualBox and Hyper-V drivers

Logs aren't really helpful:

Starting cluster components...
E0807 15:24:02.028641 10532 start.go:300] Error starting cluster: kubeadm init error
sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI &&
sudo /usr/bin/kubeadm alpha phase addon kube-dns
running command: : running command:
sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI &&
sudo /usr/bin/kubeadm alpha phase addon kube-dns

@exu
Copy link

exu commented Aug 10, 2018

Same here but when starting with logs I've got multiple hundreds of messages like this:

minikube start --logtostderr --loglevel 0                                                     

I0810 10:49:23.576654    8694 kubernetes.go:119] error getting Pods with label selector "k8s-app=kube-proxy" [Get https://192.168.99.100:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy: dial tcp 192.168.99.100:8443: getsockopt: connection refused]
I0810 10:49:24.076413    8694 kubernetes.go:119] error getting Pods with label selector "k8s-app=kube-proxy" [Get https://192.168.99.100:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy: dial tcp 192.168.99.100:8443: getsockopt: connection refused]
I0810 10:49:24.576264    8694 kubernetes.go:119] error getting Pods with label selector "k8s-app=kube-proxy" [Get https://192.168.99.100:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy: dial tcp 192.168.99.100:8443: getsockopt: connection refused]
I0810 10:49:25.076410    8694 kubernetes.go:119] error getting Pods with label selector "k8s-app=kube-proxy" [Get https://192.168.99.100:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy: dial tcp 192.168.99.100:8443: getsockopt: connection refused]
I0810 10:49:25.576426    8694 kubernetes.go:119] error getting Pods with label selector "k8s-app=kube-proxy" [Get https://192.168.99.100:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy: dial tcp 192.168.99.100:8443: getsockopt: connection refused]
I0810 10:49:26.076719    8694 kubernetes.go:119] error getting Pods with label selector "k8s-app=kube-proxy" [Get https://192.168.99.100:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy: dial tcp 192.168.99.100:8443: getsockopt: connection refused]
I0810 10:49:26.577313    8694 kubernetes.go:119] error getting Pods with label selector "k8s-app=kube-proxy" [Get https://192.168.99.100:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy: dial tcp 192.168.99.100:8443: getsockopt: connection refused]
I0810 10:49:27.077177    8694 kubernetes.go:119] error getting Pods with label selector "k8s-app=kube-proxy" [Get https://192.168.99.100:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy: dial tcp 192.168.99.100:8443: getsockopt: connection refused]
I0810 10:49:27.577812    8694 kubernetes.go:119] error getting Pods with label selector "k8s-app=kube-proxy" [Get https://192.168.99.100:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy: dial tcp 192.168.99.100:8443: getsockopt: connection refused]
I0810 10:49:28.078182    8694 kubernetes.go:119] error getting Pods with label selector "k8s-app=kube-proxy" [Get https://192.168.99.100:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy: dial tcp 192.168.99.100:8443: getsockopt: connection refused]
I0810 10:49:28.577055    8694 kubernetes.go:119] error getting Pods with label selector "k8s-app=kube-proxy" [Get https://192.168.99.100:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy: dial tcp 192.168.99.100:8443: getsockopt: connection refused]
I0810 10:49:39.076599    8694 kubernetes.go:119] error getting Pods with label selector "k8s-app=kube-proxy" [Get https://192.168.99.100:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy: net/http: TLS handshake timeout]
I0810 10:49:43.551134    8694 kubernetes.go:124] Found 0 Pods for label selector k8s-app=kube-proxy

and it finally hangs and not displaying anything

@frastel
Copy link

frastel commented Aug 10, 2018 •
edited
Loading

I'm also not able to boot a Kubernetes 1.10 Cluster with a Minikube v0.28.2 on a ubuntu xenial vm with "none" driver/localkube.
If I should open a new issue for this as I am using a different driver than @hsebastian please let me know.

After one hour I got this output

==> default: Starting local Kubernetes v1.10.0 cluster...
==> default: Starting VM...
==> default: Getting VM IP address...
==> default: Moving files into cluster...
==> default: Downloading kubeadm v1.10.0
==> default: Downloading kubelet v1.10.0
==> default: Finished Downloading kubelet v1.10.0
==> default: Finished Downloading kubeadm v1.10.0
==> default: Setting up certs...
==> default: Connecting to cluster...
==> default: Setting up kubeconfig...
==> default: Starting cluster components...
==> default: E0810 08:31:32.155043    4024 start.go:300] Error starting cluster:  kubeadm init error 
==> default: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI  &&
==> default: sudo /usr/bin/kubeadm alpha phase addon kube-dns
==> default:  running command: : running command: 
==> default: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI  &&
==> default: sudo /usr/bin/kubeadm alpha phase addon kube-dns
==> default: 
==> default:  output: [init] Using Kubernetes version: v1.10.0
==> default: [init] Using Authorization modes: [Node RBAC]
==> default: [preflight] Running pre-flight checks.
==> default: 	[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 17.09.1-ce. Max validated version: 17.03
==> default: 	[WARNING Hostname]: hostname "minikube" could not be reached
==> default: 	[WARNING Hostname]: hostname "minikube" lookup minikube on 10.0.2.3:53: no such host
==> default: 	[WARNING FileExisting-ebtables]: ebtables not found in system path
==> default: 	[WARNING FileExisting-socat]: socat not found in system path
==> default: 	[WARNING FileExisting-crictl]: crictl not found in system path
==> default: Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl
==> default: Flag --admission-control has been deprecated, Use --enable-admission-plugins or --disable-admission-plugins instead. Will be removed in a future version.
==> default: 	[WARNING ExtraArgs]: kube-apiserver: failed to parse extra argument --ServiceNodePortRange=80-32767
==> default: [certificates] Using the existing ca certificate and key.
==> default: [certificates] Using the existing apiserver certificate and key.
==> default: [certificates] Generated apiserver-kubelet-client certificate and key.
==> default: [certificates] Generated etcd/ca certificate and key.
==> default: [certificates] Generated etcd/server certificate and key.
==> default: [certificates] etcd/server serving cert is signed for DNS names [localhost] and IPs [127.0.0.1]
==> default: [certificates] Generated etcd/peer certificate and key.
==> default: [certificates] etcd/peer serving cert is signed for DNS names [minikube] and IPs [10.0.2.15]
==> default: [certificates] Generated etcd/healthcheck-client certificate and key.
==> default: [certificates] Generated apiserver-etcd-client certificate and key.
==> default: [certificates] Generated sa key and public key.
==> default: [certificates] Generated front-proxy-ca certificate and key.
==> default: [certificates] Generated front-proxy-client certificate and key.
==> default: [certificates] Valid certificates and keys now exist in "/var/lib/localkube/certs/"
==> default: [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
==> default: [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
==> default: [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
==> default: [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
==> default: [controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
==> default: [controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
==> default: [controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
==> default: [etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
==> default: [init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
==> default: [init] This might take a minute or longer if the control plane images have to be pulled.
==> default: 
==> default: Unfortunately, an error has occurred:
==> default: 	timed out waiting for the condition
==> default: 
==> default: This error is likely caused by:
==> default: 	- The kubelet is not running
==> default: 	- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
==> default: 	- Either there is no internet connection, or imagePullPolicy is set to "Never",
==> default: 	  so the kubelet cannot pull or find the following control plane images:
==> default: 		- k8s.gcr.io/kube-apiserver-amd64:v1.10.0
==> default: 		- k8s.gcr.io/kube-controller-manager-amd64:v1.10.0
==> default: 		- k8s.gcr.io/kube-scheduler-amd64:v1.10.0
==> default: 		- k8s.gcr.io/etcd-amd64:3.1.12 (only if no external etcd endpoints are configured)
==> default: 
==> default: If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
==> default: 	- 'systemctl status kubelet'
==> default: 	- 'journalctl -xeu kubelet'
==> default: couldn't initialize a Kubernetes cluster
==> default: : running command: 
==> default: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI  &&
==> default: sudo /usr/bin/kubeadm alpha phase addon kube-dns
==> default: 
==> default: .: exit status 1

In detail this call causes the problem

sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI  &&
==> default: sudo /usr/bin/kubeadm alpha phase addon kube-dns

With pinned Minikube version v0.25.2 everything works just fine.

@frastel
Copy link

frastel commented Aug 10, 2018

To reproduce this issue I have created a simple vagrant environment.

@hsebastian
Copy link
Author

FYI, this problem went away for me a few days ago after upgrading Virtualbox to Version 5.2.16 r123759 (Qt5.6.1).

@frastel
Copy link

frastel commented Aug 10, 2018

Mmh, can't confirm this for my setup. Virtualbox 5.2.16 r123759 (Qt5.6.3) and Vagrant 2.1.2 installed.
I don't think the Virtualbox version has something to do with my setup as I am using localkube.

@Misoder
Copy link

Misoder commented Aug 15, 2018

I'm also experiencing this on MacOS Sierra with minikube v0.28.2 trying to start a Kubernetes v1.10.0 cluster. I've tried with hyperkit as well as virtualbox with the same outcome that @frastel posted above. Process exiting with status 1 after sudo /usr/bin/kubeadm alpha phase addon kube-dns

@imjoey
Copy link

imjoey commented Aug 24, 2018

The same issue on on MacOS High Sierra 10.13.6 with minikube v0.28.2 and the driver of virtualbox v5.1.20 for kubernetes v1.10.0.

Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Downloading Minikube ISO
 160.27 MB / 160.27 MB [============================================] 100.00% 0s
Getting VM IP address...
Moving files into cluster...
Downloading kubeadm v1.10.0
Downloading kubelet v1.10.0
Finished Downloading kubeadm v1.10.0
Finished Downloading kubelet v1.10.0
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...

E0824 13:13:57.554917   80481 start.go:300] Error starting cluster:  kubeadm init error
sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI  &&
sudo /usr/bin/kubeadm alpha phase addon kube-dns
 running command: : running command:
sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI  &&
sudo /usr/bin/kubeadm alpha phase addon kube-dns

.: Process exited with status 1

@Custodia
Copy link

Custodia commented Aug 27, 2018 •
edited
Loading

@Arnavion

Is the only solution then to downgrade Kubernetes to an earlier version?

edit:

I actually tried downgrading my kubernetes cluster to v1.9.4 but it still hangs indefinitely on Starting cluster components.

This is the command I'm trying to run: sudo minikube start --bootstrapper kubeadm start --extra-config=kubelet.cgroup-driver=systemd --vm-driver=none --kubernetes-version=v1.9.4

Anywhere where I could look at logs or something of the sort to try to solve the problem?

@dkushner
Copy link

dkushner commented Sep 1, 2018

Experiencing the same issue on Ubuntu 18.04 with the following command:

minikube start --vm-driver=kvm2 --extra-config=apiserver.Authorization.Mode=RBAC

My minikube version is 0.28.2. As others have mentioned, I've tried specifying the kubernetes version and rolling back to previous versions after completely removing the ~/.minikube directory. No result.

Have to say, I can't remember the last time I was able to upgrade my minikube installation without incident.

@dkushner
Copy link

dkushner commented Sep 1, 2018 •
edited
Loading

As usual, immediately after posting my addition to this issue,I somehow stumbled upon a solution. After stripping off the --extra-config=apiserver.Authorization.Mode=RBAC from my command line and manually deleting the KVM2 network default that minikube had created in a previous run, I was able to complete the cluster bootstrapping. I'm still unclear on which of these was causing the issue so I'm going to run a few trials of destroying the entire cluster and all associated KVM2 resources.

EDIT: Turns out it was a mix of both, I had the --extra-config option incorrect as mentioned here but even after changing that and simply using minikube delete && rm -rf ~/.minikube I still had issues rebuilding the cluster under I manually deleted the kvm2 network via virsh.

@tstromberg tstromberg changed the title When running "minikube start" it hangs at "Starting cluster components...." Mystery hang during /usr/bin/kubeadm init Sep 19, 2018
@tstromberg
Copy link
Contributor

FYI, this step should timeout within 30 minutes and provide more insight into what happened.

That's no excuse for the lack of usable output in this case, however.

@tstromberg tstromberg added the kind/bug Categorizes issue or PR as related to a bug. label Sep 19, 2018
@tstromberg tstromberg changed the title Mystery hang during /usr/bin/kubeadm init Mystery hang during /usr/bin/kubeadm init (with no --extra-config options) Sep 19, 2018
@ghost
Copy link

ghost commented Sep 25, 2018 •
edited by ghost
Loading

I'm experiencing the same hang. Which ends after 10 minutes with:

E0925 13:25:33.415269   87898 start.go:305] Error restarting cluster:  restarting kube-proxy: waiting for kube-proxy to be up for configmap update: timed out waiting for the condition

kubectl shows some kube-system pods in a crash loop:

$ kubectl get --all-namespaces pods
NAMESPACE     NAME                                        READY     STATUS             RESTARTS   AGE
kube-system   default-http-backend-59868b7dd6-f8kjn       1/1       Running            4          23m
kube-system   etcd-minikube                               1/1       Running            0          4m
kube-system   heapster-9thf5                              1/1       Running            4          23m
kube-system   influxdb-grafana-nkrk8                      2/2       Running            8          23m
kube-system   kube-addon-manager-minikube                 1/1       Running            4          22m
kube-system   kube-apiserver-minikube                     1/1       Running            0          4m
kube-system   kube-controller-manager-minikube            1/1       Running            0          4m
kube-system   kube-scheduler-minikube                     1/1       Running            0          4m
kube-system   kubernetes-dashboard-5498ccf677-kkc4d       0/1       CrashLoopBackOff   11         23m
kube-system   nginx-ingress-controller-5984b97644-lm4d7   0/1       CrashLoopBackOff   14         23m
kube-system   registry-dpgj8                              1/1       Running            4          23m
kube-system   storage-provisioner                         0/1       CrashLoopBackOff   11         23m

Reading logs for those pods shows:

$ kubectl logs -n kube-system storage-provisioner
F0925 11:19:28.285587       1 main.go:37] Error getting server version: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout

$ kubectl logs -n kube-system nginx-ingress-controller-5984b97644-lm4d7
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:    0.16.2
  Build:      git-26eacf4
  Repository: https://github.com/kubernetes/ingress-nginx
-------------------------------------------------------------------------------

nginx version: nginx/1.13.12
W0925 11:19:40.063017      12 client_config.go:533] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0925 11:19:40.063507      12 main.go:183] Creating API client for https://10.96.0.1:443

$ kubectl logs -n kube-system  kubernetes-dashboard-5498ccf677-kkc4d
2018/09/25 11:19:00 Starting overwatch
2018/09/25 11:19:00 Using in-cluster config to connect to apiserver
2018/09/25 11:19:00 Using service account token for csrf signing
2018/09/25 11:19:00 No request provided. Skipping authorization
2018/09/25 11:19:30 Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout
Refer to our FAQ and wiki pages for more information: https://github.com/kubernetes/dashboard/wiki/FAQ

@tstromberg
Copy link
Contributor

tstromberg commented Sep 25, 2018 •
edited
Loading

@roelal - Those messages sound very similar to those mentioned in #3031 (networking issue?), so you should follow-up there. Any chance there is a firewall or proxy installed on your machine?

@rliuyi
Copy link

rliuyi commented Dec 2, 2018

Same issue here with:
MasOS: 10.14.1
minikube: 0.30.0
VM Driver: virtualbox

Finally solved this issue with minikube: 0.25.0, vm-driver: hyperkit

@xingbo
Copy link

xingbo commented Dec 5, 2018

Same issue here with:
MasOS: 10.14.1
minikube: 0.30.0
VM Driver: virtualbox

After some google, and then try to install driver hyperkit by the instruction here

curl -LO https://storage.googleapis.com/minikube/releases/latest/docker-machine-driver-hyperkit
&& sudo install -o root -g wheel -m 4755 docker-machine-driver-hyperkit /usr/local/bin/

Homebrew also has a formulae here, I believe it also works fine to install the driver.

And for convenient, set the default driver to hyperkit by minikube config set vm-driver hyperkit.

Finally this issue has gone with minikube: 0.30.0, vm-driver: hyperkit

@tstromberg
Copy link
Contributor

minikube v0.33.1 should now remove mystery hangs. You can watch the output streamed from kubeadm by using --alsologtostderr -v=1.

We'll also be dividing up the pull & run sections in the UI shortly, which should help, so I'm not closing this issue as obsolete just yet.

@tstromberg tstromberg self-assigned this Jan 24, 2019
@tstromberg tstromberg added the priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. label Jan 24, 2019
@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 29, 2019
@tstromberg
Copy link
Contributor

I believe this issue was resolved in the v1.1.0 release. Please try upgrading to the latest release of minikube, and if the same issue occurs, please re-open this bug. Thank you opening this bug report, and for your patience!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tstromberg tstromberg

Labels
ev/hung-start kind/bug Categorizes issue or PR as related to a bug. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. os/linux priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests