Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

minikube start hung: etcd CrashLoopBackoff #2765

Closed
vinceskahan opened this issue Apr 23, 2018 · 65 comments
Closed

minikube start hung: etcd CrashLoopBackoff #2765

vinceskahan opened this issue Apr 23, 2018 · 65 comments
Labels
co/etcd startup failures where etcd may be involved co/virtualbox ev/CrashLoopBackOff Crash Loop Backoff events ev/hung-start kind/bug Categorizes issue or PR as related to a bug. os/macos

Comments

@vinceskahan
Copy link

vinceskahan commented Apr 23, 2018

BUG REPORT

Environment:

Minikube version
v0.26.1

  • OS
    MacOS High Sierra 10.13.4

  • VM Driver
    Virtual Box

  • ISO version
    Latest as of today

  • Install tools:

  • Others:

What happened:

Initial minikube start hangs at 'Starting cluster components...' and never completes. Note that the vm 'is' running, and I can log into it from the Virtual Box console. From another terminal window, 'minikube ip' does return an ip address.

$ minikube start
Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Downloading kubeadm v1.10.0
Downloading kubelet v1.10.0
Finished Downloading kubelet v1.10.0
Finished Downloading kubeadm v1.10.0
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...

What you expected to happen:

Ummm - for it to complete, of course...

How to reproduce it (as minimally and precisely as possible):

minikube start

Output of minikube logs (if applicable):

(attached)
minikube-logs-output.txt

@davidnewman
Copy link
Contributor

I have the same issue except that I am using the hyperkit driver, and I'm installing a k8s 1.9.0 cluster. Gets stuck at "Starting cluster components"

This is on MacOS High Sierra (10.13.3)

@mgdevstack
Copy link

Same issue for me.

@eldada
Copy link

eldada commented Apr 24, 2018

If you have an old ~/.minikube, try

$ minikube delete
$ rm -rf ~/.minikube

Try starting up again.

@playniuniu
Copy link

same issue here

Also try minikube delete && rm -rf ~/.minikube

But it not worked

Use minikube logs -f and see the error is:

Apr 24 15:44:59 minikube kubelet[3287]: E0424 15:44:59.532988 3287 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://192.168.99.100:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.99.100:8443: getsockopt: connection refused
Apr 24 15:44:59 minikube kubelet[3287]: E0424 15:44:59.533844 3287 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.99.100:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&limit=500&resourceVersion=0: dial tcp 192.168.99.100:8443: getsockopt: connection refused

@slmagus
Copy link

slmagus commented Apr 24, 2018

Similar problem on minikube for windows. I did a minikube delete && rm ~/.minikube and reinstalled v0.25.2 and minikube started in <30 seconds.

@mgdevstack
Copy link

@slmagus v0.25.2 is running well as expected. But seems there is an issue #2753 with 0.26.0 and 0.26.1. Since last 10 days, I'm facing issue, before that 0.26.* were working.

@ukclivecox
Copy link

Same issue on Linux Mint (ubuntu derived distro).

@lokeshmadan
Copy link

Same issue with me : Apr 26 21:37:10 minikube kubelet[3154]: E0426 21:37:10.657560 3154 kubelet_node_status.go:106] Unable to register node "minikube" with API server: Post https://192.168.64.22:8443/api/v1/nodes: dial tcp 192.168.64.22:8443: getsockopt: connection refused
Apr 26 21:37:11 minikube kubelet[3154]: E0426 21:37:11.445468 3154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.64.22:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&limit=500&resourceVersion=0: dial tcp 192.168.64.22:8443: getsockopt: connection refused
Apr 26 21:37:11 minikube kubelet[3154]: E0426 21:37:11.446275 3154 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://192.168.64.22:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.64.22:8443: getsockopt: connection refused

@IBMRob
Copy link

IBMRob commented Apr 27, 2018

Same issue with same config as @vinceskahan also getting the getsockopt: connection refused errors although I do have some extra cmdline parms minikube start --cpus 4 --disk-size 60g --memory 6144 --vm-driver=virtualbox --extra-config=apiserver.ServiceNodePortRange=80-32000

@alanbrent
Copy link

alanbrent commented Apr 27, 2018

Same issue here, on MacOS w/ hyperkit driver, kubeadm bootstrapper, k8s version v1.9.6. minikube ssh does work, and using that I see that it appears that the kube-apiserver container keeps unexpectedly exiting.

$ minikube ssh 'docker ps -a'
CONTAINER ID        IMAGE                                                    COMMAND                  CREATED              STATUS                          PORTS               NAMES
601c48f3bfb8        gcr.io/google_containers/kube-apiserver-amd64            "kube-apiserver --..."   About a minute ago   Exited (2) About a minute ago                       k8s_kube-apiserver_kube-apiserver-minikube_kube-system_268917328163745905396fe3f66a8905_5
000fe7a44e2b        gcr.io/google_containers/etcd-amd64                      "etcd --listen-cli..."   4 minutes ago        Up 4 minutes                                        k8s_etcd_etcd-minikube_kube-system_d2c35e6cc76ad8c4c16b9f75c1f14a60_2
de44ce926d64        gcr.io/google_containers/kube-controller-manager-amd64   "kube-controller-m..."   4 minutes ago        Up 4 minutes                                        k8s_kube-controller-manager_kube-controller-manager-minikube_kube-system_ab782b037996db13fa5d60b54d0fdc0c_0
709611ac3ea6        k8s.gcr.io/kube-addon-manager                            "/opt/kube-addons.sh"    4 minutes ago        Up 4 minutes                                        k8s_kube-addon-manager_kube-addon-manager-minikube_kube-system_c9760ab157812e306964c796adcfaefb_1
70498c0aea0d        gcr.io/google_containers/kube-scheduler-amd64            "kube-scheduler --..."   4 minutes ago        Up 4 minutes                                        k8s_kube-scheduler_kube-scheduler-minikube_kube-system_0f95caae4a7fffab092dfa6db3c27347_3
a3f9ff1d712d        gcr.io/google_containers/pause-amd64:3.0                 "/pause"                 4 minutes ago        Up 4 minutes                                        k8s_POD_etcd-minikube_kube-system_d2c35e6cc76ad8c4c16b9f75c1f14a60_2
0b12b0502878        gcr.io/google_containers/pause-amd64:3.0                 "/pause"                 4 minutes ago        Up 4 minutes                                        k8s_POD_kube-scheduler-minikube_kube-system_0f95caae4a7fffab092dfa6db3c27347_3
acc562ef0a02        gcr.io/google_containers/pause-amd64:3.0                 "/pause"                 4 minutes ago        Up 4 minutes                                        k8s_POD_kube-addon-manager-minikube_kube-system_c9760ab157812e306964c796adcfaefb_1
ade51feb9df4        gcr.io/google_containers/pause-amd64:3.0                 "/pause"                 4 minutes ago        Up 4 minutes                                        k8s_POD_kube-controller-manager-minikube_kube-system_ab782b037996db13fa5d60b54d0fdc0c_0
aed6f830a8ef        gcr.io/google_containers/pause-amd64:3.0                 "/pause"                 4 minutes ago        Up 4 minutes                                        k8s_POD_kube-apiserver-minikube_kube-system_268917328163745905396fe3f66a8905_0

Looking into it a bit farther, it looks like --require-kubeconfig is being passed to kube-apiserver but the flag is unknown/unsupported.

$ docker logs 601c48f3bfb8
unknown flag: --require-kubeconfig
# REMOVED THE REST OF THE OUTPUT #

Updating the k8s version to v1.10.2 appears to fix the issue for me, but I'm sure that's not really much of a fix for many people.


TLDR looks like minikube is passing unsupported flags to some versions of the k8s API server.

Here's a reference link re: the --require-kubeconfig flag and its recent deprecation/removal.

@gsaslis
Copy link

gsaslis commented Apr 29, 2018

just to share that on minikube version: v0.26.1, --extra-config=apiserver.service-node-port-range=90-32000 worked for me.

p.s. the minikube logs -f helped me identify what the problem was.

@vinceskahan
Copy link
Author

update - the '--extra-config' that gsaslis mentioned above did 'not' work for me. Total hang to the point where the computer almost totally locked up and I had to reset it.

Downgrading minikube to 0.25.2 'did' work just fine, so the problem appears to be something in the 0.26.x codebase.

@dpacierpnik
Copy link

dpacierpnik commented Apr 30, 2018

I have the same issue with hyperkit v0.20171204-60-g0e5b6b driver and MacOS High Sierra 10.13.4. In minikube logs got:

Apr 30 11:37:55 minikube kubelet[2977]: E0430 11:37:55.570213    2977 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://192.168.64.100:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.64.100:8443: getsockopt: connection refused
Apr 30 11:37:55 minikube kubelet[2977]: E0430 11:37:55.572007    2977 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.64.100:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&limit=500&resourceVersion=0: dial tcp 192.168.64.100:8443: getsockopt: connection refused
Apr 30 11:37:55 minikube kubelet[2977]: E0430 11:37:55.572993    2977 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://192.168.64.100:8443/api/v1/nodes?fieldSelector=metadata.name%3Dminikube&limit=500&resourceVersion=0: dial tcp 192.168.64.100:8443: getsockopt: connection refused

Neither removing .minikube directory nor --extra-config=apiserver.service-node-port-range=90-32000 works for me

@alanbrent
Copy link

alanbrent commented May 1, 2018

I'm pretty sure the issue is in this line, which was added as part of this commit in the lead up to the 0.26 release.

AFAICT, the --require-kubeconfig flag stopped working in 1.9, so what we'd need is to only enable it for versions less than 1.9.0-alpha.0.

@krancour
Copy link
Member

krancour commented May 1, 2018

You can work around this by using the localkube bootstrapper.

I hope a fix for this is forthcoming soon.

@ttripp
Copy link

ttripp commented May 1, 2018

Temporary workaround:

Using these instructions as an example, I forced an install of 0.25.2 in the meantime:

minikube stop
minikube delete
brew cask uninstall --force minikube
brew cask edit minikube
#See below for file
brew cask install minikube

Using this file:

cask 'minikube' do
  version '0.25.2'
  sha256 'dc5b00c4a06e8160bd607732c9a2294598d803716e353293b4463cc2c9539eec'

  # storage.googleapis.com/minikube was verified as official when first introduced to the cask
  url "https://storage.googleapis.com/minikube/releases/v#{version}/minikube-darwin-amd64"
  appcast 'https://github.com/kubernetes/minikube/releases.atom',
          checkpoint: 'dc5b00c4a06e8160bd607732c9a2294598d803716e353293b4463cc2c9539eec'
  name 'Minikube'
  homepage 'https://github.com/kubernetes/minikube'

  depends_on formula: 'kubernetes-cli'
  container type: :naked

  binary 'minikube-darwin-amd64', target: 'minikube'

  zap trash: '~/.minikube'
end

@dlorenc
Copy link
Contributor

dlorenc commented May 1, 2018

The fix still isn't clear to me. What versions of Kubernetes is this reproducing with?

It looks to me like the require-kubeconfig flag was only removed for 1.10, and should still have been present in 1.9:
kubernetes/kubernetes#58367

@Cryptophobia
Copy link

This happens on Linux (Fedora), not just mac and seems to be a problem with versions > 0.26.x...

@reghuramrv
Copy link

As @dlorenc said it looks like bug caused by removing the require-kubeconfig flag. It worked perfectly fine v1.10.0

@dlorenc
Copy link
Contributor

dlorenc commented May 3, 2018

This happens on Linux (Fedora), not just mac and seems to be a problem with versions > 0.26.x...

Which versions of Kubernetes (not minikube) are you seeing this with?

@Cryptophobia
Copy link

@dlorenc

Which versions of Kubernetes (not minikube) are you seeing this with?

I was seeing this when trying to run k8s v1.9.4 and v1.9.7 locally. I had to downgrade to 0.25.2 in order to make it work.

ssalaues pushed a commit to scality/Zenko that referenced this issue May 3, 2018
There are known issues with the current latest versions of helm and minikube
which is causing failures for new users trying to deploy zenko on minikube and
therefore explicitly declaring known working versions for future reference.

Issues with new versions of minikube
kubernetes/minikube#2765
Issues with new versions of helm
helm/helm#3991
@AnoyiX
Copy link

AnoyiX commented May 4, 2018

same issues , failed to run minikube start,minikube version: v0.26.1 , OSX

➜  ~ minikube start
Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...


@bitfield
Copy link

bitfield commented May 4, 2018

$ minikube version
minikube version: v0.26.1

$ minikube start
Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Downloading kubelet v1.10.0
Downloading kubeadm v1.10.0
Finished Downloading kubelet v1.10.0
Finished Downloading kubeadm v1.10.0
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
E0504 15:02:46.676882   24473 start.go:281] Error restarting cluster:  restarting kube-proxy: waiting for kube-proxy to be up for configmap update: timed out waiting for the condition

$ minikube status
minikube: Running
cluster: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.100

$ minikube stop
Stopping local Kubernetes cluster...
Machine stopped.

$ minikube start
Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
[hang]

@AnoyiX
Copy link

AnoyiX commented May 5, 2018

try to run minikube start --bootstrapper=localkube , it works well...

➜  ~ minikube start --bootstrapper=localkube
Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Getting VM IP address...
WARNING: The localkube bootstrapper is now deprecated and support for it
will be removed in a future release. Please consider switching to the kubeadm bootstrapper, which
is intended to replace the localkube bootstrapper. To disable this message, run
[minikube config set ShowBootstrapperDeprecationNotification false]
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
Kubectl is now configured to use the cluster.
Loading cached images from config file.

@Robot684
Copy link

@playniuniu 不行,我测试了从0.26以上的版本都不可以,我现在在测试你说的启动方式

@Robot684
Copy link

Robot684 commented May 18, 2018

@playniuniu 我试了下,为什么cluster是关闭的呢?

`
➜ ~ kubectl get nodes
NAME STATUS ROLES AGE VERSION
minikube Ready 1m v1.10.0

➜ ~ minikube status
minikube: Running
cluster: Stopped
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.64.3
`

@playniuniu
Copy link

@Robot684 我的也是,应该不影响使用吧

@Robot684
Copy link

@playniuniu 你可以进行后续的模拟创建pod测试吗?

@gsaslis
Copy link

gsaslis commented May 18, 2018

@playniuniu / @Robot684 please note that when you write a comment on a GitHub issue (which has so far 25+ participants), by default, a notification email is sent to each and everyone of them.

when your comment is not in english, you are essentially sending us messages in your own language, which we all can't understand. that doesn't feel very nice.

please reconsider using just English. ;)

@Hujun
Copy link

Hujun commented May 19, 2018

same problem on macOS High Sierra 10.13.4 with minikube v0.27.0

maybe a different cause

log:

May 19 15:59:00 minikube kubelet[2893]: I0519 15:59:00.888564    2893 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 19 15:59:00 minikube kubelet[2893]: I0519 15:59:00.893989    2893 kubelet_node_status.go:82] Attempting to register node minikube
May 19 15:59:00 minikube kubelet[2893]: E0519 15:59:00.894672    2893 kubelet_node_status.go:106] Unable to register node "minikube" with API server: Post https://192.168.64.2:8443/api/v1/nodes: dial tcp 192.168.64.2:8443: getsockopt: connection refused
May 19 15:59:01 minikube kubelet[2893]: E0519 15:59:01.023553    2893 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause-amd64:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
May 19 15:59:01 minikube kubelet[2893]: E0519 15:59:01.023654    2893 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-apiserver-minikube_kube-system(6c3bcd23aafaaad42909185679ecdb62)" failed: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause-amd64:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
May 19 15:59:01 minikube kubelet[2893]: E0519 15:59:01.023696    2893 kuberuntime_manager.go:646] createPodSandbox for pod "kube-apiserver-minikube_kube-system(6c3bcd23aafaaad42909185679ecdb62)" failed: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause-amd64:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
May 19 15:59:01 minikube kubelet[2893]: E0519 15:59:01.023865    2893 pod_workers.go:186] Error syncing pod 6c3bcd23aafaaad42909185679ecdb62 ("kube-apiserver-minikube_kube-system(6c3bcd23aafaaad42909185679ecdb62)"), skipping: failed to "CreatePodSandbox" for "kube-apiserver-minikube_kube-system(6c3bcd23aafaaad42909185679ecdb62)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-apiserver-minikube_kube-system(6c3bcd23aafaaad42909185679ecdb62)\" failed: rpc error: code = Unknown desc = failed pulling image \"k8s.gcr.io/pause-amd64:3.1\": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
May 19 15:59:01 minikube kubelet[2893]: E0519 15:59:01.033510    2893 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause-amd64:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
May 19 15:59:01 minikube kubelet[2893]: E0519 15:59:01.033594    2893 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "etcd-minikube_kube-system(41069a7dedf9b86c8d5dd0a50a0ac538)" failed: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause-amd64:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
May 19 15:59:01 minikube kubelet[2893]: E0519 15:59:01.033616    2893 kuberuntime_manager.go:646] createPodSandbox for pod "etcd-minikube_kube-system(41069a7dedf9b86c8d5dd0a50a0ac538)" failed: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause-amd64:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
May 19 15:59:01 minikube kubelet[2893]: E0519 15:59:01.033736    2893 pod_workers.go:186] Error syncing pod 41069a7dedf9b86c8d5dd0a50a0ac538 ("etcd-minikube_kube-system(41069a7dedf9b86c8d5dd0a50a0ac538)"), skipping: failed to "CreatePodSandbox" for "etcd-minikube_kube-system(41069a7dedf9b86c8d5dd0a50a0ac538)" with CreatePodSandboxError: "CreatePodSandbox for pod \"etcd-minikube_kube-system(41069a7dedf9b86c8d5dd0a50a0ac538)\" failed: rpc error: code = Unknown desc = failed pulling image \"k8s.gcr.io/pause-amd64:3.1\": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
May 19 15:59:01 minikube kubelet[2893]: E0519 15:59:01.410607    2893 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://192.168.64.2:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.64.2:8443: getsockopt: connection refused
May 19 15:59:01 minikube kubelet[2893]: E0519 15:59:01.410629    2893 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://192.168.64.2:8443/api/v1/nodes?fieldSelector=metadata.name%3Dminikube&limit=500&resourceVersion=0: dial tcp 192.168.64.2:8443: getsockopt: connection refused
May 19 15:59:01 minikube kubelet[2893]: E0519 15:59:01.411414    2893 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.64.2:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&limit=500&resourceVersion=0: dial tcp 192.168.64.2:8443: getsockopt: connection refused
May 19 15:59:02 minikube kubelet[2893]: E0519 15:59:02.414657    2893 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://192.168.64.2:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.64.2:8443: getsockopt: connection refused
May 19 15:59:02 minikube kubelet[2893]: E0519 15:59:02.414683    2893 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://192.168.64.2:8443/api/v1/nodes?fieldSelector=metadata.name%3Dminikube&limit=500&resourceVersion=0: dial tcp 192.168.64.2:8443: getsockopt: connection refused
May 19 15:59:02 minikube kubelet[2893]: E0519 15:59:02.416531    2893 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.64.2:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&limit=500&resourceVersion=0: dial tcp 192.168.64.2:8443: getsockopt: connection refused
May 19 15:59:02 minikube kubelet[2893]: I0519 15:59:02.699420    2893 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 19 15:59:03 minikube kubelet[2893]: E0519 15:59:03.416798    2893 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://192.168.64.2:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.64.2:8443: getsockopt: connection refused
May 19 15:59:03 minikube kubelet[2893]: E0519 15:59:03.418399    2893 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://192.168.64.2:8443/api/v1/nodes?fieldSelector=metadata.name%3Dminikube&limit=500&resourceVersion=0: dial tcp 192.168.64.2:8443: getsockopt: connection refused
May 19 15:59:03 minikube kubelet[2893]: E0519 15:59:03.420191    2893 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.64.2:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&limit=500&resourceVersion=0: dial tcp 192.168.64.2:8443: getsockopt: connection refused
May 19 15:59:04 minikube kubelet[2893]: E0519 15:59:04.424624    2893 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://192.168.64.2:8443/api/v1/nodes?fieldSelector=metadata.name%3Dminikube&limit=500&resourceVersion=0: dial tcp 192.168.64.2:8443: getsockopt: connection refused
May 19 15:59:04 minikube kubelet[2893]: E0519 15:59:04.424757    2893 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://192.168.64.2:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.64.2:8443: getsockopt: connection refused
May 19 15:59:04 minikube kubelet[2893]: E0519 15:59:04.424864    2893 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.64.2:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&limit=500&resourceVersion=0: dial tcp 192.168.64.2:8443: getsockopt: connection refused
May 19 15:59:05 minikube kubelet[2893]: E0519 15:59:05.431137    2893 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.64.2:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&limit=500&resourceVersion=0: dial tcp 192.168.64.2:8443: getsockopt: connection refused
May 19 15:59:05 minikube kubelet[2893]: E0519 15:59:05.431632    2893 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://192.168.64.2:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.64.2:8443: getsockopt: connection refused
May 19 15:59:05 minikube kubelet[2893]: E0519 15:59:05.431799    2893 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://192.168.64.2:8443/api/v1/nodes?fieldSelector=metadata.name%3Dminikube&limit=500&resourceVersion=0: dial tcp 192.168.64.2:8443: getsockopt: connection refused
May 19 15:59:05 minikube kubelet[2893]: E0519 15:59:05.687566    2893 eviction_manager.go:246] eviction manager: failed to get get summary stats: failed to get node info: node "minikube" not found
May 19 15:59:06 minikube kubelet[2893]: E0519 15:59:06.434120    2893 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://192.168.64.2:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.64.2:8443: getsockopt: connection refused
May 19 15:59:06 minikube kubelet[2893]: E0519 15:59:06.435347    2893 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://192.168.64.2:8443/api/v1/nodes?fieldSelector=metadata.name%3Dminikube&limit=500&resourceVersion=0: dial tcp 192.168.64.2:8443: getsockopt: connection refused
May 19 15:59:06 minikube kubelet[2893]: E0519 15:59:06.436853    2893 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.64.2:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&limit=500&resourceVersion=0: dial tcp 192.168.64.2:8443: getsockopt: connection refused

problem solved with localkube

minikube start --vm-driver=hyperkit --boostrapper=localkube

@vinaynair
Copy link

I too am encountering the same set of problems :-

Configuration

  • Minikube: Both 0.26.1 as well as 0.27
  • OS: osx 10.13.4

Issues:

See attempts below that did not work :-

minikube start --vm-driver=hyperkit --bootstrapper=localkube

$> kubectl get all --all-namespaces
NAMESPACE     NAME                              READY     STATUS              RESTARTS   AGE
kube-system   pod/kube-addon-manager-minikube   0/1       ContainerCreating   0          8m

NAMESPACE   NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
default     service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   8m

minikube start

May 19 19:57:08 minikube kubelet[2986]: E0519 19:57:08.559046    2986 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://192.168.64.5:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.64.5:8443: getsockopt: connection refused
May 19 19:57:09 minikube kubelet[2986]: E0519 19:57:09.561264    2986 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.64.5:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&limit=500&resourceVersion=0: dial tcp 192.168.64.5:8443: getsockopt: connection refused
May 19 19:57:09 minikube kubelet[2986]: E0519 19:57:09.561264    2986 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://192.168.64.5:8443/api/v1/nodes?fieldSelector=metadata.name%3Dminikube&limit=500&resourceVersion=0: dial tcp 192.168.64.5:8443: getsockopt: connection refused
May 19 19:57:09 minikube kubelet[2986]: E0519 19:57:09.561585    2986 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://192.168.64.5:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.64.5:8443: getsockopt: connection refused
May 19 19:57:09 minikube kubelet[2986]: E0519 19:57:09.736862    2986 eviction_manager.go:246] eviction manager: failed to get get summary stats: failed to get node info: node "minikube" not found

@pstinghua
Copy link

+1

@DickChesterwood
Copy link

DickChesterwood commented May 21, 2018

Still hanging for me - mac, linux and windows - on the new 0.27 (unless the deprecated --bootstrapper=localkube is applied)

@AnoyiX
Copy link

AnoyiX commented May 22, 2018

@vinaynair Maybe you cannot pull the images, try to run kubectl get events -n kube-system get more details

@sc68cal
Copy link

sc68cal commented May 22, 2018

Same here - High Sierra and had to use --bootstrapper=localkube to get a working cluster.

@ningning54321
Copy link

ningning54321 commented May 24, 2018

Similar error as @vinaynair, I had thought it was my environment issue, while it seems a common one.

$ minikube start --vm-driver=hyperkit --bootstrapper=localkube
$ kubectl get po --all-namespaces
NAMESPACE     NAME                                READY     STATUS              RESTARTS   AGE
default       samplecontroller-5686f7649f-c2r42   0/1       ContainerCreating   0          13d
kube-system   kube-addon-manager-minikube         0/1       ContainerCreating   0          13d

$ kubectl get events -n kube-system

12d         12d          1         kube-scheduler-minikube.152d9942416d2a42            Pod                     Normal    SuccessfulMountVolume    kubelet, minikube    MountVolume.SetUp succeeded for volume "kubeconfig" 
12d         12d          346       kube-scheduler-minikube.152d99465a77c2d5            Pod                     Warning   FailedCreatePodSandBox   kubelet, minikube    Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause-amd64:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
12d         12d          70        kube-scheduler-minikube.152d99f6e5e6fedf            Pod                     Warning   FailedCreatePodSandBox   kubelet, minikube    Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause-amd64:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 10.0.2.3:53: server misbehaving

@dlorenc
Copy link
Contributor

dlorenc commented May 24, 2018

I think this was fixed with #2791

@dlorenc dlorenc closed this as completed May 24, 2018
@sc68cal
Copy link

sc68cal commented May 24, 2018

It does not appear that fixed the issue, because I encountered this issue two days ago and only was able to solve it by using the workaround.

scolli572 » minikube version                                                                                                                                                                        1 ↵
minikube version: v0.27.0

@vinceskahan
Copy link
Author

agree - this is 'not' fixed in 0.27.0 without using the workaround

@arbourd
Copy link

arbourd commented May 24, 2018

Ya, and localkube isn't really a workaround -- it's a deprecated bootstrapper. This needs to work with kubeadm as well.

@dlorenc
Copy link
Contributor

dlorenc commented May 24, 2018

I think there might have been several different underlying issues being tracked in here. Would someone still experiencing this mind opening another issue with all the info from the template?

@sc68cal
Copy link

sc68cal commented May 24, 2018

I think closing an issue and asking a user to create a brand new issue from a template is .... perhaps not ideal.

@dlorenc
Copy link
Contributor

dlorenc commented May 24, 2018

Unfortunately there's just not enough information in any of these comments beyond the initial ones (involving the --require-kubeconfig flag, which was fixed in the linked PR). I'm sure there are other issues, but the best way to help us find and fix them is to include the debugging information asked for in the template.

@vinceskahan
Copy link
Author

vinceskahan commented May 24, 2018

@dlorenc - I have to call baloney on that one. Please read 'my' original problem description for this ticket at the very top. The expectation is that I can run "minikube start" per the docs, with no options or flags or workarounds and have the blasted thing actually start. If that's somehow unclear, I can't help you.

All the other comments, in my reading, are people mentioning:

  • it's broken for me too on (insert platform description here)
  • but it used to work in 0.25.2 for me too on (insert platform description here)
  • and it seems it can be worked around with version x.xx.x with this set of deprecated flags

It's broken if it used to work before. It's broken if deprecated workaround flags are required. So it's still broken for me (the original poster) on the same High Sierra Macbook that I reported it on.

And I 'did' try to use 0.27.0 and did verify it is still broken here on that platform.

@dlorenc
Copy link
Contributor

dlorenc commented May 24, 2018

I'm going to lock this issue in the interest of keeping things productive. If you'd like to help see issues get fixed, please open a new issue with the output of "minikube info" and "minikube logs" attached. Without that we can't really debug anything.

@kubernetes kubernetes locked and limited conversation to collaborators May 24, 2018
@tstromberg tstromberg added kind/bug Categorizes issue or PR as related to a bug. os/macos co/virtualbox ev/hung-start labels Sep 20, 2018
@tstromberg tstromberg changed the title minikube start hangs forever on mac minikube start hung: etcd CrashLoopBackoff Sep 20, 2018
@tstromberg tstromberg added co/etcd startup failures where etcd may be involved ev/CrashLoopBackOff Crash Loop Backoff events labels Sep 20, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
co/etcd startup failures where etcd may be involved co/virtualbox ev/CrashLoopBackOff Crash Loop Backoff events ev/hung-start kind/bug Categorizes issue or PR as related to a bug. os/macos
Projects
None yet
Development

No branches or pull requests