Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HTTP_PROXY set: error getting Pods with label selector "k8s-app=kube-proxy" ... kube-proxy: Service Unavailable #2726

Closed
kevinbin opened this issue Apr 16, 2018 · 17 comments
Closed

HTTP_PROXY set: error getting Pods with label selector "k8s-app=kube-proxy" ... kube-proxy: Service Unavailable #2726

kevinbin opened this issue Apr 16, 2018 · 17 comments
Labels
area/networking networking issues cause/firewall-or-proxy When firewalls or proxies seem to be interfering ev/apiserver-refused failures due to the master on 8443 returning "Connection Refused" help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/bug Categorizes issue or PR as related to a bug. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.

Comments

@kevinbin
Copy link

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Please provide the following details:
minikube start not listen port 8443

kubectl cluster-info
Kubernetes master is running at https://192.168.99.106:8443

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Unable to connect to the server: Service Unavailable

Environment:

Minikube version (use minikube version): 0.26.0

  • OS (e.g. from /etc/os-release): macOS 10.13.4
  • VM Driver: vbox
  • ISO version : minikube-v0.26.0.iso
  • Install tools: homebrew
  • Others:
    The above can be generated in one go with the following commands (can be copied and pasted directly into your terminal):
hongbin@MBP ~> kubectl cluster-info
Kubernetes master is running at https://192.168.99.106:8443

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Unable to connect to the server: Service Unavailable

hongbin@MBP ~> minikube start  --docker-env HTTP_PROXY=http://<myproxy>:21089 --docker-env HTTPS_PROXY=https://<myproxy>:21089 --docker-env no_proxy=192.168.99.0/24 --v=9 --logtostderr
...

Setting up kubeconfig...
I0416 15:57:31.012657   32868 config.go:101] Using kubeconfig:  /Users/hongbin/.kube/config
Starting cluster components...
I0416 15:57:31.015960   32868 ssh_runner.go:58] Run:
sudo kubeadm alpha phase certs all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase kubeconfig all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase controlplane all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase etcd local --config /var/lib/kubeadm.yaml

I0416 15:57:34.998529   32868 loader.go:357] Config loaded from file /Users/hongbin/.kube/config
I0416 15:57:35.002515   32868 round_trippers.go:386] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: minikube/v1.10.0 (darwin/amd64) kubernetes/fc32d2f" https://192.168.99.106:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy
I0416 15:57:36.008773   32868 round_trippers.go:405] GET https://192.168.99.106:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy  in 1006 milliseconds
I0416 15:57:36.008798   32868 round_trippers.go:411] Response Headers:
I0416 15:57:36.008874   32868 kubernetes.go:119] error getting Pods with label selector "k8s-app=kube-proxy" [Get https://192.168.99.106:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy: Service Unavailable]
I0416 15:57:36.510272   32868 round_trippers.go:386] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: minikube/v1.10.0 (darwin/amd64) kubernetes/fc32d2f" https://192.168.99.106:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy
I0416 15:57:37.519401   32868 round_trippers.go:405] GET https://192.168.99.106:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy  in 1009 milliseconds
I0416 15:57:37.519430   32868 round_trippers.go:411] Response Headers:
I0416 15:57:37.519470   32868 kubernetes.go:119] error getting Pods with label selector "k8s-app=kube-proxy" [Get https://192.168.99.106:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy: Service Unavailable]
I0416 15:57:38.009508   32868 round_trippers.go:386] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: minikube/v1.10.0 (darwin/amd64) kubernetes/fc32d2f" https://192.168.99.106:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy
I0416 15:57:39.014992   32868 round_trippers.go:405] GET https://192.168.99.106:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy  in 1005 milliseconds
I0416 15:57:39.015020   32868 round_trippers.go:411] Response Headers:
I0416 15:57:39.015070   32868 kubernetes.go:119] error getting Pods with label selector "k8s-app=kube-proxy" [Get https://192.168.99.106:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy: Service Unavailable]
I0416 15:57:39.510294   32868 round_trippers.go:386] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: minikube/v1.10.0 (darwin/amd64) kubernetes/fc32d2f" https://192.168.99.106:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy
I0416 15:57:40.517726   32868 round_trippers.go:405] GET https://192.168.99.106:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy  in 1007 milliseconds
I0416 15:57:40.517755   32868 round_trippers.go:411] Response Headers:
I0416 15:57:40.517796   32868 kubernetes.go:119] error getting Pods with label selector "k8s-app=kube-proxy" [Get https://192.168.99.106:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy: Service Unavailable]
I

enter minikube ssh
$ netstat -natl
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 127.0.0.1:15441         0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:42097           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 10.0.2.15:10010         0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:59515           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:38463           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:18112           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:2049            0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:37291           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:5355            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:48235           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN
tcp        0   1656 10.0.2.15:22            10.0.2.2:54982          ESTABLISHED
tcp        0      0 10.0.2.15:44634         23.99.111.35:21089      ESTABLISHED
tcp        0      0 127.0.0.1:41266         127.0.0.1:15441         ESTABLISHED
tcp        0      0 127.0.0.1:15441         127.0.0.1:41266         ESTABLISHED
tcp        0      0 :::22                   :::*                    LISTEN
tcp        0      0 :::46661                :::*                    LISTEN
tcp        0      0 :::2376                 :::*                    LISTEN
tcp        0      0 :::10250                :::*                    LISTEN
tcp        0      0 :::5355                 :::*                    LISTEN
tcp        0      0 :::10255                :::*                    LISTEN
tcp        0      0 :::111                  :::*                    LISTEN

What happened:

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Output of minikube logs (if applicable):

Anything else do we need to know:
@IlluYing
Copy link

@kevinbin I'm meeting the same issue, do you have any idea?

@lexuszhi1990
Copy link

we have the same problems, any guys can help?

@cute-angelia
Copy link

+1

@sergeytkachenko
Copy link

+1 .(((((

@77720616
Copy link

+1

1 similar comment
@emandere
Copy link

+1

@pouyaesm
Copy link

pouyaesm commented Jul 5, 2018 •
edited
Loading

Same here.

  • OS: Windows 10
  • VM Driver: Hyper-V
  • ISO version : minikube-v0.28.0.iso
$ netstat -natl
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:43241           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:5355            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:15441         0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:39571           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 192.168.43.187:10010    0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:18112           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:2049            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:42307           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:59267           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:54467           0.0.0.0:*               LISTEN
tcp        0      1 192.168.43.187:33056    173.194.76.82:443       SYN_SENT
tcp        0      0 192.168.43.187:32862    173.194.76.82:443       TIME_WAIT
tcp        0      0 127.0.0.1:58354         127.0.0.1:15441         ESTABLISHED
tcp        0   1252 192.168.43.187:22       192.168.43.252:53678    ESTABLISHED
tcp        0      0 192.168.43.187:32908    173.194.76.82:443       TIME_WAIT
tcp        0      0 192.168.43.187:22       192.168.43.252:53020    ESTABLISHED
tcp        0      0 192.168.43.187:60490    173.194.76.82:443       TIME_WAIT
tcp        0      0 192.168.43.187:60654    173.194.76.82:443       TIME_WAIT
tcp        0      0 192.168.43.187:60614    173.194.76.82:443       TIME_WAIT
tcp        0      0 127.0.0.1:15441         127.0.0.1:58354         ESTABLISHED
tcp        0      0 :::2376                 :::*                    LISTEN
tcp        0      0 :::10250                :::*                    LISTEN
tcp        0      0 :::5355                 :::*                    LISTEN
tcp        0      0 :::10255                :::*                    LISTEN
tcp        0      0 :::111                  :::*                    LISTEN
tcp        0      0 :::22                   :::*                    LISTEN
tcp        0      0 :::34361                :::*                    LISTEN

I solved my problem. It was related to the internet connection of docker inside minikube
Here is my solution

@tstromberg tstromberg changed the title kubectl Unable to connect to the server: Service Unavailable error getting Pods with label selector "k8s-app=kube-proxy" ... kube-proxy: Service Unavailable Sep 19, 2018
@tstromberg tstromberg added ev/apiserver-refused failures due to the master on 8443 returning "Connection Refused" kind/bug Categorizes issue or PR as related to a bug. drivers/virtualbox/osx area/networking networking issues failed/local-networking startup failures due to networking issues cause/firewall-or-proxy When firewalls or proxies seem to be interfering co/virtualbox co/kube-proxy issues relating to kube-proxy in some way os/macos and removed ev/apiserver-refused failures due to the master on 8443 returning "Connection Refused" drivers/virtualbox/osx labels Sep 19, 2018
@notjames
Copy link

Was this ever fixed? I'm getting the same problem: minikube 0.28.0 with kube* 1.10.0 on mac as host and Virtualbox 5.2.20 r125813 (Qt5.6.3) as hypervisor.

@paksboda
Copy link

Getting same problem in Windows 7, Kube* 1.10.0 and VirtualBox V 5.2.22 r126460 (Qt5.6.2). Anybody has any solution? Please update here.

@dgorczyca
Copy link

dgorczyca commented Nov 24, 2018 •
edited
Loading

I don't think this has been fixed yet, at least it still does not work with VirtualBox, but I found a workaround that seems to work, just for clarity, here are my environment's brief specs:
VirtualBox version: 5.2.18
minikube version: v0.30.0
MacOS Mojave: version 10.14.1

What seems to work instead is to use different driver instead: https://github.com/kubernetes/minikube/blob/master/docs/drivers.md#hyperkit-driver

$ curl -LO https://storage.googleapis.com/minikube/releases/latest/docker-machine-driver-hyperkit \ && sudo install -o root -g wheel -m 4755 docker-machine-driver-hyperkit /usr/local/bin/

after running the above I was able to see:
/usr/local/bin/hyperkit -> /Applications/Docker.app/Contents/Resources/bin/com.docker.hyperkit

and I was able to start minikube:
$ minikube start --vm-driver=hyperkit

@ernaveenverma
Copy link

I am getting same issue even with hyperkit.

minikube start --vm-driver=hyperkit --logtostderr -v=7

I1125 08:19:29.622240 27422 kubernetes.go:119] error getting Pods with label selector "k8s-app=kube-proxy" [Get https://192.168.64.2:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy: dial tcp 192.168.64.2:8443: connect: connection refused]

@tstromberg tstromberg added the ev/apiserver-refused failures due to the master on 8443 returning "Connection Refused" label Jan 23, 2019
@tstromberg tstromberg changed the title error getting Pods with label selector "k8s-app=kube-proxy" ... kube-proxy: Service Unavailable HTTP_PROXY set: error getting Pods with label selector "k8s-app=kube-proxy" ... kube-proxy: Service Unavailable Jan 23, 2019
@tstromberg tstromberg removed failed/local-networking startup failures due to networking issues co/kube-proxy issues relating to kube-proxy in some way co/virtualbox os/macos labels Jan 23, 2019
@tstromberg
Copy link
Contributor

Related issues: #3145 and #3242

@tstromberg tstromberg added help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. labels Jan 24, 2019
@tstromberg
Copy link
Contributor

I've updated the HTTP proxy docs, which gives guidance to "no_proxy" values which will likely help with this: https://github.com/kubernetes/minikube/blob/master/docs/http_proxy.md

Does someone who is setting HTTP_*PROXY mind trying this with minikube v0.34.1, and include the output of minikube logs if they run into a failure?

@kuchaguangjie
Copy link

kuchaguangjie commented Feb 27, 2019 •
edited
Loading

Still got the same error with minikube v0.34.1, any idea ???

@kuchaguangjie
Copy link

kuchaguangjie commented Feb 27, 2019 •
edited
Loading

With v0.34.1, removing ~/.minikube, first, then delete & start again seems to fix it.
And, backup & restore ~/.minikube/cache by hand, would be helpful to avoid download the caches again.

Refer: https://stackoverflow.com/questions/54903740/minikube-start-get-error-k8s-app-kube-proxy-connection-refused

@tstromberg tstromberg mentioned this issue Mar 1, 2019
Merged
@tstromberg
Copy link
Contributor

I believe this issue was resolved in the v1.1.0 release. Please try upgrading to the latest release of minikube, and if the same issue occurs, please re-open this bug. Thank you opening this bug report, and for your patience!

@yangyangpig
Copy link

mark

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/networking networking issues cause/firewall-or-proxy When firewalls or proxies seem to be interfering ev/apiserver-refused failures due to the master on 8443 returning "Connection Refused" help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/bug Categorizes issue or PR as related to a bug. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests