Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kubectl api-resources broken after disabling metrics-server addon: server is currently unable to handle the request #3571

Closed
dongwenjuan opened this issue Jan 23, 2019 · 10 comments
Closed

kubectl api-resources broken after disabling metrics-server addon: server is currently unable to handle the request #3571

dongwenjuan opened this issue Jan 23, 2019 · 10 comments
Labels
addon/metrics-server help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/bug Categorizes issue or PR as related to a bug. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. priority/backlog Higher priority than priority/awaiting-more-evidence. r/2019q2 Issue was last reviewed 2019q2

Comments

@dongwenjuan
Copy link

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
Bug report

Please provide the following details:

Environment: minikube version: v0.33.1 on Linux ubuntu16 4.4.0-133

Minikube version (use minikube version): minikube version: v0.33.1 on Linux ubuntu16 4.4.0-133

  • OS (e.g. from /etc/os-release): ubuntu16 4.4.0-133
  • VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): None
  • ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION):
  • Install tools:
  • Others:
    $kubectl version
    Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.2", GitCommit:"cff46ab41ff0bb44d8584413b598ad8360ec1def", GitTreeState:"clean", BuildDate:"2019-01-10T23:35:51Z", GoVersion:"go1.11.4", Compiler:"gc", Platform:"linux/amd64"}
    Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.2", GitCommit:"cff46ab41ff0bb44d8584413b598ad8360ec1def", GitTreeState:"clean", BuildDate:"2019-01-10T23:28:14Z", GoVersion:"go1.11.4", Compiler:"gc", Platform:"linux/amd64"}

What happened:
I use minikube addons enable metrics-server to deploy metrics-server in my env, but can't get the metrics.

$ sudo kubectl get pods.metrics.k8s.io
error: the server doesn't have a resource type "pods"

$ kubectl get –raw apis/metrics.k8s.io/v1beta1/nodes
error: there is no need to specify a resource type as a separate argument when passing arguments in resource/name form (e.g. 'kubectl get resource/<resource_name>' instead of 'kubectl get resource resource/<resource_name>'

$ kubectl get pods -n kube-system
NAME                                       READY   STATUS             RESTARTS   AGE
coredns-86c58d9df4-5wbgj                   0/1     CrashLoopBackOff   4          2m23s
coredns-86c58d9df4-zl5bm                   0/1     CrashLoopBackOff   4          2m23s
default-http-backend-6f855497bd-j8fm6      1/1     Running            0          2m18s
etcd-minikube                              1/1     Running            0          91s
kube-addon-manager-minikube                1/1     Running            0          98s
kube-apiserver-minikube                    1/1     Running            0          87s
kube-controller-manager-minikube           1/1     Running            0          100s
kube-proxy-5jmdk                           1/1     Running            0          2m23s
kube-scheduler-minikube                    1/1     Running            0          84s
kubernetes-dashboard-85b7755d8f-4g5dc      1/1     Running            0          2m19s
metrics-server-6fc4b7bcff-2crq7            1/1     Running            0          22s
nginx-ingress-controller-c69f7cdd7-mp4q8   1/1     Running            0          2m18s
storage-provisioner                        1/1     Running            0          2m16s

$ kubectl logs pods/metrics-server-6fc4b7bcff-2crq7    -n kube-system
I0123 06:25:20.905890       1 heapster.go:71] /metrics-server --source=kubernetes.summary_api:''
I0123 06:25:20.906021       1 heapster.go:72] Metrics Server version v0.2.1
I0123 06:25:20.906276       1 configs.go:61] Using Kubernetes client with master "https://10.96.0.1:443" and version
I0123 06:25:20.906309       1 configs.go:62] Using kubelet port 10255
I0123 06:25:20.907962       1 heapster.go:128] Starting with Metric Sink
I0123 06:25:21.887668       1 serving.go:308] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0123 06:25:22.561804       1 heapster.go:101] Starting Heapster API server...
[restful] 2019/01/23 06:25:22 log.go:33: [restful/swagger] listing is available at https:///swaggerapi
[restful] 2019/01/23 06:25:22 log.go:33: [restful/swagger] https:///swaggerui/ is mapped to folder /swagger-ui/
I0123 06:25:22.564940       1 serve.go:85] Serving securely on 0.0.0.0:443

There is no error log in metrics-server.

And after I disable metrics-server addon, there raise an error when I use kubectl api-resources.

$ kubectl api-resources
error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request

What you expected to happen:
It shoudn't throw an error when user disable addons.

How to reproduce it (as minimally and precisely as possible):

  1. create a cluster: sudo minikube start --vm-driver=none --kubernetes-version=v1.13.2
  2. enable 'metrics-server' addon: sudo minikube addons enable metrics-server
  3. get the metrics: sudo kubectl get pods.metrics.k8s.io
  4. disable the addon: sudo minikube addons disable metrics-server
  5. see the resources: sudo kubectl api-resources

Output of minikube logs (if applicable):

$ kubectl logs pods/kube-apiserver-minikube -n kube-system
I0123 06:23:10.557896 1 controller.go:608] quota admission added evaluator for: deployments.apps
I0123 06:23:10.646166 1 controller.go:608] quota admission added evaluator for: daemonsets.apps
I0123 06:23:16.069174 1 controller.go:608] quota admission added evaluator for: controllerrevisions.apps
I0123 06:23:16.142450 1 controller.go:608] quota admission added evaluator for: replicasets.apps
I0123 06:23:21.188746 1 controller.go:608] quota admission added evaluator for: deployments.extensions
I0123 06:25:21.864048 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:25:22.347592 1 available_controller.go:316] v1beta1.metrics.k8s.io failed with: Get https://10.97.40.73:443: dial tcp 10.97.40.73:443: connect: connection refused
E0123 06:25:22.368621 1 available_controller.go:316] v1beta1.metrics.k8s.io failed with: Get https://10.97.40.73:443: dial tcp 10.97.40.73:443: connect: connection refused
E0123 06:25:22.373397 1 available_controller.go:316] v1beta1.metrics.k8s.io failed with: Get https://10.97.40.73:443: dial tcp 10.97.40.73:443: connect: connection refused
E0123 06:25:22.381806 1 available_controller.go:316] v1beta1.metrics.k8s.io failed with: Get https://10.97.40.73:443: dial tcp 10.97.40.73:443: connect: connection refused
E0123 06:25:22.459431 1 available_controller.go:316] v1beta1.metrics.k8s.io failed with: Get https://10.97.40.73:443: dial tcp 10.97.40.73:443: connect: connection refused
E0123 06:25:22.544807 1 available_controller.go:316] v1beta1.metrics.k8s.io failed with: Get https://10.97.40.73:443: dial tcp 10.97.40.73:443: connect: connection refused
E0123 06:25:24.039650 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0123 06:25:24.039673 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0123 06:25:28.225301 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:25:28.229348 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0123 06:25:28.229374 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0123 06:26:28.229649 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:26:28.237167 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0123 06:26:28.237233 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0123 06:27:20.119495 1 controller.go:608] quota admission added evaluator for: replicasets.extensions
I0123 06:27:22.079911 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:27:22.080127 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[X-Content-Type-Options:[nosniff] Content-Type:[text/plain; charset=utf-8]]
I0123 06:27:22.080156 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:27:27.310802 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:27:57.333088 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:28:01.208934 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:28:01.209130 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:28:01.209162 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:28:27.363134 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:28:57.393069 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:29:01.209377 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:29:01.209532 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:29:01.209551 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:29:27.423372 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:29:57.450537 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:30:27.480751 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:30:57.511436 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:31:01.209873 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:31:01.210132 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:31:01.210148 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:31:27.539124 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:31:57.568052 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:32:27.593809 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:32:57.615577 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:33:01.194325 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:33:01.194779 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[X-Content-Type-Options:[nosniff] Content-Type:[text/plain; charset=utf-8]]
I0123 06:33:01.194866 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:33:27.645871 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:33:57.679229 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:34:01.195242 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:34:01.195429 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:34:01.195469 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:34:27.710700 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:34:57.739270 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:35:27.768806 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:35:57.799318 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:36:01.195847 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:36:01.196262 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:36:01.196340 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:36:27.829623 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:36:57.856002 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:37:27.887110 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:37:57.912339 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:38:01.421432 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:38:01.421771 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[X-Content-Type-Options:[nosniff] Content-Type:[text/plain; charset=utf-8]]
I0123 06:38:01.421826 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:38:27.941176 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:38:57.972746 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:39:01.422093 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:39:01.422267 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:39:01.422296 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:39:28.000507 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:39:58.030544 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:40:28.062406 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:40:58.090693 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:41:01.422587 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:41:01.422810 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:41:01.422835 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:41:28.118852 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:41:58.147632 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:42:28.176996 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:42:58.200169 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:43:01.438219 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:43:01.438607 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[X-Content-Type-Options:[nosniff] Content-Type:[text/plain; charset=utf-8]]
I0123 06:43:01.438632 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:43:28.227598 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:43:58.257716 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:44:01.439085 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:44:01.439341 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:44:01.439368 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:44:28.286862 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:44:58.316254 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:45:28.345326 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:45:58.374960 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:46:01.439720 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:46:01.439912 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:46:01.439937 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:46:28.401303 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:46:58.432436 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:47:28.461957 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:47:58.515028 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:48:01.179057 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:48:01.179478 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:48:01.179505 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:48:28.544684 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:48:58.574627 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:49:01.179946 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:49:01.180426 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:49:01.180455 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:49:28.605946 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:49:58.634988 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:50:28.662338 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:50:58.693018 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:51:01.180981 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:51:01.181470 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:51:01.181493 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:51:28.721102 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:51:58.751810 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:52:28.784909 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:52:58.806793 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:53:01.158125 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:53:01.158545 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[X-Content-Type-Options:[nosniff] Content-Type:[text/plain; charset=utf-8]]
I0123 06:53:01.158591 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:53:28.836887 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:53:58.869002 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:54:01.158997 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:54:01.159173 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:54:01.159195 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:54:28.899156 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:54:58.932305 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:55:28.963440 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:55:58.991959 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:56:01.159565 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:56:01.160042 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:56:01.160067 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:56:29.020089 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:56:59.050326 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:57:29.078128 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:57:59.101355 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:58:01.153808 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:58:01.153947 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:58:01.153966 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:58:29.129283 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:58:59.153660 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:59:01.154228 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:59:01.154350 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:59:01.154489 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:59:29.184314 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:59:59.212270 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 07:00:29.238526 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 07:00:59.264794 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 07:01:01.154957 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 07:01:01.155168 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 07:01:01.155194 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 07:01:29.291611 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 07:01:59.321175 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 07:02:29.351060 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 07:02:59.378184 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 07:03:01.243614 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 07:03:01.243967 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 07:03:01.244024 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 07:03:29.405791 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 07:03:59.431544 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 07:04:01.244431 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 07:04:01.244576 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 07:04:01.244590 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 07:04:29.457090 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 07:04:59.484954 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 07:05:29.505677 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 07:05:59.532492 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 07:06:01.244888 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 07:06:01.245533 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 07:06:01.245716 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 07:06:29.559155 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 07:06:59.583618 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 07:07:29.609731 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 07:07:59.636070 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 07:08:01.252289 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 07:08:01.252474 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[X-Content-Type-Options:[nosniff] Content-Type:[text/plain; charset=utf-8]]
I0123 07:08:01.252498 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 07:08:29.660252 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request

Anything else do we need to know:
@tstromberg tstromberg added co/none-driver area/addons addon/metrics-server priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. and removed area/addons labels Jan 23, 2019
@tstromberg tstromberg changed the title metrics-server addon: can't get pods and nodes metrics metrics-server addon: metrics.k8s.io/v1beta1: the server is currently unable to handle the request Jan 23, 2019
@tstromberg
Copy link
Contributor

I haven't tried the metrics-server addon yet, but the CoreDNS pods in CrashLoopBackoff looks suspicious. Perhaps it is trying to reference metrics.k8s.io internally, but DNS isn't available. Do you mind looking at the logs for the CoreDNS pods?

This might just be due to #3511

@tstromberg tstromberg added the kind/bug Categorizes issue or PR as related to a bug. label Jan 23, 2019
@dongwenjuan
Copy link
Author

@tsmetana Hi, thanks for your reply. I followed the guide: delete the loop in coredns configmap and restart the coredns pods, it works! Thanks a lot~

@tstromberg tstromberg added the help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. label Jan 24, 2019
@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 29, 2019
@tstromberg
Copy link
Contributor

I can confirm that this is still an issue in v1.1:

minikube addons enable metrics-server
kubectl get pods --all-namespaces --show-labels | grep metric
minikube addons disable metrics-server
kubectl get pods --all-namespaces --show-labels | grep metric
kubectl api-resources

output:

error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request

@tstromberg tstromberg added priority/backlog Higher priority than priority/awaiting-more-evidence. and removed co/none-driver lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. labels May 22, 2019
@tstromberg tstromberg changed the title metrics-server addon: metrics.k8s.io/v1beta1: the server is currently unable to handle the request kubectl api-resources broken after disabling metrics-server addon: server is currently unable to handle the request May 22, 2019
@tstromberg tstromberg added the r/2019q2 Issue was last reviewed 2019q2 label May 23, 2019
@ryancalhoun
Copy link

I had the same issue on a "regular" (non-minikube) cluster. I found this comment
kubernetes/kubernetes#72051 (comment)

Once I ran kubectl delete apiservice v1beta1.metrics.k8s.io the errors stopped on their own.

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 10, 2019
@fejta-bot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Dec 10, 2019
@fejta-bot
Copy link

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@k8s-ci-robot
Copy link
Contributor

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@mbaeuerle
Copy link

I had the same issue on a "regular" (non-minikube) cluster. I found this comment
kubernetes/kubernetes#72051 (comment)

Once I ran kubectl delete apiservice v1beta1.metrics.k8s.io the errors stopped on their own.

Keep in mind the command will delete the metrics service and doesn't fix the underlying problem, possibly introducing other issues. I don't recommend doing that. In our case one master node could not connect to the metrics server but after deleting this master and waiting for a new one to spin up the issue was gone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
addon/metrics-server help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/bug Categorizes issue or PR as related to a bug. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. priority/backlog Higher priority than priority/awaiting-more-evidence. r/2019q2 Issue was last reviewed 2019q2
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@tstromberg @mbaeuerle @ryancalhoun @dongwenjuan @k8s-ci-robot @fejta-bot