-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request #157
Comments
Check that your ControlPlane can reach your DataPlane on 443 (I had to modify SG for both to allow this and it worked) |
We are also seeing this issue on Kubernetes version 1.10.11, metrics-server v0.3.1. The error doesn't occur all the time, but seemingly randomly. HPA is also not working:
Seeing a lot of these errors in the metrics-server logs:
And about the time the 'unable to handle request' error gets thrown, we see this in the API server logs:
|
I had this problem, in my case i am using
and after this, |
Coming back around to this, I am still seeing these errors with metrics-server. Here is my config:
If I spam
I'm seeing this in the apiserver logs:
|
The issue still exists in V 1.13.3 |
As an update to this, all my issues with metrics-server went away after I set
in the stable helm chart https://github.com/helm/charts/tree/master/stable/metrics-server |
@ag237 Thanks for sharing this . Any reasons on how did this got fix when you enabled host network |
did you solve the problem? I have a same issue.
The master node's api-server log says: Any ideas? |
@abizake I think you are also unable to reach pods on the other nodes. |
In the above mentioned step , I am actually using calico. The respective ports of calico are open and pods from other nodes are reachable. |
Having the same issue with Kubernetes kubectl logs metrics-server-ddd54b5c5-mxxb7
I0812 20:56:27.161337 1 serving.go:273] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
I0812 20:56:32.059998 1 manager.go:95] Scraping metrics from 0 sources
I0812 20:56:32.060020 1 manager.go:150] ScrapeMetrics: time: 1.003µs, nodes: 0, pods: 0
[restful] 2019/08/12 20:56:32 log.go:33: [restful/swagger] listing is available at https://:8443/swaggerapi
[restful] 2019/08/12 20:56:32 log.go:33: [restful/swagger] https://:8443/swaggerui/ is mapped to folder /swagger-ui/
I0812 20:56:32.467070 1 serve.go:96] Serving securely on [::]:8443
I0812 20:57:32.060888 1 manager.go:95] Scraping metrics from 4 sources
I0812 20:57:32.063876 1 manager.go:120] Querying source: kubelet_summary:master-node2.internal
I0812 20:57:32.068071 1 manager.go:120] Querying source: kubelet_summary:worker-node1.internal
I0812 20:57:32.068425 1 manager.go:120] Querying source: kubelet_summary:worker-node2.internal
I0812 20:57:32.088240 1 manager.go:120] Querying source: kubelet_summary:master-node1.internal
I0812 20:57:32.256135 1 manager.go:150] ScrapeMetrics: time: 195.03251ms, nodes: 4, pods: 23 kubectl top node
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io) |
I can confirm #157 (comment) helps Not using the helm chart, I added to the manifest under spec/template/spec Also I am using the flags
|
I've been having several problems in the cluster because of this, including HPA. #157 (comment) seems to nail it, indeed, but I'm still wondering what's the actual problem. Setting |
i added hostNetwork: true but my problem no fix the apiserver report log "kube-controller-manager: E1011 13:37:24.015616 33182 resource_quota_controller.go:407] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request“” |
I don't think metrics-server was meant to run in host network. I think it's a problem with particular overlay network, but it's not my expertise. Metrics Server uses https://github.com/kubernetes/kube-aggregator to register into Apiserver maybe you could find answers there? Still it would be useful to document on how metrics server provides Metrics API and what requirements it poses on Network |
The comment in values.yaml https://github.com/helm/charts/blob/master/stable/metrics-server/values.yaml mentions that might be required if we use Weave network on EKS. We faced a similar problem in EKS using AWS CNI and this issue seems to fix the problem. I believe this is more a band-aid solution and the root is somewhere else.
|
Thanks, this is gold!!! |
What is SG? |
Probably "security group," in the context of AWS EC2. |
How to check this? I have my cluster hosted on Azure AKS |
Closing per Kubernetes issue triage policy GitHub is not the right place for support requests. |
It works, thanks! |
Note that if you are using GKE (Google Cloud Kubernetes Engine) and that your cluster has been without containers for a long time (multiple days), then GKE decommissions the nodes from the cluster (so as to save you costs). As such, without nodes, the control plane processes cannot start. So if that's your case, all is good! |
@philippefutureboy I'm having this problem in GKE, and yes, my cluster was idle, but I've run two DAGs over the last hour and still it does not work. Is there any other way to revive it? |
No unfortunately the issue has started persisting through spinning new pods on my side as well 😕 |
Oh. I'm trying to delete a namespace and it can't be deleted because of
|
Ah, there's a GKE trouble-shooting guide here: https://cloud.google.com/kubernetes-engine/docs/troubleshooting#namespace_stuck_in_terminating_state |
Thanks for sharing this. It worked for me |
Same issue: turning off the firewall worked for me (...yeah, quite overkilling but I have no time for fine tuning right now...) |
My solution was this:
I did not have the metrics server installed, nor did I need it. At some point somebody installed it and uninstalled it. But the uninstallation was not complete. We had these lingering resources:
|
API Server Logs :-
1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E1012 08:23:25.282353 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I1012 08:23:25.282377 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E1012 08:23:25.396126 1 memcache.go:134] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E1012 08:23:25.991550 1 available_controller.go:311] v1beta1.metrics.k8s.io failed with: Get https://10.105.54.184:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E1012 08:23:46.469237 1 available_controller.go:311] v1beta1.metrics.k8s.io failed with: Get https://10.105.54.184:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E1012 08:23:55.440941 1 memcache.go:134] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E1012 08:23:55.789103 1 available_controller.go:311] v1beta1.metrics.k8s.io failed with: Get https://10.105.54.184:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E1012 08:24:25.477704 1 memcache.go:134] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E1012 08:24:25.705399 1 available_controller.go:311] v1beta1.metrics.k8s.io failed with: Get https://10.105.54.184:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E1012 08:24:55.516394 1 memcache.go:134] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E1012 08:24:55.719712 1 available_controller.go:311] v1beta1.metrics.k8s.io failed with: Get https://10.105.54.184:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E1012 08:25:13.395961 1 available_controller.go:311] v1beta1.metrics.k8s.io failed with: Get https://10.105.54.184:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I1012 08:25:25.282682 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E1012 08:25:25.282944 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[X-Content-Type-Options:[nosniff] Content-Type:[text/plain; charset=utf-8]]
I1012 08:25:25.282969 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E1012 08:25:25.563266 1 memcache.go:134] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
Controller Logs :-
E1012 08:26:57.910695 1 memcache.go:134] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E1012 08:27:13.214427 1 resource_quota_controller.go:430] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
W1012 08:27:17.126343 1 garbagecollector.go:647] failed to discover some groups: map[metrics.k8s.io/v1beta1:the server is currently unable to handle the request]
Metric Server Logs :-
I1012 08:22:11.248135 1 serving.go:273] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
[restful] 2018/10/12 08:22:12 log.go:33: [restful/swagger] listing is available at https://:443/swaggerapi
[restful] 2018/10/12 08:22:12 log.go:33: [restful/swagger] https://:443/swaggerui/ is mapped to folder /swagger-ui/
I1012 08:22:12.537437 1 serve.go:96] Serving securely on [::]:443
Kubernetes Version :- 1.12.1
Metric Server Deployment YAML :-
Any help is appreciated.
The text was updated successfully, but these errors were encountered: