-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
unable to fetch metrics from external metrics API: no matches for kind "ScaledObject" in version "keda.k8s.io/v1alpha1" (Scaler : Prometheus) #868
Comments
Strange, is your Keda Metrics Server pod running correctly? |
They are working fine and autoscaling is also working fine but not sure about the error.
Logs for Metric Server |
I also have the same problem , it works as expected in HPA but , lot of spam in logs. We are using promethus in another name space. Our promethus is centralized. |
@akramwaseem @zroubalik I was able to resolve it by using the setting in new operator. metricsServer: so it will not use your vnet DNS server to loop the prometheus host |
@kalyan123i great! Thanks for sharing :) |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions. |
This issue has been automatically closed due to inactivity. |
@kalyan123i @zroubalik - where did you add that?? In keda-operator pod? if yes what line or after which line?? Pls help |
Autoscaling is working okay but I keep seeing error in hpa when describing it
Warning FailedComputeMetricsReplicas 44m (x12 over 47m) horizontal-pod-autoscaler invalid metrics (1 invalid out of 1), first error is: failed to get nginx_ingress_controller_requests external metric: unable to get external metric demo/nginx_ingress_controller_requests/&LabelSelector{MatchLabels:map[string]string{deploymentName: autoscalingdemo,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: no matches for kind "ScaledObject" in version "keda.k8s.io/v1alpha1" Warning FailedGetExternalMetric 37m (x40 over 47m) horizontal-pod-autoscaler unable to get external metric demo/nginx_ingress_controller_requests/&LabelSelector{MatchLabels:map[string]string{deploymentName: autoscalingdemo,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: no matches for kind "ScaledObject" in version "keda.k8s.io/v1alpha1" Warning FailedGetExternalMetric 31m horizontal-pod-autoscaler unable to get external metric demo/nginx_ingress_controller_requests/&LabelSelector{MatchLabels:map[string]string{deploymentName: autoscalingdemo,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: the server is currently unable to handle the request (get nginx_ingress_controller_requests.external.metrics.k8s.io) Normal SuccessfulRescale 21m horizontal-pod-autoscaler New size: 4; reason: external metric nginx_ingress_controller_requests(&LabelSelector{MatchLabels:map[string]string{deploymentName: autoscalingdemo,},MatchExpressions:[]LabelSelectorRequirement{},}) above target Normal SuccessfulRescale 21m horizontal-pod-autoscaler New size: 8; reason: external metric nginx_ingress_controller_requests(&LabelSelector{MatchLabels:map[string]string{deploymentName: autoscalingdemo,},MatchExpressions:[]LabelSelectorRequirement{},}) above target Normal SuccessfulRescale 8m51s horizontal-pod-autoscaler New size: 8; reason: All metrics below target Normal SuccessfulRescale 8m5s horizontal-pod-autoscaler New size: 7; reason: All metrics below target Normal SuccessfulRescale 38s horizontal-pod-autoscaler New size: 4; reason: external metric nginx_ingress_requests(&LabelSelector{MatchLabels:map[string]string{deploymentName: autoscalingdemo,},MatchExpressions:[]LabelSelectorRequirement{},}) above target Normal SuccessfulRescale 22s horizontal-pod-autoscaler New size: 8; reason: external metric nginx_ingress_requests(&LabelSelector{MatchLabels:map[string]string{deploymentName: autoscalingdemo,},MatchExpressions:[]LabelSelectorRequirement{},}) above target
Expected Behavior
Working as expected but would be more clear to understand what actual error is for on the hpa side.
Ignoring the error did not sound like a good option.
Actual Behavior
scaling is working as expected as mentioned below
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE keda-hpa-autoscalingdemo Deployment/autoscalingdemo 45/3 (avg) 1 10 4 26m keda-hpa-autoscalingdemo Deployment/autoscalingdemo 22500m/3 (avg) 1 10 8 26m keda-hpa-autoscalingdemo Deployment/autoscalingdemo 16600m/3 (avg) 1 10 10 27m keda-hpa-autoscalingdemo Deployment/autoscalingdemo 13800m/3 (avg) 1 10 10 27m keda-hpa-autoscalingdemo Deployment/autoscalingdemo 18/3 (avg) 1 10 10 27m keda-hpa-autoscalingdemo Deployment/autoscalingdemo 8700m/3 (avg) 1 10 10 28m keda-hpa-autoscalingdemo Deployment/autoscalingdemo 0/3 (avg) 1 10 10 28m keda-hpa-autoscalingdemo Deployment/autoscalingdemo 1900m/3 (avg) 1 10 10 29m keda-hpa-autoscalingdemo Deployment/autoscalingdemo 17900m/3 (avg) 1 10 10 29m keda-hpa-autoscalingdemo Deployment/autoscalingdemo 18/3 (avg) 1 10 10 30m keda-hpa-autoscalingdemo Deployment/autoscalingdemo 17900m/3 (avg) 1 10 10 30m keda-hpa-autoscalingdemo Deployment/autoscalingdemo 6200m/3 (avg) 1 10 10 30m keda-hpa-autoscalingdemo Deployment/autoscalingdemo 6200m/3 (avg) 1 10 10 30m keda-hpa-autoscalingdemo Deployment/autoscalingdemo 0/3 (avg) 1 10 10 30m
In this case when hpa is created throws me an error saying
unable to fetch metrics from external metrics API: no matches for kind "ScaledObject" in version "keda.k8s.io/v1alpha1"
unable to fetch metrics from external metrics API: the server is currently unable to handle the request (get nginx_ingress_controller_requests.external.metrics.k8s.io)
and when you hit the application as you can see above autoscaling works.
Is this an expected behaviour when first creating hpa ?
Are there any additional troubleshooting steps which can be performed to see why this is happening ?
scaled-objects yaml file
Using prometheus version : 2.16.0
Steps to Reproduce the Problem
1.deploy any demo app with ingress using helm and deploy it on a desired namespace(In this case used nginx alpine image)
2. create a scaled object and deploy it on a namespace desired(should be same as helm demo app)
3.kubectl describe hpa should show you above error message when you trigger a autoscale (used hey and vegeta)
Let me know if any additional information is needed ?
Other things
Installed keda using kubectl apply.
Specifications
The text was updated successfully, but these errors were encountered: