HPA not working, unknown target CPU #4456

devarajchidambaram · 2019-06-10T08:21:53Z

Im trying to auto scale the kuber pods but not able to get the hpa cpu metrics . so its failing to auto scale.

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubernetes-tutorial-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubernetes-tutorial-deployment
  template:
    metadata:
      labels:
        app: kubernetes-tutorial-deployment
    spec:
      containers:
      - name: kubernetes-tutorial-application
        image: devarajc/tomcat_helloworld
        resources:
          requests:
             cpu: 500m
        ports:
          - containerPort: 8080

Ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kubernetes-tutorial-ingress
spec:
  rules:
  - http:
      paths:
      - path: /
        backend:
          serviceName: kubernetes-tutorial-cluster-ip
          servicePort: 80

Services.yaml

apiVersion: v1
kind: Service
metadata:
  name: kubernetes-tutorial-cluster-ip
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: kubernetes-tutorial-deployment
  type: ClusterIP

CMD output

D:\docker\kubernetes-tutorial>kubectl get all
NAME                                                  READY   STATUS    RESTARTS   AGE
pod/kubernetes-tutorial-deployment-7b78fb65c7-5dvz2   1/1     Running   0          2d18h

NAME                                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/kubernetes                       ClusterIP   10.96.0.1        <none>        443/TCP   2d21h
service/kubernetes-tutorial-cluster-ip   ClusterIP   10.108.184.150   <none>        80/TCP    2d20h

NAME                                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kubernetes-tutorial-deployment   1/1     1            1           2d20h

NAME                                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/kubernetes-tutorial-deployment-7b78fb65c7   1         1         1       2d18h

NAME                                                                 REFERENCE                                   TARGETS        MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/kubernetes-tutorial-deployment   Deployment/kubernetes-tutorial-deployment   <unknown>/2%   1         4         1          122m

Here horizontalpodautoscaler.autoscaling/kubernetes-tutorial-deployment showing target is UNKNOWN.

This is the error I got

D:\docker\kubernetes-tutorial>kubectl describe hpa kubernetes-tutorial-deployment
Name:                                                  kubernetes-tutorial-deployment
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Mon, 10 Jun 2019 11:46:48 +0530
Reference:                                             Deployment/kubernetes-tutorial-deployment
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 2%
Min replicas:                                          1
Max replicas:                                          4
Deployment pods:                                       1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)
Events:
  Type     Reason                   Age                      From                       Message
  ----     ------                   ----                     ----                       -------
  Warning  FailedGetResourceMetric  3m3s (x1009 over 4h18m)  horizontal-pod-autoscaler  unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)

Im new to the kubernetes . so I don't know , how to proceed further.

The text was updated successfully, but these errors were encountered:

afbjorklund · 2019-06-10T17:24:23Z

Did you deploy metrics-server ?

devarajchidambaram · 2019-06-11T08:19:30Z

Yes @afbjorklund ...

devarajchidambaram · 2019-06-11T08:20:47Z

Finally I did like this ...now its working

https://stackoverflow.com/questions/54106725/docker-kubernetes-mac-autoscaler-unable-to-find-metrics

tstromberg · 2019-07-16T22:55:02Z

It's unclear what the issue is with minikube is here? Are we providing a broken metrics-server?

afbjorklund · 2019-08-08T13:51:43Z

Seems like there is some kind of issue with certificates and/or IP, workaround is disabling those:

--kubelet-insecure-tls

--kubelet-preferred-address-types=InternalIP

Could be related to a broken DNS (somehow), or could be our broken InternalIP (vs. hostname) ?

salsaverde123 · 2019-08-10T20:16:28Z

I have same problem but the solution of @devarajchidambaram didn't work for me.

ns0092 · 2019-08-23T22:12:15Z

The deployment file should contain resource limits in order for the metrics-server to calculate the CPU utilization.
https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/
Edit your deployment.yaml and add the resource limits part in order to fix this.

To read more on this, refer to the autoscaler algorithm:
Please note that if some of the pod’s containers do not have the relevant resource request set, CPU utilization for the pod will not be defined and the autoscaler will not take any action for that metric. See the algorithm details section below for more information about how the autoscaling algorithm works.
https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

highway900 · 2019-11-26T02:57:59Z

@nikhils5501 Thanks, this solved it for me. More specifically I added the metrics section to the config for service-pod.yml

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: {{ .Values.name }}
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ .Values.name }}
  minReplicas: {{ .Values.replicaCount }}
  maxReplicas: 16
  targetCPUUtilizationPercentage: 70
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 70

cmarquezrusso · 2019-12-04T16:07:57Z

I can reproduce this on:
minikube version: v1.5.2
Running k8s v1.16.2

I tried adding

--kubelet-insecure-tls
--kubelet-preferred-address-types=InternalIP

but it didn't work. Minikube keeps restoring metrics-server to the default values.

medyagh · 2019-12-16T22:33:10Z

@Cristian04
does the solution mentioned above help ?
#4456 (comment)

cmarquezrusso · 2019-12-17T05:07:41Z

@Cristian04
does the solution mentioned above help ?
#4456 (comment)

Not really. I am having this issue when I add metrics into the hpa.

* : Invalid value: "The edited file failed validation": ValidationError(HorizontalPodAutoscaler): unknown field "metrics" in io.k8s.api.autoscaling.v1.HorizontalPodAutoscaler

I think is related to this bug

ns0092 · 2019-12-17T10:37:11Z

@Cristian04 You need to have metrics-server installed on your cluster. I hope that should solve it for you. Metrics-server is the alternative for Heapster.

cmarquezrusso · 2019-12-17T21:19:07Z

@Cristian04 You need to have metrics-server installed on your cluster. I hope that should solve it for you. Metrics-server is the alternative for Heapster.

Yes, I am using metric-server and it's running on my cluster. Not using heapster as it is deprecated.

Updates:

I've retested on:

Minikube v1.6.1 + metric-server
K8S= 1.17.0
Result: ~~Error~~ Working 😄
Error: ~~Issue starting minikube to start a 1.7.0 cluster "💣 Unable to parse "": Version string empty"~~

Metrics started working after 5 minutes of seeing this:

W1217 19:09:09.037972   12866 top_pod.go:266] Metrics not available for pod default/nginx-f5f948997-
6cw7s, age: 2m28.037948s
error: Metrics not available for pod default/nginx
-f5f948997-6cw7s, age: 2m28.037948s

Minikube v1.5.2 + metric-server:
K8S: v1.16.2

Result: Not working
Error: Metrics are not available:

kubectl top pod
W1217 18:13:05.325281   40088 top_pod.go:266] Metrics not available for pod default/nginx-f5f948997-9
qn9f, age: 7m7.32526s
error: Metrics not available for pod default/nginx-f5f948997-9qn9f, age: 7m7.32526s

HPA with unknown status:

kubectl get hpa 
NAME    REFERENCE          TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
nginx   Deployment/nginx   <unknown>/50%   2         10        2          6m54s

metric-server log

I1217 21:18:13.159257       1 reststorage.go:93] No metrics for pod default/nginx-f5f948997-76zrd

Minikube v1.3.0 + metrics-server
K8s v1.15.2

Result: Working without workarounds

Steps

minikube start --memory 4096 --cpus 2 --vm-driver=hyperkit
minikube addons enable metrics-server
kubectl run nginx --image nginx --port 80
kubectl set resources deployment nginx -c=nginx --limits=cpu=100m,memory=64Mi
kubectl autoscale deployment nginx --max=10 --cpu-percent=50 --min=5

Verify with:

kubectl top pod && kubectl top node && kubectl get hpa

cmarquezrusso · 2019-12-30T05:37:51Z

Fix for Minikube v1.5.2 + K8S v1.16.2

Disable the metric-server addon and install from https://github.com/kubernetes-sigs/metrics-server

$ minikube addons disable metrics-server
$ kubectl create -f deploy/1.8+/
# edit metric-server deployment to add the flags
# args:
# - --kubelet-insecure-tls
# - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
$ kubectl edit deploy -n kube-system metrics-server

I think we can close the issue with that

medyagh · 2020-01-29T19:42:47Z

@Cristian04 @devarajchidambaram
@DShepherd @infra-ddeore

we just merged a PR that improves the metrics-server addon, (it is on the master and will be in the next release)

do you mind checking the minikube that would help with this issue ?

#6322

cmarquezrusso · 2020-01-30T02:23:05Z

@medyagh Yeah, Im fine with that. Feel free to assign this ticket to me if required.

govargo · 2020-03-09T13:59:29Z

I think this issue had been resolved.

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubernetes-tutorial-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubernetes-tutorial-deployment
  template:
    metadata:
      labels:
        app: kubernetes-tutorial-deployment
    spec:
      containers:
      - name: kubernetes-tutorial-application
        image: devarajc/tomcat_helloworld
        resources:
          requests:
             cpu: 500m
        ports:
          - containerPort: 8080 
EOF

and I confirmed HPA is working.

$ kubectl autoscale deployment kubernetes-tutorial-deployment --cpu-percent=50 --min=1 --max=4
horizontalpodautoscaler.autoscaling/kubernetes-tutorial-deployment autoscaled

$ kubectl get hpa
NAME                             REFERENCE                                   TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
kubernetes-tutorial-deployment   Deployment/kubernetes-tutorial-deployment   1%/50%    1         4         1          22s

ryuheechul · 2020-06-03T10:40:35Z

In case it's not obvious how to fix unknown, what you most likely be missing is following:

kubectl set resources deployment nginx -c=nginx --limits=cpu=100m,memory=64Mi

thelman · 2023-03-03T20:16:51Z

In my case, the only deployment of metrics-server that worked for hpa, is this one that I share with you, I tried it with docker-desktop

full deployment of metrics server, work for hpa

tstromberg added the kind/support Categorizes issue or PR as a support question. label Sep 20, 2019

tstromberg closed this as completed Apr 16, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HPA not working, unknown target CPU #4456

HPA not working, unknown target CPU #4456

devarajchidambaram commented Jun 10, 2019 •

edited

Loading

afbjorklund commented Jun 10, 2019

devarajchidambaram commented Jun 11, 2019

devarajchidambaram commented Jun 11, 2019

tstromberg commented Jul 16, 2019

afbjorklund commented Aug 8, 2019

salsaverde123 commented Aug 10, 2019

ns0092 commented Aug 23, 2019 •

edited

Loading

highway900 commented Nov 26, 2019

cmarquezrusso commented Dec 4, 2019

medyagh commented Dec 16, 2019

cmarquezrusso commented Dec 17, 2019

ns0092 commented Dec 17, 2019

cmarquezrusso commented Dec 17, 2019 •

edited

Loading

cmarquezrusso commented Dec 30, 2019 •

edited

Loading

medyagh commented Jan 29, 2020

cmarquezrusso commented Jan 30, 2020 •

edited

Loading

govargo commented Mar 9, 2020

ryuheechul commented Jun 3, 2020

thelman commented Mar 3, 2023 •

edited

Loading

HPA not working, unknown target CPU #4456

HPA not working, unknown target CPU #4456

Comments

devarajchidambaram commented Jun 10, 2019 • edited Loading

afbjorklund commented Jun 10, 2019

devarajchidambaram commented Jun 11, 2019

devarajchidambaram commented Jun 11, 2019

tstromberg commented Jul 16, 2019

afbjorklund commented Aug 8, 2019

salsaverde123 commented Aug 10, 2019

ns0092 commented Aug 23, 2019 • edited Loading

highway900 commented Nov 26, 2019

cmarquezrusso commented Dec 4, 2019

medyagh commented Dec 16, 2019

cmarquezrusso commented Dec 17, 2019

ns0092 commented Dec 17, 2019

cmarquezrusso commented Dec 17, 2019 • edited Loading

Updates:

I've retested on:

Steps

cmarquezrusso commented Dec 30, 2019 • edited Loading

medyagh commented Jan 29, 2020

cmarquezrusso commented Jan 30, 2020 • edited Loading

govargo commented Mar 9, 2020

ryuheechul commented Jun 3, 2020

thelman commented Mar 3, 2023 • edited Loading

devarajchidambaram commented Jun 10, 2019 •

edited

Loading

ns0092 commented Aug 23, 2019 •

edited

Loading

cmarquezrusso commented Dec 17, 2019 •

edited

Loading

cmarquezrusso commented Dec 30, 2019 •

edited

Loading

cmarquezrusso commented Jan 30, 2020 •

edited

Loading

thelman commented Mar 3, 2023 •

edited

Loading