cron sometimes report 1 even when out of the cron windows

[loicmathieu]

Report

cron scaler sometimes report 1 even inside the cron windows. We saw it report 0, and 1 when out of the cron window.

Expected Behavior

It should always report 0 (or unknown when the HPA is deactivated)

Actual Behavior

It sometimes report 0, sometimes 1, sometimes

Steps to Reproduce the Problem

Logs from KEDA operator

example

KEDA Version

2.6.0

Kubernetes Version

1.19

Platform

Google Cloud

Scaler Details

prometheus, kafka, cron

Anything else?

We define a cron scaler like this:

  - type: cron
    metadata:
      timezone: Europe/Brussels
      start: 30 * * * *
      end: 59 * * * *
      desiredReplicas: "1"

We define the following scalled object spec:

  idleReplicaCount: 0
  # we want to scale to 1 min in case of activity, to always have one replica available
  minReplicaCount:  1
  # we want to scale up to the number of partitions
  maxReplicaCount:  6

Inside the cron window, we can see that the scaler report 1/1 as it should:

Name:                                                              keda-hpa-vcstream-gen-anomaly
Namespace:                                                         vcstream-dev
Labels:                                                            app.kubernetes.io/managed-by=Helm
                                                                   app.kubernetes.io/name=keda-hpa-vcstream-gen-anomaly
                                                                   app.kubernetes.io/part-of=vcstream-gen-anomaly
                                                                   app.kubernetes.io/version=2.6.0
                                                                   scaledobject.keda.sh/name=vcstream-gen-anomaly
Annotations:                                                       <none>
CreationTimestamp:                                                 Mon, 10 Jan 2022 16:19:18 +0100
Reference:                                                         Deployment/vcstream-gen-anomaly
Metrics:                                                           ( current / target )
  "s0-cron-Europe-Brussels-30xxxx-59xxxx" (target average value):  1 / 1
  "s1-prometheus-process_cpu_usage" (target average value):        0 / 75
  "s2-kafka-private_aoaa_anomaly_v1" (target average value):       0 / 100
  "s3-kafka-private_aoaa_deadletter_v1" (target average value):    0 / 100
Min replicas:                                                      1
Max replicas:                                                      6
Deployment pods:                                                   1 current / 1 desired
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    ReadyForNewScale    recommended size matches current size
  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from external metric s0-cron-Europe-Brussels-30xxxx-59xxxx(&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: vcstream-gen-anomaly,},MatchExpressions:[]LabelSelectorRequirement{},})
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:           <none>

But outside the cron window we sometimes saw 1 like in the following:

Name:                                                              keda-hpa-vcstream-gen-anomaly
Namespace:                                                         vcstream-dev
Labels:                                                            app.kubernetes.io/managed-by=Helm
                                                                   app.kubernetes.io/name=keda-hpa-vcstream-gen-anomaly
                                                                   app.kubernetes.io/part-of=vcstream-gen-anomaly
                                                                   app.kubernetes.io/version=2.6.0
                                                                   scaledobject.keda.sh/name=vcstream-gen-anomaly
Annotations:                                                       <none>
CreationTimestamp:                                                 Mon, 10 Jan 2022 16:19:18 +0100
Reference:                                                         Deployment/vcstream-gen-anomaly
Metrics:                                                           ( current / target )
  "s0-cron-Europe-Brussels-30xxxx-59xxxx" (target average value):  1 / 1
  "s1-prometheus-process_cpu_usage" (target average value):        0 / 75
  "s3-kafka-private_aoaa_anomaly_v1" (target average value):       <unknown> / 100
  "s3-kafka-private_aoaa_deadletter_v1" (target average value):    0 / 100
Min replicas:                                                      1
Max replicas:                                                      6
Deployment pods:                                                   1 current / 1 desired
Conditions:
  Type            Status  Reason                   Message
  ----            ------  ------                   -------
  AbleToScale     True    ReadyForNewScale         recommended size matches current size
  ScalingActive   False   FailedGetExternalMetric  the HPA was unable to compute the replica count: unable to get external metric vcstream-dev/s3-kafka-private_aoaa_anomaly_v1/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: vcstream-gen-anomaly,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: no matching metrics found for s3-kafka-private_aoaa_anomaly_v1
  ScalingLimited  False   DesiredWithinRange       the desired count is within the acceptable range
Events:
  Type     Reason                   Age                 From                       Message
  ----     ------                   ----                ----                       -------
  Warning  FailedGetExternalMetric  31m                 horizontal-pod-autoscaler  unable to get external metric vcstream-dev/s3-kafka-private_aoaa_deadletter_v1/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: vcstream-gen-anomaly,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: no matching metrics found for s3-kafka-private_aoaa_deadletter_v1
  Warning  FailedGetExternalMetric  75s (x77 over 46m)  horizontal-pod-autoscaler  unable to get external metric vcstream-dev/s3-kafka-private_aoaa_anomaly_v1/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: vcstream-gen-anomaly,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: no matching metrics found for s3-kafka-private_aoaa_anomaly_v1

And sometimes 0 or like in the following:

Name:                                                              keda-hpa-vcstream-gen-anomaly
Namespace:                                                         vcstream-dev
Labels:                                                            app.kubernetes.io/managed-by=Helm
                                                                   app.kubernetes.io/name=keda-hpa-vcstream-gen-anomaly
                                                                   app.kubernetes.io/part-of=vcstream-gen-anomaly
                                                                   app.kubernetes.io/version=2.6.0
                                                                   scaledobject.keda.sh/name=vcstream-gen-anomaly
Annotations:                                                       <none>
CreationTimestamp:                                                 Mon, 10 Jan 2022 16:19:18 +0100
Reference:                                                         Deployment/vcstream-gen-anomaly
Metrics:                                                           ( current / target )
  "s0-cron-Europe-Brussels-30xxxx-59xxxx" (target average value):  <unknown> / 1
  "s1-prometheus-process_cpu_usage" (target average value):        <unknown> / 75
  "s3-kafka-private_aoaa_anomaly_v1" (target average value):       <unknown> / 100
  "s3-kafka-private_aoaa_deadletter_v1" (target average value):    <unknown> / 100
Min replicas:                                                      1
Max replicas:                                                      6
Deployment pods:                                                   0 current / 0 desired
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    ReadyForNewScale    recommended size matches current size
  ScalingActive   False   ScalingDisabled     scaling is disabled since the replica count of the target is zero
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:
  Type     Reason                   Age                   From                       Message
  ----     ------                   ----                  ----                       -------
  Warning  FailedGetExternalMetric  57m (x2 over 117m)    horizontal-pod-autoscaler  unable to get external metric vcstream-dev/s3-kafka-private_aoaa_deadletter_v1/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: vcstream-gen-anomaly,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: no matching metrics found for s3-kafka-private_aoaa_deadletter_v1
  Warning  FailedGetExternalMetric  12m (x259 over 132m)  horizontal-pod-autoscaler  unable to get external metric vcstream-dev/s3-kafka-private_aoaa_anomaly_v1/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: vcstream-gen-anomaly,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: no matching metrics found for s3-kafka-private_aoaa_anomaly_v1

The scaled objet is described as follow:

Name:         vcstream-gen-anomaly
Namespace:    vcstream-dev
[...]
Status:
  Conditions:
    Message:  ScaledObject is defined correctly and is ready for scaling
    Reason:   ScaledObjectReady
    Status:   True
    Type:     Ready
    Message:  Scaler cooling down because triggers are not active
    Reason:   ScalerCooldown
    Status:   False
    Type:     Active
    Message:  No fallbacks are active on this scaled object
    Reason:   NoFallbackFound
    Status:   False
    Type:     Fallback
  External Metric Names:
    s0-cron-Europe-Brussels-30xxxx-59xxxx
    s1-prometheus-process_cpu_usage
    s3-kafka-private_aoaa_anomaly_v1
    s3-kafka-private_aoaa_deadletter_v1
  Health:
    s0-cron-europe-brussels-30xxxx-59xxxx:
      Number Of Failures:  0
      Status:              Happy
    s1-prometheus-process_cpu_usage:
      Number Of Failures:  0
      Status:              Happy
    s3-kafka-private_aoaa_deadletter_v1:
      Number Of Failures:  0
      Status:              Happy
  Last Active Time:        2022-02-14T13:15:43Z
  Original Replica Count:  1
  Scale Target GVKR:
    Group:            apps
    Kind:             Deployment
    Resource:         deployments
    Version:          v1
  Scale Target Kind:  apps/v1.Deployment
Events:
  Type    Reason                      Age                  From           Message
  ----    ------                      ----                 ----           -------
  Normal  KEDAScaleTargetDeactivated  14m (x11 over 136m)  keda-operator  Deactivated apps/v1.Deployment vcstream-dev/vcstream-gen-anomaly from 1 to 0
  Normal  KEDAScaleTargetActivated    42s (x10 over 135m)  keda-operator  Scaled apps/v1.Deployment vcstream-dev/vcstream-gen-anomaly from 0 to 1

[loicmathieu]

This seems to also be linked to #2592 and fixed when upgrading to 2.6.1.