SQS scaleOnInFlight not being respected

[miklapko]

Report

We have our ScaledObject configured as such:

spec:
  cooldownPeriod: 180
  maxReplicaCount: 2000
  minReplicaCount: 100
  pollingInterval: 5
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: $DEPLOYMENT
triggers:
    - metadata:
        awsRegion: $REGION
        identityOwner: operator
        queueLength: '2'
        queueURL: >-
          $QUEUE_URL
        scaleOnInFlight: 'true'
      type: aws-sqs-queue

KEDA is not scaling the deployment based on in-flight message count (AWS API is reporting the value correctly). It only looks at messages available, and we verified it by changing queueLength - 1 instead of 2 indeed causes KEDA to scale twice as high, but only if there are messages available.

As soon as messages available reach 0, no matter how many messages are in-flight (tested with 200-2000), KEDA will not scale unless messages available starts going up again.

This results in frequent and steep peaks and lows in scaling.

What is also weird is that it was working as expected before, and started behaving incorrectly when we added more ScaledObjects / ScaledJobs (not many, around 8) to this cluster.

We have tried recreating ScaledObject and restarting / redeploying KEDA completely, no dice. We're running EKS K8S 1.22 and are planning to upgrade, however, again, it worked as expected for some time. I haven't seen any specific fixes in newer versions / any similar issues open or closed.

Expected Behavior

Expected to see KEDA keeping up with in-flight message counter and keep, for example, deployment replicas at 250 for ~500 messages in-flight and 0 messages available.

Actual Behavior

KEDA does not keep up with messages in-flight and scales deployment to minimum.

Steps to Reproduce the Problem

1. Deploy KEDA 2.8.1
2. Deploy a ScaledObject with aforementioned config and deployment to scale.
3. Have another ScaledObject / Deployment with scaleOnInFlight set to false.
4. Have an SQS queue with constant ~200-2000 messages in-flight and coming in consistently.

Logs from KEDA operator

Nothing suspicious / no errors in logs, tried debug level as well.

KEDA Version

2.8.1

Kubernetes Version

< 1.23

Platform

Amazon Web Services

Scaler Details

SQS

Anything else?

No response

[miklapko]

After some experiments and looking at the scaler code, it turns out if only one of many ScaledObjects in the cluster has scaleOnInFlight set to false, true isn't being respected for any other ScaledObject and metrics server doesn't count in-flight messages. I guess it's not the intended behaviour? Our dev tells me scaleOnInFlight is a global variable and that's the cause (I'm not much of a golanger myself).

[JorTurFer]

Our dev tells me scaleOnInFlight is a global variable and that's the cause (I'm not much of a golanger myself).

What do you mean? It's a global variable inside the AWS SDK? inside the SQS Backend? I KEDA we store it as part of the scaler but each trigger has its own scaler instance. This behaviour you are describing sounds weird :( If you set scaleOnInFlight to true in all your scaledobjects , does it work? 🤔

[miklapko]

Yes, if it's explicitly set to true in all ScaledObjects, it works as expected, same if it's not set at all and uses the default. However, just one trigger having it set to false seemingly overrides all the others. Speaking about "global variable" - I guess it's global for the scaler? Idk how go / go instances work, sorry.

[JorTurFer]

I guess it's global for the scaler? Idk how go / go instances work, sorry.

Each trigger inside each ScaledObject generates its own scaler instance, so from KEDA code pov, it isn't global. I can't guarantee that internally the client doesn't use it globally somehow (maybe storing the value at scope level). I have to check it deeper to be 100% sure but this is interesting as this could be a strong limitation to share in docs.

[JorTurFer]

Hi,
I think that I have found the problem. I'll open a PR soon, but you were right, once a ScaledObject sets scaleOnInFlight: false, all the ScaledObjects has that value set. Thanks for reporting