KEDA Scaling Jobs using Azure Service Bus scaler - queuelength configuration parameter

[fisapras]

I have a queue which holds incoming messages from an application (producer). I have a consumer application in an container which is deployed to AKS. I have configured KEDA scaledobject jobtype with Azure Service bus as the trigger to scale the consumer instances. As the messages are queued KEDA creates consumer instances which read a single message from the queue and start processing.

However what I have noticed is that the number of consumer instances created by KEDA is always equal to the queuelength parameter set for the Azure Service Bus trigger. I want KEDA to scale the consumer instances equal to the number of messages in the queue and not to the queuelength parameter configuration of Azure Service Bus trigger. This is so that all the messages can be processed in parallel for performance.

Is this known behavior? How can I change the behavior so that KEDA scales based on number of messages and not the queuelenghth configuration parameter?

[tomkerkhove]

I get your understanding but I think there is a bigger issue which I'm addressing in #653.

You will need to increase the maximum count to a high number as we keep that in check.

[fisapras]

@tomkerkhove, I already have set maxReplicaCount to 100. Below is my script that I use to deploy. What I have noticed when I queue more than 400 messages in the queue KEDA creates only 5 (since default queue lenght is 5) consumer instances. I would expect it to scale to 100 instances equal to maxReplicaCount. Have I configured something incorrectly?

data:
SB_CONN_STR: BASE64_SOME_VALUE
apiVersion: v1
kind: Secret
metadata:
name: secrets-mounthood-jobs
namespace: default

apiVersion: keda.k8s.io/v1alpha1
kind: TriggerAuthentication
metadata:
name: trigger-auth-service-bus-mounthood
spec:
secretTargetRef:

• parameter: connection
  name: secrets-mounthood-jobs
  key: SB_CONN_STR

apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
name: mounthood-jobs
namespace: default
spec:
scaleType: job
maxReplicaCount: 100
jobTargetRef:
parallelism: 1
completions: 1
activeDeadlineSeconds: 5000
backoffLimit: 3
template:
spec:
containers:
- name: mounthood
image: mounthood:latest
resources:
requests:
memory: 2G
cpu: 2000m
limits:
cpu: 2000m
memory: 2G
imagePullSecrets:
- name: kubecontainers-secret
tolerations:
- operator: Exists
triggers:

• type: azure-servicebus
  metadata:
  queueName: mh-request
  authenticationRef:
  name: trigger-auth-service-bus-mounthood

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.

[sarathkrishna-spherex]

I have a similar issue, not sure though. I am using a scaledJob. When there are 5 ( or x number of )messages in the queue in service bus, it scales up 5 (or x number of ) jobs ( inturn pods ). But when a new message arrives in the queue, the scaledJob doesn't create one more to process it. I read a lot and felt like I should set "accurate" strategy. But that also didn't work. My messages are just waiting in the queue. Can someone help?