KEDA scales my Az Function pods to my number of Event Hub partitions

[nansravn]

I'm using KEDA to scale Azure Functions (running on AKS) consuming events from Event Hub. I have a conceptual doubt about how to configure properly the pod autoscaler.

• Based on my tests, when I create an Event Hub with 10 partitions, KEDA will scale by default my number of pods running my Azure Function to 10.
• If i have an Event Hub with 20 partitions, KEDA will scale the number of pods to 20.
• How can I force KEDA to start only 1 pod, running my Azure Functions image. I should be able to have only 1 pod consuming from all my 20 Event Hub partitions, right?

This is my manifest:

kind: ScaledObject
metadata:
  name: dropai
  labels: {}
spec:
  scaleTargetRef:
    name: dropai
  pollingInterval: 60
  cooldownPeriod: 300
  minReplicaCount: 3
  maxReplicaCount: 20
  triggers:
  - type: azure-eventhub
    metadata:
      direction: in
      eventHubName: evh-newvideos-dev
      cardinality: one
      consumerGroup: $default
      dataType: binary
      connectionFromEnv: EVH_CONNECTION_STRING
      storageConnectionFromEnv: ADLS_CONNECTION_STRING
      unprocessedEventThreshold: '64'

[tomkerkhove]

How can I force KEDA to start only 1 pod, running my Azure Functions image. I should be able to have only 1 pod consuming from all my 20 Event Hub partitions, right?

Here lies the misunderstanding, you'll need a pod per partition to process the events as it cannot listen to all partitions at once but I'll leave this up to @ahmelsayed to verify.

[nansravn]

If the number of pods always need to be the same as the number of partitions, the horizontal pod autoscaler would be useless, wouldn't it?

• On my understanding, I can't have more than 1 pod consuming from the same partition, as an event hub partition is assigned for only 1 consumer at a given time.
• On the other hand, it should be possible to have multiple event hub partitions assigned to a single consumer. As an example, it should be possible to have 2 consumers (2 pods), each one targeting events from 10 different partitions, in a way all the 20 event hub partitions are covered by one of the 2 pods.

Please, feel free to correct me if I'm wrong on these understandings.

The only Azure documentation I've found about KEDA + Azure Functions was this one. We may need to improve these docs to discuss the ScaleObject configuration and how the events are routed from each event hub partition to Azure Functions when using KEDA. This isn't clear by now on that documentation.

[tomkerkhove]

If the number of pods always need to be the same as the number of partitions, the horizontal pod autoscaler would be useless, wouldn't it?

No because instance for partition 1 could go to 0 instances while the others are still up.

• On my understanding, I can't have more than 1 pod consuming from the same partition, as an event hub partition is assigned for only 1 consumer at a given time.

Correct

• On the other hand, it should be possible to have multiple event hub partitions assigned to a single consumer. As an example, it should be possible to have 2 consumers (2 pods), each one targeting events from 10 different partitions, in a way all the 20 event hub partitions are covered by one of the 2 pods.

We don't support this AFAIK because we assume the above given we don't know how many pods you're using to process partitions. If we'd add this then there would need to be a amountOfPartitionsPerPod like field which is definitely possible.

The only Azure documentation I've found about KEDA + Azure Functions was this one. We may need to improve these docs to discuss the ScaleObject configuration and how the events are routed from each event hub partition to Azure Functions when using KEDA. This isn't clear by now on that documentation.

I'll leave this up to @jeffhollan @ahmelsayed

[Cameron-Nott]

@tomkerkhove, is this still a limitation that the eventhub scaler only scales to the partion count?

We have implmented the partitionbasedloadbalancer for the eventhub so that a pod can take ownership of the partition and then relisquish it. Given that the event stream comes though faster than the pod processing of the event with the various complexity of the business process we are implmenting I would (similar to @nansravn ) like to have X partions and have either 2X or 3X the pod count to process the incoming events.

If I take your approach of scaling the pod's to Zero then I would need to have 32 partions to have from 0 to 32 pods scaling at any given time.

Microsofts doc for your reference https://learn.microsoft.com/en-us/azure/event-hubs/event-processor-balance-partition-load

The alternative would be to create a pod that checked the message count of the partitions and use the storage queue scaler or similar to orchestrate scaling.

So basically I have the same requriement/issue as @nansravn