Support 'startThreshold' for queue scalers like Kafka

[kellybirr]

When using queue based scalers like EventHubs, Kafka, RabbitMQ it would be great to have a low threshold (of 1 message) when the HPA has scaled to zero, but keep the normal, larger lagThreshold when scaling up above one instance.

I'm mostly talking about Kafka a the moment, as that's what I use. I'd like to let my ScaledObject drop to zero for topics that don't see much traffic and start when a single message needs processing. However, I only want to scale one instance per 100 messages behind the head of the topic. This is especially useful when messages come infrequently but in batches when they do.

Something like this:

apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
  name: my-kafka-processor-keda
spec:
  scaleTargetRef:
    deploymentName: my-kafka-processor
  pollingInterval: 30
  minReplicaCount: 0
  maxReplicaCount: 10
  triggers:
  - type: kafka
    metadata:
      bootstrapServers: awesome.servicebus.windows.net:9093
      consumerGroup: my-kafka-worker
      topic: my-topic
      lagThreshold: "100"
      startThreshold: "1"

I think for this to work, the scaler would have to lie to the HPA about the metric value when it was greater or equal to the startThreshold but less than the lagThreshold and claim that it was the lagThreshold, or reconfigure the HPA when it scaled to zero and then back when scaled above zero.

[tomkerkhove]

Interesting. Question is, should 1 be configurable or just how it behaves because AFAIK it was spinning up an instance once there is a message already; no @jeffhollan ?

[kellybirr]

One, as the default, would be ideal. I’m pretty sure that’s what 99% if use cases would call for. Being able to configure it to wait for more than one would be a benefit in some special cases.

[zroubalik]

I like this idea to enable this option (keeping the default as it is currently of course).

@ppatierno WDYT?

[shashiik]

#1303 @zroubalik In response to our standup discussion yesterday, this is what I had to accomplish the minimumThreshold requirement.

[knepe]

+1 on this feature. It would be a great improvement I think 👍

[zroubalik]

Discussed this with @knepe and he is willing to contribute this for Rabbit MQ scaler 🎉

We will do the POC on Kafka and then follow up with other scalers.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[tomkerkhove]

Is this still WIP @knepe ?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.