Support "QueueLength/Value" metric type for Kafka scaler

[PieterVanZyl-Dev]

Proposal

Would like to be able to use "QueueLength" metric type for the Kafka scaler, instead of "LagOffset" as it is now.
This would be useful for HPA and for anyone who isn't trying to scale on a lag value.

Use-Case

Similar to #1314
We want to be able to have access to QueueLength in the KafkaScaler as an option.
For our use case we want to scale replicas on the total amount of messages in the queue instead of the current "lagOffset".

Want to have min replicas set to 0, so it will have 0 replicas when queue is empty, but as soon as the queue for ex contains 1 to 300 messages it should scale to 1 replica, and between 300-600 it should scale to 2 replicas and so on.

I'd like to help with this if someone isn't already working on this for kafka.

Anything else?

Related to : #1314 (rabbitMQ implementation of this) #692 (This would be benefit from this feature I believe)

I'd prefer some umbrella issue where we can track this effort, because imho it should be done for all scalers.

Originally posted by @zroubalik in #1314 (comment)

Is there an umbrella issue I can tag this one onto ?

Discussed this with @knepe and he is willing to contribute this for Rabbit MQ scaler 🎉
We will do the POC on Kafka and then follow up with other scalers.

Originally posted by @zroubalik in #692 (comment)

Is there a POC on kafka ? or is this something I can help with in the future when I have capacity.

[PieterVanZyl-Dev]

@zroubalik is it possible there is already a POC somewhere in the PRs/Issues for Kafka that I can't find anywhere ?

[zroubalik]

I am not sure if I fully understand your request. IMHO there are mutliple different features/concepts mixed in this.

• Kafka & queueLength - is that even something that could be pulled from Kafka as a metric?

• Want to have min replicas set to 0, so it will have 0 replicas when queue is empty, but as soon as the queue for ex contains 1 to 300 messages it should scale to 1 replica, and between 300-600 it should scale to 2 replicas and so on."

  To cover this scenario, you would probably need something like startThreshold as proposed in the Support 'startThreshold' for queue scalers like Kafka #692, queueLength will not help with this

• support for Value type - that's another concept, it's about the target's replica count calculation, it's a type of metrics consumed by HPA