KEDA ScaledObject with minReplicaCount: 0 and Kafka Consumer Groups

[aryabharat]

Query:

When using KEDA to create a ScaledObject with minReplicaCount: 0, issues arise with Kafka consumers. If the Kafka topic has no messages or the retention period lapses, Kafka marks the consumer group as dead. Consequently, when a producer sends a new message, KEDA fails to scale the consumer from 0 to 1 because it cannot find the consumer group. Is there any kind of heartbeat mechanism for Kafka consumers in KEDA to prevent this issue.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: <name>
  
spec:
  minReplicaCount: 0
  maxReplicaCount: 18
  pollingInterval: 10
  scaleTargetRef:
    name: <name>
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: kafka-broker.svc.cluster.local:9092
        consumerGroup: CONSUMER_GROUP_ID
        lagThreshold: 1000
        topic: <topic>
        offsetResetPolicy: 'latest'

Key Points:

Configuration Issue:

• After the consumer group is marked dead, a new message sent to the topic does not trigger KEDA to scale the consumer pods up from 0.
• KEDA cannot locate the dead consumer group, preventing it from scaling the necessary consumers.

[zroubalik]

How would you refresh the consumer group? Any ideas what can be done?