Cannot authenticate against confluent hosted kafka

[dodizzle]

Report

This appears to have been addressed in the past but I am now experiencing the same problems as:
#2136
&
#1241

We should be able to connect using the following configs:
sasl=plaintext
username=api key
password=api secret
However this fails to authenticate.
If I enable tls ( which I believe was a previous workaround ) it complains that I also need to supply a valid ca, cert and key and fails to authenticate.

Expected Behavior

I expected keda-operator to connect to the specified topic.

Actual Behavior

It fails to authenticate.

Steps to Reproduce the Problem

1. Install keda via helm chart(version 2.6.2)
2. Create a TriggerAuthentication as documented here: https://keda.sh/docs/2.6/scalers/apache-kafka/
3. Create a ScaledObject that references the TriggerAuthentication above.
4. Check the logs.

Logs from KEDA operator

ERROR   controller.scaledobject Reconciler error        {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "kafka-scaled-object-spin", "namespace": "helix-apps", "error": "error creating kafka client: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)"}

KEDA Version

2.6.1

Kubernetes Version

1.21

Platform

Google Cloud

Scaler Details

kafka

Anything else?

I tried the work around in the 2 bugs that I referenced but that is not working for me.

[JorTurFer]

hey @dodizzle ,
Could you share your ScaledObject? Is the Kafka cluster reachable from KEDA?

[dodizzle]

I confirmed that the kafka cluster is reachable from the pod.
Here is the SharedObject:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: kafka-scaled-object-spin
  namespace: apps
spec:
  scaleTargetRef:
    name: test-deployment
    namespace: apps
  pollingInterval: 30
  triggers:
  - type: kafka
    metadata:
      bootstrapServers: pkc-lgk0v.us-west1.gcp.confluent.cloud:9092
      consumerGroup: my-group 
      topic: testing01
      lagThreshold: "50"
      offsetResetPolicy: latest
    authenticationRef:
      name: keda-kafka-trigger-auth

[JorTurFer]

any idea @zroubalik ?

[zroubalik]

If I enable tls ( which I believe was a previous workaround ) it complains that I also need to supply a valid ca, cert and key and fails to authenticate.

@dodizzle would you mind sharing the error message?

Sarama client that we use to communicate with Kafka doesn't have the best error messages :) So it is sometimes hard to debug. There's a proposal to improve this though: #3056

[JorTurFer]

Could this vulnerability be related?

[akhan-2020]

Hello, I am experiencing the same issue using the latest 2.7.1 Keda version integrating with confluent cloud, I also confirmed that the cluster has connectivity to the the broker on Confluent cloud, note: TLS is disabled.

Here is the error
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
1.6543820732464771e+09 ERROR controller.scaledobject Reconciler error {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "kafka-scaledobject", "namespace": "default", "error": "error creating kafka client: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)"}

SacledObject and associated secret/authentication objects are attached
kafka-scaler-bug.txt

[zroubalik]

We should really have #3056 implemented, so we can inspect those errors in details. Not it is very hard to tell, what is the actuall problem :(

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[v-shenoy]

@raorugan was trying the Kafka scaler recently and ran into the same problem. The workaround with setting tls: enable fixed it, without specifying cert, key.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.