Strimzi kafka connect connection to confluent kafka broker

[sai9920]

Hey Team,

I am setting up the kafka connect on Kubernetes and using the confluent kafka for it. We use API Key and secret with SASL, plain authentication.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect-cluster
spec:
  version: 3.6.0
  replicas: 3
  image: 386077922041.dkr.ecr.ap-southeast-1.amazonaws.com/kafkaconnect:latest
  bootstrapServers: aws.confluent.cloud:9092
  authentication:
    type: plain
    username: ABCD
    passwordSecret:
      secretName: poc-kafka
      password: password
  readinessProbe:
    initialDelaySeconds: 200
    timeoutSeconds: 20
  livenessProbe:
    initialDelaySeconds: 200
    timeoutSeconds: 20

  config:
    group.id: connect-cluster
    offset.storage.topic: connect-cluster-offsets
    config.storage.topic: connect-cluster-configs
    status.storage.topic: connect-cluster-status
    # -1 means it will use the default replication factor configured in the broker
    config.storage.replication.factor: 1
    offset.storage.replication.factor: 1
    status.storage.replication.factor: 1

After after applying this manifest, i am seeing below errors

2023-11-06 08:31:45,146 WARN [AdminClient clientId=adminclient-1] Connection to node -1 terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-1]

Note: I have created secret using base64 did not use same password. Any help is highly appreciated.

[ppatierno]

The first thing I can say is that Strimzi doesn't work with any random image for Kafka Connect ... it's not going to work with a Confluent one. Unless you built such an image on the top of Strimzi provided Kafka Connect image.
Also ... isn't this aws.confluent.cloud:9092 too much generic as a bootstrap server to connect to your own cluster? Tbh I don't know how Confluent cloud offering works.

[sai9920]

yes we are building it. the image is working with local kafka.. but failing with confluent kafka. The endpoint i just removed since posting here

[ppatierno]

How you are building it? What do you mean by local Kafka?

[sai9920]

i deployed a local kafka for testing the connect in same kubernetes cluster

[ppatierno]

So the problem is understanding what the Confluent Cloud needs in terms of encryption and authentication, something I cannot help with, because I don't know it.

[scholzj]

I think this error normally means you are missing TLS. Are you sure your Kafka cluster does not use TLS? If it does, yu have to configure it in the custom resource.KafkaConnect (check docs and examples for the exact details).

[sai9920]

The team confirmed, they dont use TLS. They informed me authentication type should be SASL SSL. Is there something i can do to make it work?

[ppatierno]

Actually SASL SSL, means you are using TLS as encryption and SASL as authentication.

[sai9920]

but they are not providing any TLS certs, also does SASL work with strimzi?
Sorry for my lack of knowledge, please guide in right path. We want to use strimzi to setup self manage kafka connect

[ppatierno]

Strimzi supports SASL-based with SCRAM-SHA 512 and 256, also PLAIN. You can get more info about how configure it on the doc (it's always the best place to start) https://strimzi.io/docs/operators/latest/configuring#type-KafkaConnectSpec-schema-reference
I don't know Confluent Cloud but I would expect it's TLS based, because I would not expose a Kafka cluster outside, on the public Internet without encryption at least.

[sai9920]

@ppatierno @scholzj For SASL_SSL. I have used. Which solved my problem of connection.

Thank you for your time.

  tls:
    trustedCertificates: []
  authentication:
    type: plain
    username: asdsafsadfsdf
    passwordSecret:
      secretName: poc-kafka
      password: password