MM2 authentication failing when connecting to Confluent Kafka

[AAR256]

I'm trying to setup MM2 between Confluent Kafka and Strimzi Kafka on AKS cloud. The MM2 YAML that I'm using is -

---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: mm2-cluster
  namespace: kafka
spec:
  version: 3.0.0
  replicas: 1
  template:
    pod:
      metadata:
        annotations:
          prometheus.io/scrape: "true"
          prometheus.io/port: "9404"
  connectCluster: dev-strimzi-kafka #"my-cluster-target"
  clusters:
    - alias: dev-ccloud-kafka #"my-cluster-source"
      authentication:
        type: plain
        username: XXXXXXXXXX
        passwordSecret:
          secretName: ccloud-admin
          password: password
      bootstrapServers: XXXXXXXX.azure.confluent.cloud:XXXX
    - alias: dev-strimzi-kafka #"my-cluster-target"
      authentication:
        type: scram-sha-512
        username: strimzi-admin
        passwordSecret:
          secretName: strimzi-admin
          password: password
      bootstrapServers: dev-strimzi-kafka-bootstrap.XXXX.net:XXX
      tls:
        trustedCertificates:
          - secretName: dev-strimzi-kafka-cluster-ca-cert
            certificate: ca.crt
      config:
        # -1 means it will use the default replication factor configured in the broker
        config.storage.replication.factor: 3
        offset.storage.replication.factor: 3
        status.storage.replication.factor: 3
  mirrors:
    - sourceCluster: dev-ccloud-kafka #"my-cluster-source"
      targetCluster: dev-strimzi-kafka #"my-cluster-target"
      sourceConnector:
        config:
          replication.factor: 3
          offset-syncs.topic.replication.factor: 3
          tasks.max: 6
      heartbeatConnector:
        config:
          heartbeats.topic.replication.factor: 3
      checkpointConnector:
        config:
          checkpoints.topic.replication.factor: 3
          sync.group.offsets.enabled: true
      topicsPattern: "topic1"
      groupsPattern: ".*"

  resources:
    requests:
      memory: 4Gi
      cpu: "1"
    limits:
      memory: 8Gi
      cpu: "2"

The error log that I'm getting is -

2023-01-30 06:53:48,102 WARN [AdminClient clientId=adminclient-14] Connection to node -1 (XXXX.azure.confluent.cloud/XX.XXX.XX.XXX:XXXX) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-14]

I have tried producing and consuming from Confluent using the same credentials and it was working fine. I also came upon a similar case on the discussion forum but couldn't find any solution - #6988.
Can someone help me out with this one? If any other information is required, do let me know.

[AAR256]

The Confluent cluster configuration is as follows -

security.protocol=SASL_SSL
sasl.mechanism=PLAIN

We're using API Key and Password provided by confluent to connect to the kafka cluster for production and consumption.

[scholzj]

2023-01-30 06:53:48,102 WARN [AdminClient clientId=adminclient-14] Connection to node -1 (XXXX.azure.confluent.cloud/XX.XXX.XX.XXX:XXXX) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-14]

Often suggests that you are trying to connect without encryption to a TLS interface. Are you sure Confluent Cloud does not use TLS?

[AAR256]

@scholzj After configuring TLS with the Let's encrypt certificate, MM2 started running fine between confluent and Strimzi cloud. One issue I saw was that if in topic whitelisting, let's assume we have the below Configuration defined -

topicsPattern: "topicABC.v1\ |topicxyz.v1"

So the above pattern contains 2 topics - one lowercase and one mixed-case. When the replication started only the lowercase topic was replicated and mixed-case was ignored entirely. I even tried creating the topic in K8s first so that MM2 might recognise and move the data but that didn't happen as well.
Any idea why it isn't working as we can create mixed-case topics in Strimzi kafka cluster by using spec.topicName in KafkaTopic definition or is there any config in MM2 that we're missing out?

[scholzj]

I do not see any reason for it. The MM2 does not care about the Topic Operator. That said, you expression above seems weird. Do you have space in the topic name?

[AAR256]

I believe you might be correct. So, based on your comment about the expression, I changed the expression to just the mixed-case topic name - topicsPattern: "topicABC.v1" and that seemed to work. As soon as I applied the changes, the messages flowed into the topic present in Strimzi cluster.

[irmiller22]

@AAR256 I'm running into this exact issue. Can you provide some guidance on how you configured this TLS cert via LetsEncrypt? And got authentication working with a Confluent-managed cluster via username / password API key and secret? I'd also be interested in seeing your working configuration, if possible. My setup is accessible here: https://github.com/irmiller22/mirrormaker2-poc/blob/replicate-confluent-to-strimzi/workloads/kafka-mirrormaker/mirrormaker.yaml.

Any help or guidance would be extremely helpful!

[romankydybets]

this is example which is working for me

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: to-sa-east-1
  namespace: strimzi-kafka-operator
spec:
  version: 3.7.0
  replicas: 1
  connectCluster: "cluster-sa-east-1" # Must be the target custer
  clusters:
  - alias: "cluster-sa-east-1" # Target cluster
    bootstrapServers: pls-kafka-cluster-kafka-bootstrap.strimzi-kafka-operator.svc.cluster.local.:9092
    config:
      # -1 means it will use the default replication factor configured in the broker
      config.storage.replication.factor: -1
      offset.storage.replication.factor: -1
      status.storage.replication.factor: -1
  - alias: "cluster-confluent" # Source cluster
    bootstrapServers: <hostname>.aws.glb.confluent.cloud:9092
    authentication:
      type: plain
      username: "<username>"
      passwordSecret:
        secretName: confluent-sa-secret
        password: password
    tls:
      trustedCertificates: [ ]
  mirrors:
  - sourceCluster: "cluster-confluent"
    targetCluster: "cluster-sa-east-1"
    sourceConnector:
      tasksMax: 1
      autoRestart:
        enabled: true
      config:
        # -1 means it will use the default replication factor configured in the broker
        replication.factor: -1
        offset-syncs.topic.replication.factor: -1
        sync.topic.acls.enabled: "false"
        refresh.topics.interval.seconds: 600
    checkpointConnector:
      tasksMax: 1
      autoRestart:
        enabled: true
      config:
        # -1 means it will use the default replication factor configured in the broker
        checkpoints.topic.replication.factor: -1
        sync.group.offsets.enabled: "false"
        refresh.groups.interval.seconds: 600
    topicsPattern: "test-strimzi-mm2"
    groupsPattern: "extra"
  template:
    pod:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node.kubernetes.io/lifecycle
                    operator: In
                    values:
                      - strimzi