[Bug]: Unable to get MirrorMaker2 to create replicated topic in target cluster #9577

ETisREAL · 2024-01-20T17:25:45Z

ETisREAL
Jan 20, 2024

Bug Description

First and foremost, thank you guys for this amazing project!

I've really been trying my head off to get MirrorMaker2 to work. I've gone through the documentation multiple times and looked at all the examples provided, but I just can't get it to work! :(

Steps to reproduce

I am creating a source cluster in GKE with internal load balancers for each broker to be reachable by mirrormaker2:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka
spec:
  kafka:
    version: 3.6.1
    replicas: 3
    authorization:
      type: simple
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
        authentication:
          type: scram-sha-512
      - name: secure
        port: 9093
        type: internal
        tls: true
        authentication:
          type: scram-sha-512
      - name: external
        port: 9094
        configuration:
          brokers:
            - broker: 0
              annotations:
                networking.gke.io/load-balancer-type: "Internal"
                networking.gke.io/internal-load-balancer-allow-global-access: "true" # to allow traffic across regions
            - broker: 1
              annotations:
                networking.gke.io/load-balancer-type: "Internal"
                networking.gke.io/internal-load-balancer-allow-global-access: "true" # to allow traffic across regions  
            - broker: 2
              annotations:
                networking.gke.io/load-balancer-type: "Internal"
                networking.gke.io/internal-load-balancer-allow-global-access: "true" # to allow traffic across regions  
          bootstrap:
            annotations:
              networking.gke.io/load-balancer-type: "Internal"
              networking.gke.io/internal-load-balancer-allow-global-access: "true" # to allow traffic across regions
        type: loadbalancer
        tls: false
        authentication:
          type: scram-sha-512
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.6"
      auto.create.topics.enable: false
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 4Gi
        deleteClaim: true
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 4Gi
      deleteClaim: true
  entityOperator:
    tlsSidecar: {}
    topicOperator: {}
    userOperator: {}

This is the target cluster:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka-mirror
spec:
  kafka:
    version: 3.6.1
    replicas: 3
    authorization:
      type: simple
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
        authentication:
          type: scram-sha-512
      - name: secure
        port: 9093
        type: internal
        tls: true
        authentication:
          type: scram-sha-512
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.6"
      auto.create.topics.enable: false
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 4Gi
        deleteClaim: true
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 4Gi
      deleteClaim: true
  entityOperator:
    tlsSidecar: {}
    topicOperator: {}
    userOperator: {}

I am creating a user in the source cluster fot mirrormaker2:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: kmm2-source-user
  labels:
    strimzi.io/cluster: kafka
    kafka-mirror-make: source
spec:
  authentication:
    type: scram-sha-512
  authorization:
    type: simple
    acls:
      - resource:
          type: topic
          name: "*"
          patternType: literal
        operations:
          - All
        host: "*"
      - resource:
          type: group
          name: "*"
          patternType: literal
        operations:
          - All
        host: "*"
      - resource:
          type: cluster
        operations:
          - All
      - resource:
          type: transactionalId
          name: "*"
          patternType: literal
        operations:
          - All

I then retrieve the password and copied over to a secret in the target cluster:

kubectl get secret kmm2-source-user -o jsonpath='{.data.password}'

# copied passsword in the source-cluster-user-secret.yaml under the password key

kubectl config use gke_k8s-409709_europe-west2-a_cluster-2

kubectl apply -f source-cluster-user-secret.yaml

I create a topic in the source cluster:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: mirrored-topic-two
  labels:
    strimzi.io/cluster: kafka
spec:
  partitions: 4
  replicas: 2
  config:
    retention.ms: 7200000
    segment.bytes: 1073741824

and finally I create mirrormaker2 in the target cluster

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: kafka-mirror-maker2
spec:
  version: 3.6.1
  replicas: 1
  connectCluster: "cluster-target"
  resources:
    requests:
      cpu: 10m
      memory: 50Mi
    limits:
      cpu: 750m
      memory: 512Mi
  clusters:
  - alias: "cluster-source"
    bootstrapServers: 10.2.0.4:9094,10.2.0.5:9094,10.2.0.6:9094
    authentication:
      type: scram-sha-512
      username: kmm2-source-user
      passwordSecret:
        password: password
        secretName: source-cluster-user-secret
  - alias: "cluster-target"
    bootstrapServers: kafka-mirror-kafka-bootstrap.default.svc:9092
    authentication:
      type: scram-sha-512
      username: kmm2-target-user
      passwordSecret:
        password: password
        secretName: kmm2-target-user
  mirrors:
  - sourceCluster: "cluster-source"
    targetCluster: "cluster-target"
    topicsPattern: "mirrored-topic,"

As far as I can tell, I am doing everything correctly... MirrorMaker2 can actually connect successfully to both clusters (as the logs on the brokers on each cluster show the metadata topic for mirrormaker are getting created). Everything looks normal on MirrorMaker's logs as well:

 2024-01-20 17:07:59,802 INFO 192.168.124.16 - - [20/Jan/2024:17:07:59 +0000] "GET /connectors HTTP/1.1" 200 2 "-" "- │
│ " 3 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp1761853129-20]                                            │
│ 2024-01-20 17:07:59,822 INFO 192.168.124.16 - - [20/Jan/2024:17:07:59 +0000] "GET /admin/loggers/ HTTP/1.1" 200 142  │
│ "-" "-" 2 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp1761853129-19]                                      │
│ 2024-01-20 17:08:01,384 INFO 192.168.126.1 - - [20/Jan/2024:17:08:01 +0000] "GET / HTTP/1.1" 200 91 "-" "kube-probe/ │
│ 1.27" 3 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp1761853129-22]

For reference, I tried using both the brokers endpoint and the bootsrapServer endpoint for the 'bootstrapServers' propery of the source cluster, but it didn't change anything.

Also, I produced records on the topic, and still the topic didn't even get created in the target cluster

As I said, everything starts without errors or anything, so I really don't know what else to try. Any help is very, very much appreciated.

Thank you in advance

Expected behavior

No response

Strimzi version

0.39.0

Kubernetes version

1.27.3-gke.100

Installation method

Helm chart, yaml manifests

Infrastructure

GKE

Configuration files and logs

No response

Additional context

No response

scholzj · 2024-01-20T18:53:18Z

scholzj
Jan 20, 2024
Maintainer

I think you would need to share the full log (at least from Mirror Maker 2 for a start I guess) to better understand what might be the issue.

What looks strange in your configuration is this: topicsPattern: "mirrored-topic,". This should be if I remember correctly a comma separate list or regular expression. I'm not sure how the trailing , will be parsed. But even if it is parsed correctly, I guess it would not match topic named mirrored-topic-two.

3 replies

ETisREAL Jan 20, 2024
Author

Right, sorry for not clarifying that as well. That was a reminiscence from another attempt.
You remember correctly and the doc says that the topicsPattern can be a comma separated list of topics.
I have tried using the "match all" regex pattern as well (.*), still no luck.

I will provide the logs of the Kafka Brokers and a longer list of logs from MM2, I guess a question I have in the meantime is if it is okay to specify the bootstrap server service for the bootstrapServers parameter or if I must specify the actual brokers themselves?

scholzj Jan 20, 2024
Maintainer

If you have a single bootstrap service, it often works better when some of the brokers are down etc. But in general, both should work.

ETisREAL Jan 20, 2024
Author

Got it, thanks for the clarification :)

ETisREAL · 2024-01-20T19:44:24Z

ETisREAL
Jan 20, 2024
Author

So, these are the startup logs from MM:

(org.apache.kafka.clients.consumer.ConsumerConfig) [DistributedHerder-connect-1-1]                                                                                       │
│ 2024-01-20 19:35:16,723 INFO These configurations '[producer.sasl.jaas.config, rest.advertised.port, plugin.path, config.providers, consumer.sasl.mechanism, admin.securi │
│ ty.protocol, metrics.context.connect.kafka.cluster.id, config.providers.file.class, offset.storage.topic, consumer.security.protocol, admin.sasl.mechanism, value.convert │
│ er, key.converter, consumer.client.rack, consumer.sasl.jaas.config, config.storage.topic, producer.security.protocol, metrics.context.connect.group.id, rest.advertised.h │
│ ost.name, status.storage.topic, header.converter, producer.sasl.mechanism, rest.port, admin.sasl.jaas.config]' were supplied but are not used yet. (org.apache.kafka.clie │
│ nts.consumer.ConsumerConfig) [DistributedHerder-connect-1-1]                                                                                                              │
│ 2024-01-20 19:35:16,723 INFO Kafka version: 3.6.1 (org.apache.kafka.common.utils.AppInfoParser) [DistributedHerder-connect-1-1]                                           │
│ 2024-01-20 19:35:16,732 INFO Kafka commitId: 5e3c2b738d253ff5 (org.apache.kafka.common.utils.AppInfoParser) [DistributedHerder-connect-1-1]                               │
│ 2024-01-20 19:35:16,739 INFO Kafka startTimeMs: 1705779316723 (org.apache.kafka.common.utils.AppInfoParser) [DistributedHerder-connect-1-1]                               │
│ 2024-01-20 19:35:16,793 INFO [Consumer clientId=mirrormaker2-cluster-configs, groupId=mirrormaker2-cluster] Cluster ID: W2O_Piw0QHebNY6OPF8tkQ (org.apache.kafka.clients. │
│ Metadata) [DistributedHerder-connect-1-1]                                                                                                                                 │
│ 2024-01-20 19:35:16,797 INFO [Consumer clientId=mirrormaker2-cluster-configs, groupId=mirrormaker2-cluster] Assigned to partition(s): mirrormaker2-cluster-configs-0 (org │
│ .apache.kafka.clients.consumer.KafkaConsumer) [DistributedHerder-connect-1-1]                                                                                             │
│ 2024-01-20 19:35:16,798 INFO [Consumer clientId=mirrormaker2-cluster-configs, groupId=mirrormaker2-cluster] Seeking to earliest offset of partition mirrormaker2-cluster- │
│ configs-0 (org.apache.kafka.clients.consumer.internals.SubscriptionState) [DistributedHerder-connect-1-1]                                                                 │
│ 2024-01-20 19:35:16,814 INFO [Producer clientId=mirrormaker2-cluster-configs] Cluster ID: W2O_Piw0QHebNY6OPF8tkQ (org.apache.kafka.clients.Metadata) [kafka-producer-netw │
│ ork-thread | mirrormaker2-cluster-configs]                                                                                                                                │
│ 2024-01-20 19:35:16,852 INFO [Consumer clientId=mirrormaker2-cluster-configs, groupId=mirrormaker2-cluster] Resetting offset for partition mirrormaker2-cluster-configs-0 │
│  to position FetchPosition{offset=0, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka-mirror-kafka-0.kafka-mirror-kafka-brokers.default.svc │
│ :9092 (id: 0 rack: null)], epoch=0}}. (org.apache.kafka.clients.consumer.internals.SubscriptionState) [DistributedHerder-connect-1-1]                                     │
│ 2024-01-20 19:35:16,852 INFO Finished reading KafkaBasedLog for topic mirrormaker2-cluster-configs (org.apache.kafka.connect.util.KafkaBasedLog) [DistributedHerder-conne │
│ ct-1-1]                                                                                                                                                                   │
│ 2024-01-20 19:35:16,853 INFO Started KafkaBasedLog for topic mirrormaker2-cluster-configs (org.apache.kafka.connect.util.KafkaBasedLog) [DistributedHerder-connect-1-1]   │
│ 2024-01-20 19:35:16,853 INFO Started KafkaConfigBackingStore (org.apache.kafka.connect.storage.KafkaConfigBackingStore) [DistributedHerder-connect-1-1]                   │
│ 2024-01-20 19:35:16,854 INFO [Worker clientId=connect-1, groupId=mirrormaker2-cluster] Herder started (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [D │
│ istributedHerder-connect-1-1]                                                                                                                                             │
│ 2024-01-20 19:35:16,964 INFO [Worker clientId=connect-1, groupId=mirrormaker2-cluster] Cluster ID: W2O_Piw0QHebNY6OPF8tkQ (org.apache.kafka.clients.Metadata) [Distribute │
│ dHerder-connect-1-1]
2024-01-20 19:35:20,967 INFO [Worker clientId=connect-1, groupId=mirrormaker2-cluster] Request joining group due to: rebalance failed due to 'The group member needs to h │
│ ave a valid member id before actually entering a consumer group.' (MemberIdRequiredException) (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [Distribut │
│ edHerder-connect-1-1]                                                                                                                                                     │
│ 2024-01-20 19:35:20,967 INFO [Worker clientId=connect-1, groupId=mirrormaker2-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator │
│ ) [DistributedHerder-connect-1-1]                                                                                                                                         │
│ 2024-01-20 19:35:23,982 INFO [Worker clientId=connect-1, groupId=mirrormaker2-cluster] Successfully joined group with generation Generation{generationId=1, memberId='con │
│ nect-1-d383c5fb-c615-471e-8ce0-3d487876cc2e', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]      │
│ 2024-01-20 19:35:24,222 INFO [Worker clientId=connect-1, groupId=mirrormaker2-cluster] Successfully synced group in generation Generation{generationId=1, memberId='conne │
│ ct-1-d383c5fb-c615-471e-8ce0-3d487876cc2e', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [kafka-coordinator-heartbeat-thread |  │
│ mirrormaker2-cluster]                                                                                                                                                     │
│ 2024-01-20 19:35:24,222 INFO [Worker clientId=connect-1, groupId=mirrormaker2-cluster] Joined group at generation 1 with protocol version 2 and got assignment: Assignmen │
│ t{error=0, leader='connect-1-d383c5fb-c615-471e-8ce0-3d487876cc2e', leaderUrl='http://kafka-mirror-maker2-mirrormaker2-0.kafka-mirror-maker2-mirrormaker2.default.svc:808 │
│ 3/', offset=-1, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.Di │
│ stributedHerder) [DistributedHerder-connect-1-1]                                                                                                                          │
│ 2024-01-20 19:35:24,223 INFO [Worker clientId=connect-1, groupId=mirrormaker2-cluster] Starting connectors and tasks using config offset -1 (org.apache.kafka.connect.run │
│ time.distributed.DistributedHerder) [DistributedHerder-connect-1-1]                                                                                                       │
│ 2024-01-20 19:35:24,223 INFO [Worker clientId=connect-1, groupId=mirrormaker2-cluster] Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distribut │
│ ed.DistributedHerder) [DistributedHerder-connect-1-1]                                                                                                                     │
│ 2024-01-20 19:35:24,443 INFO [Worker clientId=connect-1, groupId=mirrormaker2-cluster] Session key updated (org.apache.kafka.connect.runtime.distributed.DistributedHerde │
│ r) [KafkaBasedLog Work Thread - mirrormaker2-cluster-configs]                                                                                                             │
│ 2024-01-20 19:35:54,828 INFO 192.168.126.1 - - [20/Jan/2024:19:35:54 +0000] "GET / HTTP/1.1" 200 91 "-" "kube-probe/1.27" 232 (org.apache.kafka.connect.runtime.rest.Rest │
│ Server) [qtp2100132485-22]                                                                                                                                                │
│ 2024-01-20 19:35:54,832 INFO 192.168.126.1 - - [20/Jan/2024:19:35:54 +0000] "GET / HTTP/1.1" 200 91 "-" "kube-probe/1.27" 239 (org.apache.kafka.connect.runtime.rest.Rest │
│ Server) [qtp2100132485-17]                                                                                                                                                │
│ 2024-01-20 19:35:55,478 INFO 192.168.126.7 - - [20/Jan/2024:19:35:55 +0000] "GET /connectors HTTP/1.1" 200 2 "-" "-" 34 (org.apache.kafka.connect.runtime.rest.RestServer │
│ ) [qtp2100132485-21]

These are the logs from a target broker:

2024-01-20 19:35:22,148 WARN [ReplicaFetcher replicaId=2, leaderId=1, fetcherId=0] Received UNKNOWN_TOPIC_ID from the leader for partition __consumer_offsets-29. This er │
│ ror may be returned transiently when the partition is being created or deleted, but it is not expected to persist. (kafka.server.ReplicaFetcherThread) [ReplicaFetcherThr │
│ ead-0-1]                                                                                                                                                                  │
│ 2024-01-20 19:35:22,148 WARN [ReplicaFetcher replicaId=2, leaderId=1, fetcherId=0] Received UNKNOWN_TOPIC_ID from the leader for partition __consumer_offsets-8. This err │
│ or may be returned transiently when the partition is being created or deleted, but it is not expected to persist. (kafka.server.ReplicaFetcherThread) [ReplicaFetcherThre │
│ ad-0-1]                                                                                                                                                                   │
│ 2024-01-20 19:35:22,148 WARN [ReplicaFetcher replicaId=2, leaderId=1, fetcherId=0] Received UNKNOWN_TOPIC_ID from the leader for partition __consumer_offsets-35. This er │
│ ror may be returned transiently when the partition is being created or deleted, but it is not expected to persist. (kafka.server.ReplicaFetcherThread) [ReplicaFetcherThr │
│ ead-0-1]                                                                                                                                                                  │
│ 2024-01-20 19:35:22,148 WARN [ReplicaFetcher replicaId=2, leaderId=1, fetcherId=0] Received UNKNOWN_TOPIC_ID from the leader for partition __consumer_offsets-2. This err │
│ or may be returned transiently when the partition is being created or deleted, but it is not expected to persist. (kafka.server.ReplicaFetcherThread) [ReplicaFetcherThre │
│ ad-0-1]                                                                                                                                                                   │
│ 2024-01-20 19:37:14,396 INFO Processing notification(s) to /config/changes (kafka.common.ZkNodeChangeNotificationListener) [/config/changes-event-process-thread]         │
│ 2024-01-20 19:37:14,404 INFO Processing override for entityPath: users/kmm2-target-user with config: HashMap(SCRAM-SHA-512 -> [hidden]) (kafka.server.ZkConfigManager) [/ │
│ config/changes-event-process-thread]                                                                                                                                      │
│ 2024-01-20 19:37:14,404 INFO Removing PRODUCE quota for user kmm2-target-user (kafka.server.ClientQuotaManager) [/config/changes-event-process-thread]                    │
│ 2024-01-20 19:37:14,405 INFO Removing FETCH quota for user kmm2-target-user (kafka.server.ClientQuotaManager) [/config/changes-event-process-thread]                      │
│ 2024-01-20 19:37:14,410 INFO Sensor for user kmm2-target-user already exists. Changing quota to 9.223372036854776E18 in MetricConfig (kafka.server.ClientQuotaManager) [/ │
│ config/changes-event-process-thread]                                                                                                                                      │
│ 2024-01-20 19:37:14,411 INFO Removing REQUEST quota for user kmm2-target-user (kafka.server.ClientRequestQuotaManager) [/config/changes-event-process-thread]             │
│ 2024-01-20 19:37:14,411 INFO Sensor for user kmm2-target-user already exists. Changing quota to 9.223372036854776E18 in MetricConfig (kafka.server.ClientRequestQuotaMana │
│ ger) [/config/changes-event-process-thread]                                                                                                                               │
│ 2024-01-20 19:37:14,411 INFO Removing CONTROLLER_MUTATION quota for user kmm2-target-user (kafka.server.ControllerMutationQuotaManager) [/config/changes-event-process-th │
│ read]                                                                                                                                                                     │
│ 2024-01-20 19:37:14,411 INFO Sensor for user kmm2-target-user already exists. Changing quota to 9.223372036854776E18 in MetricConfig (kafka.server.ControllerMutationQuot │
│ aManager) [/config/changes-event-process-thread]

and another target broker:

2024-01-20 19:35:24,004 INFO [GroupCoordinator 0]: Assignment received from leader connect-1-d383c5fb-c615-471e-8ce0-3d487876cc2e for group mirrormaker2-cluster for gene │
│ ration 1. The group has 1 members, 0 of which are static. (kafka.coordinator.group.GroupCoordinator) [data-plane-kafka-request-handler-7]                                 │
│ 2024-01-20 19:37:07,169 INFO [Controller id=0] Processing automatic preferred replica leader election (kafka.controller.KafkaController) [controller-event-thread]        │
│ 2024-01-20 19:37:07,170 TRACE [Controller id=0] Checking need to trigger auto leader balancing (kafka.controller.KafkaController) [controller-event-thread]               │
│ 2024-01-20 19:37:07,174 DEBUG [Controller id=0] Topics not in preferred replica for broker 0 HashMap() (kafka.controller.KafkaController) [controller-event-thread]       │
│ 2024-01-20 19:37:07,175 TRACE [Controller id=0] Leader imbalance ratio for broker 0 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]                   │
│ 2024-01-20 19:37:07,175 DEBUG [Controller id=0] Topics not in preferred replica for broker 1 HashMap() (kafka.controller.KafkaController) [controller-event-thread]       │
│ 2024-01-20 19:37:07,175 TRACE [Controller id=0] Leader imbalance ratio for broker 1 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]                   │
│ 2024-01-20 19:37:07,176 DEBUG [Controller id=0] Topics not in preferred replica for broker 2 HashMap() (kafka.controller.KafkaController) [controller-event-thread]       │
│ 2024-01-20 19:37:07,176 TRACE [Controller id=0] Leader imbalance ratio for broker 2 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]                   │
│ 2024-01-20 19:37:14,400 INFO Processing notification(s) to /config/changes (kafka.common.ZkNodeChangeNotificationListener) [/config/changes-event-process-thread]         │
│ 2024-01-20 19:37:14,406 INFO Processing override for entityPath: users/kmm2-target-user with config: HashMap(SCRAM-SHA-512 -> [hidden]) (kafka.server.ZkConfigManager) [/ │
│ config/changes-event-process-thread]                                                                                                                                      │
│ 2024-01-20 19:37:14,406 INFO Removing PRODUCE quota for user kmm2-target-user (kafka.server.ClientQuotaManager) [/config/changes-event-process-thread]                    │
│ 2024-01-20 19:37:14,406 INFO Sensor for user kmm2-target-user already exists. Changing quota to 9.223372036854776E18 in MetricConfig (kafka.server.ClientQuotaManager) [/ │
│ config/changes-event-process-thread]                                                                                                                                      │
│ 2024-01-20 19:37:14,406 INFO Removing FETCH quota for user kmm2-target-user (kafka.server.ClientQuotaManager) [/config/changes-event-process-thread]                      │
│ 2024-01-20 19:37:14,407 INFO Sensor for user kmm2-target-user already exists. Changing quota to 9.223372036854776E18 in MetricConfig (kafka.server.ClientQuotaManager) [/ │
│ config/changes-event-process-thread]                                                                                                                                      │
│ 2024-01-20 19:37:14,407 INFO Removing REQUEST quota for user kmm2-target-user (kafka.server.ClientRequestQuotaManager) [/config/changes-event-process-thread]             │
│ 2024-01-20 19:37:14,407 INFO Sensor for user kmm2-target-user already exists. Changing quota to 9.223372036854776E18 in MetricConfig (kafka.server.ClientRequestQuotaMana │
│ ger) [/config/changes-event-process-thread]                                                                                                                               │
│ 2024-01-20 19:37:14,407 INFO Removing CONTROLLER_MUTATION quota for user kmm2-target-user (kafka.server.ControllerMutationQuotaManager) [/config/changes-event-process-th │
│ read]                                                                                                                                                                     │
│ 2024-01-20 19:37:14,407 INFO Sensor for user kmm2-target-user already exists. Changing quota to 9.223372036854776E18 in MetricConfig (kafka.server.ControllerMutationQuot │
│ aManager) [/config/changes-event-process-thread]

on the source side, here are some logs from a broker (after a restart to check whether something gets mentioned about the connector):

 (kafka.server.KafkaConfig) [/config/changes-event-process-thread]                                                                                                        │
│ 2024-01-20 19:43:15,340 INFO [SocketServer listenerType=ZK_BROKER, nodeId=2] Enabling request processing. (kafka.network.SocketServer) [main]                             │
│ 2024-01-20 19:43:15,406 INFO Awaiting socket connections on 0.0.0.0:9090. (kafka.network.ControlPlaneAcceptor) [main]                                                     │
│ 2024-01-20 19:43:15,413 INFO Processing override for entityPath: brokers/1 with config: HashMap() (kafka.server.ZkConfigManager) [/config/changes-event-process-thread]   │
│ 2024-01-20 19:43:15,429 INFO Awaiting socket connections on 0.0.0.0:9093. (kafka.network.DataPlaneAcceptor) [main]                                                        │
│ 2024-01-20 19:43:15,508 INFO Awaiting socket connections on 0.0.0.0:9094. (kafka.network.DataPlaneAcceptor) [main]                                                        │
│ 2024-01-20 19:43:15,611 INFO Processing notification(s) to /config/changes (kafka.common.ZkNodeChangeNotificationListener) [/config/changes-event-process-thread]         │
│ 2024-01-20 19:43:15,621 INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.DataPlaneAcceptor) [main]                                                        │
│ 2024-01-20 19:43:15,715 INFO Awaiting socket connections on 0.0.0.0:9091. (kafka.network.DataPlaneAcceptor) [main]                                                        │
│ 2024-01-20 19:43:15,811 INFO Kafka version: 3.6.1 (org.apache.kafka.common.utils.AppInfoParser) [main]                                                                    │
│ 2024-01-20 19:43:15,813 INFO Kafka commitId: 5e3c2b738d253ff5 (org.apache.kafka.common.utils.AppInfoParser) [main]                                                        │
│ 2024-01-20 19:43:15,815 INFO Kafka startTimeMs: 1705779795808 (org.apache.kafka.common.utils.AppInfoParser) [main]                                                        │
│ 2024-01-20 19:43:15,905 INFO [KafkaServer id=2] started (kafka.server.KafkaServer) [main]                                                                                 │
│ 2024-01-20 19:43:16,123 INFO [zk-broker-2-to-controller-alter-partition-channel-manager]: Recorded new controller, from now on will use node monitored-kafka-kafka-1.moni │
│ tored-kafka-kafka-brokers.default.svc:9090 (id: 1 rack: null) (kafka.server.BrokerToControllerRequestThread) [zk-broker-2-to-controller-alter-partition-channel-manager]  │
│ 2024-01-20 19:43:16,204 INFO [zk-broker-2-to-controller-forwarding-channel-manager]: Recorded new controller, from now on will use node monitored-kafka-kafka-1.monitored │
│ -kafka-kafka-brokers.default.svc:9090 (id: 1 rack: null) (kafka.server.BrokerToControllerRequestThread) [zk-broker-2-to-controller-forwarding-channel-manager]            │
│```

4 replies

scholzj Jan 20, 2024
Maintainer

I do not think there is much that can be said from these short snippets. The UNKNOWN_TOPIC_ID might indicate some issues if it persists.

ETisREAL Jan 21, 2024
Author

Ok, but what could be the issue causing UNKNOWN_TOPIC_ID ?? Is there something blatantly wrong or omitted in my configuration? I tried to leave as much as possible as default specifically just to make it work for now

scholzj Jan 21, 2024
Maintainer

Well, as the warning message says, this might be completely fine when new topics are created. Depends if it persists or not.

ETisREAL Jan 21, 2024
Author

Checking back on the logs, the error message persisted even hours after the deployment. Unfortunately, no additional error or working logs appear, and the rest are all INFO logs identical to the ones I provided :/

ETisREAL · 2024-01-27T15:59:22Z

ETisREAL
Jan 27, 2024
Author

Hi @scholzj sorry for the delay.. had a very busy week. I dumped all the logs of the three brokers and the MM2 pod and attached them as a zip file to this message.

If you'd be so kind to take a look at them it would mean the world to me. I really don't know what else to try

logs.zip

3 replies

scholzj Jan 27, 2024
Maintainer

I had a look at the logs, but I do not see anything significant there to be honest. The UNKNOWN_TOPIC_ID warnings are there only once for each partition. So that would idicte that they are just being created. So that should not be an issue.

ETisREAL Jan 27, 2024
Author

Gosh, then why does it not show up when using the kafkatopics command? Also, the topic doesn't actually exist in the cluster because if I try to consume with the "schema" <source_cluser>.<topic_name> it fails saying the topic doesn't exist... I really have no clue what's going wrong

scholzj Jan 27, 2024
Maintainer

I'm afraid I have no idea. Anything in the operator logs? Anything in the status section of the Mirror Maker 2 custom resource?

ETisREAL · 2024-01-27T20:15:16Z

ETisREAL
Jan 27, 2024
Author

As far as I can remember nothing noticeable there either unfortunately. Will keep looking, still thank you very, very much for all the support. It was truly appreciated

…

-------- Messaggio originale -------- Oggetto: Re: [strimzi/strimzi-kafka-operator] [Bug]: Unable to get MirrorMaker2 to create replicated topic in target cluster (Discussion #9577) Da: Jakub Scholz A: strimzi/strimzi-kafka-operator CC: Ettore Pelosato ,Author I'm afraid I have no idea. Anything in the operator logs? Anything in the status section of the Mirror Maker 2 custom resource? — Reply to this email directly, view it on GitHub<#9577 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A3AO7NQUVU5DOWAGVAJYPWDYQVNHTAVCNFSM6AAAAABCDMRCCWVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DENRXGI3DE>. You are receiving this because you authored the thread.Message ID: ***@***.***>

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

[Bug]: Unable to get MirrorMaker2 to create replicated topic in target cluster #9577

{{title}}

Replies: 4 comments 10 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Strimzi

[Bug]: Unable to get MirrorMaker2 to create replicated topic in target cluster #9577

ETisREAL Jan 20, 2024

Bug Description

Steps to reproduce

Expected behavior

Strimzi version

Kubernetes version

Installation method

Infrastructure

Configuration files and logs

Additional context

Replies: 4 comments · 10 replies

scholzj Jan 20, 2024 Maintainer

ETisREAL Jan 20, 2024 Author

scholzj Jan 20, 2024 Maintainer

ETisREAL Jan 20, 2024 Author

ETisREAL Jan 20, 2024 Author

scholzj Jan 20, 2024 Maintainer

ETisREAL Jan 21, 2024 Author

scholzj Jan 21, 2024 Maintainer

ETisREAL Jan 21, 2024 Author

ETisREAL Jan 27, 2024 Author

scholzj Jan 27, 2024 Maintainer

ETisREAL Jan 27, 2024 Author

scholzj Jan 27, 2024 Maintainer

ETisREAL Jan 27, 2024 Author

ETisREAL
Jan 20, 2024

Replies: 4 comments 10 replies

scholzj
Jan 20, 2024
Maintainer

ETisREAL Jan 20, 2024
Author

scholzj Jan 20, 2024
Maintainer

ETisREAL Jan 20, 2024
Author

ETisREAL
Jan 20, 2024
Author

scholzj Jan 20, 2024
Maintainer

ETisREAL Jan 21, 2024
Author

scholzj Jan 21, 2024
Maintainer

ETisREAL Jan 21, 2024
Author

ETisREAL
Jan 27, 2024
Author

scholzj Jan 27, 2024
Maintainer

ETisREAL Jan 27, 2024
Author

scholzj Jan 27, 2024
Maintainer

ETisREAL
Jan 27, 2024
Author