Use mirror make 2 instances with multiple source clusters and one destination cluster

[rose36]

Hi guys,

I've seen that it's possible to configure mirror make 2 to synchronize data from multiple source clusters and one target cluster, similar to what is shown in this “Data aggregation” use case: https://developers.redhat.com/articles/2023/11/13/demystifying-kafka-mirrormaker-2-use-cases-and-architecture#use_cases.

I would like to know if it is possible to instantiate a mirror make 2 for each source cluster and with only one common target cluster, for example:

sourceClusterA -> mirrorMakeA -> targetClusterC
sourceClusterB -> mirrorMakeB -> targetClusterC

[scholzj]

Multiple source cluster and one target cluster can be done both through a single Mirror Maker or through multiple Mirror Makers. For the second case, please make sure to follow https://strimzi.io/docs/operators/latest/full/deploying.html#con-config-mm2-multiple-instances-str to make them really independent.

[rose36]

Thanks, @scholzj!

I added the fields group.id, offset.storage.topic, config.storage.topic, and status.storage.topic and it worked for me. My mirror make configuration file looks like this:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: mirror-maker-2
  namespace: kafka
spec:
  version: 3.6.0
  replicas: 1
  connectCluster: {{ .Values.targetCluster.name }} # Must be the target custer
  clusters:
  - alias: {{ .Values.sourceCluster.name }} # Source cluster
    bootstrapServers: {{ .Values.sourceCluster.bootstrapServers }}
  - alias: {{ .Values.targetCluster.name }} # Target cluster
    bootstrapServers: {{ .Values.targetCluster.bootstrapServers }}
    config:
      # -1 means it will use the default replication factor configured in the broker
      config.storage.replication.factor: -1
      offset.storage.replication.factor: -1
      status.storage.replication.factor: -1
      group.id: {{ .Values.groupID }}
      offset.storage.topic: {{ .Values.offsetStorageTopic }}
      config.storage.topic: {{ .Values.configStorageTopic }}
      status.storage.topic: {{ .Values.statusStorageTopic }}
  mirrors:
  - sourceCluster: {{ .Values.sourceCluster.name }}
    targetCluster: {{ .Values.targetCluster.name }}
    sourceConnector:
      tasksMax: 1
      config:
        # -1 means it will use the default replication factor configured in the broker
        replication.factor: -1
        offset-syncs.topic.replication.factor: -1
        sync.topic.acls.enabled: "false"
    checkpointConnector:
      tasksMax: 1
      config:
        # -1 means it will use the default replication factor configured in the broker
        checkpoints.topic.replication.factor: -1
    topicsPattern: ".*"
    groupsPattern: ".*"

For the scenario I mentioned these fields need to have different values.