Error when running two MirrorMakers, each with different sources, to one target.

[skas-zilo]

Hi,

I'm managing three EKS clusters running Kafka. I have two MirrorMakers in two different EKS clusters, and one EKS cluster as the target.

One MirrorMaker always fails while the other succeeds in syncing. When I restart MirrorMaker-1, MirrorMaker-2 will run successfully, and vice versa. It seems like it doesn't allow me to connect both MirrorMakers to one target simultaneously.

When I sync topics, I get the following error. It seems like it doesn't allow me to connect these two MirrorMakers to one target.

Topic names are quite similar in the two sources, separated by environment.

Please suggest. Thank you.

2024-08-16 13:02:46,030 ERROR IO error forwarding REST request: (org.apache.kafka.connect.runtime.rest.RestClient) [ForwardRequestExecutor-connect-1-1] java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: Connect Timeout at org.eclipse.jetty.client.util.FutureResponseListener.getResult(FutureResponseListener.java:118) at org.eclipse.jetty.client.util.FutureResponseListener.get(FutureResponseListener.java:101) at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:732) at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:148) at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:110) at org.apache.kafka.connect.runtime.distributed.DistributedHerder.lambda$reconfigureConnector$42(DistributedHerder.java:1955) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: java.net.SocketTimeoutException: Connect Timeout at org.eclipse.jetty.io.ManagedSelector$Connect.run(ManagedSelector.java:955) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ... 3 more 2024-08-16 13:02:46,034 ERROR [Worker clientId=connect-1, groupId=mirrormaker2-cluster] Request to leader to reconfigure connector tasks failed (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [ForwardRequestExecutor-connect-1-1] org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: IO Error trying to forward REST request: java.net.SocketTimeoutException: Connect Timeout at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:167) at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:110) at org.apache.kafka.connect.runtime.distributed.DistributedHerder.lambda$reconfigureConnector$42(DistributedHerder.java:1955) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: Connect Timeout at org.eclipse.jetty.client.util.FutureResponseListener.getResult(FutureResponseListener.java:118) at org.eclipse.jetty.client.util.FutureResponseListener.get(FutureResponseListener.java:101) at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:732) at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:148) ... 7 more Caused by: java.net.SocketTimeoutException: Connect Timeout at org.eclipse.jetty.io.ManagedSelector$Connect.run(ManagedSelector.java:955) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ... 3 more 2024-08-16 13:02:46,034 ERROR [Worker clientId=connect-1, groupId=mirrormaker2-cluster] Failed to reconfigure connector's tasks (source-kafka-cluster->target-kafka-cluster.MirrorSourceConnector), retrying after backoff: (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [ForwardRequestExecutor-connect-1-1] org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: IO Error trying to forward REST request: java.net.SocketTimeoutException: Connect Timeout at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:167) at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:110) at org.apache.kafka.connect.runtime.distributed.DistributedHerder.lambda$reconfigureConnector$42(DistributedHerder.java:1955) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: Connect Timeout at org.eclipse.jetty.client.util.FutureResponseListener.getResult(FutureResponseListener.java:118) at org.eclipse.jetty.client.util.FutureResponseListener.get(FutureResponseListener.java:101) at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:732) at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:148) ... 7 more Caused by: java.net.SocketTimeoutException: Connect Timeout at org.eclipse.jetty.io.ManagedSelector$Connect.run(ManagedSelector.java:955) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ... 3 more

2024-08-16 12:40:52,184 WARN [source-kafka-cluster->target-kafka-cluster.MirrorSourceConnector|task-0] [Consumer clientId=consumer-null-1, groupId=null] Error while fetching metadata with correlation id 1348 : {env.mesaage-1=UNKNOWN_TOPIC_OR_PARTITION, env.message-2=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [task-thread-source-kafka-cluster->target-kafka-cluster.MirrorSourceConnector-0]

[scholzj]

The error is more or less impossible to read because it seems to be badly formatted. But as you say that you are running multiple MM2s connected to the same target Kafka cluster, did you made sure to follow this: https://strimzi.io/docs/operators/latest/full/deploying.html#con-config-mm2-multiple-instances-str?

[skas-zilo]

After following your links, the error is gone. However, MirrorMaker replicates only the topics but not the messages, and there are no errors. I'm not sure what I did wrong.

[skas-zilo]

I managed to get it work already by remove topics in source and destination. Restart mirrormaker. Thank you for help answer al questions.