[Bug]: ERROR IO error forwarding REST request: (org.apache.kafka.connect.runtime.rest.RestClient) [qtp2025444884-163671]

[areebemumba]

Bug Description

I deployed Kafka MirrorMaker2 on Kubernetes. It's not able to mirror messages from the src Kafka to the target Kafka. These are the error logs from the pod: my-mirror-maker-2-mirrormaker2-0:

024-04-01 07:55:15,934 ERROR IO error forwarding REST request: (org.apache.kafka.connect.runtime.rest.RestClient) [qtp2025444884-163671]
java.util.concurrent.ExecutionException: java.net.UnknownHostException: my-mirror-maker-2-mirrormaker2-0.my-mirror-maker-2-mirrormaker2.strimzi-kafka.svc
at org.eclipse.jetty.client.util.FutureResponseListener.getResult(FutureResponseListener.java:118)
at org.eclipse.jetty.client.util.FutureResponseListener.get(FutureResponseListener.java:101)
at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:732)
at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:154)
at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:116)
at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:80)
at org.apache.kafka.connect.runtime.rest.HerderRequestHandler.completeOrForwardRequest(HerderRequestHandler.java:131)
at org.apache.kafka.connect.runtime.rest.HerderRequestHandler.completeOrForwardRequest(HerderRequestHandler.java:148)
at org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource.putConnectorConfig(ConnectorsResource.java:232)
at jdk.internal.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:134)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:177)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:81)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.net.UnknownHostException: my-mirror-maker-2-mirrormaker2-0.my-mirror-maker-2-mirrormaker2.strimzi-kafka.svc
at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:801)
at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1385)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)
at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:170)
... 3 more

k get svc -n kafka

my-mirror-maker-2-mirrormaker2 ClusterIP None 8083/TCP 58m
my-mirror-maker-2-mirrormaker2-api ClusterIP xxx.xx.xxx.xxx 8083/TCP 58m

k get pods -n kafka

my-mirror-maker-2-mirrormaker2-0 1/1 Running 0 50m
strimzi-cluster-operator-79fb75cb8d-zglkp 1/1 Running 0 56m

Steps to reproduce

1. Deploy Strimzi operator
   kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka
2. Deploy KafkaMirror Maker 2 with the below mentioned manifest.

Expected behavior

Messages should be replicated from src to target kafka topic.

Strimzi version

quay.io/strimzi/operator:0.40.0

Kubernetes version

1.27

Installation method

Helm Chart

Infrastructure

Amazon EKS

Configuration files and logs

Deploy Kafka MirrorMaker2:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: my-mirror-maker-2
  namespace: kafka
spec:
  version: 3.7.0
  replicas: 1
  connectCluster: "xiq-kafka"
  clusters:
  - alias: "msk-kafka" # Source cluster
    bootstrapServers: "b-1.abcmskcluster.1uc516.k6.kafka.eu-central-1.amazonaws.com:9092,b-1.abcmskcluster.1uc516.k6.kafka.eu-central-1.amazonaws.com:9092"
  - alias: "xiq-kafka" # Target cluster
    bootstrapServers: "mtxb9g8-okl-kafka.xy.abc.com:30092,mtxb9g8-okl-kafka.xy.abc.com:30094:30093,mtxb9g8-okl-kafka.xy.abc.com:30094"
    config:
      # -1 means it will use the default replication factor configured in the broker
      config.storage.replication.factor: -1
      offset.storage.replication.factor: -1
      status.storage.replication.factor: -1
  mirrors:
  - sourceCluster: "msk-kafka"
    targetCluster: "xiq-kafka"
    sourceConnector:
      tasksMax: 1
      config:
        # -1 means it will use the default replication factor configured in the broker
        replication.factor: -1
        offset-syncs.topic.replication.factor: -1
        sync.topic.acls.enabled: "false"
        replication.policy.class: "org.apache.kafka.connect.mirror.IdentityReplicationPolicy"
        refresh.topics.interval.seconds: 600
    checkpointConnector:
      tasksMax: 1
      config:
        # -1 means it will use the default replication factor configured in the broker
        checkpoints.topic.replication.factor: -1
        sync.group.offsets.enabled: "false"
        replication.policy.class: "org.apache.kafka.connect.mirror.IdentityReplicationPolicy"
        refresh.groups.interval.seconds: 600
    topicsPattern: "events-topic"
    groupsPattern: "dev-replicator"

Additional context

No response

[scholzj]

Please share the full logs. You should probably also share what you have in the strimzi-kafka namespase.

[areebemumba]

Sorry @scholzj, it was a typo I have made the configuration valid.

Full Error log: https://gist.github.com/areebemumba/d71d6cc345ea8185ff48e07b0bcbe239

Also I don't have a strimzi-kafka namespace.
k get all -n strimzi-kafka

No resources found in strimzi-kafka namespace.

[scholzj]

That is not a full log as it is not from the start. Also, the log suggests there is such namespace somewhere:

java.util.concurrent.ExecutionException: java.net.UnknownHostException: my-mirror-maker-2-mirrormaker2-0.my-mirror-maker-2-mirrormaker2.strimzi-kafka.svc

Maybe it is in a different cluster? You should make sure you follow this: https://strimzi.io/docs/operators/latest/full/deploying.html#con-config-mm2-multiple-instances-str

[areebemumba]

Full Logs: https://gist.github.com/areebemumba/136588020452f5993b07abb90938c1b6

Yes it's weird, the namespace doesn't exist also I am using the same cluster:

I have followed the example form official strimzi repo for deployment: https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/mirror-maker/kafka-mirror-maker-2.yaml

[scholzj]

Well, as I said, the namespace might be in a different Kubernetes cluster as well.

[scholzj]

The log seems to match that assumption -> you have somewhere else running another MM2 with the same config. You have to follow the docs I linked to make them independent.