[Bug]: Can't delete connetor using connect-api cluster >2 replicas

[calebebrim]

Bug Description

Kafka-connect cluster (>2 replicas) fail to delete connectors using rest-api.

connector-logs shows this error:

2024-01-16 02:25:39,769 ERROR IO error forwarding REST request:  (org.apache.kafka.connect.runtime.rest.RestClient) [qtp1714241157-31]
java.util.concurrent.ExecutionException: java.net.UnknownHostException: kafka-connect-connect-5479d4db88-5k2v2: Name or service not known
	at org.eclipse.jetty.client.util.FutureResponseListener.getResult(FutureResponseListener.java:118)
	at org.eclipse.jetty.client.util.FutureResponseListener.get(FutureResponseListener.java:101)
	at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:732)
	at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:154)
	at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:116)
	at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:80)
	at org.apache.kafka.connect.runtime.rest.HerderRequestHandler.completeOrForwardRequest(HerderRequestHandler.java:112)
	at org.apache.kafka.connect.runtime.rest.HerderRequestHandler.completeOrForwardRequest(HerderRequestHandler.java:129)
	at org.apache.kafka.connect.runtime.rest.HerderRequestHandler.completeOrForwardRequest(HerderRequestHandler.java:134)
	at org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource.destroyConnector(ConnectorsResource.java:341)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:134)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:177)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:81)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
	at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.net.UnknownHostException: kafka-connect-connect-5479d4db88-5k2v2: Name or service not known
	at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:934)
	at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1543)
	at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852)
	at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1385)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)
	at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:170)
	... 3 more
2024-01-16 02:25:39,781 INFO 127.0.0.1 - - [16/Jan/2024:02:25:39 +0000] "DELETE /connectors/operator HTTP/1.1" 500 168 "-" "PostmanRuntime/7.36.0" 155 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp1714241157-31]

and the rest API responds this error:

`
{
"error_code": 500,
"message": "IO Error trying to forward REST request: java.net.UnknownHostException: kafka-connect-connect-5479d4db88-5k2v2: Name or service not known"
}

`

I'm using Strimzi version 0.32

Steps to reproduce

1-deploy kafka cluster operator
2-deploy kafka cluster
3-deploy kafka-connect cluster
4-create a jdbc connector
5-delete jdbc connector

This is the topic configuration:

topics: "customer"
  tasks.max: "1"
  quote.sql.identifiers: "never"
  insert.mode: "upsert"
  connector.class: "io.confluent.connect.jdbc.JdbcSinkConnector"
  connection.url: "jdbc:postgresql://postgres-postgresql:5432/data"
  connection.user: "postgres"
  connection.password: "I7VWDj4zFX"
  auto.create: true
  auto.evolve: true
  delete.enabled: false
  _transforms: "ValueToKey"
  _transforms.ValueToKey.type: "org.apache.kafka.connect.transforms.ValueToKey"
  _transforms.ValueToKey.fields: "key"
  pk.mode: "record_value"
  pk.fields: "id"

Expected behavior

the connector should be deleted when using

curl -XDELETE http://{{connector-server}}/connectors/{{connector-name}}

Strimzi version

0.32

Kubernetes version

k3s version v1.27.4-k3s1 (default)

Installation method

helm chart

Infrastructure

Bare-metal

Configuration files and logs

No response

Additional context

No response

[scholzj]

I guess you should start by sharing the full log and configurations.

[calebebrim]

Changed Base image to:

FROM quay.io/strimzi/kafka:0.32.0-kafka-3.2.1
USER root:root
COPY ./plugins/ /opt/kafka/plugins/
USER 1001

logs from both pods:

kafka-connect-connect-6d95f68667-6btm8.txt
kafka-connect-connect-6d95f68667-h4k8w.txt

[calebebrim]

The test was the same create and delete kafka connectors.

[scholzj]

And did it helped? The logs are not complete, so I do not see what the configuration is.

[calebebrim]

Hum... I think so :)

I was testing here... that helps.

One more question....
This kafka-connector operator works with other kafka cluster like msk?
or just with strimzy-operator deployed cluster?

[scholzj]

No, the Connector operator works only with Strimzi managed Kafka Connect clusters.

[calebebrim]

Ok I feel that fixed my issue.
Thanks for your attention @scholzj!