Connection timeout to schema registry

[mahauke]

Hi,

we are using AKHQ 0.20.0 as developer tool for Strimzi Kafka in Kubernetes with a Confluent Schema Registry 6.2.2 also hosted in Kubernetes.

We currently encounter issues using the "Schema Registry" part of AKHQ, since we get an empty error message in the UI after a long time of loading. I digged into the logs and found the following error is related to this behavior:

ERROR thread-243 o.a.c.ErrorController connect timed out
java.net.SocketTimeoutException: connect timed out
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.base/java.net.SocksSocketImpl.connect(Unknown Source)
at java.base/java.net.Socket.connect(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.connect(Unknown Source)
at java.base/sun.net.NetworkClient.doConnect(Unknown Source)
at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)
at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)
at java.base/sun.net.www.protocol.https.HttpsClient.<init>(Unknown Source)
at java.base/sun.net.www.protocol.https.HttpsClient.New(Unknown Source)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.base/java.net.HttpURLConnection.getResponseCode(Unknown Source)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown Source)
at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:277)
at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:367)
at io.confluent.kafka.schemaregistry.client.rest.RestService.getAllSubjects(RestService.java:994)
at io.confluent.kafka.schemaregistry.client.rest.RestService.getAllSubjects(RestService.java:979)
at org.akhq.repositories.SchemaRegistryRepository.all(SchemaRegistryRepository.java:90)
at org.akhq.repositories.SchemaRegistryRepository.list(SchemaRegistryRepository.java:42)
at org.akhq.controllers.SchemaController.list(SchemaController.java:51)
at org.akhq.controllers.$SchemaController$Definition$Exec.dispatch(Unknown Source)
at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:351)
at io.micronaut.context.DefaultBeanContext$4.invoke(DefaultBeanContext.java:583)
at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:303)
at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:111)
at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:103)
at io.micronaut.http.server.RouteExecutor.lambda$executeRoute$14(RouteExecutor.java:656)
at reactor.core.publisher.FluxDeferContextual.subscribe(FluxDeferContextual.java:49)
at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62)
at reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194)
at io.micronaut.reactive.reactor.instrument.ReactorInstrumentation.lambda$null$0(ReactorInstrumentation.java:62)
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)
at io.micronaut.scheduling.instrument.InvocationInstrumenterWrappedCallable.call(InvocationInstrumenterWrappedCallable.java:53)
at io.micrometer.core.instrument.composite.CompositeTimer.recordCallable(CompositeTimer.java:68)
at io.micrometer.core.instrument.Timer.lambda$wrap$1(Timer.java:171)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

The schemaregistry seems to run fine, manual API calls and also consumers and producers of our applications are running fine. Thus, it seems to be either a network issue, a certificate issue (we are using an internal CA) or an AKHQ issue.

I already tested connecting to the registry using the specified connection URL of the schema registry from the config file, by using curl from the AKHQ container. I had to apply --insecure, since I cannot use the truststore of AKHQ with curl and the certificate is signed by an internal CA. Still, the correct Root CA is in the Truststore used by AKHQ. And connecting with --insecure worked.

Since I have no idea where to investigate further into this issue, my question is: How can I get more information on the error logged by AKHQ? Is there any way to find out whether this is a network issue, a certificate issue or an AKHQ issue?

Just as a note: It seemed to work fine at the beginning when no schemas had been stored in the Registry. At least, no error came up - as far as I remember, but I cannot reproduce now...

Any ideas or suggestions?

Thank you in advance
Mario Hauke