"ERROR Failed to verify hostname" - Is there a resolution for this?

[kdavid76]

I have a very similar problem described here.

I used the latest version of pstrimzi 0.33.2 as I remember. I tried to deploy a simple cluster with 2 Kafka nodes and 2 Zookeeper nodes. But the Zookeeper instances don't want to come up and they are constantly restarting. They try to match certificates, for the host machine, and obviously, that is failing.

Here is the error message, but the full log is also attached:
2023-03-06 21:13:01,289 ERROR Failed to verify hostname: apps.besztercekk.hu (org.apache.zookeeper.common.ZKTrustManager) [ListenerHandler-tao-zookeeper-0.tao-zookeeper-nodes.kafka.svc/192.168.29.161:3888]

report-06-03-2023_22-28-20.zip

Any ideas are appreciated. How can this be fixed? Probably something is wrong in my configuration, but what? I followed the step by step guides from the web page, I just changed the number of replicas and persistent volume sizes. Those, I believe, can't cause this issue.

Using the fix suggested in the similar issue kind of worked, but that's just a workaround. I don't really want to do similar things in production.

Best regards,
Krisztian

[scholzj]

This is essentially an issue with how your DNS is configured. ZooKeeper does TLS hostname verification through a reverse DNS lookup. So essentially:

1. It is told to connect to something like tao-zookeeper-0.tao-zookeeper-nodes.kafka.svc
2. It resolves it to the IP address 192.168.29.161
3. It connects to this address and gets the certificate
4. Then it does a reverse lookup of the IP address 192.168.29.161 and checks if the DNS name it gets back is in the TLS certificate subject alternative names. And in your case, the DNS tells it that this IP belongs to apps.besztercekk.hu => and that is obviously in the certificate as it has nothing to do with the actual address of the ZooKeeper nodes. So it fails.

You have two options how to fix it:

• Fix your DNS to return something more reasonable that would match the certificate. (I'm afraid I don't have idea how - some of the old discussions might give you some idea)
• Disable the TLS hostname verification in ZooKeeper in the Kafka custom resource:

  apiVersion: kafka.strimzi.io/v1beta1
  kind: Kafka
  metadata:
    name: test-kafka
  spec:
    ...
    zookeeper:
      replicas: 3
      jvmOptions:
        javaSystemProperties:
          - name: zookeeper.ssl.hostnameVerification
            value: "false"
          - name: zookeeper.ssl.quorum.hostnameVerification
            value: "false"
    ...

[kdavid76]

Hi @scholzj !

Thanks for the swift and detailed answer around what could have gone wrong with DNS. The workaround is fine, both Zookeeper and Kafka nodes are up and running, they are pretty "green" on the dashboard without restarts. :-)

But as you suggested, the main issue here is how to fix the DNS...as per I haven't really touched that, since the K8S cluster was deployed. It's in the state, as it was. It's just a dev environment, where I can live with the workaround, but when the software evolves to prod level, workaround won't be so fine.

Thanks for the help again!

[LawrenceB5477]

I was literally just learning about Strimzi today and this post saved me. Thanks so much! In my free time I'll dig deeper to see what's going on with the TLS validation for my personal learning.

[ravihiremani]

This configuration works. Thanks @kdavid76