[Bug]: Zookeeper doesn't work with custom DNS

[mruncleleon]

Bug Description

Adding KUBERNETES_SERVICE_DNS_NAME in strimzi-cluster-operator didn't make Zookeeper work as expect. So, only after running kill command inside the Zookeeper container worked.
k exec -it monitoracao-chat-zookeeper-0 -c zookeeper -n kafka -- /bin/bash -c "kill 1"
I'm running kubernetes on GKE with Cloud DNS.

strimzi-cluster-operator.txt

zookeeper.txt

Steps to reproduce

gke-cloud-dns-test.tf.txt
After deploy GKE with Cloud DNS, run quickstart for strimzi-cluster-operator 0.36.0 with kafka-single-persistent.yaml.

Zookeeper should recognized your own fqdn after boot.

Expected behavior

No response

Strimzi version

0.36.0

Kubernetes version

1.27.5-gke.200

Installation method

YAML

Infrastructure

Google Cloud GKE + Cloud DNS

Configuration files and logs

Add useServiceDnsDomain didnt help. It was needed to run kill command.

    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
        configuration:
          useServiceDnsDomain: true

Additional context

I did find some other issues related to deploy strimzi with custom DNS name but didnt help me:

#2824
#2656
#5519
#5621

[strimzi@strimzi-cluster-operator-5fb6564998-vdchx strimzi]$ cat /etc/resolv.conf
search kafka.svc.dev svc.dev dev southamerica-east1-a.c.kafka-temp-project.internal c.kafka-temp-project.internal google.internal
nameserver 169.254.169.254
options ndots:5`
[strimzi@strimzi-cluster-operator-5fb6564998-vdchx strimzi]$ curl -Iv aviso-zookeeper-0.aviso-zookeeper-nodes.kafka.svc:2181
* Rebuilt URL to: aviso-zookeeper-0.aviso-zookeeper-nodes.kafka.svc:2181/
*   Trying 10.188.6.64...
* TCP_NODELAY set
* Connected to aviso-zookeeper-0.aviso-zookeeper-nodes.kafka.svc (10.188.6.64) port 2181 (#0)
> HEAD / HTTP/1.1
> Host: aviso-zookeeper-0.aviso-zookeeper-nodes.kafka.svc:2181
> User-Agent: curl/7.61.1
> Accept: */*
> 
* Empty reply from server
* Connection #0 to host aviso-zookeeper-0.aviso-zookeeper-nodes.kafka.svc left intact
curl: (52) Empty reply from server

root@4c38b7fe8a91:/strimzi# k exec -it  pod/aviso-zookeeper-0 -n kafka -- bash -c "getent hosts"
127.0.0.1       localhost
127.0.0.1       localhost ip6-localhost ip6-loopback
10.188.6.67     aviso-zookeeper-0.aviso-zookeeper-nodes.kafka.svc.dev aviso-zookeeper-0```

[scholzj]

I had a look at the logs:

• It does not seem you configured KUBERNETES_SERVICE_DNS_NAME => the log suggests it was autodetected as dev. Not sure if that is correct or not, you would need to confirm it based on your cluster setup. The DNs name new-cluster-zookeeper-0.new-cluster-zookeeper-nodes.kafka.svc should anyway resolve regardless of your DNS suffix as that does not use it.
• The error in the ZooKeeper cluster refers to <unresolved> DNS address. So it seems to be related to ZooKeeper DNS issues (it seems to happen more often when the DNS in the Kube cluster is slow which might be why are you seeing it, but overall it seems more a ZooKeeper issue rather than a Kubernetes issue). The related discussions are for example here: ZooKeeper 3.6.4 quorum failing with Kafka 3.4.1 #8675
• The useServiceDnsDomain option has no relation to ZooKeeper. It only controls if the advertised names in the KAfka listeners will use the full DNS name or not. That could impact some Kafka clients, but not ZooKeeper nodes.

[mruncleleon]

It looks like you didn't configure KUBERNETES_SERVICE_DNS_NAME => the log suggests it was automatically detected as dev. I'm not sure if this is correct or not, you'll need to confirm based on your cluster configuration. The DNs name new-cluster-zookeeper-0.new-cluster-zookeeper-nodes.kafka.svc should resolve anyway, regardless of its DNS suffix, as it does not use it.

Yes, new-cluster-zookeeper-0.new-cluster-zookeeper-nodes.kafka.svc can be resolved within the Zookeeper container while generating logs with the DNS address <unresolved>.
But after running the kill command inside the Zookeeper container, it made Zookeeper restart inside the pod without recreating one and keeping the same hostname and IP address which made it work correctly.

[scholzj]

Well, it might resolve when you try it, but it did not resolve when ZooKeeper started as you see from the log. My understanding of the ZooKeeper bug is that it should try to re-resolve, but it does not and basically remains in some weird state. That is basically what the issue I linked discusses.