Time taking for Kafka for being non-operational when zk is down.

[m2khosravi]

Hi Team,

I conducted a test scenario by shutting down the zk node, and I observed that the following command received a response:

kafka-topics --bootstrap-server localhost:9091 --list strimzi.cruisecontrol.metrics strimzi.cruisecontrol.modeltrainingsamples strimzi.cruisecontrol.partitionmetricsamples

However, after a couple of minutes, the execution failed, and recognized the zk issue. The time it takes for this failure varies. Could you please help me understand the main reason for this variability? Additionally, can we tune or modify parameters to enable Kafka to detect zk failures more efficiently and fail such commands?

Your assistance would be greatly appreciated,
Mahdi

[scholzj]

In general, ZooKeeper failure might not cause any immediate issues to the Kafka cluster. But might break it for example when a failover happens, creation of new topics etc. I'm not sure how exactly do you run the command you shared above, but if you do it from the Kafka pod, I would never expect it to work ZooKeeper or not as it does not authenticate etc.

[m2khosravi]

the command ran from the Kafka pods, there was no problem with the command execution itself. the only question is why it is taking time to realize there is an issue with ZK and why is not immediately failed.

[scholzj]

That command would never work because it is missing authentication etc.

[m2khosravi]

here is a full command with output and as we can see it's work fine and I get the answer while zk node is down and it takes time to failed
bash-4.4$ kafka-topics --bootstrap-server strimzi-test-kafka-bootstrap:9091 --list --command-config /kafka/kafka_ssl_config.propertieserties ERROR StatusLogger Reconfiguration failed: No configuration found for '2cdf8d8a' at 'null' in 'null' topic-1 topic-2 topic-3

After a couple of mins Kafka is able to detect problem time might be difference in such as more replicas

bash-4.4$ kafka-topics --bootstrap-server strimzi-test-kafka-bootstrap:9091 --list --command-config /kafka/kafka_ssl_config.propertieses ERROR StatusLogger Reconfiguration failed: No configuration found for '2cdf8d8a' at 'null' in 'null' Error while executing topic command : Timed out waiting for a node assignment. Call: listTopics 14:24:52.931 [main] ERROR kafka.admin.TopicCommand$ - org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listTopics

[m2khosravi]

Any thoughts on that?