The Kafka pods are not came to running mode which I used with strimzi operator Kafka in Azure Kubernetes

[damodarreddy08]

Hi..,

As part of POC preparation I trying to install 3 node Kafka using the Strimzi operators after enabled with external ingress concept for calling to cluster from external client, but here zookeeper and services are came to running mode with following details in zookeeper pod logs along with broker also not came to running mode.

cluster yaml file:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: sims-kafka-cluster
spec:
kafka:
version: 3.5.0
replicas: 3
listeners:
- name: plain
port: 9092
type: internal
tls: false
- name: tls
port: 9093
type: internal
tls: true
- name: external
port: 9094
type: ingress
tls: true
configuration:
bootstrap:
annotations:
external-dns.alpha.kubernetes.io/hostname: kafkabootstrap-pilot.xxx.xxx.net
external-dns.alpha.kubernetes.io/ttl: "60"
host: kafkabootstrap-pilot.xxx.xxx.net
brokers:
- broker: 0
annotations:
external-dns.alpha.kubernetes.io/hostname: kafkabroker-pilot-0.xxxx.xxx.net
external-dns.alpha.kubernetes.io/ttl: "60"
host: kafkabroker-pilot-0.xxxx.xxx.net
- broker: 1
annotations:
external-dns.alpha.kubernetes.io/hostname: kafkabroker-pilot-1.xxxx.xxxx.net
external-dns.alpha.kubernetes.io/ttl: "60"
host: kafkabroker-pilot-1.xxxx.xxxx.net
- broker: 2
annotations:
external-dns.alpha.kubernetes.io/hostname: kafkabroker-pilot-2.xxxx.xxxx.net
external-dns.alpha.kubernetes.io/ttl: "60"
host: kafkabroker-pilot-2.xxxx.xxx.net
class: nginx
brokerCertChainAndKey:
secretName: kafka-ingress-tls
certificate: tls.crt
key: tls.key
config:
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
default.replication.factor: 3
min.insync.replicas: 2
inter.broker.protocol.version: "3.5"
storage:
type: jbod
volumes:
- id: 0
type: persistent-claim
size: 32Gi
deleteClaim: false
class: managed-csi-premium
zookeeper:
replicas: 3
storage:
type: persistent-claim
size: 32Gi
deleteClaim: false
class: managed-csi-premium
entityOperator:
topicOperator: {}
userOperator: {}

zookeeper pod logs:
[acusdre@apprdaz0031 scripts]$ kubectl logs pod/sims-kafka-cluster-zookeeper-0 -n kafka
Detected Zookeeper ID 1
Preparing truststore
Adding /opt/kafka/cluster-ca-certs/ca.crt to truststore /tmp/zookeeper/cluster.truststore.p12 with alias ca
Certificate was added to keystore
Preparing truststore is complete
Looking for the right CA
Found the right CA: /opt/kafka/cluster-ca-certs/ca.crt
Preparing keystore for client and quorum listeners
Preparing keystore for client and quorum listeners is complete
Starting Zookeeper with configuration:

The directory where the snapshot is stored.

dataDir=/var/lib/zookeeper/data

Other options

4lw.commands.whitelist=*
standaloneEnabled=false
reconfigEnabled=true

TLS options

serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
ssl.clientAuth=need
ssl.quorum.clientAuth=need
secureClientPort=2181
sslQuorum=true

ssl.trustStore.location=/tmp/zookeeper/cluster.truststore.p12
ssl.trustStore.password=[hidden]
ssl.trustStore.type=PKCS12
ssl.quorum.trustStore.location=/tmp/zookeeper/cluster.truststore.p12
ssl.quorum.trustStore.password=[hidden]
ssl.quorum.trustStore.type=PKCS12

ssl.keyStore.location=/tmp/zookeeper/cluster.keystore.p12
ssl.keyStore.password=[hidden]
ssl.keyStore.type=PKCS12
ssl.quorum.keyStore.location=/tmp/zookeeper/cluster.keystore.p12
ssl.quorum.keyStore.password=[hidden]
ssl.quorum.keyStore.type=PKCS12

Provided configuration

tickTime=2000
initLimit=5
syncLimit=2
autopurge.purgeInterval=1
admin.enableServer=false

Zookeeper nodes configuration

server.1=0.0.0.0:2888:3888:participant;127.0.0.1:12181
server.2=sims-kafka-cluster-zookeeper-1.sims-kafka-cluster-zookeeper-nodes.kafka.svc:2888:3888:participant;127.0.0.1:12181
server.3=sims-kafka-cluster-zookeeper-2.sims-kafka-cluster-zookeeper-nodes.kafka.svc:2888:3888:participant;127.0.0.1:12181

• exec /usr/bin/tini -w -e 143 -- /opt/kafka/bin/zookeeper-server-start.sh /tmp/zookeeper.properties
  2024-03-01 04:21:02,880 INFO Reading configuration from: /tmp/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [main]
  2024-03-01 04:21:02,886 INFO clientPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [main]
  2024-03-01 04:21:02,890 INFO secureClientPortAddress is 0.0.0.0:2181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [main]
  2024-03-01 04:21:02,905 INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util) [main]
  2024-03-01 04:21:02,912 INFO observerMasterPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [main]
  2024-03-01 04:21:02,912 INFO metricsProvider.className is org.apache.zookeeper.metrics.impl.DefaultMetricsProvider (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [main]
  2024-03-01 04:21:03,042 INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager) [main]
  2024-03-01 04:21:03,043 INFO autopurge.purgeInterval set to 1 (org.apache.zookeeper.server.DatadirCleanupManager) [main]
  2024-03-01 04:21:03,046 INFO Log4j 1.2 jmx support not found; jmx disabled. (org.apache.zookeeper.jmx.ManagedUtil) [main]
  2024-03-01 04:21:03,056 INFO Starting quorum peer, myid=1 (org.apache.zookeeper.server.quorum.QuorumPeerMain) [main]
  2024-03-01 04:21:03,068 INFO Purge task started. (org.apache.zookeeper.server.DatadirCleanupManager) [PurgeTask]
  2024-03-01 04:21:03,101 INFO zookeeper.snapshot.trust.empty : false (org.apache.zookeeper.server.persistence.FileTxnSnapLog) [PurgeTask]
  2024-03-01 04:21:03,137 INFO Purge task completed. (org.apache.zookeeper.server.DatadirCleanupManager) [PurgeTask]
  2024-03-01 04:21:03,167 INFO ServerMetrics initialized with provider org.apache.zookeeper.metrics.impl.DefaultMetricsProvider@37918c79 (org.apache.zookeeper.server.ServerMetrics) [main]
  2024-03-01 04:21:03,336 INFO zookeeper.client.portUnification=false (org.apache.zookeeper.server.NettyServerCnxnFactory) [main]
  2024-03-01 04:21:03,336 INFO zookeeper.netty.advancedFlowControl.enabled = false (org.apache.zookeeper.server.NettyServerCnxnFactory) [main]
  2024-03-01 04:21:03,336 INFO handshakeThrottlingEnabled = false, zookeeper.netty.server.outstandingHandshake.limit = -1 (org.apache.zookeeper.server.NettyServerCnxnFactory) [main]
  2024-03-01 04:21:03,402 INFO Using org.apache.zookeeper.server.NettyServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory) [main]
  2024-03-01 04:21:03,403 WARN maxCnxns is not configured, using default value 0. (org.apache.zookeeper.server.ServerCnxnFactory) [main]
  2024-03-01 04:21:03,404 INFO zookeeper.client.portUnification=false (org.apache.zookeeper.server.NettyServerCnxnFactory) [main]
  2024-03-01 04:21:03,404 INFO zookeeper.netty.advancedFlowControl.enabled = false (org.apache.zookeeper.server.NettyServerCnxnFactory) [main]
  2024-03-01 04:21:03,404 INFO handshakeThrottlingEnabled = false, zookeeper.netty.server.outstandingHandshake.limit = -1 (org.apache.zookeeper.server.NettyServerCnxnFactory) [main]
  2024-03-01 04:21:03,408 INFO Using org.apache.zookeeper.server.NettyServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory) [main]
  2024-03-01 04:21:03,408 WARN maxCnxns is not configured, using default value 0. (org.apache.zookeeper.server.ServerCnxnFactory) [main]
  2024-03-01 04:21:03,414 INFO zookeeper.quorumCnxnTimeoutMs=-1 (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
  2024-03-01 04:21:03,418 INFO zookeeper.snapshot.trust.empty : false (org.apache.zookeeper.server.persistence.FileTxnSnapLog) [main]
  2024-03-01 04:21:03,418 INFO Local sessions disabled (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
  2024-03-01 04:21:03,418 INFO Local session upgrading disabled (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
  2024-03-01 04:21:03,418 INFO tickTime set to 2000 (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
  2024-03-01 04:21:03,418 INFO minSessionTimeout set to 4000 (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
  2024-03-01 04:21:03,418 INFO maxSessionTimeout set to 40000 (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
  2024-03-01 04:21:03,419 INFO initLimit set to 5 (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
  2024-03-01 04:21:03,419 INFO syncLimit set to 2 (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
  2024-03-01 04:21:03,419 INFO connectToLearnerMasterLimit set to 0 (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
  2024-03-01 04:21:03,430 INFO (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,430 INFO ______ _ (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,431 INFO |___ / | | (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,431 INFO / / ___ ___ | | __ ___ ___ _ __ ___ _ __ (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,431 INFO / / / _ \ / _ \ | |/ / / _ \ / _ \ | '_ \ / _ \ | '| (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,431 INFO / / | () | | () | | < | / | / | |_) | | / | | (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,431 INFO /| _/ _/ |_|_\ _| _| | ./ __| || (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,431 INFO | | (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,431 INFO || (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,431 INFO (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,432 INFO Server environment:zookeeper.version=3.6.4--d65253dcf68e9097c6e95a126463fd5fdeb4521c, built on 12/18/2022 18:10 GMT (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,432 INFO Server environment:host.name=sims-kafka-cluster-zookeeper-0.sims-kafka-cluster-zookeeper-nodes.kafka.svc.cluster.local (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,433 INFO Server environment:java.version=17.0.9 (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,433 INFO Server environment:java.vendor=Red Hat, Inc. (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,433 INFO Server environment:java.home=/usr/lib/jvm/java-17-openjdk-17.0.9.0.9-2.el8.x86_64 (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,433 INFO Server environment:java.class.path=/opt/kafka/bin/../libs/accessors-smart-2.4.9.jar:/opt/kafka/bin/../libs/activation-1.1.1.jar:/opt/kafka/bin/../libs/annotations-13.0.jar:/opt/kafka/bin/../libs/aopalliance-repackaged-2.6.1.jar:/opt/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/bin/../libs/audience-annotations-0.13.0.jar:/opt/kafka/bin/../libs/checker-qual-3.5.0.jar:/opt/kafka/bin/../libs/commons-cli-1.4.jar:/opt/kafka/bin/../libs/commons-lang-2.6.jar:/opt/kafka/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka/bin/../libs/connect-api-3.5.0.jar:/opt/kafka/bin/../libs/connect-basic-auth-extension-3.5.0.jar:/opt/kafka/bin/../libs/connect-json-3.5.0.jar:/opt/kafka/bin/../libs/connect-mirror-3.5.0.jar:/opt/kafka/bin/../libs/connect-mirror-client-3.5.0.jar:/opt/kafka/bin/../libs/connect-runtime-3.5.0.jar:/opt/kafka/bin/../libs/connect-transforms-3.5.0.jar:/opt/kafka/bin/../libs/cruise-control-metrics-reporter-2.5.128.jar:/opt/kafka/bin/../libs/error_prone_annotations-2.3.4.jar:/opt/kafka/bin/../libs/failureaccess-1.0.1.jar:/opt/kafka/bin/../libs/guava-30.1-jre.jar:/opt/kafka/bin/../libs/hk2-api-2.6.1.jar:/opt/kafka/bin/../libs/hk2-locator-2.6.1.jar:/opt/kafka/bin/../libs/hk2-utils-2.6.1.jar:/opt/kafka/bin/../libs/j2objc-annotations-1.3.jar:/opt/kafka/bin/../libs/jackson-annotations-2.13.5.jar:/opt/kafka/bin/../libs/jackson-core-2.13.5.jar:/opt/kafka/bin/../libs/jackson-databind-2.13.5.jar:/opt/kafka/bin/../libs/jackson-dataformat-csv-2.13.5.jar:/opt/kafka/bin/../libs/jackson-dataformat-yaml-2.15.2.jar:/opt/kafka/bin/../libs/jackson-datatype-jdk8-2.13.5.jar:/opt/kafka/bin/../libs/jackson-datatype-jsr310-2.15.2.jar:/opt/kafka/bin/../libs/jackson-jaxrs-base-2.13.5.jar:/opt/kafka/bin/../libs/jackson-jaxrs-json-provider-2.13.5.jar:/opt/kafka/bin/../libs/jackson-module-jaxb-annotations-2.13.5.jar:/opt/kafka/bin/../libs/jackson-module-scala_2.13-2.13.5.jar:/opt/kafka/bin/../libs/jakarta.activation-api-1.2.2.jar:/opt/kafka/bin/../libs/jakarta.annotation-api-1.3.5.jar:/opt/kafka/bin/../libs/jakarta.inject-2.6.1.jar:/opt/kafka/bin/../libs/jakarta.validation-api-2.0.2.jar:/opt/kafka/bin/../libs/jakarta.ws.rs-api-2.1.6.jar:/opt/kafka/bin/../libs/jakarta.xml.bind-api-2.3.3.jar:/opt/kafka/bin/../libs/javassist-3.29.2-GA.jar:/opt/kafka/bin/../libs/javax.activation-api-1.2.0.jar:/opt/kafka/bin/../libs/javax.annotation-api-1.3.2.jar:/opt/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/bin/../libs/jaxb-api-2.3.1.jar:/opt/kafka/bin/../libs/jersey-client-2.39.1.jar:/opt/kafka/bin/../libs/jersey-common-2.39.1.jar:/opt/kafka/bin/../libs/jersey-container-servlet-2.39.1.jar:/opt/kafka/bin/../libs/jersey-container-servlet-core-2.39.1.jar:/opt/kafka/bin/../libs/jersey-hk2-2.39.1.jar:/opt/kafka/bin/../libs/jersey-server-2.39.1.jar:/opt/kafka/bin/../libs/jetty-client-9.4.51.v20230217.jar:/opt/kafka/bin/../libs/jetty-continuation-9.4.51.v20230217.jar:/opt/kafka/bin/../libs/jetty-http-9.4.51.v20230217.jar:/opt/kafka/bin/../libs/jetty-io-9.4.51.v20230217.jar:/opt/kafka/bin/../libs/jetty-security-9.4.51.v20230217.jar:/opt/kafka/bin/../libs/jetty-server-9.4.51.v20230217.jar:/opt/kafka/bin/../libs/jetty-servlet-9.4.51.v20230217.jar:/opt/kafka/bin/../libs/jetty-servlets-9.4.51.v20230217.jar:/opt/kafka/bin/../libs/jetty-util-9.4.51.v20230217.jar:/opt/kafka/bin/../libs/jetty-util-ajax-9.4.51.v20230217.jar:/opt/kafka/bin/../libs/jline-3.22.0.jar:/opt/kafka/bin/../libs/jmx_prometheus_javaagent-0.20.0.jar:/opt/kafka/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/bin/../libs/jose4j-0.9.3.jar:/opt/kafka/bin/../libs/json-path-2.8.0.jar:/opt/kafka/bin/../libs/json-smart-2.4.9.jar:/opt/kafka/bin/../libs/jsonevent-layout-1.7.jar:/opt/kafka/bin/../libs/jsr305-3.0.2.jar:/opt/kafka/bin/../libs/kafka-agent-0.38.0.jar:/opt/kafka/bin/../libs/kafka-clients-3.5.0.jar:/opt/kafka/bin/../libs/kafka-env-var-config-provider-1.1.0.jar:/opt/kafka/bin/../libs/kafka-group-coordinator-3.5.0.jar:/opt/kafka/bin/../libs/kafka-kubernetes-config-provider-1.1.2.jar:/opt/kafka/bin/../libs/kafka-log4j-appender-3.5.0.jar:/opt/kafka/bin/../libs/kafka-metadata-3.5.0.jar:/opt/kafka/bin/../libs/kafka-oauth-client-0.14.0.jar:/opt/kafka/bin/../libs/kafka-oauth-common-0.14.0.jar:/opt/kafka/bin/../libs/kafka-oauth-keycloak-authorizer-0.14.0.jar:/opt/kafka/bin/../libs/kafka-oauth-server-0.14.0.jar:/opt/kafka/bin/../libs/kafka-oauth-server-plain-0.14.0.jar:/opt/kafka/bin/../libs/kafka-quotas-plugin-0.2.0.jar:/opt/kafka/bin/../libs/kafka-raft-3.5.0.jar:/opt/kafka/bin/../libs/kafka-server-common-3.5.0.jar:/opt/kafka/bin/../libs/kafka-shell-3.5.0.jar:/opt/kafka/bin/../libs/kafka-storage-3.5.0.jar:/opt/kafka/bin/../libs/kafka-storage-api-3.5.0.jar:/opt/kafka/bin/../libs/kafka-streams-3.5.0.jar:/opt/kafka/bin/../libs/kafka-streams-examples-3.5.0.jar:/opt/kafka/bin/../libs/kafka-streams-scala_2.13-3.5.0.jar:/opt/kafka/bin/../libs/kafka-streams-test-utils-3.5.0.jar:/opt/kafka/bin/../libs/kafka-tools-3.5.0.jar:/opt/kafka/bin/../libs/kafka-tools-api-3.5.0.jar:/opt/kafka/bin/../libs/kafka_2.13-3.5.0.jar:/opt/kafka/bin/../libs/kotlin-stdlib-1.6.20.jar:/opt/kafka/bin/../libs/kotlin-stdlib-common-1.5.31.jar:/opt/kafka/bin/../libs/kotlin-stdlib-jdk7-1.5.31.jar:/opt/kafka/bin/../libs/kotlin-stdlib-jdk8-1.5.31.jar:/opt/kafka/bin/../libs/kubernetes-client-6.8.1.jar:/opt/kafka/bin/../libs/kubernetes-client-api-6.8.1.jar:/opt/kafka/bin/../libs/kubernetes-httpclient-jdk-6.8.1.jar:/opt/kafka/bin/../libs/kubernetes-model-apps-6.8.1.jar:/opt/kafka/bin/../libs/kubernetes-model-batch-6.8.1.jar:/opt/kafka/bin/../libs/kubernetes-model-certificates-6.8.1.jar:/opt/kafka/bin/../libs/kubernetes-model-common-6.8.1.jar:/opt/kafka/bin/../libs/kubernetes-model-core-6.8.1.jar:/opt/kafka/bin/../libs/kubernetes-model-extensions-6.8.1.jar:/opt/kafka/bin/../libs/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/opt/kafka/bin/../libs/lz4-java-1.8.0.jar:/opt/kafka/bin/../libs/maven-artifact-3.8.4.jar:/opt/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/bin/../libs/metrics-core-4.1.12.1.jar:/opt/kafka/bin/../libs/mirror-maker-2-extensions-1.2.0.jar:/opt/kafka/bin/../libs/mirror-maker-agent-0.38.0.jar:/opt/kafka/bin/../libs/netty-buffer-4.1.92.Final.jar:/opt/kafka/bin/../libs/netty-codec-4.1.92.Final.jar:/opt/kafka/bin/../libs/netty-common-4.1.92.Final.jar:/opt/kafka/bin/../libs/netty-handler-4.1.92.Final.jar:/opt/kafka/bin/../libs/netty-resolver-4.1.92.Final.jar:/opt/kafka/bin/../libs/netty-transport-4.1.92.Final.jar:/opt/kafka/bin/../libs/netty-transport-classes-epoll-4.1.92.Final.jar:/opt/kafka/bin/../libs/netty-transport-native-epoll-4.1.92.Final.jar:/opt/kafka/bin/../libs/netty-transport-native-unix-common-4.1.92.Final.jar:/opt/kafka/bin/../libs/nimbus-jose-jwt-9.31.jar:/opt/kafka/bin/../libs/okhttp-4.10.0.jar:/opt/kafka/bin/../libs/okio-jvm-3.0.0.jar:/opt/kafka/bin/../libs/opa-authorizer-1.5.1.jar:/opt/kafka/bin/../libs/opentelemetry-api-1.19.0.jar:/opt/kafka/bin/../libs/opentelemetry-api-logs-1.19.0-alpha.jar:/opt/kafka/bin/../libs/opentelemetry-context-1.19.0.jar:/opt/kafka/bin/../libs/opentelemetry-exporter-common-1.19.0.jar:/opt/kafka/bin/../libs/opentelemetry-exporter-otlp-1.19.0.jar:/opt/kafka/bin/../libs/opentelemetry-exporter-otlp-common-1.19.0.jar:/opt/kafka/bin/../libs/opentelemetry-instrumentation-api-1.19.0.jar:/opt/kafka/bin/../libs/opentelemetry-instrumentation-api-semconv-1.19.0-alpha.jar:/opt/kafka/bin/../libs/opentelemetry-kafka-clients-2.6-1.19.0-alpha.jar:/opt/kafka/bin/../libs/opentelemetry-kafka-clients-common-1.19.0-alpha.jar:/opt/kafka/bin/../libs/opentelemetry-sdk-1.19.0.jar:/opt/kafka/bin/../libs/opentelemetry-sdk-common-1.19.0.jar:/opt/kafka/bin/../libs/opentelemetry-sdk-extension-autoconfigure-1.19.0-alpha.jar:/opt/kafka/bin/../libs/opentelemetry-sdk-extension-autoconfigure-spi-1.19.0.jar:/opt/kafka/bin/../libs/opentelemetry-sdk-logs-1.19.0-alpha.jar:/opt/kafka/bin/../libs/opentelemetry-sdk-metrics-1.19.0.jar:/opt/kafka/bin/../libs/opentelemetry-sdk-trace-1.19.0.jar:/opt/kafka/bin/../libs/opentelemetry-semconv-1.19.0-alpha.jar:/opt/kafka/bin/../libs/osgi-resource-locator-1.0.3.jar:/opt/kafka/bin/../libs/paranamer-2.8.jar:/opt/kafka/bin/../libs/plexus-utils-3.3.0.jar:/opt/kafka/bin/../libs/reflections-0.9.12.jar:/opt/kafka/bin/../libs/reload4j-1.2.25.jar:/opt/kafka/bin/../libs/rocksdbjni-7.1.2.jar:/opt/kafka/bin/../libs/scala-collection-compat_2.13-2.10.0.jar:/opt/kafka/bin/../libs/scala-java8-compat_2.13-1.0.2.jar:/opt/kafka/bin/../libs/scala-library-2.13.10.jar:/opt/kafka/bin/../libs/scala-logging_2.13-3.9.4.jar:/opt/kafka/bin/../libs/scala-reflect-2.13.10.jar:/opt/kafka/bin/../libs/slf4j-api-1.7.36.jar:/opt/kafka/bin/../libs/slf4j-reload4j-1.7.36.jar:/opt/kafka/bin/../libs/snakeyaml-2.0.jar:/opt/kafka/bin/../libs/snakeyaml-engine-2.6.jar:/opt/kafka/bin/../libs/snappy-java-1.1.10.0.jar:/opt/kafka/bin/../libs/swagger-annotations-2.2.8.jar:/opt/kafka/bin/../libs/tracing-agent-0.38.0.jar:/opt/kafka/bin/../libs/trogdor-3.5.0.jar:/opt/kafka/bin/../libs/zjsonpatch-0.3.0.jar:/opt/kafka/bin/../libs/zookeeper-3.6.4.jar:/opt/kafka/bin/../libs/zookeeper-jute-3.6.4.jar:/opt/kafka/bin/../libs/zstd-jni-1.5.5-1.jar (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,433 INFO Server environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,433 INFO Server environment:java.io.tmpdir=/tmp (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,433 INFO Server environment:java.compiler= (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,433 INFO Server environment:os.name=Linux (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,433 INFO Server environment:os.arch=amd64 (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,433 INFO Server environment:os.version=5.15.0-1053-azure (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,433 INFO Server environment:user.name=kafka (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,433 INFO Server environment:user.home=/home/kafka (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,434 INFO Server environment:user.dir=/opt/kafka (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,434 INFO Server environment:os.memory.free=119MB (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,434 INFO Server environment:os.memory.max=1986MB (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,434 INFO Server environment:os.memory.total=130MB (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,434 INFO zookeeper.enableEagerACLCheck = false (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,434 INFO zookeeper.skipACL=="yes", ACL checks will be skipped (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,434 INFO zookeeper.digest.enabled = true (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,434 INFO zookeeper.closeSessionTxn.enabled = true (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,434 INFO zookeeper.flushDelay=0 (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,434 INFO zookeeper.maxWriteQueuePollTime=0 (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,434 INFO zookeeper.maxBatchSize=1000 (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,434 INFO zookeeper.intBufferStartingSizeBytes = 1024 (org.apache.zookeeper.server.ZooKeeperServer) [main]
  2024-03-01 04:21:03,439 INFO Using org.apache.zookeeper.server.watch.WatchManager as watch manager (org.apache.zookeeper.server.watch.WatchManagerFactory) [main]
  2024-03-01 04:21:03,439 INFO Using org.apache.zookeeper.server.watch.WatchManager as watch manager (org.apache.zookeeper.server.watch.WatchManagerFactory) [main]
  2024-03-01 04:21:03,440 INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase) [main]
  2024-03-01 04:21:03,440 INFO zookeeper.commitLogCount=500 (org.apache.zookeeper.server.ZKDatabase) [main]
  2024-03-01 04:21:03,448 INFO Using TLS encrypted quorum communication (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
  2024-03-01 04:21:03,448 INFO Port unification disabled (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
  2024-03-01 04:21:03,448 INFO multiAddress.enabled set to false (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
  2024-03-01 04:21:03,448 INFO multiAddress.reachabilityCheckEnabled set to true (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
  2024-03-01 04:21:03,448 INFO multiAddress.reachabilityCheckTimeoutMs set to 1000 (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
  2024-03-01 04:21:03,448 INFO QuorumPeer communication is not secured! (SASL auth disabled) (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
  2024-03-01 04:21:03,448 INFO quorum.cnxn.threads.size set to 20 (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
  2024-03-01 04:21:03,456 INFO zookeeper.snapshot.compression.method = CHECKED (org.apache.zookeeper.server.persistence.SnapStream) [main]
  2024-03-01 04:21:03,457 INFO Snapshotting: 0x0 to /var/lib/zookeeper/data/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog) [main]
  2024-03-01 04:21:03,462 INFO Snapshot loaded in 14 ms, highest zxid is 0x0, digest is 1371985504 (org.apache.zookeeper.server.ZKDatabase) [main]
  2024-03-01 04:21:03,464 INFO currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
  2024-03-01 04:21:03,488 INFO acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
  2024-03-01 04:21:03,492 INFO binding to port /127.0.0.1:12181 (org.apache.zookeeper.server.NettyServerCnxnFactory) [main]
  2024-03-01 04:21:03,646 INFO bound to port 12181 (org.apache.zookeeper.server.NettyServerCnxnFactory) [main]
  2024-03-01 04:21:03,646 INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NettyServerCnxnFactory) [main]
  2024-03-01 04:21:03,649 INFO bound to port 2181 (org.apache.zookeeper.server.NettyServerCnxnFactory) [main]
  2024-03-01 04:21:03,650 INFO Using 4000ms as the quorum cnxn socket timeout (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
  2024-03-01 04:21:03,666 INFO Election port bind maximum retries is infinite (org.apache.zookeeper.server.quorum.QuorumCnxManager) [main]
  2024-03-01 04:21:03,669 INFO zookeeper.fastleader.minNotificationInterval=200 (org.apache.zookeeper.server.quorum.FastLeaderElection) [main]
  2024-03-01 04:21:03,669 INFO zookeeper.fastleader.maxNotificationInterval=60000 (org.apache.zookeeper.server.quorum.FastLeaderElection) [main]
  2024-03-01 04:21:03,688 INFO ZooKeeper audit is disabled. (org.apache.zookeeper.audit.ZKAuditProvider) [main]
  2024-03-01 04:21:03,706 INFO Creating TLS-only quorum server socket (org.apache.zookeeper.server.quorum.QuorumCnxManager) [ListenerHandler-/0.0.0.0:3888]
  2024-03-01 04:21:03,709 INFO 1 is accepting connections now, my election bind port: /0.0.0.0:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [ListenerHandler-/0.0.0.0:3888]
  2024-03-01 04:21:03,723 INFO LOOKING (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:03,724 INFO New election. My id = 1, proposed zxid=0x0 (org.apache.zookeeper.server.quorum.FastLeaderElection) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:03,737 INFO Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:1, n.round:0x1, n.peerEpoch:0x0, n.zxid:0x0, message format version:0x2, n.config version:0x0 (org.apache.zookeeper.server.quorum.FastLeaderElection) [WorkerReceiver[myid=1]]
  2024-03-01 04:21:03,949 INFO Notification time out: 400 (org.apache.zookeeper.server.quorum.FastLeaderElection) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:04,230 WARN Cannot open channel to 2 at election address sims-kafka-cluster-zookeeper-1.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.16.32:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumConnectionThread-[myid=1]-1]
  java.net.ConnectException: Connection refused
  at java.base/sun.nio.ch.Net.pollConnect(Native Method)
  at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
  at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:554)
  at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:602)
  at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
  at java.base/java.net.Socket.connect(Socket.java:633)
  at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304)
  at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:383)
  at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:457)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  at java.base/java.lang.Thread.run(Thread.java:840)
  2024-03-01 04:21:04,231 WARN Cannot open channel to 3 at election address sims-kafka-cluster-zookeeper-2.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.13.66:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumConnectionThread-[myid=1]-2]
  java.net.ConnectException: Connection refused
  at java.base/sun.nio.ch.Net.pollConnect(Native Method)
  at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
  at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:554)
  at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:602)
  at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
  at java.base/java.net.Socket.connect(Socket.java:633)
  at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304)
  at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:383)
  at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:457)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  at java.base/java.lang.Thread.run(Thread.java:840)
  2024-03-01 04:21:04,351 INFO Notification time out: 800 (org.apache.zookeeper.server.quorum.FastLeaderElection) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:04,355 WARN Cannot open channel to 2 at election address sims-kafka-cluster-zookeeper-1.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.16.32:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumConnectionThread-[myid=1]-3]
  java.net.ConnectException: Connection refused
  at java.base/sun.nio.ch.Net.pollConnect(Native Method)
  at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
  at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:554)
  at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:602)
  at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
  at java.base/java.net.Socket.connect(Socket.java:633)
  at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304)
  at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:383)
  at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:457)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  at java.base/java.lang.Thread.run(Thread.java:840)
  2024-03-01 04:21:04,943 INFO Received connection request from /192.168.13.66:47628 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [ListenerHandler-/0.0.0.0:3888]
  2024-03-01 04:21:05,189 INFO Notification time out: 1600 (org.apache.zookeeper.server.quorum.FastLeaderElection) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:05,281 WARN Cannot open channel to 2 at election address sims-kafka-cluster-zookeeper-1.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.16.32:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumConnectionThread-[myid=1]-3]
  java.net.ConnectException: Connection refused
  at java.base/sun.nio.ch.Net.pollConnect(Native Method)
  at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
  at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:554)
  at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:602)
  at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
  at java.base/java.net.Socket.connect(Socket.java:633)
  at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304)
  at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:383)
  at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:457)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  at java.base/java.lang.Thread.run(Thread.java:840)
  2024-03-01 04:21:05,582 INFO SSL handshake complete with sims-kafka-cluster-zookeeper-2.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.13.66:3888 - TLSv1.2 - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumConnectionThread-[myid=1]-2]
  2024-03-01 04:21:05,584 INFO Have smaller server identifier, so dropping the connection: (myId:1 --> sid:3) (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumConnectionThread-[myid=1]-2]
  2024-03-01 04:21:05,787 INFO Accepted TLS connection from sims-kafka-cluster-zookeeper-2.sims-kafka-cluster-zookeeper-nodes.kafka.svc.cluster.local/192.168.13.66:47628 - TLSv1.2 - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (org.apache.zookeeper.server.quorum.UnifiedServerSocket) [ListenerHandler-/0.0.0.0:3888]
  2024-03-01 04:21:05,801 INFO Notification: my state:LOOKING; n.sid:3, n.state:LOOKING, n.leader:3, n.round:0x1, n.peerEpoch:0x0, n.zxid:0x0, message format version:0x2, n.config version:0x0 (org.apache.zookeeper.server.quorum.FastLeaderElection) [WorkerReceiver[myid=1]]
  2024-03-01 04:21:05,808 INFO Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:3, n.round:0x1, n.peerEpoch:0x0, n.zxid:0x0, message format version:0x2, n.config version:0x0 (org.apache.zookeeper.server.quorum.FastLeaderElection) [WorkerReceiver[myid=1]]
  2024-03-01 04:21:06,010 INFO Peer state changed: following (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,011 INFO FOLLOWING (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,020 INFO leaderConnectDelayDuringRetryMs: 100 (org.apache.zookeeper.server.quorum.Learner) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,020 INFO TCP NoDelay set to: true (org.apache.zookeeper.server.quorum.Learner) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,020 INFO zookeeper.learner.asyncSending = false (org.apache.zookeeper.server.quorum.Learner) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,022 INFO Weighed connection throttling is disabled (org.apache.zookeeper.server.BlueThrottle) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,024 INFO minSessionTimeout set to 4000 (org.apache.zookeeper.server.ZooKeeperServer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,024 INFO maxSessionTimeout set to 40000 (org.apache.zookeeper.server.ZooKeeperServer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,025 INFO Response cache size is initialized with value 400. (org.apache.zookeeper.server.ResponseCache) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,025 INFO Response cache size is initialized with value 400. (org.apache.zookeeper.server.ResponseCache) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,026 INFO zookeeper.pathStats.slotCapacity = 60 (org.apache.zookeeper.server.util.RequestPathMetricsCollector) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,027 INFO zookeeper.pathStats.slotDuration = 15 (org.apache.zookeeper.server.util.RequestPathMetricsCollector) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,027 INFO zookeeper.pathStats.maxDepth = 6 (org.apache.zookeeper.server.util.RequestPathMetricsCollector) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,028 INFO zookeeper.pathStats.initialDelay = 5 (org.apache.zookeeper.server.util.RequestPathMetricsCollector) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,028 INFO zookeeper.pathStats.delay = 5 (org.apache.zookeeper.server.util.RequestPathMetricsCollector) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,029 INFO zookeeper.pathStats.enabled = false (org.apache.zookeeper.server.util.RequestPathMetricsCollector) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,031 INFO The max bytes for all large requests are set to 104857600 (org.apache.zookeeper.server.ZooKeeperServer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,031 INFO The large request threshold is set to -1 (org.apache.zookeeper.server.ZooKeeperServer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,031 INFO Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 clientPortListenBacklog -1 datadir /var/lib/zookeeper/data/version-2 snapdir /var/lib/zookeeper/data/version-2 (org.apache.zookeeper.server.ZooKeeperServer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,033 INFO FOLLOWING - LEADER ELECTION TOOK - 2308 MS (org.apache.zookeeper.server.quorum.Learner) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,053 INFO Peer state changed: following - discovery (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,156 INFO Received connection request from /192.168.16.32:49910 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [ListenerHandler-/0.0.0.0:3888]
  2024-03-01 04:21:06,201 INFO Successfully connected to leader, using address: sims-kafka-cluster-zookeeper-2.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.13.66:2888 (org.apache.zookeeper.server.quorum.Learner) [LeaderConnector-sims-kafka-cluster-zookeeper-2.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.13.66:2888]
  2024-03-01 04:21:06,234 INFO Peer state changed: following - synchronization (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,242 INFO Getting a diff from the leader 0x0 (org.apache.zookeeper.server.quorum.Learner) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,242 INFO Peer state changed: following - synchronization - diff (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,247 INFO Learner received NEWLEADER message (org.apache.zookeeper.server.quorum.Learner) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,259 INFO Peer state changed: following - synchronization (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,266 INFO Configuring CommitProcessor with readBatchSize -1 commitBatchSize 1 (org.apache.zookeeper.server.quorum.CommitProcessor) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,267 INFO Configuring CommitProcessor with 2 worker threads. (org.apache.zookeeper.server.quorum.CommitProcessor) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,271 INFO zookeeper.request_throttler.shutdownTimeout = 10000 (org.apache.zookeeper.server.RequestThrottler) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,340 INFO Learner received UPTODATE message (org.apache.zookeeper.server.quorum.Learner) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,345 WARN Restarting Leader Election (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:06,810 WARN Send worker leaving thread id 3 my id = 1 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [SendWorker:3]
  2024-03-01 04:21:06,811 WARN Connection broken for id 3, my id = 1 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [RecvWorker:3]
  java.net.SocketException: Socket closed
  at java.base/sun.nio.ch.NioSocketImpl.endRead(NioSocketImpl.java:253)
  at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:332)
  at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:355)
  at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:808)
  at java.base/java.net.Socket$SocketInputStream.read(Socket.java:966)
  at java.base/java.io.SequenceInputStream.read(SequenceInputStream.java:198)
  at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:484)
  at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
  at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
  at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1465)
  at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1069)
  at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.read(UnifiedServerSocket.java:693)
  at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:244)
  at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:263)
  at java.base/java.io.DataInputStream.readInt(DataInputStream.java:381)
  at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1384)
  2024-03-01 04:21:06,812 WARN Interrupting SendWorker thread from RecvWorker. sid: 3. myId: 1 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [RecvWorker:3]
  2024-03-01 04:21:06,977 INFO Accepted TLS connection from sims-kafka-cluster-zookeeper-1.sims-kafka-cluster-zookeeper-nodes.kafka.svc.cluster.local/192.168.16.32:49910 - TLSv1.2 - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (org.apache.zookeeper.server.quorum.UnifiedServerSocket) [ListenerHandler-/0.0.0.0:3888]
  2024-03-01 04:21:06,997 INFO Leaving listener (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumPeerListener]
  2024-03-01 04:21:06,998 WARN Interrupting SendWorker thread from RecvWorker. sid: 2. myId: 1 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [RecvWorker:2]
  2024-03-01 04:21:06,999 WARN Send worker leaving thread id 2 my id = 1 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [SendWorker:2]
  2024-03-01 04:21:06,999 INFO Using 4000ms as the quorum cnxn socket timeout (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,000 INFO Election port bind maximum retries is infinite (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,001 WARN Clobbering already-set QuorumCnxManager (restarting leader election?) (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,002 WARN Exception when following the leader (org.apache.zookeeper.server.quorum.Learner) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  java.lang.Exception: changes proposed in reconfig
  at org.apache.zookeeper.server.quorum.Learner.syncWithLeader(Learner.java:711)
  at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:107)
  at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1467)
  2024-03-01 04:21:07,004 INFO Disconnected from leader (with address: sims-kafka-cluster-zookeeper-2.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.13.66:2888). Was connected for 799ms. Sync state: false (org.apache.zookeeper.server.quorum.Learner) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,005 INFO shutdown Follower (org.apache.zookeeper.server.quorum.Learner) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,006 INFO Peer state changed: looking (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,006 WARN PeerState set to LOOKING (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,006 INFO LOOKING (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,007 INFO New election. My id = 1, proposed zxid=0x0 (org.apache.zookeeper.server.quorum.FastLeaderElection) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,020 INFO Creating TLS-only quorum server socket (org.apache.zookeeper.server.quorum.QuorumCnxManager) [ListenerHandler-sims-kafka-cluster-zookeeper-0.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.12.221:3888]
  2024-03-01 04:21:07,021 INFO 1 is accepting connections now, my election bind port: sims-kafka-cluster-zookeeper-0.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.12.221:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [ListenerHandler-sims-kafka-cluster-zookeeper-0.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.12.221:3888]
  2024-03-01 04:21:07,022 INFO Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:1, n.round:0x1, n.peerEpoch:0x1, n.zxid:0x0, message format version:0x2, n.config version:0x100000000 (org.apache.zookeeper.server.quorum.FastLeaderElection) [WorkerReceiver[myid=1]]
  2024-03-01 04:21:07,023 INFO Received connection request from /192.168.12.221:37878 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [ListenerHandler-sims-kafka-cluster-zookeeper-0.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.12.221:3888]
  2024-03-01 04:21:07,149 INFO Accepted TLS connection from sims-kafka-cluster-zookeeper-0.sims-kafka-cluster-zookeeper-nodes.kafka.svc.cluster.local/192.168.12.221:37878 - TLSv1.2 - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (org.apache.zookeeper.server.quorum.UnifiedServerSocket) [ListenerHandler-sims-kafka-cluster-zookeeper-0.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.12.221:3888]
  2024-03-01 04:21:07,150 INFO SSL handshake complete with sims-kafka-cluster-zookeeper-0/192.168.12.221:3888 - TLSv1.2 - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumConnectionThread-[myid=1]-2]
  2024-03-01 04:21:07,152 WARN We got a connection request from a server with our own ID. This should be either a configuration error, or a bug. (org.apache.zookeeper.server.quorum.QuorumCnxManager) [ListenerHandler-sims-kafka-cluster-zookeeper-0.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.12.221:3888]
  2024-03-01 04:21:07,152 INFO Have smaller server identifier, so dropping the connection: (myId:1 --> sid:3) (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumConnectionThread-[myid=1]-2]
  2024-03-01 04:21:07,205 WARN Cannot open channel to 2 at election address sims-kafka-cluster-zookeeper-1.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.16.32:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumConnectionThread-[myid=1]-1]
  java.net.SocketException: Connection reset
  at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:328)
  at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:355)
  at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:808)
  at java.base/java.net.Socket$SocketInputStream.read(Socket.java:966)
  at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:484)
  at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
  at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160)
  at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
  at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1510)
  at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1425)
  at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)
  at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426)
  at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:386)
  at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:457)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  at java.base/java.lang.Thread.run(Thread.java:840)
  2024-03-01 04:21:07,224 INFO Notification time out: 400 (org.apache.zookeeper.server.quorum.FastLeaderElection) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,228 INFO Received connection request from /192.168.12.221:37886 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [ListenerHandler-sims-kafka-cluster-zookeeper-0.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.12.221:3888]
  2024-03-01 04:21:07,253 INFO SSL handshake complete with sims-kafka-cluster-zookeeper-0/192.168.12.221:3888 - TLSv1.2 - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumConnectionThread-[myid=1]-1]
  2024-03-01 04:21:07,253 INFO Have smaller server identifier, so dropping the connection: (myId:1 --> sid:3) (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumConnectionThread-[myid=1]-1]
  2024-03-01 04:21:07,257 INFO Accepted TLS connection from /192.168.12.221:37886 - TLSv1.2 - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (org.apache.zookeeper.server.quorum.UnifiedServerSocket) [ListenerHandler-sims-kafka-cluster-zookeeper-0.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.12.221:3888]
  2024-03-01 04:21:07,257 WARN We got a connection request from a server with our own ID. This should be either a configuration error, or a bug. (org.apache.zookeeper.server.quorum.QuorumCnxManager) [ListenerHandler-sims-kafka-cluster-zookeeper-0.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.12.221:3888]
  2024-03-01 04:21:07,264 WARN Cannot open channel to 2 at election address sims-kafka-cluster-zookeeper-1.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.16.32:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumConnectionThread-[myid=1]-3]
  java.net.ConnectException: Connection refused
  at java.base/sun.nio.ch.Net.pollConnect(Native Method)
  at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
  at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:554)
  at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:602)
  at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
  at java.base/java.net.Socket.connect(Socket.java:633)
  at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304)
  at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:383)
  at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:457)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  at java.base/java.lang.Thread.run(Thread.java:840)
  2024-03-01 04:21:07,541 INFO SSL handshake complete with sims-kafka-cluster-zookeeper-1.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.16.32:3888 - TLSv1.2 - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumConnectionThread-[myid=1]-2]
  2024-03-01 04:21:07,543 INFO Have smaller server identifier, so dropping the connection: (myId:1 --> sid:2) (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumConnectionThread-[myid=1]-2]
  2024-03-01 04:21:07,547 INFO Received connection request from /192.168.16.32:49920 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [ListenerHandler-sims-kafka-cluster-zookeeper-0.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.12.221:3888]
  2024-03-01 04:21:07,628 INFO Notification time out: 800 (org.apache.zookeeper.server.quorum.FastLeaderElection) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,668 INFO Accepted TLS connection from sims-kafka-cluster-zookeeper-1.sims-kafka-cluster-zookeeper-nodes.kafka.svc.cluster.local/192.168.16.32:49920 - TLSv1.2 - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (org.apache.zookeeper.server.quorum.UnifiedServerSocket) [ListenerHandler-sims-kafka-cluster-zookeeper-0.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.12.221:3888]
  2024-03-01 04:21:07,685 INFO SSL handshake complete with sims-kafka-cluster-zookeeper-1.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.16.32:3888 - TLSv1.2 - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumConnectionThread-[myid=1]-3]
  2024-03-01 04:21:07,686 INFO Have smaller server identifier, so dropping the connection: (myId:1 --> sid:2) (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumConnectionThread-[myid=1]-3]
  2024-03-01 04:21:07,687 INFO Received connection request from /192.168.16.32:49928 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [ListenerHandler-sims-kafka-cluster-zookeeper-0.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.12.221:3888]
  2024-03-01 04:21:07,696 INFO Accepted TLS connection from /192.168.16.32:49928 - TLSv1.2 - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (org.apache.zookeeper.server.quorum.UnifiedServerSocket) [ListenerHandler-sims-kafka-cluster-zookeeper-0.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.12.221:3888]
  2024-03-01 04:21:07,697 WARN Connection broken for id 2, my id = 1 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [RecvWorker:2]
  java.io.EOFException
  at java.base/java.io.DataInputStream.readInt(DataInputStream.java:386)
  at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1384)
  2024-03-01 04:21:07,697 WARN Interrupted while waiting for message on queue (org.apache.zookeeper.server.quorum.QuorumCnxManager) [SendWorker:2]
  java.lang.InterruptedException
  at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1679)
  at org.apache.zookeeper.util.CircularBlockingQueue.poll(CircularBlockingQueue.java:105)
  at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1447)
  at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$900(QuorumCnxManager.java:98)
  at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1271)
  2024-03-01 04:21:07,707 WARN Send worker leaving thread id 2 my id = 1 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [SendWorker:2]
  2024-03-01 04:21:07,707 WARN Interrupting SendWorker thread from RecvWorker. sid: 2. myId: 1 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [RecvWorker:2]
  2024-03-01 04:21:07,715 INFO Received connection request from /192.168.12.221:37900 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [ListenerHandler-sims-kafka-cluster-zookeeper-0.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.12.221:3888]
  2024-03-01 04:21:07,717 INFO Notification: my state:LOOKING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x1, n.peerEpoch:0x0, n.zxid:0x0, message format version:0x2, n.config version:0x100000000 (org.apache.zookeeper.server.quorum.FastLeaderElection) [WorkerReceiver[myid=1]]
  2024-03-01 04:21:07,723 INFO SSL handshake complete with sims-kafka-cluster-zookeeper-0/192.168.12.221:3888 - TLSv1.2 - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumConnectionThread-[myid=1]-1]
  2024-03-01 04:21:07,723 INFO Have smaller server identifier, so dropping the connection: (myId:1 --> sid:3) (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumConnectionThread-[myid=1]-1]
  2024-03-01 04:21:07,725 INFO Notification: my state:LOOKING; n.sid:2, n.state:LOOKING, n.leader:1, n.round:0x1, n.peerEpoch:0x1, n.zxid:0x0, message format version:0x2, n.config version:0x100000000 (org.apache.zookeeper.server.quorum.FastLeaderElection) [WorkerReceiver[myid=1]]
  2024-03-01 04:21:07,726 INFO Accepted TLS connection from /192.168.12.221:37900 - TLSv1.2 - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (org.apache.zookeeper.server.quorum.UnifiedServerSocket) [ListenerHandler-sims-kafka-cluster-zookeeper-0.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.12.221:3888]
  2024-03-01 04:21:07,726 WARN We got a connection request from a server with our own ID. This should be either a configuration error, or a bug. (org.apache.zookeeper.server.quorum.QuorumCnxManager) [ListenerHandler-sims-kafka-cluster-zookeeper-0.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.12.221:3888]
  2024-03-01 04:21:07,925 INFO Peer state changed: leading (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,926 INFO LEADING (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,935 INFO zookeeper.leader.maxConcurrentSnapSyncs = 10 (org.apache.zookeeper.server.quorum.LearnerMaster) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,935 INFO zookeeper.leader.maxConcurrentDiffSyncs = 100 (org.apache.zookeeper.server.quorum.LearnerMaster) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,938 INFO TCP NoDelay set to: true (org.apache.zookeeper.server.quorum.Leader) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,938 INFO zookeeper.leader.ackLoggingFrequency = 1000 (org.apache.zookeeper.server.quorum.Leader) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,938 INFO zookeeper.leader.maxTimeToWaitForEpoch = -1ms (org.apache.zookeeper.server.quorum.Leader) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,940 INFO minSessionTimeout set to 4000 (org.apache.zookeeper.server.ZooKeeperServer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,940 INFO maxSessionTimeout set to 40000 (org.apache.zookeeper.server.ZooKeeperServer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,940 INFO Response cache size is initialized with value 400. (org.apache.zookeeper.server.ResponseCache) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,940 INFO Response cache size is initialized with value 400. (org.apache.zookeeper.server.ResponseCache) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,940 INFO zookeeper.pathStats.slotCapacity = 60 (org.apache.zookeeper.server.util.RequestPathMetricsCollector) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,940 INFO zookeeper.pathStats.slotDuration = 15 (org.apache.zookeeper.server.util.RequestPathMetricsCollector) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,940 INFO zookeeper.pathStats.maxDepth = 6 (org.apache.zookeeper.server.util.RequestPathMetricsCollector) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,940 INFO zookeeper.pathStats.initialDelay = 5 (org.apache.zookeeper.server.util.RequestPathMetricsCollector) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,941 INFO zookeeper.pathStats.delay = 5 (org.apache.zookeeper.server.util.RequestPathMetricsCollector) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,941 INFO zookeeper.pathStats.enabled = false (org.apache.zookeeper.server.util.RequestPathMetricsCollector) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,941 INFO The max bytes for all large requests are set to 104857600 (org.apache.zookeeper.server.ZooKeeperServer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,941 INFO The large request threshold is set to -1 (org.apache.zookeeper.server.ZooKeeperServer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,941 INFO Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 clientPortListenBacklog -1 datadir /var/lib/zookeeper/data/version-2 snapdir /var/lib/zookeeper/data/version-2 (org.apache.zookeeper.server.ZooKeeperServer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,945 INFO LEADING - LEADER ELECTION TOOK - 937 MS (org.apache.zookeeper.server.quorum.Leader) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,954 INFO Peer state changed: leading - discovery (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,955 INFO Snapshotting: 0x0 to /var/lib/zookeeper/data/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:07,955 INFO Snapshot taken in 1 ms (org.apache.zookeeper.server.ZooKeeperServer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:08,079 INFO Accepted TLS connection from sims-kafka-cluster-zookeeper-1.sims-kafka-cluster-zookeeper-nodes.kafka.svc.cluster.local/192.168.16.32:46718 - TLSv1.2 - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (org.apache.zookeeper.server.quorum.UnifiedServerSocket) [LearnerHandler-/192.168.16.32:46718]
  2024-03-01 04:21:08,098 INFO Follower sid: 2 : info : sims-kafka-cluster-zookeeper-1.sims-kafka-cluster-zookeeper-nodes.kafka.svc:2888:3888:participant;127.0.0.1:12181 (org.apache.zookeeper.server.quorum.LearnerHandler) [LearnerHandler-/192.168.16.32:46718]
  2024-03-01 04:21:08,122 INFO On disk txn sync enabled with snapshotSizeFactor 0.33 (org.apache.zookeeper.server.ZKDatabase) [LearnerHandler-/192.168.16.32:46718]
  2024-03-01 04:21:08,123 INFO Synchronizing with Learner sid: 2 maxCommittedLog=0x0 minCommittedLog=0x0 lastProcessedZxid=0x0 peerLastZxid=0x0 (org.apache.zookeeper.server.quorum.LearnerHandler) [LearnerHandler-/192.168.16.32:46718]
  2024-03-01 04:21:08,123 INFO Sending DIFF zxid=0x0 for peer sid: 2 (org.apache.zookeeper.server.quorum.LearnerHandler) [LearnerHandler-/192.168.16.32:46718]
  2024-03-01 04:21:08,127 INFO Peer state changed: leading - synchronization (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:08,193 INFO Have quorum of supporters, sids: [[1, 2]]; starting up and setting last processed zxid: 0x200000000 (org.apache.zookeeper.server.quorum.Leader) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:08,196 INFO Configuring CommitProcessor with readBatchSize -1 commitBatchSize 1 (org.apache.zookeeper.server.quorum.CommitProcessor) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:08,196 INFO Configuring CommitProcessor with 2 worker threads. (org.apache.zookeeper.server.quorum.CommitProcessor) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:08,200 INFO PrepRequestProcessor (sid:1) started, reconfigEnabled=true (org.apache.zookeeper.server.PrepRequestProcessor) [ProcessThread(sid:1 cport:-1):]
  2024-03-01 04:21:08,202 INFO Using checkIntervalMs=60000 maxPerMinute=10000 maxNeverUsedIntervalMs=0 (org.apache.zookeeper.server.ContainerManager) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:08,207 INFO Peer state changed: leading - broadcast (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)]
  2024-03-01 04:21:08,356 INFO Received connection request from /192.168.13.66:47636 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [ListenerHandler-sims-kafka-cluster-zookeeper-0.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.12.221:3888]
  2024-03-01 04:21:08,369 INFO Accepted TLS connection from /192.168.13.66:47636 - TLSv1.2 - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (org.apache.zookeeper.server.quorum.UnifiedServerSocket) [ListenerHandler-sims-kafka-cluster-zookeeper-0.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.12.221:3888]
  2024-03-01 04:21:08,373 INFO Notification: my state:LEADING; n.sid:3, n.state:LOOKING, n.leader:3, n.round:0x2, n.peerEpoch:0x1, n.zxid:0x100000000, message format version:0x2, n.config version:0x100000000 (org.apache.zookeeper.server.quorum.FastLeaderElection) [WorkerReceiver[myid=1]]
  2024-03-01 04:21:08,396 INFO Notification: my state:LEADING; n.sid:3, n.state:LOOKING, n.leader:3, n.round:0x2, n.peerEpoch:0x1, n.zxid:0x100000000, message format version:0x2, n.config version:0x100000000 (org.apache.zookeeper.server.quorum.FastLeaderElection) [WorkerReceiver[myid=1]]
  2024-03-01 04:21:08,526 INFO Accepted TLS connection from sims-kafka-cluster-zookeeper-2.sims-kafka-cluster-zookeeper-nodes.kafka.svc.cluster.local/192.168.13.66:58498 - TLSv1.2 - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (org.apache.zookeeper.server.quorum.UnifiedServerSocket) [LearnerHandler-/192.168.13.66:58498]
  2024-03-01 04:21:08,530 INFO Follower sid: 3 : info : 0.0.0.0:2888:3888:participant;127.0.0.1:12181 (org.apache.zookeeper.server.quorum.LearnerHandler) [LearnerHandler-/192.168.13.66:58498]
  2024-03-01 04:21:08,537 INFO On disk txn sync enabled with snapshotSizeFactor 0.33 (org.apache.zookeeper.server.ZKDatabase) [LearnerHandler-/192.168.13.66:58498]
  2024-03-01 04:21:08,537 INFO Synchronizing with Learner sid: 3 maxCommittedLog=0x0 minCommittedLog=0x0 lastProcessedZxid=0x200000000 peerLastZxid=0x100000000 (org.apache.zookeeper.server.quorum.LearnerHandler) [LearnerHandler-/192.168.13.66:58498]
  2024-03-01 04:21:08,539 INFO Sending snapshot last zxid of peer is 0x100000000, zxid of leader is 0x200000000, send zxid of db as 0x200000000, 1 concurrent snapshot sync, snapshot sync was exempt from throttle (org.apache.zookeeper.server.quorum.LearnerHandler) [LearnerHandler-/192.168.13.66:58498]
  2024-03-01 04:21:08,806 INFO WorkerSender is down (org.apache.zookeeper.server.quorum.FastLeaderElection) [WorkerSender[myid=1]]
  2024-03-01 04:21:08,810 INFO WorkerReceiver is down (org.apache.zookeeper.server.quorum.FastLeaderElection) [WorkerReceiver[myid=1]]
  2024-03-01 04:21:19,722 INFO The list of known four letter word commands is : [{1936881266=srvr, 1937006964=stat, 2003003491=wchc, 1685417328=dump, 1668445044=crst, 1936880500=srst, 1701738089=envi, 1668247142=conf, -720899=telnet close, 1751217000=hash, 2003003507=wchs, 2003003504=wchp, 1684632179=dirs, 1668247155=cons, 1835955314=mntr, 1769173615=isro, 1920298859=ruok, 1735683435=gtmk, 1937010027=stmk}] (org.apache.zookeeper.server.command.FourLetterCommands) [nioEventLoopGroup-4-1]
  2024-03-01 04:21:19,723 INFO The list of enabled four letter word commands is : [[wchs, stat, wchp, dirs, stmk, conf, ruok, mntr, srvr, wchc, envi, srst, isro, dump, gtmk, telnet close, crst, hash, cons]] (org.apache.zookeeper.server.command.FourLetterCommands) [nioEventLoopGroup-4-1]
  2024-03-01 04:21:19,724 INFO Processing ruok command from /127.0.0.1:52352 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
  2024-03-01 04:21:19,728 INFO Processing ruok command from /127.0.0.1:52354 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]
  2024-03-01 04:21:23,677 INFO Creating new log file: log.200000001 (org.apache.zookeeper.server.persistence.FileTxnLog) [SyncThread:1]
  2024-03-01 04:21:23,692 INFO Committing global session 0x300257d4eb60000 (org.apache.zookeeper.server.quorum.LeaderSessionTracker) [CommitProcessor:1]
  2024-03-01 04:21:23,954 WARN skipACL is set, reconfig operation will skip ACL checks! (org.apache.zookeeper.server.PrepRequestProcessor) [ProcessThread(sid:1 cport:-1):]
  2024-03-01 04:21:23,955 INFO Non-incremental reconfig (org.apache.zookeeper.server.PrepRequestProcessor) [ProcessThread(sid:1 cport:-1):]
  2024-03-01 04:21:23,967 WARN Restarting Leader Election (org.apache.zookeeper.server.quorum.QuorumPeer) [LearnerHandler-/192.168.16.32:46718]
  2024-03-01 04:21:23,967 INFO Leaving listener (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumPeerListener]
  2024-03-01 04:21:23,969 WARN Connection broken for id 2, my id = 1 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [RecvWorker:2]
  java.net.SocketException: Socket closed
  at java.base/sun.nio.ch.NioSocketImpl.endRead(NioSocketImpl.java:253)
  at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:332)
  at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:355)
  at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:808)
  at java.base/java.net.Socket$SocketInputStream.read(Socket.java:966)
  at java.base/java.io.SequenceInputStream.read(SequenceInputStream.java:198)
  at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:484)
  at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
  at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
  at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1465)
  at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1069)
  at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.read(UnifiedServerSocket.java:693)
  at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:244)
  at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:263)
  at java.base/java.io.DataInputStream.readInt(DataInputStream.java:381)
  at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1384)
  2024-03-01 04:21:23,969 WARN Interrupting SendWorker thread from RecvWorker. sid: 2. myId: 1 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [RecvWorker:2]
  2024-03-01 04:21:23,970 WARN Connection broken for id 3, my id = 1 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [RecvWorker:3]
  java.net.SocketException: Socket closed
  at java.base/sun.nio.ch.NioSocketImpl.endRead(NioSocketImpl.java:253)
  at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:332)
  at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:355)
  at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:808)
  at java.base/java.net.Socket$SocketInputStream.read(Socket.java:966)
  at java.base/java.io.SequenceInputStream.read(SequenceInputStream.java:198)
  at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:484)
  at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
  at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
  at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1465)
  at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1069)
  at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.read(UnifiedServerSocket.java:693)
  at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:244)
  at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:263)
  at java.base/java.io.DataInputStream.readInt(DataInputStream.java:381)
  at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1384)
  2024-03-01 04:21:23,969 WARN Interrupted while waiting for message on queue (org.apache.zookeeper.server.quorum.QuorumCnxManager) [SendWorker:2]
  java.lang.InterruptedException
  at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1679)
  at org.apache.zookeeper.util.CircularBlockingQueue.poll(CircularBlockingQueue.java:105)
  at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1447)
  at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$900(QuorumCnxManager.java:98)
  at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1271)
  2024-03-01 04:21:23,971 WARN Send worker leaving thread id 2 my id = 1 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [SendWorker:2]
  2024-03-01 04:21:23,972 WARN Interrupted while waiting for message on queue (org.apache.zookeeper.server.quorum.QuorumCnxManager) [SendWorker:3]
  java.lang.InterruptedException
  at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1679)
  at org.apache.zookeeper.util.CircularBlockingQueue.poll(CircularBlockingQueue.java:105)
  at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1447)
  at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$900(QuorumCnxManager.java:98)
  at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1271)
  2024-03-01 04:21:23,972 INFO Using 4000ms as the quorum cnxn socket timeout (org.apache.zookeeper.server.quorum.QuorumPeer) [LearnerHandler-/192.168.16.32:46718]
  2024-03-01 04:21:23,973 WARN Send worker leaving thread id 3 my id = 1 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [SendWorker:3]
  2024-03-01 04:21:23,973 INFO Election port bind maximum retries is infinite (org.apache.zookeeper.server.quorum.QuorumCnxManager) [LearnerHandler-/192.168.16.32:46718]
  2024-03-01 04:21:23,973 WARN Interrupting SendWorker thread from RecvWorker. sid: 3. myId: 1 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [RecvWorker:3]
  2024-03-01 04:21:23,973 WARN Clobbering already-set QuorumCnxManager (restarting leader election?) (org.apache.zookeeper.server.quorum.QuorumPeer) [LearnerHandler-/192.168.16.32:46718]
  2024-03-01 04:21:23,976 INFO Creating TLS-only quorum server socket (org.apache.zookeeper.server.quorum.QuorumCnxManager) [ListenerHandler-sims-kafka-cluster-zookeeper-0.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.12.221:3888]
  2024-03-01 04:21:23,976 INFO 1 is accepting connections now, my election bind port: sims-kafka-cluster-zookeeper-0.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.12.221:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [ListenerHandler-sims-kafka-cluster-zookeeper-0.sims-kafka-cluster-zookeeper-nodes.kafka.svc/192.168.12.221:3888]
  2024-03-01 04:21:26,400 INFO WorkerReceiver is down (org.apache.zookeeper.server.quorum.FastLeaderElection) [WorkerReceiver[myid=1]]
  2024-03-01 04:21:26,400 INFO WorkerSender is down (org.apache.zookeeper.server.quorum.FastLeaderElection) [WorkerSender[myid=1]]
  2024-03-01 04:21:29,681 INFO Processing ruok command from /127.0.0.1:46040 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-3]
  2024-03-01 04:21:29,706 INFO Processing ruok command from /127.0.0.1:46044 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-4]
  2024-03-01 04:21:39,581 INFO Processing ruok command from /127.0.0.1:49834 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
  2024-03-01 04:21:39,606 INFO Processing ruok command from /127.0.0.1:49842 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]
  2024-03-01 04:21:49,578 INFO Processing ruok command from /127.0.0.1:58870 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-3]
  2024-03-01 04:21:49,592 INFO Processing ruok command from /127.0.0.1:58876 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-4]
  2024-03-01 04:21:59,587 INFO Processing ruok command from /127.0.0.1:50956 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
  2024-03-01 04:21:59,592 INFO Processing ruok command from /127.0.0.1:50960 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]
  2024-03-01 04:22:09,602 INFO Processing ruok command from /127.0.0.1:52890 (org.apache.zookeeper.server.NettyServerCnxn)
  [nioEventLoopGroup-4-3]
  2024-03-01 04:22:09,610 INFO Processing ruok command from /127.0.0.1:52906 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-4]
  2024-03-01 04:22:19,661 INFO Processing ruok command from /127.0.0.1:38358 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
  2024-03-01 04:22:19,663 INFO Processing ruok command from /127.0.0.1:38368 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]
  2024-03-01 04:22:29,589 INFO Processing ruok command from /127.0.0.1:46378 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-3]
  2024-03-01 04:22:29,604 INFO Processing ruok command from /127.0.0.1:46382 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-4]
  2024-03-01 04:22:39,587 INFO Processing ruok command from /127.0.0.1:45450 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
  2024-03-01 04:22:39,603 INFO Processing ruok command from /127.0.0.1:45454 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]
  2024-03-01 04:22:49,581 INFO Processing ruok command from /127.0.0.1:36320 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-3]
  2024-03-01 04:22:49,608 INFO Processing ruok command from /127.0.0.1:36336 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-4]
  2024-03-01 04:22:59,655 INFO Processing ruok command from /127.0.0.1:40680 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]
  2024-03-01 04:22:59,666 INFO Processing ruok command from /127.0.0.1:40676 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
  2024-03-01 04:23:09,587 INFO Processing ruok command from /127.0.0.1:42000 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-3]
  2024-03-01 04:23:09,601 INFO Processing ruok command from /127.0.0.1:42010 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-4]

Is there any help how to fix the socket connection refused issue in zookeepers

[scholzj]

For a start, you should properly format the YAML and the logs. You should also share the operator logs.

[damodarreddy08]

Thanks for your quick response..

Now able to started the Kafka cluster after modify the some changes and proceeding for further modules on cluster and will get touch with you incase need...

[damodarreddy08]

closing discussion