Cluster failed when I try to use local storage #10365

al66 · 2024-07-20T09:38:42Z

al66
Jul 20, 2024

Hello,

I have problems when using local storage for my Kafka cluster on Kubernetes.
It's running in kraft mode with 3 nodes - each node with the roles controller & broker.
The nodes are connected via a private network - therefore the node IP's are 10.0.0.2, 10.0.0.4, 10.0.0.5

The error I'm getting was also mentioned here: 9629, but I don't see a solution for my current problem.

The operater is 5 days old - installed via kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka.

Using ephemeral storage type instead, the cluster runs w/o any errors.

The error occurs only with the local storage.
The volumes are found and bound sucessfully, but the pods are still pending.

Any help is very welcome!

My cluster with local storage is configuared as follows:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: dev-cluster
  namespace: kafka
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
spec:
  kafka:
    version: 3.7.1
    metadataVersion: 3.7-IV4
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 1
      default.replication.factor: 3
      min.insync.replicas: 2
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                  - key: kubernetes.io/hostname
                    operator: In
                    values:
                    - imicros-dev-2
                    - imicros-dev-3
                    - imicros-dev-4
  entityOperator:
    topicOperator: {}
    userOperator: {}
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: dev-pool
  namespace: kafka
  labels:
    strimzi.io/cluster: dev-cluster
spec:
  replicas: 3
  roles:
    - controller
    - broker
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        #type: ephemeral
        size: 10Gi
        class: local-storage
        deleteClaim: false
        kraftMetadata: shared
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: data-0-dev-cluster-dev-pool-0 
  labels:
    strimzi.io/cluster: dev-cluster
spec:
  claimRef:
    name: data-0-dev-cluster-dev-pool-0 
    namespace: kafka
  capacity:
    storage: 10Gi
  accessModes:
  - ReadWriteOnce
  - ReadWriteMany
  volumeMode: Filesystem
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /data/volumes/pv1
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - dev-imicros-2
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: data-0-dev-cluster-dev-pool-1
  labels:
    strimzi.io/cluster: dev-cluster
spec:
  claimRef:
    name: data-0-dev-cluster-dev-pool-1
    namespace: kafka
  capacity:
    storage: 10Gi
  accessModes:
  - ReadWriteOnce
  volumeMode: Filesystem
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /data/volumes/pv1
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - dev-imicros-3
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: data-0-dev-cluster-dev-pool-2
  labels:
    strimzi.io/cluster: dev-cluster
spec:
  claimRef:
    name: data-0-dev-cluster-dev-pool-2
    namespace: kafka
  capacity:
    storage: 10Gi
  accessModes:
  - ReadWriteOnce
  volumeMode: Filesystem
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /data/volumes/pv1
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - dev-imicros-4

The log of the operator shows the a connection error:

2024-07-20 09:10:09 WARN  JdkWebSocketImpl:141 - Aborting WebSocket due to a write error or failure with sendClose
2024-07-20 09:10:09 WARN  JdkWebSocketImpl:128 - Queued close did not succeed
java.lang.IllegalArgumentException: statusCode
 at jdk.internal.net.http.websocket.WebSocketImpl.sendClose(WebSocketImpl.java:309) ~[java.net.http:?]
 at io.fabric8.kubernetes.client.jdkhttp.JdkWebSocketImpl.sendClose(JdkWebSocketImpl.java:125) ~[io.fabric8.kubernetes-httpclient-jdk-6.13.0.jar:?]
 at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onClose(WatcherWebSocketListener.java:67) ~[io.fabric8.kubernetes-client-6.13.0.jar:?]
 at io.fabric8.kubernetes.client.jdkhttp.JdkWebSocketImpl.onClose(JdkWebSocketImpl.java:83) ~[io.fabric8.kubernetes-httpclient-jdk-6.13.0.jar:?]
 at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.processClose(WebSocketImpl.java:528) ~[java.net.http:?]
 at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.run(WebSocketImpl.java:459) ~[java.net.http:?]
 at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:149) ~[java.net.http:?]
 at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:230) ~[java.net.http:?]
 at jdk.internal.net.http.HttpClientImpl$DelegatingExecutor.execute(HttpClientImpl.java:157) ~[java.net.http:?]
 at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:305) ~[java.net.http:?]
 at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:274) ~[java.net.http:?]
 at jdk.internal.net.http.websocket.WebSocketImpl.trySetState(WebSocketImpl.java:845) ~[java.net.http:?]
 at jdk.internal.net.http.websocket.WebSocketImpl.signalClose(WebSocketImpl.java:771) ~[java.net.http:?]
 at jdk.internal.net.http.websocket.WebSocketImpl$SignallingMessageConsumer.onComplete(WebSocketImpl.java:828) ~[java.net.http:?]
 at jdk.internal.net.http.websocket.TransportImpl$ReceiveTask.run(TransportImpl.java:709) ~[java.net.http:?]
 at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:149) ~[java.net.http:?]
 at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:230) ~[java.net.http:?]
 at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:303) ~[java.net.http:?]
 at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:256) ~[java.net.http:?]
 at jdk.internal.net.http.websocket.TransportImpl$ReadEvent.handle(TransportImpl.java:766) ~[java.net.http:?]
 at jdk.internal.net.http.RawChannelTube$ReadSubscriber.checkEvents(RawChannelTube.java:174) ~[java.net.http:?]
 at jdk.internal.net.http.RawChannelTube$ReadSubscriber.onComplete(RawChannelTube.java:223) ~[java.net.http:?]
 at jdk.internal.net.http.common.SSLTube$DelegateWrapper.onComplete(SSLTube.java:276) ~[java.net.http:?]
 at jdk.internal.net.http.common.SSLTube$SSLSubscriberWrapper.complete(SSLTube.java:440) ~[java.net.http:?]
 at jdk.internal.net.http.common.SSLTube$SSLSubscriberWrapper.onComplete(SSLTube.java:541) ~[java.net.http:?]
 at jdk.internal.net.http.common.SubscriberWrapper.checkCompletion(SubscriberWrapper.java:472) ~[java.net.http:?]
 at jdk.internal.net.http.common.SubscriberWrapper$DownstreamPusher.run1(SubscriberWrapper.java:334) ~[java.net.http:?]
 at jdk.internal.net.http.common.SubscriberWrapper$DownstreamPusher.run(SubscriberWrapper.java:259) ~[java.net.http:?]
 at jdk.internal.net.http.common.SequentialScheduler$LockingRestartableTask.run(SequentialScheduler.java:205) ~[java.net.http:?]
 at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:149) ~[java.net.http:?]
 at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:230) ~[java.net.http:?]
 at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:303) ~[java.net.http:?]
 at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:256) ~[java.net.http:?]
 at jdk.internal.net.http.common.SubscriberWrapper.outgoing(SubscriberWrapper.java:232) ~[java.net.http:?]
 at jdk.internal.net.http.common.SSLFlowDelegate$Reader.processData(SSLFlowDelegate.java:513) ~[java.net.http:?]
 at jdk.internal.net.http.common.SSLFlowDelegate$Reader$ReaderDownstreamPusher.run(SSLFlowDelegate.java:268) ~[java.net.http:?]
 at jdk.internal.net.http.common.SequentialScheduler$LockingRestartableTask.run(SequentialScheduler.java:205) ~[java.net.http:?]
 at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:149) ~[java.net.http:?]
 at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:230) ~[java.net.http:?]
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
 at java.lang.Thread.run(Thread.java:840) ~[?:?]

And here is the configuration the operator has started with:

2024-07-20 07:27:29 INFO  Main:60 - ClusterOperator 0.42.0 is starting
2024-07-20 07:27:29 INFO  Util:196 - Using config:
 PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
 container: oci
 DEV_CLUSTER_KAFKA_BOOTSTRAP_PORT_9091_TCP_PORT: 9091
 STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE: quay.io/strimzi/operator:0.42.0
 JAVA_OPTS:  -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom  -XX:+ExitOnOutOfMemoryError -XX:MinRAMPercentage=15 -XX:MaxRAMPercentage=20
 DEV_CLUSTER_KAFKA_BOOTSTRAP_SERVICE_HOST: 10.152.183.191
 STRIMZI_HOME: /opt/strimzi
 PWD: /opt/strimzi
 STRIMZI_LEADER_ELECTION_IDENTITY: strimzi-cluster-operator-6948497896-t9vd4
 JAVA_MAIN: io.strimzi.operator.cluster.Main
 KUBERNETES_PORT_443_TCP: tcp://10.152.183.1:443
 STRIMZI_LEADER_ELECTION_ENABLED: true
 DEV_CLUSTER_KAFKA_BOOTSTRAP_PORT_9093_TCP_ADDR: 10.152.183.191
 STRIMZI_VERSION: 0.42.0
 STRIMZI_DEFAULT_KANIKO_EXECUTOR_IMAGE: quay.io/strimzi/kaniko-executor:0.42.0
 STRIMZI_NAMESPACE: kafka
 TINI_SHA256_S390X: 931b70a182af879ca249ae9de87ef68423121b38d235c78997fafc680ceab32d
 STRIMZI_DEFAULT_MAVEN_BUILDER: quay.io/strimzi/maven-builder:0.42.0
 STRIMZI_FEATURE_GATES: 
 DEV_CLUSTER_KAFKA_BOOTSTRAP_PORT_9091_TCP_PROTO: tcp
 DEV_CLUSTER_KAFKA_BOOTSTRAP_PORT: tcp://10.152.183.191:9091
 TINI_SHA256_AMD64: 93dcc18adc78c65a028a84799ecf8ad40c936fdfc5f2a57b1acda5a8117fa82c
 STRIMZI_LEADER_ELECTION_LEASE_NAME: strimzi-cluster-operator
 KUBERNETES_SERVICE_PORT_HTTPS: 443
 SHLVL: 0
 STRIMZI_DEFAULT_KAFKA_BRIDGE_IMAGE: quay.io/strimzi/kafka-bridge:0.29.0
 DEV_CLUSTER_KAFKA_BOOTSTRAP_SERVICE_PORT: 9091
 STRIMZI_KAFKA_MIRROR_MAKER_IMAGES: 3.6.0=quay.io/strimzi/kafka:0.42.0-kafka-3.6.0
3.6.1=quay.io/strimzi/kafka:0.42.0-kafka-3.6.1
3.6.2=quay.io/strimzi/kafka:0.42.0-kafka-3.6.2
3.7.0=quay.io/strimzi/kafka:0.42.0-kafka-3.7.0
3.7.1=quay.io/strimzi/kafka:0.42.0-kafka-3.7.1
 KUBERNETES_PORT: tcp://10.152.183.1:443
 JAVA_HOME: /usr/lib/jvm/jre-17
 DEV_CLUSTER_KAFKA_BOOTSTRAP_PORT_9092_TCP_ADDR: 10.152.183.191
 STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 120000
 KUBERNETES_SERVICE_HOST: 10.152.183.1
 DEV_CLUSTER_KAFKA_BOOTSTRAP_SERVICE_PORT_TCP_CLIENTS: 9092
 JAVA_CLASSPATH: :lib/io.strimzi.cluster-operator-0.42.0.jar:lib/io.strimzi.api-0.42.0.jar:lib/io.fabric8.kubernetes-model-apiextensions-6.13.0.jar:lib/io.strimzi.config-model-0.42.0.jar:lib/io.strimzi.operator-common-0.42.0.jar:lib/io.strimzi.crd-annotations-0.42.0.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar:lib/org.eclipse.jetty.jetty-server-9.4.54.v20240208.jar:lib/org.eclipse.jetty.jetty-http-9.4.54.v20240208.jar:lib/org.eclipse.jetty.jetty-io-9.4.54.v20240208.jar:lib/javax.servlet.javax.servlet-api-3.1.0.jar:lib/io.fabric8.openshift-client-6.13.0.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.fabric8.openshift-client-api-6.13.0.jar:lib/io.fabric8.openshift-model-clusterautoscaling-6.13.0.jar:lib/io.fabric8.openshift-model-operator-6.13.0.jar:lib/io.fabric8.openshift-model-operatorhub-6.13.0.jar:lib/io.fabric8.openshift-model-machine-6.13.0.jar:lib/io.fabric8.openshift-model-whereabouts-6.13.0.jar:lib/io.fabric8.openshift-model-monitoring-6.13.0.jar:lib/io.fabric8.openshift-model-storageversionmigrator-6.13.0.jar:lib/io.fabric8.openshift-model-tuned-6.13.0.jar:lib/io.fabric8.openshift-model-console-6.13.0.jar:lib/io.fabric8.openshift-model-config-6.13.0.jar:lib/io.fabric8.openshift-model-machineconfig-6.13.0.jar:lib/io.fabric8.openshift-model-miscellaneous-6.13.0.jar:lib/io.fabric8.openshift-model-hive-6.13.0.jar:lib/io.fabric8.openshift-model-installer-6.13.0.jar:lib/io.fabric8.kubernetes-client-6.13.0.jar:lib/io.fabric8.kubernetes-httpclient-jdk-6.13.0.jar:lib/io.fabric8.kubernetes-client-api-6.13.0.jar:lib/io.fabric8.kubernetes-model-gatewayapi-6.13.0.jar:lib/io.fabric8.kubernetes-model-resource-6.13.0.jar:lib/io.fabric8.kubernetes-model-admissionregistration-6.13.0.jar:lib/io.fabric8.kubernetes-model-autoscaling-6.13.0.jar:lib/io.fabric8.kubernetes-model-batch-6.13.0.jar:lib/io.fabric8.kubernetes-model-certificates-6.13.0.jar:lib/io.fabric8.kubernetes-model-discovery-6.13.0.jar:lib/io.fabric8.kubernetes-model-extensions-6.13.0.jar:lib/io.fabric8.kubernetes-model-flowcontrol-6.13.0.jar:lib/io.fabric8.kubernetes-model-metrics-6.13.0.jar:lib/io.fabric8.kubernetes-model-scheduling-6.13.0.jar:lib/io.fabric8.kubernetes-model-node-6.13.0.jar:lib/org.slf4j.slf4j-api-1.7.36.jar:lib/org.snakeyaml.snakeyaml-engine-2.7.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.16.2.jar:lib/io.fabric8.kubernetes-model-core-6.13.0.jar:lib/io.fabric8.kubernetes-model-common-6.13.0.jar:lib/io.fabric8.kubernetes-model-events-6.13.0.jar:lib/io.fabric8.kubernetes-model-policy-6.13.0.jar:lib/io.fabric8.kubernetes-model-rbac-6.13.0.jar:lib/io.fabric8.kubernetes-model-apps-6.13.0.jar:lib/io.fabric8.kubernetes-model-storageclass-6.13.0.jar:lib/io.fabric8.kubernetes-model-networking-6.13.0.jar:lib/io.fabric8.kubernetes-model-coordination-6.13.0.jar:lib/io.fabric8.openshift-model-6.13.0.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/com.fasterxml.jackson.core.jackson-core-2.16.2.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.16.2.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.16.2.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.16.2.jar:lib/org.yaml.snakeyaml-2.2.jar:lib/io.vertx.vertx-core-4.5.8.jar:lib/io.netty.netty-common-4.1.110.Final.jar:lib/io.netty.netty-buffer-4.1.110.Final.jar:lib/io.netty.netty-handler-4.1.110.Final.jar:lib/io.netty.netty-handler-proxy-4.1.110.Final.jar:lib/io.netty.netty-codec-http-4.1.110.Final.jar:lib/io.netty.netty-codec-http2-4.1.110.Final.jar:lib/io.netty.netty-resolver-4.1.110.Final.jar:lib/io.netty.netty-resolver-dns-4.1.110.Final.jar:lib/io.netty.netty-codec-dns-4.1.110.Final.jar:lib/io.strimzi.certificate-manager-0.42.0.jar:lib/org.apache.logging.log4j.log4j-core-2.17.2.jar:lib/org.apache.logging.log4j.log4j-api-2.17.2.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar:lib/io.strimzi.kafka-oauth-server-0.15.0.jar:lib/io.strimzi.kafka-oauth-server-plain-0.15.0.jar:lib/io.strimzi.kafka-oauth-client-0.15.0.jar:lib/io.strimzi.kafka-oauth-common-0.15.0.jar:lib/com.nimbusds.nimbus-jose-jwt-9.37.2.jar:lib/com.github.stephenc.jcip.jcip-annotations-1.0-1.jar:lib/com.jayway.jsonpath.json-path-2.9.0.jar:lib/net.minidev.json-smart-2.5.0.jar:lib/net.minidev.accessors-smart-2.5.0.jar:lib/org.eclipse.jetty.jetty-util-9.4.54.v20240208.jar:lib/org.apache.kafka.kafka-clients-3.7.1.jar:lib/com.github.luben.zstd-jni-1.5.6-3.jar:lib/org.lz4.lz4-java-1.8.0.jar:lib/org.apache.kafka.kafka-server-common-3.7.1.jar:lib/com.yammer.metrics.metrics-core-2.2.0.jar:lib/net.sf.jopt-simple.jopt-simple-5.0.4.jar:lib/org.pcollections.pcollections-4.0.1.jar:lib/org.apache.zookeeper.zookeeper-3.8.4.jar:lib/org.apache.yetus.audience-annotations-0.12.0.jar:lib/commons-io.commons-io-2.11.0.jar:lib/org.apache.zookeeper.zookeeper-jute-3.8.4.jar:lib/io.netty.netty-transport-4.1.110.Final.jar:lib/io.netty.netty-transport-native-epoll-4.1.110.Final-linux-x86_64.jar:lib/io.netty.netty-transport-native-unix-common-4.1.110.Final.jar:lib/io.netty.netty-transport-classes-epoll-4.1.110.Final.jar:lib/io.netty.netty-transport-native-epoll-4.1.110.Final-linux-aarch_64.jar:lib/io.micrometer.micrometer-core-1.12.3.jar:lib/io.micrometer.micrometer-commons-1.12.3.jar:lib/io.micrometer.micrometer-observation-1.12.3.jar:lib/org.hdrhistogram.HdrHistogram-2.1.12.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.micrometer.micrometer-registry-prometheus-1.12.3.jar:lib/io.prometheus.simpleclient_common-0.16.0.jar:lib/io.prometheus.simpleclient-0.16.0.jar:lib/io.prometheus.simpleclient_tracer_otel-0.16.0.jar:lib/io.prometheus.simpleclient_tracer_common-0.16.0.jar:lib/io.prometheus.simpleclient_tracer_otel_agent-0.16.0.jar:lib/io.vertx.vertx-micrometer-metrics-4.5.8.jar:lib/io.netty.netty-codec-4.1.110.Final.jar:lib/io.netty.netty-codec-socks-4.1.110.Final.jar
 DEV_CLUSTER_KAFKA_BOOTSTRAP_PORT_9093_TCP_PROTO: tcp
 STRIMZI_DEFAULT_CRUISE_CONTROL_IMAGE: quay.io/strimzi/kafka:0.42.0-kafka-3.7.1
 TINI_VERSION: v0.19.0
 STRIMZI_OPERATION_TIMEOUT_MS: 300000
 KUBERNETES_PORT_443_TCP_ADDR: 10.152.183.1
 DEV_CLUSTER_KAFKA_BOOTSTRAP_PORT_9092_TCP: tcp://10.152.183.191:9092
 STRIMZI_LEADER_ELECTION_LEASE_NAMESPACE: kafka
 DEV_CLUSTER_KAFKA_BOOTSTRAP_PORT_9093_TCP: tcp://10.152.183.191:9093
 STRIMZI_KAFKA_MIRROR_MAKER_2_IMAGES: 3.6.0=quay.io/strimzi/kafka:0.42.0-kafka-3.6.0
3.6.1=quay.io/strimzi/kafka:0.42.0-kafka-3.6.1
3.6.2=quay.io/strimzi/kafka:0.42.0-kafka-3.6.2
3.7.0=quay.io/strimzi/kafka:0.42.0-kafka-3.7.0
3.7.1=quay.io/strimzi/kafka:0.42.0-kafka-3.7.1
 DEV_CLUSTER_KAFKA_BOOTSTRAP_SERVICE_PORT_TCP_REPLICATION: 9091
 DEV_CLUSTER_KAFKA_BOOTSTRAP_PORT_9093_TCP_PORT: 9093
 KUBERNETES_PORT_443_TCP_PROTO: tcp
 STRIMZI_DEFAULT_USER_OPERATOR_IMAGE: quay.io/strimzi/operator:0.42.0
 KUBERNETES_SERVICE_PORT: 443
 STRIMZI_DEFAULT_KAFKA_EXPORTER_IMAGE: quay.io/strimzi/kafka:0.42.0-kafka-3.7.1
 DEV_CLUSTER_KAFKA_BOOTSTRAP_PORT_9092_TCP_PORT: 9092
 TINI_SHA256_ARM64: 07952557df20bfd2a95f9bef198b445e006171969499a1d361bd9e6f8e5e0e81
 STRIMZI_DEFAULT_KAFKA_INIT_IMAGE: quay.io/strimzi/operator:0.42.0
 DEV_CLUSTER_KAFKA_BOOTSTRAP_PORT_9092_TCP_PROTO: tcp
 DEV_CLUSTER_KAFKA_BOOTSTRAP_PORT_9091_TCP_ADDR: 10.152.183.191
 TINI_SHA256_PPC64LE: 3f658420974768e40810001a038c29d003728c5fe86da211cff5059e48cfdfde
 HOSTNAME: strimzi-cluster-operator-6948497896-t9vd4
 DEV_CLUSTER_KAFKA_BOOTSTRAP_PORT_9091_TCP: tcp://10.152.183.191:9091
 STRIMZI_KAFKA_CONNECT_IMAGES: 3.6.0=quay.io/strimzi/kafka:0.42.0-kafka-3.6.0
3.6.1=quay.io/strimzi/kafka:0.42.0-kafka-3.6.1
3.6.2=quay.io/strimzi/kafka:0.42.0-kafka-3.6.2
3.7.0=quay.io/strimzi/kafka:0.42.0-kafka-3.7.0
3.7.1=quay.io/strimzi/kafka:0.42.0-kafka-3.7.1
 DEV_CLUSTER_KAFKA_BOOTSTRAP_SERVICE_PORT_TCP_CLIENTSTLS: 9093
 STRIMZI_KAFKA_IMAGES: 3.6.0=quay.io/strimzi/kafka:0.42.0-kafka-3.6.0
3.6.1=quay.io/strimzi/kafka:0.42.0-kafka-3.6.1
3.6.2=quay.io/strimzi/kafka:0.42.0-kafka-3.6.2
3.7.0=quay.io/strimzi/kafka:0.42.0-kafka-3.7.0
3.7.1=quay.io/strimzi/kafka:0.42.0-kafka-3.7.1
 KUBERNETES_PORT_443_TCP_PORT: 443
 STRIMZI_OPERATOR_NAME: strimzi-cluster-operator-6948497896-t9vd4
 HOME: /home/strimzi
 STRIMZI_OPERATOR_NAMESPACE: kafka
 MALLOC_ARENA_MAX: 2

Answered by scholzj

Jul 20, 2024

No, the scheduling and binding is done by Kubernetes. That said, it seems strange if the PVCs are bound to the volumes on these nodes and Kubernetes says it cannot schedule the pods there because of them 🤔.

View full answer

scholzj · 2024-07-20T14:11:36Z

scholzj
Jul 20, 2024
Maintainer

If the PVC and the volumes are bound and the Pods are still Pending, you would have to check why are they Pending. The Pods should have some more details in its events. It could be something with scheduling, but the events should tell it for sure.

As for the operator error - it is hard to say from a single exception. It could be some temporary networking glitch, or lasting networking issue or something more. But that is hard to determine even from a full log.

4 replies

al66 Jul 20, 2024
Author

Yes, the events are:

Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  3m6s                 default-scheduler  0/4 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 3 node(s) didn't find available persistent volumes to bind. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  2m6s (x2 over 3m5s)  default-scheduler  0/4 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 3 node(s) didn't find available persistent volumes to bind. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.

..but the pvc's are bound:

admin@imicros-dev-1:~$ kubectl describe pvc data-0-dev-cluster-dev-pool-0 -n kafka
Name:          data-0-dev-cluster-dev-pool-0
Namespace:     kafka
StorageClass:  local-storage
Status:        Bound
Volume:        data-0-dev-cluster-dev-pool-0
Labels:        app.kubernetes.io/instance=dev-cluster
               app.kubernetes.io/managed-by=strimzi-cluster-operator
               app.kubernetes.io/name=kafka
               app.kubernetes.io/part-of=strimzi-dev-cluster
               strimzi.io/cluster=dev-cluster
               strimzi.io/component-type=kafka
               strimzi.io/kind=Kafka
               strimzi.io/name=dev-cluster-kafka
               strimzi.io/pool-name=dev-pool
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               strimzi.io/delete-claim: false
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      10Gi
Access Modes:  RWO,RWX
VolumeMode:    Filesystem
Used By:       dev-cluster-dev-pool-0
Events:        <none>

therefore I assumed, the error is coming from the operator. But maybe I'm wrong?

scholzj Jul 20, 2024
Maintainer

No, the scheduling and binding is done by Kubernetes. That said, it seems strange if the PVCs are bound to the volumes on these nodes and Kubernetes says it cannot schedule the pods there because of them 🤔.

Answer selected by al66

al66 Jul 24, 2024
Author

@scholzj
Thanks for your time!
It was just a damned typo in one of my hostnames...

Now everythings works fine as it should! :-)

scholzj Jul 24, 2024
Maintainer

Those things are sometimes the hardest to find :-). Glad you solved it.

al66 · 2024-07-20T14:50:16Z

al66
Jul 20, 2024
Author

By the way... it also kills my nodes

Events:
  Type    Reason                   Age    From        Message
  ----    ------                   ----   ----        -------
  Normal  Starting                 2m2s   kube-proxy
  Normal  Starting                 5m9s   kube-proxy
  Normal  Starting                 5m9s   kubelet     Starting kubelet.
  Normal  NodeHasSufficientMemory  5m9s   kubelet     Node imicros-dev-2 status is now: NodeHasSufficientMemory
  Normal  NodeHasNoDiskPressure    5m9s   kubelet     Node imicros-dev-2 status is now: NodeHasNoDiskPressure
  Normal  NodeHasSufficientPID     5m9s   kubelet     Node imicros-dev-2 status is now: NodeHasSufficientPID
  Normal  NodeNotReady             5m9s   kubelet     Node imicros-dev-2 status is now: NodeNotReady
  Normal  NodeAllocatableEnforced  5m8s   kubelet     Updated Node Allocatable limit across pods
  Normal  NodeReady                4m58s  kubelet     Node imicros-dev-2 status is now: NodeReady
  Normal  Starting                 2m3s   kubelet     Starting kubelet.
  Normal  NodeHasSufficientMemory  2m3s   kubelet     Node imicros-dev-2 status is now: NodeHasSufficientMemory
  Normal  NodeHasNoDiskPressure    2m3s   kubelet     Node imicros-dev-2 status is now: NodeHasNoDiskPressure
  Normal  NodeHasSufficientPID     2m3s   kubelet     Node imicros-dev-2 status is now: NodeHasSufficientPID
  Normal  NodeNotReady             2m2s   kubelet     Node imicros-dev-2 status is now: NodeNotReady
  Normal  NodeAllocatableEnforced  2m2s   kubelet     Updated Node Allocatable limit across pods
  Normal  NodeReady                2m2s   kubelet     Node imicros-dev-2 status is now: NodeReady

3 replies

scholzj Jul 20, 2024
Maintainer

I do not think there is any way how Strimzi could kill the node. If the cluster was running, I would say full disk can do that. But if the Pods are not scheduled, that should not be happeninf.

al66 Jul 20, 2024
Author

Ok, it does not really a kill the node - I think it is just the connection whith the node, which is lost for a moment.
There must be somehow a difference in the startup phase between ephemeral storage and local storage.

Starting with ephemeral storage runs smooth w/o any events or errors.

scholzj Jul 20, 2024
Maintainer

The ephemeral storage has simpler scheduling as it is just a temp dir essentially.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

Cluster failed when I try to use local storage #10365

{{title}}

Replies: 2 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Strimzi

Cluster failed when I try to use local storage #10365

al66 Jul 20, 2024

Replies: 2 comments · 7 replies

scholzj Jul 20, 2024 Maintainer

al66 Jul 20, 2024 Author

scholzj Jul 20, 2024 Maintainer

al66 Jul 24, 2024 Author

scholzj Jul 24, 2024 Maintainer

al66 Jul 20, 2024 Author

scholzj Jul 20, 2024 Maintainer

al66 Jul 20, 2024 Author

scholzj Jul 20, 2024 Maintainer

al66
Jul 20, 2024

Replies: 2 comments 7 replies

scholzj
Jul 20, 2024
Maintainer

al66 Jul 20, 2024
Author

scholzj Jul 20, 2024
Maintainer

al66 Jul 24, 2024
Author

scholzj Jul 24, 2024
Maintainer

al66
Jul 20, 2024
Author

scholzj Jul 20, 2024
Maintainer

al66 Jul 20, 2024
Author

scholzj Jul 20, 2024
Maintainer