Repair broker , zookeeper by scaling replicas ?

[jackchuong]

Hi all,
I have deployed kafka and zookeeper using strimzi-operator 0.36 in self managed k8s cluster.
k8s cluster storage provider is nfs-subdir-external-provisioner , a QNAP device as NFS server.
This is yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-kafka
  namespace: kafka
spec:
  kafka:
    version: 3.4.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    config:
      num.partitions: 3
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.4.0"
      log.cleanup.policy: compact,delete
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        class: nfs-idcnas002
        size: 10Gi
        deleteClaim: true
    resources:
      requests:
        memory: 1Gi
        cpu: "0.5"
      limits:
        memory: 4Gi
        cpu: "2"
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: kubernetes.io/hostname
                  operator: NotIn
                  values:
                  - master
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      class: nfs-idcnas002
      size: 10Gi
      deleteClaim: true
    resources:
      requests:
        memory: 1Gi
        cpu: "0.5"
      limits:
        memory: 4Gi
        cpu: "2"
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: kubernetes.io/hostname
                  operator: NotIn
                  values:
                  - master
  entityOperator:
    topicOperator: {}
    userOperator: {}

Sometimes there is a problem with the folder attached to the pvc of 1 of the 3 broker or zookeeper nodes, which causes the data in it to be corrupted and 1 pod in the 3 pod statefulsets to be corrupted as well.

Can I fix it by this way : scale down replicas to 2 , remove pod and pvc , then scale up replicas to 3 again so new pod and pvc will run ok ?
I read this article https://strimzi.io/blog/2024/01/03/prevent-broker-scale-down-if-containing-paritition-replicas/ but it only for broker and I have to upgrade strimzi operator and crd to 0.38
How about zookeeper ?

[scholzj]

I think that in the first place, you should stop using NFS. NFS is not the right file system for Kafka and ZooKeeper and has known issues. You should use block storage instead.

---

If you have a broken storage, and you want to delete and recreate it (the PVC), you can use an annotation for it. I do not remember it out of my head, but it will be described int he docs. But keep in mind that every operation like that weakens at least for some time until things sync up again your availability and reliability and can lead to data loss.

[jackchuong]

@scholzj thank you for your reply, I don't have block storage and kafka in k8s self managed is only for testing purpose so data loss is acceptable.
What are the specific steps? If I just delete the corresponding pod and pvc, it will automatically recreate a new pod and pvc, then the pod and data in the new pvc will be synchronized with the 2 running pods and pvc ?
I don't understand the "use an annotation for it" part above.

[scholzj]

thank you for your reply, I don't have block storage and kafka in k8s self managed is only for testing purpose so data loss is acceptable.

Well, up to you I guess. But as I said, NFS does not work in general. So not sure what will you really get out of such a cluster.

What are the specific steps? If I just delete the corresponding pod and pvc, it will automatically recreate a new pod and pvc, then the pod and data in the new pvc will be synchronized with the 2 running pods and pvc ? I don't understand the "use an annotation for it" part above.

You annotate the Pod you want to delete together with the PVC with the annotation and wait. in the next reconciliation, the operator will delete the Pod and the PVC. Then it will recreate the PVC and the Pod again. You need to have your provisioner running to create the new PV and bind it to the new PVC.

[jackchuong]

@scholzj thank you very much.
I found documentation here : https://strimzi.io/docs/operators/0.29.0/full/configuring , will try it next time I get problem instead of recreate cluster.