How to Achieve HA with Replicas on Different Storage Classes in Strimzi Kafka Cluster on K8s?

[willyaako]

Hello,

I would like to deploy a Kafka cluster on my Kubernetes (K8s) cluster. This K8s cluster has 3 storage classes for geo-redundancy. I have created 3 KafkaNodePools and specified the storage class names for each. Each node pool has created 3 brokers.

To achieve high availability (HA) for the topics, I would like to ask if it is possible to specify replicas to be placed on brokers with different storage classes when creating a topic.

Here is my YAML configuration:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: {NODE-DC1} ~ {NODE-DC3}
  labels:
    strimzi.io/cluster: my-cluster
spec:
  replicas: 3
  roles:
    - controller
    - broker
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 10Gi
        deleteClaim: true
        class: {DC1}~{DC3}
        kraftMetadata: shared

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
spec:
  kafka:
    version: 3.7.0
    metadataVersion: 3.7-IV4
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false

    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2

[scholzj]

For that, you should use the rack awareness feature. Check the docs for more details. If you have already a running cluster, you should also use Cruise Control to redistribute any existing topics.

On a side note, 9 controller nodes are probably not a good idea. You should likely have only 3 controller nodes and 9 brokers, or 3 mixed nodes and 6 broker only nodes or something like that.

[willyaako]

Thanks for the reply,

I understand I need permission to get the node list to use the rack awareness feature. However, our Kubernetes infrastructure operator does not allow to create cluster role binding permissions. Because of this, I removed the create cluster role binding permission from the strimzi-cluster-operator-global role, which prevents me from creating a cluster role binding.

Can I pre-create this cluster role binding?

[scholzj]

No, you need to allow the operator to list the nodes and create the cluster role bindings. I do not think there is any reason for not allowing this. The operator only needs to be able to get/list the nodes which normally does not mean any security risk.

[willyaako]

They believe that granting "create" permissions too easily can lead to management difficulties. However, I will try to communicate with them to obtain the necessary permissions.

Thank you very much for your response and guidance. I appreciate your help.