[Bug]: kafkaNodePool migration fails when using gp3 storage class

[mafitconsulting]

Bug Description

Following the documented instructions here. I am trying to migrate an existing cluster to use KafkaNodePools. The storage class which is currently used is gp3 pv. I create a kafkaNodePool manifest with following storage configuration

 storage:
   type: jbod (also tried persistent-claim)
   volumes:
   - class: gp3 (also tried gp3-ebs)
     id: 0
     size: 100Gi
     type: persistent-claim

This configuration matches the existing storage configuration in the kafka resource

 Storage:
    Class:         gp3 
    Delete Claim:  false
    Size:          100Gi
    Type:          persistent-claim 

So, same storage class, same size and same type. I have the following annotations on the kafka resource

strimzi.io/node-pools: enabled

Steps to reproduce

1. Create KafkaNodePools resource
2. Apply KafkaNodePools resource
3. Update kafka resource with strimzi.io/node-pools: enabled annotation
4. See the following error

 Message:               The desired Kafka storage configuration in the KafkaNodePool resource kafka-clusters/kafka contains changes which are not allowed. As a result, all storage changes will be ignored. Use DEBUG level logging for more information about the detected changes.
    Reason:                KafkaStorage
    Status:                True
    Type:                  Warning
    Last Transition Time:  2024-07-04T18:59:17.385367162Z
    Status:                True
    Type:                  Ready

Expected behavior

Expected behaviour is for KafkaNodePools to take over the storage management of the Kafka node and not see the error message

Strimzi version

Operator 0.41 Kafka 3.7.0

Kubernetes version

1.28

Installation method

Helm Chart

Infrastructure

EKS

Configuration files and logs

[
report-05-07-2024_08-44-23.zip
](url)

Additional context

No response

[scholzj]

The migration instructions tell you to use the exactly same storage configuration in the KafkaNodePool as you have in the Kafka resource. Now your Kafka resource uses this:

    storage:
      class: gp3
      deleteClaim: false
      size: 100Gi
      type: persistent-claim

While your KafkaNodePool resource uses this:

  storage:
    type: jbod
    volumes:
    - class: gp3-ebs
      id: 0
      size: 100Gi
      type: persistent-claim

So, apart from changing the storage class, you also changed the storage type to JBOD storage. You need to do the migration with the exact same configuration.

If you want to move to a different storage class or to use JBOD storage, you should first migrate to the node pools using the old storage, than create a new node pool with the new storage configuration, move all data to the new nodes and then delete the old nodes.

[mafitconsulting]

Hey thanks for reply. I also tried (as documented in the ticket) using persistent-claim as type instead of JBOD and gp3 instead of gp3-ebs, but had the same issue. its also says deleteClaim is optional

storage:
    type: persistent-claim
    volumes:
    - class: gp3
      id: 0
      size: 100Gi
      type: persistent-claim
      deleteClaim: false

I even tried dropping the ID to try get perfect match

[mafitconsulting]

Actually, this configuration worked

 spec:
    replicas: 3
    roles:
    - broker
      storage:
        class: gp3
        size: 100Gi
        type: persistent-claim

Completely ditching volumes.

[scholzj]

Yeah, as I said, you have to use the same configuration. The issue was the type: jbod you used not the deleteClaim.

[mafitconsulting]

Closing