[Bug]: Scaling down kafka cluster nodes with kraft fails

[PriyankaGG96]

Bug Description

Hello,

I am currently trying Strimzi with Kraft on AKS cluster. I am testing scaling down scenario where i have created KafkaNode pools which acts as both controller and broker. First I did a cluster rebalance to remove the broker from the cluster and then i try to scale KafkaNodePool down. In this case i am getting this error constantly in cluster operator.

Initially i had 3 nodes. I scaled it to 5 and rebalanced the cluster. After that i rebalanced to remove broker, to test scale down behaviour and scaled the replicas back to 3, then i am getting this

2024-07-19 00:05:01 INFO  KafkaRoller:397 - Reconciliation #4779(timer) Kafka(strimzi-kafka/kafka-cluster): Will temporarily skip verifying pod kafka-cluster-kafka-nodepool-2/2 is up-to-date due to ForceableProblem: Pod kafka-cluster-kafka-nodepool-2 is the active controller and there are other pods to verify first, retrying after at least 64000ms
2024-07-19 00:05:01 INFO  KafkaRoller:397 - Reconciliation #4779(timer) Kafka(strimzi-kafka/kafka-cluster): Will temporarily skip verifying pod kafka-cluster-kafka-nodepool-1/1 is up-to-date due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod kafka-cluster-kafka-nodepool-1 cannot be updated right now., retrying after at least 64000ms


io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod kafka-cluster-kafka-nodepool-0 cannot be updated right now.
2024-07-19 00:06:05 WARN  KafkaQuorumCheck:99 - Reconciliation #4779(timer) Kafka(strimzi-kafka/kafka-cluster): No valid lastCaughtUpTimestamp is found for controller 3 
2024-07-19 00:06:05 WARN  KafkaQuorumCheck:99 - Reconciliation #4779(timer) Kafka(strimzi-kafka/kafka-cluster): No valid lastCaughtUpTimestamp is found for controller 4 
2024-07-19 00:06:05 INFO  KafkaRoller:390 - Reconciliation #4779(timer) Kafka(strimzi-kafka/kafka-cluster): Could not verify pod kafka-cluster-kafka-nodepool-2/2 is up-to-date, giving up after 10 attempts. Total delay between attempts 127750ms

Regards,
Priyanka

Steps to reproduce

No response

Expected behavior

No response

Strimzi version

0.41.0

Kubernetes version

1.28.9

Installation method

No response

Infrastructure

No response

Configuration files and logs

No response

Additional context

No response

[scholzj]

Apache Kafka does not support the scaling of controller nodes right now. So if I understood you correctly and the nodes were controllers or mixed nodes, this is a known issue.