UseStrimziPodSets feature leads to "no.scale.down.node.pod.not.backed.by.controller" on GKE node autoscaler

[nils-dehn-db]

The UseStrimziPodSets feature can prevent GKE node autoscaler to scale down nodes.

The "no.scale.down.node.pod.not.backed.by.controller" error is issued and the node is not scaled down.
This can block node upgrades.

https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-autoscaler-visibility#noscaledown-node-level-reasons

The linked doc suggests to add a custom annotation "cluster-autoscaler.kubernetes.io/safe-to-evict": "true"
to allow eviction.

Is this a safe setting for Strimzi ?

[scholzj]

It is not really clear what the annotation does. So it is hard to say how safe it is or isn't I'm afraid. But you can try it - I think it is likely to be ok. You should be able to add the annotation in the Kafka CR (e.g. in `.soec.kafka.template). Since 0.32, the pods also have the controller flag set to true in their owner reference - but I'm not sure if GKE reads that or not.

[mukul-a-gupta-db]

Was this applied? is it safe we are also facing the same issue.