[Bug]: Strimzi Trying to change LoadbalancerClass to "null"

[purohit14]

Bug Description

Hi Team,

When we are trying to expose our strimzi kafka cluster using listener of "type: Loadbalancer" on port 9094. This creates nlb of internal type. Once the NLB's are created, strimzi operator gets into an error loop where it tries to set the LoadbalancerClass to "null". Once the external Listener config is removed from the kafka-cluster -> The nlb's get's created -> operator becomes healthy.

When the Operator is in error mode, then too the nlb's keep serving the connection to kafka cluster via the bootstrap external loadbalancer.

Error in the strimzi operator pod:

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://10.100.0.1/api/v1/namespaces/kafka/services/k8s-master-kafka-external-bootstrap. Message: Service "k8s-master-kafka-external-bootstrap" is invalid: spec.loadBalancerClass: Invalid value: "null": may not change once set. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec.loadBalancerClass, message=Invalid value: "null": may not change once set, reason=FieldValueInvalid, additionalProperties={})], group=null, kind=Service, name=k8s-master-kafka-external-bootstrap, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Service "k8s-master-kafka-external-bootstrap" is invalid: spec.loadBalancerClass: Invalid value: "null": may not change once set, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).

Steps to reproduce

1. Install operator Version 0.29.0
2. Create a kafka cluster of 3.0.0 version
3. add an external listerner with type loadbalancer.
4. Once the load balancer are created, operator starts logging error

Expected behavior

Strimzi operator must not try to reset the external bootstrap service loadbalancerclass to "null". This will make strimzi operator in healthy condition

Strimzi version

0.29.0

Kubernetes version

1.25

Installation method

YAML Files

Infrastructure

Amazon EKS

Configuration files and logs

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
labels:
project: kafka
team: infra
name: k8s-master
namespace: kafka
spec:
entityOperator:
tlsSidecar: {}
topicOperator: {}
userOperator: {}
kafka:
authorization:
superUsers:
- admin
type: simple
config:
auto.create.topics.enable: "true"
inter.broker.protocol.version: "3.0"
log.message.format.version: "3.0"
offsets.topic.replication.factor: 3
transaction.state.log.min.isr: 2
transaction.state.log.replication.factor: 3
listeners:
- authentication:
type: tls
name: tls
port: 9093
tls: true
type: internal
- authentication:
type: tls
configuration:
bootstrap:
alternativeNames:
- xyz.my-domain.com
annotations:
external-dns.alpha.kubernetes.io/hostname: xyz.my-domain.com
name: external
port: 9094
tls: true
type: loadbalancer
logging:
loggers:
kafka.root.logger.level: DEBUG
type: inline
metricsConfig:
type: jmxPrometheusExporter
valueFrom:
configMapKeyRef:
key: k8s-master-kafka-jmx-exporter-configuration.yaml
name: k8s-master-kafka-jmx-exporter-configuration
replicas: 5
storage:
type: jbod
volumes:
- id: 0
size: 200Gi
type: persistent-claim
template:
pod:
metadata:
annotations:
cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
version: 3.0.0
kafkaExporter:
groupRegex: .*
topicRegex: .*
zookeeper:
metricsConfig:
type: jmxPrometheusExporter
valueFrom:
configMapKeyRef:
key: k8s-master-zookeeper-jmx-exporter-configuration.yaml
name: k8s-master-zookeeper-jmx-exporter-configuration
replicas: 3
storage:
size: 100Gi
type: persistent-claim

Additional context

No response

[scholzj]

Your Kafka CR is not readable, but basically, you have to set the load balancer class in it:https://strimzi.io/docs/operators/latest/full/configuring.html#type-GenericKafkaListenerConfiguration-schema-reference (the class field).

[scholzj]

Then you have to upgrade Strimzi I guess.

[purohit14]

Thanks for the confirmation.

Can We create an NLB Type Loadbalancer manually and point it to the bootstrap service [ Pointing to 9093 internal TLS listener. ] ?

If yes, then by binding a Route 53 DNS entry with NLB Load balancer can be a work around for this ?

[scholzj]

In general, yes. But it is more complicated than that.

[scholzj]

PS: You can check this blog post to get a better idea how it would need to work: https://strimzi.io/blog/2020/01/02/using-strimzi-with-amazon-nlb-loadbalancers/ (as it is now quite old, not sure you can follow it word-by-word, but the core principles should be the same). This blog post series might help you understand how to do it as well: https://strimzi.io/blog/2019/04/17/accessing-kafka-part-1/. But in a short, upgrading Strimzi is probably easier.

[purohit14]

Thanks @scholzj for the quick help. I was able to create manual load balancers and connect with the kafka cluster via there hostname.

I will go through these docs for reference.