[Bug]: Kafka not responding on Port 9094 with type Loadbalancer

[orbi-ci-bot]

Bug Description

I am trying to deploy Strimzi + Kafka. Following the official Documentation and the custom Example resource https://github.com/strimzi/strimzi-kafka-operator/blob/0.42.0/examples/kafka/kafka-with-node-pools.yaml. When I try to Make the type as loadbalancer to expose the service then there seems to be no response from the Application. I tried downgrading the version to 3.7.0 and the issue prevails.

[kafka@eka-dev-kafka-node-pool-1-1 kafka]$ ./bin/kafka-topics.sh --list --bootstrap-server localhost:9094

Error while executing topic command : Timed out waiting for a node assignment. Call: listTopics
[2024-08-21 10:37:37,783] ERROR org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listTopics
 (org.apache.kafka.tools.TopicCommand)

Steps to reproduce

I have the same config in two of my k8s clusters and the issue seems to be persistent on both with the below Config.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: kafka-node-pool-1
  namespace: kafka
  labels:
    strimzi.io/cluster: dev
spec:
  replicas: 2
  roles:
    - controller
    - broker
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        class: gp3
        size: 200Gi
        deleteClaim: false
---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: dev
  namespace: kafka

  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
spec:
  kafka:
    version: 3.7.1
    metadataVersion: 3.7-IV4
    template:
      externalBootstrapService:
        metadata:
          annotations:
            # NLB Support
            service.beta.kubernetes.io/aws-load-balancer-type: nlb
            service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-1234, subnet-1234
            service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
      perPodService:
        metadata:
          annotations:
            # NLB Support
            service.beta.kubernetes.io/aws-load-balancer-type: nlb
            service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-1234, subnet-1234
            service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip


    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: external
        port: 9094
        type: loadbalancer
        tls: false
   
    config:
      offsets.topic.replication.factor: 2
      transaction.state.log.replication.factor: 2
      transaction.state.log.min.isr: 1
      default.replication.factor: 2
      min.insync.replicas: 2
      # inter.broker.protocol.version: "3.7"
      num.partitions: 1
      num.recovery.threads.per.data.dir: 1
      log.retention.hours: 48
      log.segment.bytes: 500000000
      log.retention.check.interval.ms: 300000
      num.network.threads: 3
      num.io.threads: 8
      socket.send.buffer.bytes: 102400
      socket.receive.buffer.bytes: 102400
      socket.request.max.bytes: 104857600
      group.initial.rebalance.delay.ms: 0
      zookeeper.connection.timeout.ms: 6000

  zookeeper:
    replicas: 1
    storage:
      type: persistent-claim
      size: 50Gi
      deleteClaim: false
    resources:
      requests:
        cpu: 1
        memory: 1Gi
      limits:
        cpu: 2
        memory: 4Gi
    logging:
      type: inline
      loggers:
        zookeeper.root.logger: "INFO"

  entityOperator:
    topicOperator: {}
    userOperator: {}

Expected behavior

The application should respond back as same as the other port

[kafka@eka-dev-kafka-node-pool-1-1 kafka]$ ./bin/kafka-topics.sh --list --bootstrap-server localhost:9092
__consumer_offsets
topic-test

Strimzi version

0.42.0

Kubernetes version

1.29

Installation method

Helm Chart

Infrastructure

EKS

Configuration files and logs

No response

Additional context

No response

[scholzj]

• You should not run the command line tools from the Kafka node Pods. You should create your own helper Pod for running them. Otherwise you can get into issues with out of memory errors etc.
• Whether the loadbalancers are accessible from inside your Kubernetes cluster or not will depend on your networking. Do they resolve? Are they routable? Is this enabled in firewals? It has not much to do with Strimzi. Also, using an external listener from inside Kubernetes will likely incur additional costs (in terms of performance but also actual costs), so you should probably not do it.

[orbi-ci-bot]

Looks like I have to still pass the certificate manually with the cli or the code for this to work. (Which was where I was struggling cause I didn't know we had to do that)
Do we have a way where TLS can be handled without adding Certificates manually from the client's end like we do in some services? (without mTLS)

[scholzj]

You always need to pass the certificate manually unless you use a certificate signed by a trusted public CA. If you wanna do that, you can follow the link I shared above. Keep in mind that you will still need to configure TLS in your clients, even if without the certificate.

[orbi-ci-bot]

Let me try that thanks. We dont want to deal with mTLS internally and want to use mTLS for use-cases outside our Private Network.

[orbi-ci-bot]

Any ideas how to go about it?

[scholzj]

We dont want to deal with mTLS internally and want to use mTLS for use-cases outside our Private Network.

That is simple and complicated at the same point ... You can create two listeners - e.g. one on 9094 external with mTLS and one on 9095 as internal with just TLS or even without TLS. But keep in mind that when you don't use authentication on all listeners, it will make it harder to use authorization. It is not impossible, ou could for example make the ANONYMOUS user a super user to work around it. But it is a strange setup.

Also, just to be clear, the certificate signed by a public CA we discussed above helps you only for TLS. For mTLS you always need to provide at least the private key for authentication in the client.