[Bug]: Connect to Kafka report error

[drivebyer]

Bug Description

I deployed Kafka on Kubernetes with a custom advertisedHost (set to the pod IP, which allows connections from outside the cluster). However, when I connect to Kafka, the client occasionally displays the following error message: Tried to send a message to a replica that is not the leader for some partition. Your metadata is out of date.

Steps to reproduce

Apply yaml:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: test-staticip
  namespace: mcamel-system
spec:
  kafka:
    image: quay.m.daocloud.io/strimzi/kafka:0.40.0-kafka-3.7.0
    listeners:
    - name: plain
      port: 9092
      type: internal
      tls: false
      configuration:
        brokers:
        - broker: 0
          advertisedHost: 10.20.114.5
        - broker: 1
          advertisedHost: 10.20.114.6
        - broker: 2
          advertisedHost: 10.20.114.7
    - name: tls
      port: 9093
      type: internal
      tls: false
    replicas: 3
    resources:
      limits:
        cpu: "1"
        memory: 1Gi
      requests:
        cpu: "1"
        memory: 1Gi
    storage:
      class: local-path
      size: 3Gi
      type: persistent-claim
    template:
      pod:
        metadata:
          annotations:
            ipam.spidernet.io/ippools: '[{"interface":"eth0","ipv4":["ippool01"]}]'
            v1.multus-cni.io/default-network: spiderpool-system/redis-cr-test
        affinity: {}
    version: 3.7.0
  kafkaExporter:
    image: quay.m.daocloud.io/strimzi/kafka:0.40.0-kafka-3.7.0
    resources:
      limits:
        cpu: "1"
        memory: 1Gi
      requests:
        cpu: 256m
        memory: 256Mi
  zookeeper:
    config:
      admin.enableServer: "false"
      autopurge.purgeInterval: "1"
      initLimit: "5"
      syncLimit: "2"
      tickTime: "2000"
    image: quay.m.daocloud.io/strimzi/kafka:0.40.0-kafka-3.7.0
    replicas: 3
    resources:
      limits:
        cpu: "1"
        memory: 1Gi
      requests:
        cpu: "1"
        memory: 1Gi
    storage:
      class: local-path
      size: 3Gi
      type: persistent-claim
    template:
      pod:
        metadata:
          annotations:
            ipam.spidernet.io/ippools: '[{"interface":"eth0","ipv4":["ippool01"]}]'
            v1.multus-cni.io/default-network: spiderpool-system/redis-cr-test
        affinity: {}

test connect to kafka, the broker list is:

	brokers := []string{
		"10.20.114.5:9092",
		"10.20.114.6:9092",
		"10.20.114.7:9092",
	}

Expected behavior

No response

Strimzi version

0.40.0

Kubernetes version

v1.29.5

Installation method

YAML files

Infrastructure

Bare-metal

Configuration files and logs

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: test-staticip
  namespace: mcamel-system
spec:
  kafka:
    image: quay.m.daocloud.io/strimzi/kafka:0.40.0-kafka-3.7.0
    listeners:
    - name: plain
      port: 9092
      type: internal
      tls: false
      configuration:
        brokers:
        - broker: 0
          advertisedHost: 10.20.114.5
        - broker: 1
          advertisedHost: 10.20.114.6
        - broker: 2
          advertisedHost: 10.20.114.7
    - name: tls
      port: 9093
      type: internal
      tls: false
    replicas: 3
    resources:
      limits:
        cpu: "1"
        memory: 1Gi
      requests:
        cpu: "1"
        memory: 1Gi
    storage:
      class: local-path
      size: 3Gi
      type: persistent-claim
    template:
      pod:
        metadata:
          annotations:
            ipam.spidernet.io/ippools: '[{"interface":"eth0","ipv4":["ippool01"]}]'
            v1.multus-cni.io/default-network: spiderpool-system/redis-cr-test
        affinity: {}
    version: 3.7.0
  kafkaExporter:
    image: quay.m.daocloud.io/strimzi/kafka:0.40.0-kafka-3.7.0
    resources:
      limits:
        cpu: "1"
        memory: 1Gi
      requests:
        cpu: 256m
        memory: 256Mi
  zookeeper:
    config:
      admin.enableServer: "false"
      autopurge.purgeInterval: "1"
      initLimit: "5"
      syncLimit: "2"
      tickTime: "2000"
    image: quay.m.daocloud.io/strimzi/kafka:0.40.0-kafka-3.7.0
    replicas: 3
    resources:
      limits:
        cpu: "1"
        memory: 1Gi
      requests:
        cpu: "1"
        memory: 1Gi
    storage:
      class: local-path
      size: 3Gi
      type: persistent-claim
    template:
      pod:
        metadata:
          annotations:
            ipam.spidernet.io/ippools: '[{"interface":"eth0","ipv4":["ippool01"]}]'
            v1.multus-cni.io/default-network: spiderpool-system/redis-cr-test
        affinity: {}

Additional context

In my environment, I added the following annotations:

          annotations:
            ipam.spidernet.io/ippools: '[{"interface":"eth0","ipv4":["ippool01"]}]'
            v1.multus-cni.io/default-network: spiderpool-system/redis-cr-test

These annotations are passed to the pod. The CNI tool (Spiderpool) allocates an IP to the pod with two key features:

• The IP remains unchanged after the pod restarts.
• The IP is accessible from outside the Kubernetes cluster.

[scholzj]

It is quite hard to comment without full logs and without fully understanding your environments. The error suggests your client has old metadata, so it can be just that. Or it could be that you did something wrong when trying to expose it the way you did.