kafka pods will not deploy with type set to "loadbalancer"

[davehouser1]

Problem:
When I deploy strimzi as a loadbalancer type, the zookeeper pods will deploy, but the kafka pods will never deploy. I see no way to check status to see why they never deploy. The External-IPs for for each "service/my-cluster-kafka-INTEGER", and the "service/my-cluster-kafka-external-boostrap" all show PENDING but never change.

Trouble shooting:

• I can create and delete strimzi as a nodePod type with no problems. This works.
• I read through the documentation here.

This section seems interesting:
"After Strimzi creates the load balancer type Kubernetes services, the load balancers will be automatically created. Most clouds will automatically assign the load balancer some DNS name and IP addresses. These will be automatically propagated into the status section of the Kubernetes service. trimzi will read it from there and use it to configure the advertised address in the Kafka brokers."
First of all, I am running this in a lab, not using cloud, so I control all DNS and configurations manually. I have done no such configurations (that I know of) Is this the issue? How would one configure these values manually?

The document goes on to read:
"for example when for whatever reason the DNS resolution doesn’t work for your Kafka clients. In that case you can override the advertised hostnames in the Kafka custom resource."
Questions: What is performing DNS resolution, and for what? The node IPs? what is the actual dig for resolution? where is it used? This maybe apparent in a cloud env, but does not make sense to me in my lab. Also its not clear what which DNS host resolver your referring to. Do you mean the internal kubernets dns driver? (I am using Calico in my case), or my actual external DNS servers for all hosts? Either way I tried to configure "advertisedHost" with FQDN of the nodes, thried their IPs, then tried unused ips and FQDN entries for those either.

Strimzi never seems to build the kafka brokers with type set to "loadbalancer". Is there someway I can check on the status of the job? What am I missing?

Here is my configuration file I am using.

piVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls 
        port: 9093
        type: internal
        tls: true
        authentication:
          type: tls 
      - name: external
        port: 9094
        type: loadbalancer
        tls: false
        configuration:
          brokers:
          - broker: 0
            advertisedHost: kafka-broker-01.domain.internal
          - broker: 1
            advertisedHost: kafka-broker-02.domain.internal
          - broker: 2
            advertisedHost: kafka-broker-03.domain.internal
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 10Gi
        class: unity-iscsi
        deleteClaim: false
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 10Gi
      class: unity-iscsi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}

[scholzj]

When the loadbalancer type services are pending, it means the loadbalancers are not created and ready. And without that, Strimzi cannot move forward. Overriding the advertised hostname would not help you. The creation of the loadbalancers is up to Kubernetes. So if they are not creating and you think your infrastructure supports them, you need to check Kube logs or talk with your cluster admin etc. The lodbalancers in general are supported in clouds, but it could be that for example on bare-metal clusters etc. you might not have them. But it can also be that you run out of Loadbalancer quota, or do not have permissions to created them etc. So this basically depends on your infrastructure.

[davehouser1]

@scholzj thanks for the reply.
I have full control over the infrastructure, full admin rights on everything so I dont believe this is a permissions issue.

" you need to check Kube logs "
Which specific logs do you recommend I check? It seems I can only check the logs of pods. Would you recommend I check some other log? how would I go about doing this? (Share some syntax if you could). --
EDIT You can check with kubectl -n kafka get events, but it wont show you much, as I don't have a LB implemented. Researching MetalLB now.

"it can also be that you run out of Loadbalancer quota"
How can I check my loadbalancer quota?
EDIT I cant check the quota, as I don't have a load balancer implemented.

Edits: Posting before I thought, updates above. Will come back here and share more info once I have a LB set up and ready. Hopefully that will allow the pending state to go away.

[scholzj]

Ok, so, if you do not have any load balancer in your infra, then it would not work. I use MetalLB on my home cluster and it works well, so might be worth trying.

I have full control over the infrastructure, full admin rights on everything so I dont believe this is a permissions issue.

Sorry if it was not clear ... what I meant was permissions between Kubernetes and infra ... for example an AWS IAM account when using AWS etc.

How can I check my loadbalancer quota?
EDIT I cant check the quota, as I don't have a load balancer implemented.

That again meant for example AWS limits.

[davehouser1]

Got it working!
It was the loadbalancer. MetalLB fixed the problem.
I would recommend someone update the documentation here.
I am new to the world of Kubernetes so I read instructions to the letter.
I would update the instructions to include a note that most Kubernetes deployments don't come with a Loadbalancer, and one is required for this type of deployment to work. The instructions do mention cloud providers usually have a Loadbalancers which is good, however I would add another note that if you are deploying in a non cloud environment look into deploying metalLB, or some other Loadbalancing solution to get the strimzi loadbalancing deployment type to work.