Running Strimzi on amd64 Kubernetes Master Node and arm64 Worker Nodes

[bitvijays]

o/ Hope you are doing well and thank you for the support.

I have a environment where

• Kubernetes Master is installed in Ubuntu 20.04 (amd64) in the cloud.
• Kubernetes Worker Node is Raspberry Pi 4 (arm64) with 4/8 GB RAM.

Just wanted to confirm the changes required in 060-Deployment-strimzi-cluster-operator.yaml file.

Now, we built the strimzi images on Raspberry Pi

docker image ls
REPOSITORY            TAG                 IMAGE ID            CREATED             SIZE
strimzi/test-client   build-kafka-2.5.0   731ea98ca092        13 hours ago        781MB
strimzi/test-client   latest              731ea98ca092        13 hours ago        781MB
strimzi/kafka         build-kafka-2.5.0   f7371987a1e6        13 hours ago        781MB
strimzi/kafka         latest              f7371987a1e6        13 hours ago        781MB
strimzi/test-client   build-kafka-2.4.0   8caf08c424bb        14 hours ago        780MB
strimzi/kafka         build-kafka-2.4.0   05242faab5f2        14 hours ago        780MB
strimzi/test-client   build-kafka-2.4.1   7f613025f0b6        14 hours ago        780MB
strimzi/kafka         build-kafka-2.4.1   ea0d22b8b004        14 hours ago        780MB
strimzi/jmxtrans      latest              1bdf4288c0d0        14 hours ago        685MB
strimzi/operator      latest              06e88ece8a28        14 hours ago        648MB
strimzi/base          latest              f4ac3eb7c9bd        14 hours ago        602MB
<none>                <none>              b744b4d52dca        4 days ago          610MB
centos                7                   e5df02c43685        7 weeks ago         301MB

Now, as we are using amd64 Kubernetes Master Node, I believe we should not make any changes for

            - name: STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE
              value: strimzi/operator:0.20.1
            - name: STRIMZI_DEFAULT_USER_OPERATOR_IMAGE
              value: strimzi/operator:0.20.1
            - name: STRIMZI_DEFAULT_KAFKA_INIT_IMAGE
              value: strimzi/operator:0.20.1
            - name: STRIMZI_DEFAULT_KAFKA_BRIDGE_IMAGE
              value: strimzi/kafka-bridge:0.19.0
            - name: STRIMZI_DEFAULT_JMXTRANS_IMAGE
              value: strimzi/jmxtrans:0.20.1

and change the below to the arm64 image

            - name: STRIMZI_DEFAULT_TLS_SIDECAR_ENTITY_OPERATOR_IMAGE
              value: strimzi/kafka:0.20.1-kafka-2.6.0
            - name: STRIMZI_DEFAULT_KAFKA_EXPORTER_IMAGE
              value: strimzi/kafka:0.20.1-kafka-2.6.0
            - name: STRIMZI_DEFAULT_CRUISE_CONTROL_IMAGE
              value: strimzi/kafka:0.20.1-kafka-2.6.0
            - name: STRIMZI_DEFAULT_TLS_SIDECAR_CRUISE_CONTROL_IMAGE
              value: strimzi/kafka:0.20.1-kafka-2.6.0
            - name: STRIMZI_KAFKA_IMAGES
              value: |
                2.5.0=strimzi/kafka:0.20.1-kafka-2.5.0
                2.5.1=strimzi/kafka:0.20.1-kafka-2.5.1
                2.6.0=strimzi/kafka:0.20.1-kafka-2.6.0
            - name: STRIMZI_KAFKA_CONNECT_IMAGES
              value: |
                2.5.0=strimzi/kafka:0.20.1-kafka-2.5.0
                2.5.1=strimzi/kafka:0.20.1-kafka-2.5.1
                2.6.0=strimzi/kafka:0.20.1-kafka-2.6.0
            - name: STRIMZI_KAFKA_CONNECT_S2I_IMAGES
              value: |
                2.5.0=strimzi/kafka:0.20.1-kafka-2.5.0
                2.5.1=strimzi/kafka:0.20.1-kafka-2.5.1
                2.6.0=strimzi/kafka:0.20.1-kafka-2.6.0
            - name: STRIMZI_KAFKA_MIRROR_MAKER_IMAGES
              value: |
                2.5.0=strimzi/kafka:0.20.1-kafka-2.5.0
                2.5.1=strimzi/kafka:0.20.1-kafka-2.5.1
                2.6.0=strimzi/kafka:0.20.1-kafka-2.6.0
            - name: STRIMZI_KAFKA_MIRROR_MAKER_2_IMAGES
              value: |
                2.5.0=strimzi/kafka:0.20.1-kafka-2.5.0
                2.5.1=strimzi/kafka:0.20.1-kafka-2.5.1
                2.6.0=strimzi/kafka:0.20.1-kafka-2.6.0

[scholzj]

The STRIMZI_DEFAULT_KAFKA_INIT_IMAGE is running as a init container in the Kafka pods. It is used when you enable rack-awareness in the Kafka cluster or when you use Nodeports for access to the cluster. So ig you use any of those, it would need to point to arm64 image as well. On the other hand, the STRIMZI_DEFAULT_TLS_SIDECAR_ENTITY_OPERATOR_IMAGE is a sidecar image in the pod with the user and topic operators. So that needs to have the same platform as they do - in the setup described above that would be amd64.

One thing I'm wondering about is what is the expected advantage from running the operators on the master nodes in the cloud and not on the RPi worker nodes. Is it just because you have more resources on the master node which is not so limited as the RPi workers? Following the same logic, one could probably easily run for example Cruise Control on the master nodes in the cloud as well. So it is not completely clear to me why did you decided differently there.

What I'm wondering about is how many problems it would cause given the operators talk with the Kafka cluster regularly (and for example the TO is basically in permanent contact). I do not have any experience with this setup - but I'm wondering whether running them somewhere else can negatively impact how the work. But I guess you would need to try it.

[scholzj]

I think the plan is good in general. But the unstable connection could be an issue - I do not know that because I never tried it, so I'm just speculating. You would need to try it I guess. The challenges I would expect:

• The operator needs connection to the operands - Kafkas, Zookeepers etc. If the connections would not work, the operators will be affected (for example by failing reconciliations in CO and UO and I would be also very careful about using the Topic Operator in this setup).
• What happens when the Master node looses connection to the RPi worker nodes? The Pods will have some unknown state which might confuse the operator and/or Kube.

As I said - not saying this would not work ... just saying what to be careful about and what to test.

I definitely heard from users doing something similar in the past. Their Kafka architecture was very similar. But AFAIK they were always using a separate Kube clusters. I.e. the edge was not just a worker node but a whole Kube cluster running not just Kafka but also the operator. This avoids the networking challenges in the operators since they run on the same node as the cluster. But I guess it has other challenges (such as more stuff which needs to run on the edge etc.).

PS: Out of curiosity ... what kind of storage would you use on the edge nodes?

[bitvijays]

@scholzj o/ Hope you are doing well and had a good weekend. Thanks you for the support. Have multiple queries:

The setup is:

• One cloud node (amd64) with public IP address running Kubernetes master .
• Multiple Raspberry Pi 4 (arm64) connected to the Kubernetes master.
• Assumption that the network connection is stable and pods are able to recover even if network is disrupted.
• Kafka cluster deployed on RPi would be using storage type as ephemeral. (Currently for deployment/testing purposes).

Q1: Is there a way to deploy Kafka cluster as a Kubernetes Daemonset ? so that it runs one Kafka cluster (one Kafka and one Zookeeper) on each of the Raspberry Pi? The requirement is that, we need each Raspberry Pi running one Kafka instance so that they can store data coming from the sensors.

Q2: Currently, the Kafka cluster is exposed using NodePort which allows access to the Kafka cluster using Kubernetes services. Is there a way we can expose Kafka to the node interface? using hostPort? as the sensors needs to connect to the Kafka and need kafka port accessible locally?

[scholzj]

Re Q1: That is not possible with Strimzi. Also, I do not think it is so easy with Kafka. The producer is always producing data to the leading replica of a partition. So you would need to build some complicated tooling how to manage topics / partitions on the DeamonSet to be able to actually produce the sensor data locally only. So I do not think this makes sense with Kafka.

Re Q2: No, that is not possible.

[bitvijays]

@scholzj My apologies. Regarding Q1: I meant broker instead of cluster

• Is there a way to deploy local Kafka broker as a Kubernetes Daemonset ? so that it runs one Kafka broker/server (one Kafka and one Zookeeper) on each of the Raspberry Pi?

I am trying to implement A prototype of utilizing Apache Kafka and Lightweight M2M protocol as the backbone for cloud/edge IoT integration

[scholzj]

Yeah, that is how I understood it. But Strimzi does not allow it. You could of course do it with Kafka in general - without Strimzi. But keep in mind the issues I described above. Kafka is not designed to be run this way. If something, then the more natural way would be to have a super small Kafka cluster with just one Zoo node and one Kafka node and its own Mirror Maker which would be all constraint to the single worker. But not running as DeamonSet => too many complications and overhead needed to build the partition / topic management to be able to produce locally.