Skip to content

Latest commit

 

History

History
163 lines (139 loc) · 5.51 KB

README.md

File metadata and controls

163 lines (139 loc) · 5.51 KB
Raw

CI Hits License: MIT Renovate enabled

Confluent Kafka Cloud Go example

This example shows how to create a Confluent Kafka Cloud Producer/Consumer in Go and deploy it to Kubernetes.

Requirements

  • Linux (Ubuntu) or Mac OS

  • gvm Go 1.23

    gvm install go1.23 --prefer-binary --with-build-tools --with-protobuf
gvm use go1.23 --default

  • [Cross compilation on Ubuntu with CGO ] Optional

    Install LLVM Compiler Infrastructure, release 17

    ./scripts/install-clang-17-ubuntu.sh

    Install libraries for cross compilation (Windows, etc.)

    ./scripts/install-cross-libs-ubuntu.sh

    Install osxcross for MacOS cross compilation

    ./scripts/install-osxcross-ubuntu.sh

  • [Build ARM Images on x86 Hosts] Optional

    sudo apt-get update && sudo apt-get install -y --no-install-recommends qemu-user-static binfmt-support
update-binfmts --enable qemu-arm
update-binfmts --display qemu-arm
docker buildx build --platform linux/arm64 --file Dockerfile.consumer -t kafka-confluent-go-consumer:latest .

  • Confluent Kafka CLI and tools

    curl -sL --http1.1 https://cnfl.io/cli | sh -s -- latest
confluent update
# log in to a Confluent Cloud organization
confluent login --save

  • docker Optional

  • GoReleaser Optional

  • kubectl Optional

Configure Confluent Kafka Go client

Following steps are required.

  • Create Environment and Cluster - https://confluent.cloud/home

  • Export Confluent Environment ID as CONFLUENT_ENV

    xdg-open https://confluent.cloud/environments
export CONFLUENT_ENV=

  • Export Confluent cluster ID as CONFLUENT_CLUSTER

    xdg-open https://confluent.cloud/environments/$CONFLUENT_ENV/clusters
export CONFLUENT_CLUSTER=

  • Select Environment

    confluent environment use $CONFLUENT_ENV

  • Select Cluster

    confluent kafka cluster use $CONFLUENT_CLUSTER
confluent login --save

  • Create a new API key and secret pair

    confluent api-key create --resource $CONFLUENT_CLUSTER

  • Export previously created KEY and SECRET

    export CONFLUENT_API_KEY=
export CONFLUENT_API_SECRET=

  • Use an API key and secret in the CLI

    confluent api-key use $CONFLUENT_API_KEY --resource $CONFLUENT_CLUSTER

  • Export Confluent Kafka Cluster Bootstrap Server - Cluster settings -> Endpoints -> Bootstrap server, create kafka.properties from the template and add it to the git repository.

    xdg-open https://confluent.cloud/environments/$CONFLUENT_ENV/clusters/$CONFLUENT_CLUSTER/settings/kafka
export CONFLUENT_BOOTSTRAP_SERVER=
sed -e "s%BTSTRP%$CONFLUENT_BOOTSTRAP_SERVER%g" ./tmpl/kafka.properties.tmpl > ./kafka.properties

  • Create .env file

    sed -e "s%BTSTRP%$CONFLUENT_BOOTSTRAP_SERVER%g" -e "s%APIKEY%$CONFLUENT_API_KEY%g" -e "s%APISECRET%$CONFLUENT_API_SECRET%g" ./tmpl/.env.tmpl > ./.env

  • Create Confluent Kafka topic

    confluent kafka topic create test-topic

Test Confluent Kafka topic

confluent kafka topic list
confluent kafka topic produce test-topic
confluent kafka topic consume -b test-topic

Deploy Confluent Kafka Consumer to Kubernetes

To deploy on Kubernetes create configmap and secret:

# create configmap from Kafka properties file
kubectl create configmap kafka-config --from-file kafka.properties -o yaml --dry-run=client >./k8s/cm.yaml

# store $CONFLUENT_API_KEY and $CONFLUENT_API_SECRET as k8s secret
sed -e"s%USR%`echo -n $CONFLUENT_API_KEY|base64 -w0`%g" -e "s%PWD%`echo -n $CONFLUENT_API_SECRET|base64 -w0`%g" ./tmpl/sc.yaml.tmpl > ./k8s/sc.yaml

and then run

make k8s-deploy

Run Confluent Kafka Consumer Docker image locally

make consumer-image-run

Run Confluent Kafka Producer locally

make runp

Help

Commands :
help                 - List available tasks
clean                - Cleanup
build                - Build
test                 - Run tests
update               - Update dependency packages to latest versions
get                  - Download and install dependency packages
release              - Create and push a new tag
version              - Print current version(tag)
consumer-image-build - Build Consumer Docker image
consumer-image-run   - Run a Docker image
consumer-image-stop  - Run a Docker image
runp                 - Run producer
runc                 - Run consumer
k8s-deploy           - Deploy to Kubernetes
k8s-undeploy         - Undeploy from Kubernetes