Bundle librdkafka to support linux/arm64 builds #805

JimMale-tt · 2022-06-23T16:10:31Z

Description

Applications that uses confluent-kafka-go cannot be built on linux/arm64 (aka aarch64) with the bundled librdkafka library.

This includes:

Building an application natively on linux/arm64
- Eg Linux on Raspberry Pi 4, Linux on Apple Silicon, Linux on ARM-based AWS instances
Building an application in a Docker container on a darwin/arm64 or linux/arm64 host
- Eg Someone using Docker on an Apple Silicon-based Mac
Building an application in a multi-arch Docker container that includes a linux/arm64 target
- Eg Someone building a Docker container for their team to use, with a mixture of Intel-based and ARM-based Macs on the team

This is caused by the included librdkafka library not supporting Linux arm64 (neither glibc-based nor musl-based).

Two other issues have previously been opened that mention ARM support: #591 and #565 .

This issue has been opened in relation to work I'm doing for my employer.

How to reproduce

Take the example application from the docs...

main.go

package main

import (
	"fmt"

	"github.com/confluentinc/confluent-kafka-go/kafka"
)

func main() {

	c, err := kafka.NewConsumer(&kafka.ConfigMap{
		"bootstrap.servers": "localhost",
		"group.id":          "myGroup",
		"auto.offset.reset": "earliest",
	})

	if err != nil {
		panic(err)
	}

	c.SubscribeTopics([]string{"myTopic", "^aRegex.*[Tt]opic"}, nil)

	for {
		msg, err := c.ReadMessage(-1)
		if err == nil {
			fmt.Printf("Message on %s: %s\n", msg.TopicPartition, string(msg.Value))
		} else {
			// The client will automatically try to recover from all errors.
			fmt.Printf("Consumer error: %v (%v)\n", err, msg)
		}
	}

	c.Close()
}

Add a Makefile...

Makefile

# build a Docker container for a single architecture
singlearch:
	docker build . -t arm64-issue:latest

# build a Docker container for linux/amd64 and linux/arm64
multiarch:
	docker buildx build --platform linux/amd64,linux/arm64 .

# build the application. Don't invoke this directly
buildapplication:
	CGO_ENABLED=1 GOOS=linux go build -o arm64-issue .

clean:
	rm -f arm64-issue
	docker rmi -f arm64-issue:latest

Add a Dockerfile...

Dockerfile

FROM golang:1.18 AS builder


ARG TARGETPLATFORM
ARG BUILDPLATFORM
RUN echo "I am running on $BUILDPLATFORM, building for $TARGETPLATFORM"

WORKDIR /go/src/github.com/JimMale-tt/arm64-issue
COPY . ./

RUN make buildapplication

Initialize go mod with go mod init, and go mod vendor.
Attempt to build the example application in a regular, single-architecture docker container. This works!

$ make singlearch
docker build . -t arm64-issue:latest
Sending build context to Docker daemon   46.2MB
Step 1/7 : FROM golang:1.18 AS builder
 ---> 76199a964a3f
Step 2/7 : ARG TARGETPLATFORM
 ---> Using cache
 ---> 767bef576898
Step 3/7 : ARG BUILDPLATFORM
 ---> Using cache
 ---> 1f52606ebf36
Step 4/7 : RUN echo "I am running on $BUILDPLATFORM, building for $TARGETPLATFORM"
 ---> Using cache
 ---> 98cfc34e0dc2
Step 5/7 : WORKDIR /go/src/github.com/JimMale-tt/arm64-issue
 ---> Using cache
 ---> ab20e422261a
Step 6/7 : COPY . ./
 ---> 2b7fb65ec12a
Step 7/7 : RUN make buildapplication
 ---> Running in e505ee7559a0
CGO_ENABLED=1 GOOS=linux go build -o arm64-issue .
Removing intermediate container e505ee7559a0
 ---> 001ec4376cb0
Successfully built 001ec4376cb0
Successfully tagged arm64-issue:latest

Attempt to build the example application in a multi-architecture docker container. This fails :(

$ make multiarch
docker buildx build --platform linux/amd64,linux/arm64 .
[+] Building 127.2s (15/15) FINISHED                                                                                                    
 => [internal] load .dockerignore                                                                                                  0.0s
 => => transferring context: 370B                                                                                                  0.0s
 => [internal] load build definition from Dockerfile                                                                               0.0s
 => => transferring dockerfile: 266B                                                                                               0.0s
 => [linux/amd64 internal] load metadata for docker.io/library/golang:1.18                                                         0.3s
 => [linux/arm64 internal] load metadata for docker.io/library/golang:1.18                                                         0.4s
 => [internal] load build context                                                                                                  0.0s
 => => transferring context: 6.70kB                                                                                                0.0s
 => [linux/arm64 1/5] FROM docker.io/library/golang:1.18@sha256:1c3d22f95ce57821fff1dcd857c54809ea62f33634e2696e0d623e077c97bb8f   0.0s
 => => resolve docker.io/library/golang:1.18@sha256:1c3d22f95ce57821fff1dcd857c54809ea62f33634e2696e0d623e077c97bb8f               0.0s
 => [linux/amd64 1/5] FROM docker.io/library/golang:1.18@sha256:1c3d22f95ce57821fff1dcd857c54809ea62f33634e2696e0d623e077c97bb8f   0.0s
 => => resolve docker.io/library/golang:1.18@sha256:1c3d22f95ce57821fff1dcd857c54809ea62f33634e2696e0d623e077c97bb8f               0.0s
 => CACHED [linux/arm64 2/5] RUN echo "I am running on linux/amd64, building for linux/arm64"                                      0.0s
 => CACHED [linux/arm64 3/5] WORKDIR /go/src/github.com/JimMale-tt/arm64-issue                                                     0.0s
 => CACHED [linux/amd64 2/5] RUN echo "I am running on linux/amd64, building for linux/amd64"                                      0.0s
 => CACHED [linux/amd64 3/5] WORKDIR /go/src/github.com/JimMale-tt/arm64-issue                                                     0.0s
 => [linux/arm64 4/5] COPY . ./                                                                                                    0.0s
 => [linux/amd64 4/5] COPY . ./                                                                                                    0.0s
 => ERROR [linux/arm64 5/5] RUN make buildapplication                                                                            126.8s
 => [linux/amd64 5/5] RUN make buildapplication                                                                                   13.1s
------                                                                                                                                                                                                   
 > [linux/arm64 5/5] RUN make buildapplication:                                                                                                                                                          
#0 0.149 CGO_ENABLED=1 GOOS=linux go build -o arm64-issue .                                                                                                                                              
#0 0.438 go: downloading github.com/confluentinc/confluent-kafka-go v1.9.0                                                                                                                               
#0 126.8 # github.com/confluentinc/confluent-kafka-go/kafka                                                                                                                                              
#0 126.8 /usr/bin/ld: /go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_error.o): Relocations in generic ELF (EM: 62)               
#0 126.8 /usr/bin/ld: /go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_error.o): Relocations in generic ELF (EM: 62)
#0 126.8 /usr/bin/ld: /go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a: error adding symbols: file in wrong format
#0 126.8 collect2: error: ld returned 1 exit status
#0 126.8 make: *** [Makefile:10: buildapplication] Error 2
------
WARNING: No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
Dockerfile:11
--------------------
   9 |     COPY . ./
  10 |     
  11 | >>> RUN make buildapplication
  12 |     
--------------------
error: failed to solve: process "/bin/sh -c make buildapplication" did not complete successfully: exit code: 2
make: *** [buildcontainer] Error 1

When running make multiarch, we can see Docker attempting to build two versions of the container:
One for linux/arm64, and one for linux/amd64. The linux/amd64 build completes successfully, but the linux/arm64 build fails.

This is because ld can't link the application against librdkafka_glibc_linux.a since it was bundled with
confluent-kafka-go, and does not include a version for the linux/arm64 build target.

Potential Solutions

Users can build librdkafka for arm64 themselves
- This takes a long time, especially if done with an emulator
- This duplicates effort for every build of a Docker container
confluent-kafka-go could include a linux/arm64 version of librdkafka

Checklist

Please provide the following information:

confluent-kafka-go and librdkafka version (LibraryVersion()): N/A, tested with 1.9.0
Apache Kafka broker version: N/A
Client configuration: ConfigMap{...} N/A
Operating system: Linux on arm64
Provide client logs (with "debug": ".." as necessary) N/A
Provide broker log excerpts N/A
Critical issue

The text was updated successfully, but these errors were encountered:

mhowlett · 2022-08-18T00:48:26Z

Hi @JimMale-tt - you will be pleased to know we have an internal user who needs really needs this, so you can expect a bundled linux/arm64 build in the not too distant future. However, it's not currently at the top of the priority queue.

Josh-ShubhamVyas · 2022-08-24T12:25:43Z

@JimMale-tt @mhowlett
Any work around for this issue. If you can respond with it would be very helpful. Stuck with this for last 1 week.
Facing the same issue when running go build musl go-file. Works in amd64 and fails in arm64
Error returned -
#66 [linux/arm64 build-env 19/19] RUN go build -tags musl consumers.go
968 #66 167.0 # github.com/confluentinc/confluent-kafka-go/kafka
969 #66 167.0 /usr/lib/gcc/aarch64-alpine-linux-musl/8.3.0/../../../../aarch64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_musl_linux.a(rdkafka_error.o): Relocations in generic ELF (EM: 62)
970 #66 167.0 /usr/lib/gcc/aarch64-alpine-linux-musl/8.3.0/../../../../aarch64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_musl_linux.a(rdkafka_error.o): Relocations in generic ELF (EM: 62)
971 #66 167.0 /usr/lib/gcc/aarch64-alpine-linux-musl/8.3.0/../../../../aarch64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_musl_linux.a: error adding symbols: file in wrong format
972 #66 167.0 collect2: error: ld returned 1 exit status
973 #66 ERROR: process "/bin/sh -c go build -tags musl consumers.go" did not complete successfully: exit code: 2

domgolonka · 2022-09-15T12:53:44Z

I am getting the same error on my M1 with -tags musl.

kimgr · 2022-09-30T17:08:52Z

I hesitate to link it if it's unrelated, but does purl rquest numero 845 work toward solving this? Thanks.

sagimedina1 · 2022-11-07T12:36:32Z

The same problem when building with docker buildx build --platform linux/arm64 .

 > [builder 10/10] RUN go build -tags musl --ldflags "-extldflags -static" ./cmd/balance-service:
 => ERROR [builder 10/10] RUN go build -tags musl --ldflags "-extldflags -static" ./cmd/balance-service                                                                                                                           11.7s
------
 > [builder 10/10] RUN go build -tags musl --ldflags "-extldflags -static" ./cmd/balance-service:
#0 11.63 # github.com/porticohomes/balance-service/cmd/balance-service
#0 11.63 /usr/local/go/pkg/tool/linux_arm64/link: running gcc failed: exit status 1
#0 11.63 /usr/lib/gcc/aarch64-alpine-linux-musl/11.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_musl_linux.a(rdkafka_error.o): Relocations in generic ELF (EM: 62)
#0 11.63 /usr/lib/gcc/aarch64-alpine-linux-musl/11.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_musl_linux.a(rdkafka_error.o): Relocations in generic ELF (EM: 62)
#0 11.63 /usr/lib/gcc/aarch64-alpine-linux-musl/11.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_musl_linux.a: error adding symbols: file in wrong format
#0 11.63 collect2: error: ld returned 1 exit status
#0 11.63 
------
Dockerfile:26
--------------------
  24 |     COPY . .
  25 |     
  26 | >>> RUN go build -tags musl --ldflags "-extldflags -static" ./cmd/balance-service
  27 |     
  28 |     FROM --platform=$BUILDPLATFORM scratch AS runner
--------------------
error: failed to solve: process "/bin/sh -c go build -tags musl --ldflags \"-extldflags -static\" ./cmd/balance-service" did not complete successfully: exit code: 2

Dockerfile:

FROM --platform=$BUILDPLATFORM golang:1.19-alpine AS builder

ARG TARGETPLATFORM
ARG BUILDPLATFORM
ARG TARGETOS
ARG TARGETARCH

ENV PATH="/go/bin:${PATH}"
ENV GO111MODULE=on
ENV CGO_ENABLED=1
ENV GOOS=${TARGETOS}
ENV GOARCH=${TARGETARCH}

WORKDIR /go/src

COPY go.mod .
COPY go.sum .
RUN go mod download

RUN apk -U add ca-certificates
RUN apk update && apk upgrade && apk add pkgconf git bash build-base sudo
RUN git clone https://github.com/edenhill/librdkafka.git && cd librdkafka && ./configure --prefix /usr && make && make install

COPY . .

RUN go build -tags musl --ldflags "-extldflags -static" ./cmd/some-service

PrasanthV454 · 2022-11-15T14:22:43Z

@JimMale-tt @sagimedina1
Please check the RC-3. You should be able to build it successfully. It will be shipped with next release linked to a librdkafka release

ferranorriols · 2022-11-22T09:51:08Z

Hi @PrasanthV454, do you have an idea when is this new next release going to be published? Thank you!

PrasanthV454 · 2022-12-06T06:04:57Z

Hi @ferranorriols, we are targeting for Jan 1st week, 2023. The version will be 2.0.0

ferranorriols · 2022-12-06T15:57:52Z

That's great @PrasanthV454, thanks for letting me know.

doyelese · 2023-01-09T23:29:01Z

Hi @ferranorriols, we are targeting for Jan 1st week, 2023. The version will be 2.0.0

Is the plan to still release this early Jan?

elmoamedeo · 2023-02-13T03:21:51Z

any updates at all on this?

daybarr · 2023-02-13T08:45:04Z

@elmoamedeo this was fixed in the 2.0.2 release

tiagosatur · 2023-02-15T14:37:20Z

When will this v 2.02 be published? I can see only v1.9.3-RC3 available on https://pkg.go.dev/github.com/confluentinc/[email protected]?tab=versions

daybarr · 2023-02-15T15:36:48Z

https://pkg.go.dev/github.com/confluentinc/confluent-kafka-go/[email protected]

jaime0815 mentioned this issue Jul 20, 2022

Upgrade confluent kafka go version milvus-io/milvus#18293

Merged

mhowlett added the enhancement label Aug 18, 2022

mhowlett added the HIGH label Nov 4, 2022

PrasanthV454 self-assigned this Nov 7, 2022

achrefbensaad mentioned this issue Nov 10, 2022

ARM builds are failing, need to deprecate kafka accuknox/discovery-engine#586

Open

PrasanthV454 closed this as completed Nov 15, 2022

MarioCarrion mentioned this issue Dec 28, 2022

Trying to run kafka sampole using docker on apple m1 macbook MarioCarrion/todo-api-microservice-example#199

Closed

yanmxa mentioned this issue Mar 1, 2023

upgrade confluent-kafka-go to v2.0.2 to be compatible with linux/aarch64 stolostron/multicluster-global-hub#331

Merged

amit-thakur-S mentioned this issue Jul 27, 2023

Issue with Dockerfile on M2 Mac epomatti/go-kafka#1

Closed

k-anshul mentioned this issue Oct 12, 2023

upgrade to kafka lib v2 rilldata/rill#3225

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bundle librdkafka to support linux/arm64 builds #805

Bundle librdkafka to support linux/arm64 builds #805

JimMale-tt commented Jun 23, 2022 •

edited

Loading

mhowlett commented Aug 18, 2022

Josh-ShubhamVyas commented Aug 24, 2022 •

edited

Loading

domgolonka commented Sep 15, 2022

kimgr commented Sep 30, 2022

sagimedina1 commented Nov 7, 2022

PrasanthV454 commented Nov 15, 2022

ferranorriols commented Nov 22, 2022

PrasanthV454 commented Dec 6, 2022

ferranorriols commented Dec 6, 2022

doyelese commented Jan 9, 2023

elmoamedeo commented Feb 13, 2023

daybarr commented Feb 13, 2023 •

edited

Loading

tiagosatur commented Feb 15, 2023

daybarr commented Feb 15, 2023

Bundle librdkafka to support linux/arm64 builds #805

Bundle librdkafka to support linux/arm64 builds #805

Comments

JimMale-tt commented Jun 23, 2022 • edited Loading

Description

How to reproduce

Potential Solutions

Checklist

mhowlett commented Aug 18, 2022

Josh-ShubhamVyas commented Aug 24, 2022 • edited Loading

domgolonka commented Sep 15, 2022

kimgr commented Sep 30, 2022

sagimedina1 commented Nov 7, 2022

PrasanthV454 commented Nov 15, 2022

ferranorriols commented Nov 22, 2022

PrasanthV454 commented Dec 6, 2022

ferranorriols commented Dec 6, 2022

doyelese commented Jan 9, 2023

elmoamedeo commented Feb 13, 2023

daybarr commented Feb 13, 2023 • edited Loading

tiagosatur commented Feb 15, 2023

daybarr commented Feb 15, 2023

JimMale-tt commented Jun 23, 2022 •

edited

Loading

Josh-ShubhamVyas commented Aug 24, 2022 •

edited

Loading

daybarr commented Feb 13, 2023 •

edited

Loading