Unable to run application with kafka-stream on aarch64 built using native builder image

[rsvoboda]

Describe the bug

Unable to run application with kafka-stream on aarch64 in native, application built using builder image (use-case of RHBQ)

application startup fails with

[hudson@ampere-mtsnow-altramax-03-vm-13 code-with-quarkus]$ target/code-with-quarkus-1.0.0-SNAPSHOT-runner
Sep 15, 2024 7:18:35 AM io.quarkus.runtime.ApplicationLifecycleManager run
ERROR: Failed to start application
java.lang.RuntimeException: Failed to start quarkus
	at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
	at io.quarkus.runtime.Application.start(Application.java:101)
	at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:119)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:124)
	at io.quarkus.runner.GeneratedMain.main(Unknown Source)
Caused by: java.lang.ExceptionInInitializerError
	at io.quarkus.kafka.streams.runtime.KafkaStreamsRecorder.loadRocksDb(KafkaStreamsRecorder.java:14)
	at io.quarkus.deployment.steps.KafkaStreamsProcessor$loadRocksDb1611413226.deploy_0(Unknown Source)
	at io.quarkus.deployment.steps.KafkaStreamsProcessor$loadRocksDb1611413226.deploy(Unknown Source)
	... 7 more
Caused by: java.lang.RuntimeException: librocksdbjni-linux-aarch64.so was not found inside JAR.
	at org.rocksdb.NativeLibraryLoader.loadLibraryFromJarToTemp(NativeLibraryLoader.java:118)
	at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:102)
	at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:82)
	at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:70)
	at org.rocksdb.RocksDB.<clinit>(RocksDB.java:39)
	... 10 more

The binary contains strings with mixture of librocksdbjni-linux-aarch64.so and librocksdbjni-linux64.so

strings target/code-with-quarkus-1.0.0-SNAPSHOT-runner | grep librocksdbjni
librocksdbjni-linux-aarch64.so
librocksdbjni-linux64.so
librocksdbjni

If the application is built using locally installed Mandrel, the app works well and strings command returns just librocksdbjni-linux-aarch64.so in 2 occurrences.

docker inspect quay.io/quarkus/ubi-quarkus-mandrel-builder-image:jdk-21 reports architecture as arm64

This was detected when working on investigation of Quarkus Quickstarts failures investigation on aarch64 platform.

CC @cescoffier @zakkak @galderz
FYI @mjurc

Expected behavior

Native application built using native builder image can start on RHEL 8 aarch64

Actual behavior

Native application built using native builder image can't start on RHEL 8 aarch64

How to Reproduce?

Using code.quarkus:

wget -O app.zip "https://code.quarkus.io/d?e=kafka-streams&cn=code.quarkus.io"
unzip -q app.zip
cd code-with-quarkus/
mvn clean package -Dnative -Dquarkus.native.container-build=true
target/code-with-quarkus-1.0.0-SNAPSHOT-runner

Using https://github.com/quarkusio/quarkus-quickstarts:

git clone https://github.com/quarkusio/quarkus-quickstarts -b development
cd quarkus-quickstarts
mvn clean verify -f kafka-streams-quickstart -Dnative -Dquarkus.native.container-build=true

Output of uname -a or ver

RHEL 8 aarch64

Output of java -version

Java 21

Mandrel or GraalVM version (if different from Java)

quay.io/quarkus/ubi-quarkus-mandrel-builder-image:jdk-21

Quarkus version or git rev

Quarkus main / Quarkus 3.14

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

[quarkus-bot]

/cc @Karm (mandrel), @alesj (kafka,kafka-streams), @cescoffier (kafka), @galderz (mandrel), @gunnarmorling (kafka-streams), @ozangunalp (kafka,kafka-streams), @rquinio (kafka-streams), @zakkak (mandrel,native-image)

[zakkak]

It looks like this is due to assuming linux AMD64 when using container builds, see

quarkus/extensions/kafka-streams/deployment/src/main/java/io/quarkus/kafka/streams/deployment/KafkaStreamsProcessor.java

Line 181 in 2b814d3

nativeLibs.produce(new NativeImageResourceBuildItem("librocksdbjni-linux64.so"));

@rsvoboda if I am right -Dquarkus.native.container-build=false should work (assuming a local Mandrel/GraalVM installation).

[cescoffier]

Yes that's a very old assumption. We never revisited it.

[rsvoboda]

@rsvoboda if I am right -Dquarkus.native.container-build=false should work (assuming a local Mandrel/GraalVM installation).

Yes, build with local Mandrel/GraalVM installation works fine. I mentioned that in the description below the strings command example.