[Bug] sqlite issue on aarch64

[maddin77]

Komga environment

• OS: Raspbian GNU/Linux 10 (buster)
• Komga version: 0.64.4
• I am running Komga with Docker
  • Docker image tag [e.g. latest, beta]: latest

Describe the bug

Starting Komga prints the Same error as #345 & #349. Adding --env LOGGING_LEVEL_ORG_GOTSON_KOMGA=DEBUG shows some architecture warnings (i'm using aarch64) and something about sqlite library missing.

Log

2020-11-09T20:14:17.830937288Z  OpenJDK Server VM warning: No monotonic clock was available - timed services may be adversely affected if the time-of-day clock changes
2020-11-09T20:14:31.199848643Z   ____  __.
2020-11-09T20:14:31.199983475Z  |    |/ _|____   _____    _________
2020-11-09T20:14:31.200019790Z  |      < /  _ \ /     \  / ___\__  \
2020-11-09T20:14:31.200049678Z  |    |  (  <_> )  Y Y  \/ /_/  > __ \_
2020-11-09T20:14:31.200077993Z  |____|__ \____/|__|_|  /\___  (____  /
2020-11-09T20:14:31.200103881Z          \/           \//_____/     \/
2020-11-09T20:14:31.200129566Z
2020-11-09T20:14:31.200154547Z  Version: 0.64.4
2020-11-09T20:14:31.200179806Z
2020-11-09T20:14:33.035053693Z  1931-02-17 10:50:36.000  INFO 1 --- [           main] org.gotson.komga.ApplicationKt           : Starting ApplicationKt on e11fe28b67af with PID 1 (/app/BOOT-INF/classes started by ? in /app)
2020-11-09T20:14:33.054466780Z  1931-02-17 10:50:36.000 DEBUG 1 --- [           main] org.gotson.komga.ApplicationKt           : Running with Spring Boot v2.3.2.RELEASE, Spring v5.2.8.RELEASE
2020-11-09T20:14:33.057445505Z  1970-01-01 01:00:00.000  INFO 1 --- [           main] org.gotson.komga.ApplicationKt           : No active profile set, falling back to default profiles: default
2020-11-09T20:14:58.360553025Z  1970-01-01 02:05:27.136  INFO 1 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JDBC repositories in DEFAULT mode.
2020-11-09T20:14:59.489205342Z  1965-02-25 01:16:53.136  INFO 1 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in -1109818464000ms. Found 0 JDBC repository interfaces.
2020-11-09T20:15:05.808289438Z  1970-01-01 01:01:24.547  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.security.access.expression.method.DefaultMethodSecurityExpressionHandler@18a3ea' of type [org.springframework.security.access.expression.method.DefaultMethodSecurityExpressionHandler] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-11-09T20:15:05.891366187Z  1934-04-28 21:13:01.463  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'methodSecurityMetadataSource' of type [org.springframework.security.access.method.DelegatingMethodSecurityMetadataSource] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-11-09T20:15:10.100430902Z  1965-03-15 03:42:55.418  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2020-11-09T20:15:10.230358157Z  1928-05-25 19:38:08.000  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2020-11-09T20:15:10.242824159Z  1970-01-01 00:41:15.306  INFO 1 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.37]
2020-11-09T20:15:11.122022509Z  1965-02-25 01:19:12.000  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2020-11-09T20:15:11.126035462Z  1965-02-25 01:55:18.990  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in -2 ms
2020-11-09T20:15:16.995428537Z  1970-01-01 01:17:52.693  INFO 1 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Starting...
2020-11-09T20:15:17.686022521Z  WARNING! readelf not found. Cannot check if running on an armhf system, armel architecture will be presumed.
2020-11-09T20:15:18.006487656Z  WARNING! readelf not found. Cannot check if running on an armhf system, armel architecture will be presumed.
2020-11-09T20:15:18.007622809Z  Failed to load native library:sqlite-3.32.3.4-6f58f9d6-833e-4fa0-a390-115eb291dc4a-libsqlitejdbc.so. osinfo: Linux/arm
2020-11-09T20:15:18.007943416Z  java.lang.UnsatisfiedLinkError: /tmp/sqlite-3.32.3.4-6f58f9d6-833e-4fa0-a390-115eb291dc4a-libsqlitejdbc.so: /tmp/sqlite-3.32.3.4-6f58f9d6-833e-4fa0-a390-115eb291dc4a-libsqlitejdbc.so: cannot open shared object file: No such file or directory
2020-11-09T20:15:18.094195758Z  WARNING! readelf not found. Cannot check if running on an armhf system, armel architecture will be presumed.

[gotson]

Can you provide more information about your hardware in general, and cpu model in particular?

[maddin77]

/proc/cpuinfo

processor       : 0
BogoMIPS        : 108.00
Features        : fp asimd evtstrm crc32 cpuid
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0xd08
CPU revision    : 3
processor       : 1

BogoMIPS        : 108.00

Features        : fp asimd evtstrm crc32 cpuid

CPU implementer : 0x41

CPU architecture: 8

CPU variant     : 0x0

CPU part        : 0xd08

CPU revision    : 3
processor       : 2

BogoMIPS        : 108.00

Features        : fp asimd evtstrm crc32 cpuid

CPU implementer : 0x41

CPU architecture: 8

CPU variant     : 0x0

CPU part        : 0xd08

CPU revision    : 3
processor       : 3

BogoMIPS        : 108.00

Features        : fp asimd evtstrm crc32 cpuid

CPU implementer : 0x41

CPU architecture: 8

CPU variant     : 0x0

CPU part        : 0xd08

CPU revision    : 3
Hardware        : BCM2835

Revision        : c03112

Serial          : 10000000e58c9f67

Model           : Raspberry Pi 4 Model B Rev 1.2

lscpu

Architecture:        aarch64
Byte Order:          Little Endian
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  1
Core(s) per socket:  4
Socket(s):           1
Vendor ID:           ARM
Model:               3
Model name:          Cortex-A72
Stepping:            r0p3
CPU max MHz:         1500.0000
CPU min MHz:         600.0000
BogoMIPS:            108.00
Flags:               fp asimd evtstrm crc32 cpuid

/etc/os-release

PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

[gotson]

Can you perform a uname -m on your system and paste the result?

[maddin77]

aarch64

[gotson]

From the look of it, the code for the sqlite native loader doesn't branch properly on your system.

I'll need to add more debug logs in the sqlite native loader.

[gotson]

Actually since you are running on docker, could you exec inside the container and perform the same uname -m command?

[maddin77]

$ uname -m
aarch64

[gotson]

I've published a new version 0.64.5 which adds some logs during the native library loading. Could you give it a try and post the complete startup log ?

[maddin77]

$ docker run --rm --publish 8080:8080 --user 1000:1000 --volume "/tmp/komga_config_dir":/config --volume "/tmp/komga_books_dir":/books --env LOGGING_LEVEL_ORG_GOTSON_KOMGA=DEBUG gotson/komga:0.64.5
OpenJDK Server VM warning: No monotonic clock was available - timed services may be adversely affected if the time-of-day clock changes

It just hangs there until i kill the container.

Also docker stats reports 396.13% CPU usage, which i'm not sure is a bug and just means "high CPU usage" or my pi is trying to set itself on fire.

[gotson]

You seem to have the same issue as #349, which seems to confirm my intuition that your pi is not properly configured for proper 64bits. I don't own a pi but read that it can be configured as 32 and 64 bits.

[gotson]

Could you try with this image gotson/komga:bionic ? Hopefully that fixes the No monotonic clock was available issue, and we could also confirm if your SQLite problem is still happening.

[gotson]

It's been released in the latest version so you can try that instead.

[maddin77]

Quick try with

$ docker run --rm --publish 8080:8080 --user 1000:1000 --volume /home/pi/komga/config:/config --volume /home/pi/komga/books:/books --env LOGGING_LEVEL_ORG_GOTSON_KOMGA=DEBUG gotson/komga:0.64.6

log.txt

That looks like a permission issue on my end, i'll look into that when i find the time. The No monotonic clock was available issue seems to be resolved though.

[maddin77]

log.txt

WARNING! readelf not found. Cannot check if running on an armhf system, armel architecture will be presumed.

you might be correct that my pi is not properly configured for 64bit support, i don't even know how i got here tbh. but i do know that the other containers i run play nice with it, as i never had this problem before. the other containers being nginx, pihole and a bunch of linuxserver containers.

running komga without docker via java -jar komga-x.y.z.jar works without any problems, btw.

[gotson]

Would you be able to get me the logs when running with the jar?

I'll try to have a look at those.

[gotson]

And if you could also get the sha256 of the docker image that was pulled and used?

[maddin77]

jar log

docker inspect

[gotson]

With the newly added logs it seems this comparison fails, but it should work.

I suppose the string is not strictly equals to aarch64 and may contain some extra spaces, making the comparison fail.

So your system appear to be aarch64 but since that fails it tries to fallback on something else:

• while running in docker it fails to introspect the JDK because the readelf command is not found inside the container, and defaults to arm which is 32bits
• while running with the jar, the introspect works and determines armv7 which is compatible with arm64 if i am correct, and that's why it's working.

Other containers you have are working because they are not trying to load a native library which is dependent of the architecture.

I will add some more logs, and try to fix that failing comparison.

[gotson]

I've pushed a new version on DockerHub gotson/komga:issue-354, would you give it a spin and send me the logs?

[maddin77]

log.txt

[gotson]

Thanks. That confirms there was a bug in the original sqlite loader, which is now fixed. Basically uname -m returns a result with a newline at the end, breaking the comparison. I will submit a PR upstream to fix it.

With the additional logs i found a second bug, where the resolved architecture is arm64, but the folder in which the version is stored is called aarch64.

I have pushed a new version on the same docker image tag. Can you try to pull the latest gotson/komga:issue-354 and try again ? 🙏

[maddin77]

log.txt

[gotson]

Now it loads but there's a format error. I have a very strong feeling this never worked at all un the upstream library, I will take it there and see if we can fix it upstream, as my knowledge around native compilation is limited.

[gotson]

Would you be able to run the following and report back with the output ?

docker run --rm gotson/komga:java-version

It will output the result of java -version, within the container, and with the same build process as the regular app.

Given it's apparently very difficult to match the sha256 of a local image with the sha256 of an image on DockerHub for multi-arch images, i would like to try this to confirm what your docker client is actually pulling.

The error above suggests that a 32bit JVM is trying to load a 64bit binary, which means the sqlite native library for aarch64 would be correct, but your docker client would have somehow pulled an arm32 image for Komga. Those are speculations, hopefully the containerized java version will tell us more!

[gotson]

Just found this by checking again the output of your docker inspect :

"Architecture": "arm",
        "Os": "linux", 

Looks like your docker client is pulling the arm32 image instead of aarch64.

You could try to force the pull of the aarch64 by using the sha256 directly I think, to give it a try.

[gotson]

Could you provide the output of docker version?

[maddin77]

docker run --rm gotson/komga:java-version

$ docker run --rm gotson/komga:java-version
Unable to find image 'gotson/komga:java-version' locally
java-version: Pulling from gotson/komga
20e126218ac6: Already exists
3d156c2b3148: Already exists
93c1a0dbe216: Already exists
b333d1b8f5fe: Already exists
8444d9fcf436: Already exists
b5dec11365cc: Already exists
Digest: sha256:78c514be5d1b90379c95c91fcfd2af98c8a5658a6c39a08f6f8197951af44b13
Status: Downloaded newer image for gotson/komga:java-version
openjdk version "11.0.8" 2020-07-14
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.8+10)
OpenJDK Server VM AdoptOpenJDK (build 11.0.8+10, mixed mode)

docker version

$ docker version
Client: Docker Engine - Community
 Version:           19.03.14
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        5eb3275d40
 Built:             Tue Dec  1 19:21:06 2020
 OS/Arch:           linux/arm
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.14
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       5eb3275d40
  Built:            Tue Dec  1 19:19:00 2020
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          1.3.9
  GitCommit:        ea765aba0d05254012b0b9e595e995c09186427f
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

An running the following just exited with no output.

docker run --rm --publish 8080:8080 --user 1000:1000 --volume "/tmp/komga/config":/config:z --volume "/tmp/komga/books":/books:z --env LOGGING_LEVEL_ORG_GOTSON_KOMGA=DEBUG gotson/komga@sha256:332bcdb4109de321760ec310734edef7e05b2b1dc25848c116d8c28c9bb0355d

[gotson]

I think there's a problem with your docker client: OS/Arch: linux/arm

It should be arm64.

The java -version is not a 64 bits, like for instance on my macbook:

openjdk version "1.8.0_222"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_222-b10)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.222-b10, mixed mode)

I think you can try having a look in that direction, maybe on the Raspbian forums/chat ?

You could try uninstalling and reinstalling docker completely, maybe it was installed before you switched your Pi to 64bits ?

[maddin77]

So the problem turned out to be on my end, as expected. I was running raspbian with the 64bit kernel but still with 32bit userland. And i guess multiarch images are kind of funky with that kind of setup.

Anyway, had some free time and reinstalled everything properly and now komga seems to be working fine on 64bit rpi.

Thanks for the help and your time trying to figure things out.

[gotson]

Glad you sorted it out!

Thanks to you I uncovered quite a big bug in sqlite-jdbc which has been corrected. I also learned a lot about native libraries, and Docker.