zig cc: Cannot load shared library cross-compiled to arm-linux-gnueabihf

[saierd]

Zig Version

0.9.1

Steps to Reproduce

Compile the program in this gist: https://gist.github.com/saierd/3ee0a898ae7fa5b469e53536851cb39c

lib.cpp:

#include <iostream>

void test()
{
    std::cout << "lib" << std::endl;
}

main.cpp:

#include <iostream>

void test();

int main()
{
    std::cout << "main" << std::endl;
    test();
    return 0;
}

Build:

zig c++ -target arm-linux-gnueabihf -shared -o libtest.so lib.cpp
zig c++ -target arm-linux-gnueabihf main.cpp -l libtest.so

Expected Behavior

Program should run on an ARM device.

Actual Behavior

Without the dynamic library

If I remove the dynamic library from the test program (just a hello world) everything works fine. The program runs in QEMU as well as on the actual device, even though there are some strange things happening on the device:

• ldd a.out outputs not a dynamic executable.
• /lib/ld-linux-armhf.so.3 --verify a.out returns 1.

With the dynamic library

• The program does run in QEMU.
• The program does not run on the actual device and fails with the error message

  ./a.out: error while loading shared libraries: libtest.so: cannot open shared object file: No such file or directory
  

• Output of ldd and ld --verify are the same as above for both a.out and libtest.so.

I tried to debug what happens with LD_DEBUG and strace and this is what I found out:

• The search for the library does find libtest.so and tries to open it. strace shows that some parts of the file get read:

  open("./libtest.so", O_RDONLY|O_CLOEXEC) = 3
  read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\360E\2\0004\0\0\0"..., 512) = 512
  lseek(3, 497736, SEEK_SET)              = 497736
  read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1040) = 1040
  lseek(3, 493664, SEEK_SET)              = 493664
  read(3, "A\35\0\0\0aeabi\0\1\23\0\0\0\6\6\10\1\t\1\30\1\31\1\"\0D\1", 30) = 30
  close(3)                                = 0
  open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
  fstat64(3, {st_mode=S_IFREG|0644, st_size=11919, ...}) = 0
  mmap2(NULL, 11919, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb6fab000
  close(3)                                = 0
  access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
  

• It seems like the file gets discarded for some reason, since after this the search continues with the system search paths.
• If the file /etc/ld.so.nohwcap exists, this removes some of the paths that were searched previously and the access no longer fails:

  open("./libtest.so", O_RDONLY|O_CLOEXEC) = 3
  read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\360E\2\0004\0\0\0"..., 512) = 512
  lseek(3, 497736, SEEK_SET)              = 497736
  read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1040) = 1040
  lseek(3, 493664, SEEK_SET)              = 493664
  read(3, "A\35\0\0\0aeabi\0\1\23\0\0\0\6\6\10\1\t\1\30\1\31\1\"\0D\1", 30) = 30
  close(3)                                = 0
  open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
  fstat64(3, {st_mode=S_IFREG|0644, st_size=11919, ...}) = 0
  mmap2(NULL, 11919, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb6fa8000
  close(3)                                = 0
  access("/etc/ld.so.nohwcap", F_OK)      = 0
  

• Other than that everything else is the same as when the file does not exist. After these lines, the search continues with the system search paths.

I know that this is probably difficult to debug without actual hardware. Do you have any idea for how I could investigate this further?

[nektro]

please include text of program in issue instead of the gist

[saierd]

This issue seems very related: https://youtrack.jetbrains.com/issue/KT-41725

I tried to collect information about the ABI headers for things compiled by zig and it seems to be exactly the same situation as in that ticket.

readelf -A libtest.so:

Attribute Section: aeabi
File Attributes
  Tag_CPU_arch: v6
  Tag_ARM_ISA_use: Yes
  Tag_THUMB_ISA_use: Thumb-1
  Tag_ABI_align_needed: 8-byte
  Tag_ABI_align_preserved: 8-byte, except leaf SP
  Tag_CPU_unaligned_access: None
  Tag_Virtualization_use: TrustZone

readelf -A for a working library produced by GCC:

Attribute Section: aeabi
File Attributes
  Tag_CPU_name: "7-A"
  Tag_CPU_arch: v7
  Tag_CPU_arch_profile: Application
  Tag_ARM_ISA_use: Yes
  Tag_THUMB_ISA_use: Thumb-2
  Tag_FP_arch: VFPv3
  Tag_Advanced_SIMD_arch: NEONv1
  Tag_ABI_PCS_wchar_t: 4
  Tag_ABI_FP_rounding: Needed
  Tag_ABI_FP_denormal: Needed
  Tag_ABI_FP_exceptions: Needed
  Tag_ABI_FP_number_model: IEEE 754
  Tag_ABI_align_needed: 8-byte
  Tag_ABI_align_preserved: 8-byte, except leaf SP
  Tag_ABI_enum_size: int
  Tag_ABI_VFP_args: VFP registers
  Tag_CPU_unaligned_access: v6
  Tag_FP_HP_extension: Allowed
  Tag_ABI_FP_16bit_format: IEEE 754

---

Also as in that ticket the culprit seems to be crti.o.

readelf -A crti.o:

Attribute Section: aeabi
File Attributes
  Tag_CPU_arch: v6
  Tag_ARM_ISA_use: Yes
  Tag_THUMB_ISA_use: Thumb-1
  Tag_ABI_align_needed: 8-byte
  Tag_ABI_align_preserved: 8-byte, except leaf SP
  Tag_CPU_unaligned_access: None
  Tag_Virtualization_use: TrustZone

readelf -A on an object file produced from main.cpp:

Attribute Section: aeabi
File Attributes
  Tag_conformance: "2.09"
  Tag_CPU_arch: v7
  Tag_CPU_arch_profile: Application
  Tag_ARM_ISA_use: Yes
  Tag_THUMB_ISA_use: Thumb-2
  Tag_FP_arch: VFPv3
  Tag_Advanced_SIMD_arch: NEONv1
  Tag_ABI_PCS_R9_use: V6
  Tag_ABI_PCS_RW_data: PC-relative
  Tag_ABI_PCS_RO_data: PC-relative
  Tag_ABI_PCS_GOT_use: GOT-indirect
  Tag_ABI_PCS_wchar_t: 4
  Tag_ABI_FP_denormal: Needed
  Tag_ABI_FP_exceptions: Unused
  Tag_ABI_FP_number_model: IEEE 754
  Tag_ABI_align_needed: 8-byte
  Tag_ABI_align_preserved: 8-byte, except leaf SP
  Tag_ABI_enum_size: int
  Tag_ABI_VFP_args: VFP registers
  Tag_ABI_optimization_goals: Prefer Speed
  Tag_CPU_unaligned_access: v6
  Tag_ABI_FP_16bit_format: IEEE 754

[saierd]

I do not really understand the compilation process, but this might be related to #10411, since crti.S gets compiled from assembly?

I also found out that -mcpu (in my case -mcpu=cortex_a9+neon) seems to be forwarded to the assembler and changes the headers of libtest.so as follows:

Attribute Section: aeabi
File Attributes
  Tag_CPU_name: "cortex-a9"
  Tag_CPU_arch: v7
  Tag_CPU_arch_profile: Application
  Tag_ARM_ISA_use: Yes
  Tag_THUMB_ISA_use: Thumb-2
  Tag_FP_arch: VFPv3
  Tag_Advanced_SIMD_arch: NEONv1
  Tag_ABI_align_needed: 8-byte
  Tag_ABI_align_preserved: 8-byte, except leaf SP
  Tag_CPU_unaligned_access: v6
  Tag_FP_HP_extension: Allowed
  Tag_MPextension_use: Allowed
  Tag_Virtualization_use: TrustZone

This doesn't seem to be enough, though, as the library still cannot be loaded.

[saierd]

Some more cases with the same(?) problem:

• libflutter so cannot be loaded meta-flutter/flutter_embedded#36
• after compression, arm so file can't be loaded upx/upx#116

These suggest that the problem is an incompatibility between LLVM tools and ld.so on older versions of Debian (assuming specification of -mcpu otherwise #10411 is probably still a problem). Debian had a patch that looks at Tag_ABI_VFP_args to decide whether the binary is compatible with the current platform and LLVM does not emit that tag:

• https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1712938
• https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=943798

I am indeed trying to run the library on an old Ubuntu 16.04 system, but did not have a chance yet to check if the problem disappears with a newer system.