[Bug]: WSL1 NDK r25 clang-14: cannot execute binary file: Exec format error

[truboxl]

Description

microsoft/WSL#8681

This is a regression in NDK r25. It is still reproducible with NDK r25b.
I have not tested NDK r24.
NDK r23c works fine on WSL1.

$ android-ndk-r23c/toolchains/llvm/prebuilt/linux-x86_64/bin/clang-12 -v
Android (8481493, based on r416183c2) clang version 12.0.9 (https://android.googlesource.com/toolchain/llvm-project c935d99d7cf2016289302412d708641d52d2f7ee)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/jylo/android-ndk-r23c/toolchains/llvm/prebuilt/linux-x86_64/bin
Found candidate GCC installation: /home/jylo/android-ndk-r23c/toolchains/llvm/prebuilt/linux-x86_64/bin/../lib/gcc/i686-linux-android/4.9.x
Found candidate GCC installation: /home/jylo/android-ndk-r23c/toolchains/llvm/prebuilt/linux-x86_64/bin/../lib/gcc/x86_64-linux-android/4.9.x
Selected GCC installation: /home/jylo/android-ndk-r23c/toolchains/llvm/prebuilt/linux-x86_64/bin/../lib/gcc/x86_64-linux-android/4.9.x
Candidate multilib: .;@m64
Selected multilib: .;@m64
$ android-ndk-r25/toolchains/llvm/prebuilt/linux-x86_64/bin/clang-14 -v
-bash: android-ndk-r25/toolchains/llvm/prebuilt/linux-x86_64/bin/clang-14: cannot execute binary file: Exec format error
$ android-ndk-r25b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang-14 -v
-bash: android-ndk-r25b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang-14: cannot execute binary file: Exec format error

Upstream bug

No response

Commit to cherry-pick

No response

Affected versions

r25

Canary version

No response

Host OS

Windows

Host OS version

Windows 10 version 21H2

Affected ABIs

armeabi-v7a, arm64-v8a, x86, x86_64

[stephenhines]

Adding @kongy since I believe this is due to our use of BOLT to optimize the Linux binaries. There is an incompatibility with WSL1, however our BOLT-ed binaries have been confirmed to work previously with WSL2. Can you switch to WSL2 instead? While we don't officially support either of the WSL implementations, we will try to help as much as we can, but I don't think that we can force Microsoft to fix this breakage on WSL1. We also would not want to deoptimize our actual supported Linux binaries, so I unfortunately don't know that there is much more Android can do for this case.

[DanAlbert]

But while we have you here, I'm curious to know why you're using the Linux NDK via WSL instead of just using the Windows NDK, or the Linux NDK on Linux. We don't support that use case because we don't have the resources to do so, but I've also never argued that we should have those resources (not that that would necessarily change anything) because we haven't understood the motivation for doing so. AIUI the main reason to use WSL is for when there is no native Windows option.

[truboxl]

Cons of Windows NDK:

1. https://android.googlesource.com/platform/ndk/+/master/docs/BuildSystemMaintainers.md#windows-specific-issues
2. No integration of Autotools (but there's the whole MSYS2 + NDK argument but nobody really do that besides for fun), even https://developer.android.com/ndk/guides/other_build_systems#autoconf suggests to use WSL

I still think it's a major oversight it's not listed anywhere in the changelog or README that it doesn't officially support (or in this case breaks in) WSL given its relative popularity and its flexibility than a Linux VM.

Let's see if I can persuade my IT dept to enable WSL2. Not sure if it interferes with some of the IT policies.

I think it's fine to have a discussion every now and then on WSL usage here. Otherwise its a ghost town over at Microsoft's thread and users have to fend for themselves. 😅

[rprichard]

Huh. I would've expected the NDK's clang to work in either WSL1 or WSL2.

The repro steps from microsoft/WSL#8681 are:

unzip android-ndk-r25-linux.zip
root@gamefunc-2550k:/mnt/c/cpps/ndk/android-ndk-r25/toolchains/llvm/prebuilt/linux-x86_64/bin# ./clang-14 --version
-bash: ./clang-14: cannot execute binary file: Exec format error

Maybe try extracting the NDK into /home/... instead of /mnt/c/cpps/ndk, and use the Debian unzip from the command-line to extract the NDK? (e.g. Maybe some of the Unix filesystem attributes aren't set correctly, somehow?)

[truboxl]

Seems like a really widespread issue: microsoft/WSL#8219

Interestingly per microsoft/WSL#8219 (comment), doing this works:

$ /lib64/ld-linux-x86-64.so.2 android-ndk-r25/toolchains/llvm/prebuilt/linux-x86_64/bin/clang-14 -v
Android (8490178, based on r450784d) clang version 14.0.6 (https://android.googlesource.com/toolchain/llvm-project 4c603efb0cca074e9238af8b4106c30add4418f6)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/jylo/android-ndk-r25/toolchains/llvm/prebuilt/linux-x86_64/bin

[DanAlbert]

Why not just use Linux?

[rprichard]

clang-14 has a PT_LOAD segment with a 2MiB p_align (it's the unusual one second from the last). It looks like it was produced by BOLT?

$ readelf -lWS /x/android-ndk-r25/toolchains/llvm/prebuilt/linux-x86_64/bin/clang-14 
There are 34 section headers, starting at offset 0x7c32d10:

Section Headers:
  [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
  [ 1] .interp           PROGBITS        00000000002002e0 0002e0 00001c 00   A  0   0  1
  [ 2] .note.ABI-tag     NOTE            00000000002002fc 0002fc 000020 00   A  0   0  4
  [ 3] .dynsym           DYNSYM          0000000000200320 000320 14d348 18   A  8   1  8
  [ 4] .gnu.version      VERSYM          000000000034d668 14d668 01bc46 02   A  3   0  2
  [ 5] .gnu.version_r    VERNEED         00000000003692b0 1692b0 000130 00   A  8   6  4
  [ 6] .gnu.hash         GNU_HASH        00000000003693e0 1693e0 065078 00   A  3   0  8
  [ 7] .hash             HASH            00000000003ce458 1ce458 06f120 04   A  3   0  4
  [ 8] .dynstr           STRTAB          000000000043d578 23d578 44b753 00   A  0   0  1
  [ 9] .rela.dyn         RELA            0000000000888cd0 688cd0 000258 18   A  3   0  8
  [10] .rela.plt         RELA            0000000000888f28 688f28 001ed8 18  AI  3  27  8
  [11] .rodata           PROGBITS        000000000088ae00 68ae00 1222a48 00 AMS  0   0 16
  [12] .bolt.org.eh_frame PROGBITS        0000000001aad848 18ad848 93e20c 00   A  0   0  8
  [13] .bolt.org.eh_frame_hdr PROGBITS        00000000023eba54 21eba54 1642cc 00   A  0   0  4
  [14] .bolt.org.text    PROGBITS        0000000002550d20 234fd20 35cde64 00  AX  0   0 16
  [15] .init             PROGBITS        0000000005b1eb84 591db84 000024 00  AX  0   0  4
  [16] .fini             PROGBITS        0000000005b1eba8 591dba8 00000e 00  AX  0   0  4
  [17] .plt              PROGBITS        0000000005b1ebc0 591dbc0 0014a0 00  AX  0   0 16
  [18] .tbss             NOBITS          0000000005b20060 591f060 000020 00 WAT  0   0  8
  [19] .ctors            PROGBITS        0000000005b21060 591f060 000010 00  WA  0   0  8
  [20] .dtors            PROGBITS        0000000005b21070 591f070 000010 00  WA  0   0  8
  [21] .jcr              PROGBITS        0000000005b21080 591f080 000008 00  WA  0   0  8
  [22] .init_array       INIT_ARRAY      0000000005b21088 591f088 000d10 00  WA  0   0  8
  [23] .dynamic          DYNAMIC         0000000005b21d98 591fd98 0001f0 10  WA  8   0  8
  [24] .got              PROGBITS        0000000005b21f88 591ff88 0000c8 00  WA  0   0  8
  [25] .data             PROGBITS        0000000005b23050 5920050 008ae0 00  WA  0   0 16
  [26] .tm_clone_table   PROGBITS        0000000005b2bb30 5928b30 000000 00  WA  0   0  8
  [27] .got.plt          PROGBITS        0000000005b2bb30 5928b30 000a60 00  WA  0   0  8
  [28] .bss              NOBITS          0000000005b2c590 5929590 062108 00  WA  0   0 16
  [29] .text             PROGBITS        0000000005c00000 5a00000 d159ee 00  AX  0   0 2097152
  [30] .text.cold        PROGBITS        0000000006915a00 6715a00 820698 00  AX  0   0 64
  [31] .eh_frame         PROGBITS        0000000007136098 6f36098 b42da4 00   A  0   0  8
  [32] .eh_frame_hdr     PROGBITS        0000000007c78e3c 7a78e3c 1b9db4 00   A  0   0  1
  [33] .shstrtab         STRTAB          0000000000000000 7c32bf0 00011a 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  D (mbind), l (large), p (processor specific)

Elf file type is EXEC (Executable file)
Entry point 0x64b2f40
There are 12 program headers, starting at offset 64

Program Headers:
  Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
  PHDR           0x000040 0x0000000000200040 0x0000000000200040 0x0002a0 0x0002a0 R   0x8
  INTERP         0x0002e0 0x00000000002002e0 0x00000000002002e0 0x00001c 0x00001c R   0x1
      [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
  LOAD           0x000000 0x0000000000200000 0x0000000000200000 0x234fd20 0x234fd20 R   0x1000
  LOAD           0x234fd20 0x0000000002550d20 0x0000000002550d20 0x35cf340 0x35cf340 R E 0x1000
  LOAD           0x591f060 0x0000000005b21060 0x0000000005b21060 0x000ff0 0x000ff0 RW  0x1000
  LOAD           0x5920050 0x0000000005b23050 0x0000000005b23050 0x009540 0x06b648 RW  0x1000
  TLS            0x591f060 0x0000000005b20060 0x0000000005b20060 0x000000 0x000020 R   0x8
  DYNAMIC        0x591fd98 0x0000000005b21d98 0x0000000005b21d98 0x0001f0 0x0001f0 RW  0x8
  GNU_RELRO      0x591f060 0x0000000005b21060 0x0000000005b21060 0x000ff0 0x001fa0 R   0x1
  GNU_EH_FRAME   0x7a78e3c 0x0000000007c78e3c 0x0000000007c78e3c 0x1b9db4 0x1b9db4 R   0x4
  LOAD           0x5a00000 0x0000000005c00000 0x0000000005c00000 0x2232bf0 0x2232bf0 R E 0x200000
  NOTE           0x0002fc 0x00000000002002fc 0x00000000002002fc 0x000020 0x000020 R   0x4

 Section to Segment mapping:
  Segment Sections...
   00     
   01     .interp 
   02     .interp .note.ABI-tag .dynsym .gnu.version .gnu.version_r .gnu.hash .hash .dynstr .rela.dyn .rela.plt .rodata .bolt.org.eh_frame .bolt.org.eh_frame_hdr 
   03     .bolt.org.text .init .fini .plt 
   04     .ctors .dtors .jcr .init_array .dynamic .got 
   05     .data .tm_clone_table .got.plt .bss 
   06     .tbss 
   07     .dynamic 
   08     .ctors .dtors .jcr .init_array .dynamic .got 
   09     .eh_frame_hdr 
   10     .text .text.cold .eh_frame .eh_frame_hdr 
   11     .note.ABI-tag 

[enh-google]

probably for transparent huge pages? looks like there's a hugify option to bolt, so maybe there's a --no-hugify?

[gamefunc]

But while we have you here, I'm curious to know why you're using the Linux NDK via WSL instead of just using the Windows NDK, or the Linux NDK on Linux. We don't support that use case because we don't have the resources to do so, but I've also never argued that we should have those resources (not that that would necessarily change anything) because we haven't understood the motivation for doing so. AIUI the main reason to use WSL is for when there is no native Windows option.

build boost and openssl use wsl1 is simple and fast;
in wsl1 + clang12(ndk-23b):
need:{
cp clang-12 clang
cp clang-12 clang++
}

like build openssl, just:
export ANDROID_NDK_ROOT=/mnt/c/cpps/ndk/linux
PATH=$ANDROID_NDK_ROOT/toolchains/llvm/prebuilt/linux-x86_64/bin:$PATH
PATH=$ANDROID_NDK_ROOT/toolchains/llvm/prebuilt/linux-x86_64/bin:$ANDROID_NDK_ROOT/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin:$PATH

./Configure android-arm64 -D__ANDROID_API__=26 --prefix="/mnt/c/cpps/libs/openssl/openssl-3.0.5/android/arm64-v8a" --openssldir="/mnt/c/cpps/libs/openssl/openssl-3.0.5/android/arm64-v8a" && make -j4 && make install

make clean && ./Configure android-arm -D__ANDROID_API__=26 --prefix="/mnt/c/cpps/libs/openssl/openssl-3.0.5/android/armeabi-v7a" --openssldir="/mnt/c/cpps/libs/openssl/openssl-3.0.5/android/armeabi-v7a" && make -j4 && make install

make clean && ./Configure android-x86 -D__ANDROID_API__=26 --prefix="/mnt/c/cpps/libs/openssl/openssl-3.0.5/android/x86" --openssldir="/mnt/c/cpps/libs/openssl/openssl-3.0.5/android/x86" && make -j4 && make install

make clean && ./Configure android-x86_64 -D__ANDROID_API__=26 --prefix="/mnt/c/cpps/libs/openssl/openssl-3.0.5/android/x86_64" --openssldir="/mnt/c/cpps/libs/openssl/openssl-3.0.5/android/x86_64" && make -j4 && make install

other cases, like build ffmpeg also use wsl1:
git clone https://github.com/FFmpeg/FFmpeg.git --recurse-submodules
cd C:\cpps\libs\ffmpeg\ffmpeg-4.4.2
./configure --target-os=win64 --arch=x86_64 --toolchain=msvc --enable-gpl --enable-version3 --enable-nonfree --disable-debug --disable-ffmpeg --disable-ffplay --disable-ffprobe --disable-doc --disable-htmlpages --disable-manpages --disable-podpages --disable-txtpages --prefix=msvc --libdir=msvc/lib --incdir=msvc/include
make -j4
make install

wsl1 yes....

[DanAlbert]

Okay but why not just use Linux?

[gamefunc]

emmm, normal times: window;
when need build something for linux_srv / build lib for android: vs code -> reopen in wsl / click wsl(debian) icon -> ...;
win <-> wsl access friendly:
win access wsl_dir: \wsl$;
wsl access win_dir: /mnt/c/...;

[DanAlbert]

emmm, normal times: window;

My question was why? A subset of the options for you are:

1. Windows
2. Linux
3. Windows with WSL1
4. Windows with WSL2
5. Prebuilt library built from Linux

1, 2, and 5 are supported. I'm asking why 3 is your preferred option. 2 seems like the easiest option to me. If you really need WSL, why not option 4? 4 is also not supported by us, but MS supports it better aiui.

Why am I asking? Because each new host environment is very expensive for us to support. If you want us to support WSL1 officially, I need to be able to tell leadership why that use case is more deserving of headcount than, say, linux/arm64.

[DanAlbert]

https://maven.google.com/web/index.html?q=openssl#com.android.ndk.thirdparty:openssl is a prebuilt openssl, btw. That is option 5.

[DanAlbert]

https://developer.android.com/studio/build/dependencies#native-dependencies-with-agp has the docs for how to use that if you're using AGP. If you're not using AGP, it's a bit of a wild ride atm, but it should be quite easy to integrate into whatever build system you're using: https://google.github.io/prefab/example-workflow.html

[DanAlbert]

Forgot to close this. As mentioned, this is not a supported configuration. We're still here to answer questions but a fix is not something that's being tracked.

[DanAlbert]

FWIW, the discussion of "should we support WSL[12]?" is still ongoing internally. If the answer to that is yes, it'll probably a lower tier of support than can be expected of linux/darwin/windows (wouldn't block a release, but we'd fix issues going forward and have QA do some rudimentary testing so we'd know to add a warning to the changelog), but would mean that things like this get fixed.

The things I need to know to make that happen are the questions I've asked above:

1. Why is using Linux not an option for you?
2. Why is WSL2 not an option for you?

As I said above, supporting additional platforms comes at a pretty high cost to us and we're already spread quite thin, so before we can support a new platform we need to be sure we understand what the benefits are as compared to its alternatives.

[truboxl]

I created this issue mainly to make NDK folks aware that there are people have this kind of use case in the public. I leave it to others to defend their use of WSL1.

You may also want to change https://developer.android.com/ndk/guides/other_build_systems#autoconf to explicitly say WSL2 and not the vaguely WSL or maybe outright remove it.

Better yet have a NDK release note that specifically mentions "WSL1 and WSL2 are not officially supported" and not for others to Google Search their way and end up here.

[DanAlbert]

You may also want to change https://developer.android.com/ndk/guides/other_build_systems#autoconf to explicitly say WSL2 and not the vaguely WSL or maybe outright remove it.

Yes, we saw yesterday that that was worded poorly. It very much reads like "we recommend WSL", whereas it ought to read "if it's literally your only option, try WSL, but YMMV". I'll hopefully find some time to update that soon, but I'm not optimistic.

[DanAlbert]

Better yet have a NDK release note that specifically mentions "WSL1 and WSL2 are not officially supported"

I'm just stumped as to how to enumerate all the environments we don't support, because I don't even know how to enumerate all environments. We support exactly 3: GNU/Linux, Windows, and macOS. Everything not in the list on our download page is not supported.

[gamefunc]

emmm, normal times: window;

My question was why? A subset of the options for you are:

1. Windows
2. Linux
3. Windows with WSL1
4. Windows with WSL2
5. Prebuilt library built from Linux

1, 2, and 5 are supported. I'm asking why 3 is your preferred option. 2 seems like the easiest option to me. If you really need WSL, why not option 4? 4 is also not supported by us, but MS supports it better aiui.

Why am I asking? Because each new host environment is very expensive for us to support. If you want us to support WSL1 officially, I need to be able to tell leadership why that use case is more deserving of headcount than, say, linux/arm64.

not must support, just support is happy;
my english not good, so excuse me;

i build boost for android use my script: https://github.com/gamefunc/Simple_Build_Android_Boost
set main; cd xxx; python3 Simple_Build_Android_Boost.py;
1: window build boost something path too long;
2: ok;
3: habit && happy;
4: i will use vmware replace("wsl2", "vmware");
5: i want to build it from self;

so if not support, i wiil use vmware or 2;

---

build gradle i use, but just set : buildToolsVersion, ndkVersion, ndk{abiFilters}, externalNativeBuild{cmake{version}};

i main cpp/python3/js, java not good so just code a gui and button listen ; private FrameLayout main_window; private widgets; get_display_width_height() / n to set main_win \ widget x/y/h/w;

other in cpp, cmake: if(ANDROID) xx elseif(WIN32) xxx elseif(UNIX) xxx endif();
https://www.gamefunc.top:9029/public/wow60/cm_wla.txt

cpp20 very python3, i like it;

i code in vscode, android_studio just build a app;
main_android_sub.cpp just #include "xxx.cpp"

[truboxl]

I'm just stumped as to how to enumerate all the environments we don't support, because I don't even know how to enumerate all environments. We support exactly 3: GNU/Linux, Windows, and macOS. Everything not in the list on our download page is not supported

Fine by me. My daily job as a tester is to get a black and white written statement on what is supported and what is not supported (and not deduce from what is supported) from devs so that we all agree on the scope.

Here are some of my perspective on Windows and WSL (maybe untrue):
A lot of people still use Windows 10
Not many use Windows 11
People who can't upgrade from Windows 10 to Windows 11 can be attributed to increased hardware requirements (which is total BS by Microsoft imo)
Default on Windows 10 is WSL1
Default on Windows 11 is WSL2
It's likely that WSL2 enhancement and bug fixes will only happen on Windows 11 and not Windows 10
Microsoft has been silent on WSL1 for a long time does not give hope

However if you search https://github.com/microsoft/WSL/issues, you will definitely see some unhappy users switching from WSL2 back to WSL1 and their reasonings

[enh-google]

i've sent out https://critique.corp.google.com/cl/471265875 (internal URL for googlers only) to improve the text on https://developer.android.com/ndk/guides/other_build_systems#autoconf .

[binsys]

workaround:
patchelf --set-interpreter /lib64/ld-linux-x86-64.so.2 /home/binsys/work/android-ndk-r25b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang-14

[josephmoresena]

workaround: patchelf --set-interpreter /lib64/ld-linux-x86-64.so.2 /home/binsys/work/android-ndk-r25b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang-14

It runs but I get the following error:

-- The C compiler identification is unknown
-- The CXX compiler identification is unknown
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Check for working C compiler: /home/user/android-ndk-r25b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang
-- Check for working C compiler: /home/user/android-ndk-r25b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang - broken
CMake Error at /usr/share/cmake-3.22/Modules/CMakeTestCCompiler.cmake:69 (message):
  The C compiler

    "/home/user/android-ndk-r25b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang"

  is not able to compile a simple test program.