Add support for compiling for Android with ndk r22+

[rminderhoud]

Godot version:
Godot 3.2.3-stable but affects other godot versions including latest master

OS/device including version:
Windows 10 x64 but impacts other platforms. I can confirm NDK package for linux and macos have also changed.

Issue description:
Currently, when attempting to build godot for android (any arch) the build will fail because it cannot find a header file:

$ scons platform=android target=release android_arch=armv7
scons: Reading SConscript files ...
Building for Android (armv7) (with NEON)
Checking for C header file mntent.h... (cached) no
scons: done reading SConscript files.
scons: Building targets ...
Compiling shared ==> platform\android\os_android.cpp
=====
b"In file included from platform\\android\\os_android.cpp:31:
In file included from platform\\android/os_android.h:34:
In file included from platform\\android/audio_driver_jandroid.h:34:
In file included from .\\servers/audio_server.h:34:
In file included from .\\core/math/audio_frame.h:34:
In file included from .\\core/math/vector2.h:34:
In file included from .\\core/math/math_funcs.h:35:
In file included from .\\core/math/random_pcg.h:34:
In file included from C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/sources/cxx-stl/llvm-libc++/include\\math.h:294:
C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/sources/cxx-stl/llvm-libc++/include\\__config:218:12: fatal error: 'features.h' file not found
#  include <features.h>
           ^~~~~~~~~~~~
1 error generated.
"
=====
scons: *** [platform\android\os_android.os] Error 1
scons: building terminated because of errors.

This is because the sysroot directory in ndk 22+ has moved from the top level to the toolchain directory, e.g.

• Previous:
  • $ANDROID_NDK_ROOT/sysroot/
• New:
  • $ANDROID_NDK_ROOT/toolchains/llvm/prebuilt/windows-x86_64/sysroot/
  • $ANDROID_NDK_ROOT/toolchains/llvm/prebuilt/linux-x86_64/sysroot/
  • $ANDROID_NDK_ROOT/toolchains/llvm/prebuilt/darwin-x86_64/sysroot

Additionally, the platform libs have also moved

• Previous:
  • $ANDROID_NDK_ROOT/platform/android-$API_NUM/$ARCH/usr/lib
• New
  • $ANDROID_NDK_ROOT/toolchains/llvm/prebuilt/windows-x86_64/usr/lib/$ABI/$API_NUM/

Currently it appears NDK 22 is in beta so not sure what timeline is on this. However it's worth noting that this occurred on my system because Android Studio installed the beta when I installed the NDK from there.

Steps to reproduce:

• Setup android NDK 22+
• Set ANDROID_NDK_ROOT to that directory
• Build for android scons platform=android target=release android_arch=armv7

Partial Solution
A possible fix for the sysroot problem is to do version detection for NDK 22+ and adjust the values in platform/android/detect.py, e.g.
Modifying

godot/platform/android/detect.py

Line 225 in 2a0de6d

lib_sysroot = env["ANDROID_NDK_ROOT"] + "/platforms/" + env["ndk_platform"] + "/" + env["ARCH"]

to something like this:

    sysroot = env["ANDROID_NDK_ROOT"] + "/toolchains/llvm/prebuilt/" + host_subpath + "/sysroot"
    lib_sysroot = sysroot + "/usr/lib/" + abi_subpath + "/" + str(android_api)

    # The sysroot files were moved in ndk v22+ so we keep this for backwards compatibility
    ndk_version = get_ndk_version(env["ANDROID_NDK_ROOT"])
    if LooseVersion(ndk_version) < LooseVersion("22"):
        lib_sysroot = env["ANDROID_NDK_ROOT"] + "/platforms/" + env["ndk_platform"] + "/" + env["ARCH"]
        sysroot = env["ANDROID_NDK_ROOT"] + "/sysroot"

This works except it causes a new error:

b"In file included from platform\\android\\os_android.cpp:31:
In file included from platform\\android/os_android.h:34:
In file included from platform\\android/audio_driver_jandroid.h:34:
In file included from .\\servers/audio_server.h:34:
In file included from .\\core/math/audio_frame.h:34:
In file included from .\\core/math/vector2.h:34:
In file included from .\\core/math/math_funcs.h:35:
In file included from .\\core/math/random_pcg.h:34:
In file included from C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/sources/cxx-stl/llvm-libc++/include\\math.h:301:
In file included from C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/sources/cxx-stl/llvm-libc++/include\\stdlib.h:20:
In file included from C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/sources/android/support/include\\stdlib.h:31:
In file included from C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/toolchains/llvm/prebuilt/windows-x86_64/sysroot/usr/include/c++/v1\\stdlib.h:97:
In file included from C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/toolchains/llvm/prebuilt/windows-x86_64/sysroot/usr/local/include\\stdlib.h:31:
In file included from C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/toolchains/llvm/prebuilt/windows-x86_64/sysroot/usr/include\\stdlib.h:34:
C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/toolchains/llvm/prebuilt/windows-x86_64/sysroot/usr/include\\malloc.h:43:14: error: unknown type name 'size_t'
void* malloc(size_t __byte_count) __mallocfunc __BIONIC_ALLOC_SIZE(1) __wur;
             ^
C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/toolchains/llvm/prebuilt/windows-x86_64/sysroot/usr/include\\malloc.h:52:14: error: unknown type name 'size_t'
void* calloc(size_t __item_count, size_t __item_size) __mallocfunc __BIONIC_ALLOC_SIZE(1,2) __wur;
             ^
C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/toolchains/llvm/prebuilt/windows-x86_64/sysroot/usr/include\\malloc.h:52:35: error: unknown type name 'size_t'
void* calloc(size_t __item_count, size_t __item_size) __mallocfunc __BIONIC_ALLOC_SIZE(1,2) __wur;
                                  ^
C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/toolchains/llvm/prebuilt/windows-x86_64/sysroot/usr/include\\malloc.h:61:28: error: unknown type name 'size_t'
void* realloc(void* __ptr, size_t __byte_count) __BIONIC_ALLOC_SIZE(2) __wur;
                           ^
C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/toolchains/llvm/prebuilt/windows-x86_64/sysroot/usr/include\\malloc.h:94:16: error: unknown type name 'size_t'
void* memalign(size_t __alignment, size_t __byte_count) __mallocfunc __BIONIC_ALLOC_SIZE(2) __wur;
               ^
C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/toolchains/llvm/prebuilt/windows-x86_64/sysroot/usr/include\\malloc.h:94:36: error: unknown type name 'size_t'
void* memalign(size_t __alignment, size_t __byte_count) __mallocfunc __BIONIC_ALLOC_SIZE(2) __wur;
                                   ^
C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/toolchains/llvm/prebuilt/windows-x86_64/sysroot/usr/include\\malloc.h:104:1: error: unknown type name 'size_t'
size_t malloc_usable_size(const void* __ptr) __INTRODUCED_IN(17);
^
C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/toolchains/llvm/prebuilt/windows-x86_64/sysroot/usr/include\\malloc.h:112:3: error: unknown type name 'size_t'
  size_t arena;
  ^
C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/toolchains/llvm/prebuilt/windows-x86_64/sysroot/usr/include\\malloc.h:114:3: error: unknown type name 'size_t'
  size_t ordblks;
  ^
C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/toolchains/llvm/prebuilt/windows-x86_64/sysroot/usr/include\\malloc.h:116:3: error: unknown type name 'size_t'
  size_t smblks;
  ^
C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/toolchains/llvm/prebuilt/windows-x86_64/sysroot/usr/include\\malloc.h:118:3: error: unknown type name 'size_t'
  size_t hblks;
  ^
C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/toolchains/llvm/prebuilt/windows-x86_64/sysroot/usr/include\\malloc.h:120:3: error: unknown type name 'size_t'
  size_t hblkhd;
  ^
C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/toolchains/llvm/prebuilt/windows-x86_64/sysroot/usr/include\\malloc.h:122:3: error: unknown type name 'size_t'
  size_t usmblks;
  ^
C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/toolchains/llvm/prebuilt/windows-x86_64/sysroot/usr/include\\malloc.h:124:3: error: unknown type name 'size_t'
  size_t fsmblks;
  ^
C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/toolchains/llvm/prebuilt/windows-x86_64/sysroot/usr/include\\malloc.h:126:3: error: unknown type name 'size_t'
  size_t uordblks;
  ^
C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/toolchains/llvm/prebuilt/windows-x86_64/sysroot/usr/include\\malloc.h:128:3: error: unknown type name 'size_t'
  size_t fordblks;
  ^
C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/toolchains/llvm/prebuilt/windows-x86_64/sysroot/usr/include\\malloc.h:130:3: error: unknown type name 'size_t'
  size_t keepcost;
  ^
In file included from platform\\android\\os_android.cpp:31:
In file included from platform\\android/os_android.h:34:
In file included from platform\\android/audio_driver_jandroid.h:34:
In file included from .\\servers/audio_server.h:34:
In file included from .\\core/math/audio_frame.h:34:
In file included from .\\core/math/vector2.h:34:
In file included from .\\core/math/math_funcs.h:35:
In file included from .\\core/math/random_pcg.h:34:
In file included from C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/sources/cxx-stl/llvm-libc++/include\\math.h:301:
In file included from C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/sources/cxx-stl/llvm-libc++/include\\stdlib.h:20:
In file included from C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/sources/android/support/include\\stdlib.h:31:
In file included from C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/toolchains/llvm/prebuilt/windows-x86_64/sysroot/usr/include/c++/v1\\stdlib.h:97:
In file included from C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/toolchains/llvm/prebuilt/windows-x86_64/sysroot/usr/local/include\\stdlib.h:31:
C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/toolchains/llvm/prebuilt/windows-x86_64/sysroot/usr/include\\stdlib.h:98:37: error: unknown type name 'size_t'
int posix_memalign(void** __memptr, size_t __alignment, size_t __size) __INTRODUCED_IN(17);
                                    ^
C:\\Users\\Ralph\\AppData\\Local\\Android\\Sdk\\ndk\\22.0.6917172/toolchains/llvm/prebuilt/windows-x86_64/sysroot/usr/include\\stdlib.h:98:57: error: unknown type name 'size_t'
int posix_memalign(void** __memptr, size_t __alignment, size_t __size) __INTRODUCED_IN(17);
                                                        ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
"

It looks like it requires more work to adapt the the toolchain changes but I was not able to a final solution. Reverting back to 21.3.6528147 fixes the issue.

[akien-mga]

The Android NDK breaks compat every other release... Does it offer any stable way to locate relevant paths instead of us having to hardcore them just to see them break whenever upstream decides to move files around?

[rminderhoud]

I'm not sure what the answer to that question is but NDK provides a CMake file for compiling using the android toolchain, it can be found at $NDK_ROOT/build/cmake/android.toolchain.cmake. I've compared v21 vs 22 and there are only minor changes in this file related to some deprecations and compiler/linker flags. There were no changes to include paths or library paths. This suggests to me that it is possible to setup the scons file in such a way that it is portable across NDK versions without having version checks. That file would be a good reference for this issue.

[m4gr3d]

Reading from the description and comments there are two highlighted issues:

1. We need to restrict the version of the ndk used by Godot to ensure that we're only building with a supported and tested version. This may be possible by leveraging the ndkVersion property but we'll need to assess how it can be accessed from the scons build.
2. Lower priority but we need to start looking into the upgrade process for the upcoming ndk update.

[m4gr3d]

Since issue 1 is being handled by #44949, I'm updating this bug's title to reflect the remaining work.

[m4gr3d]

Follow up information - The changelog for r22 provides a build system maintainers guide to assist with handling migration away from the deprecated directories.