Linking Native AOT for android complains about -fPIC

[JCash]

Description

I'm linking our executable, and I get this error:

ld.lld: error: relocation R_AARCH64_ADR_PREL_PG_HI21 cannot be used against symbol 'RhpCidResolve'; recompile with -fPIC
>>> defined in /tmp/job17052320106867802331/build/libExtensionCSharp_3.a(libExtensionCSharp_3.o)
>>> referenced by StubDispatch.S:114 (/__w/1/s/src/coreclr/nativeaot/Runtime/arm64/StubDispatch.S:114)
>>>               StubDispatch.S.o:(RhpInterfaceDispatchSlow) in archive /tmp/.nuget/microsoft.netcore.app.runtime.nativeaot.linux-bionic-arm64/9.0.0-rc.2.24473.5/runtimes/linux-bionic-arm64/native/libRuntime.WorkstationGC.a

Reproduction Steps

I compile for android (linux-bionic-arm64), and link a shared library for android mobile.
The toolchain requires -fPIC to link a shared executable for android, which is supplied on the command line.
However, if the "-shared" flag is omitted, the command succeeds. So it's nothing wrong with our setup (our builds work for macOS/iOS/Windows).

/opt/platformsdk/android/android-ndk-r25b/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android21-clang++ -fuse-ld=/opt/platformsdk/android/android-ndk-r25b/toolchains/llvm/prebuilt/linux-x86_64/bin/ld.lld -O2 -g -isysroot=/opt/platformsdk/android/android-ndk-r25b/toolchains/llvm/prebuilt/linux-x86_64/sysroot -static-libstdc++ build/main_tmp4.o  -o build/libdmengine.so -Wl,-soname=libdmengine.so -DDLIB_LOG_DOMAIN="UNKNOWN" -DDDF_EXPOSE_DESCRIPTORS -DDM_PLATFORM_ANDROID -D__aarch64__ -DDM_RELEASE  -Wl,--build-id=uuid  -fpic /tmp/.nuget/microsoft.netcore.app.runtime.nativeaot.linux-bionic-arm64/9.0.0-rc.2.24473.5/runtimes/linux-bionic-arm64/native/libbootstrapperdll.o /tmp/.nuget/microsoft.netcore.app.runtime.nativeaot.linux-bionic-arm64/9.0.0-rc.2.24473.5/runtimes/linux-bionic-arm64/native/libRuntime.WorkstationGC.a /tmp/.nuget/microsoft.netcore.app.runtime.nativeaot.linux-bionic-arm64/9.0.0-rc.2.24473.5/runtimes/linux-bionic-arm64/native/libeventpipe-enabled.a /tmp/.nuget/microsoft.netcore.app.runtime.nativeaot.linux-bionic-arm64/9.0.0-rc.2.24473.5/runtimes/linux-bionic-arm64/native/libstandalonegc-enabled.a /tmp/.nuget/microsoft.netcore.app.runtime.nativeaot.linux-bionic-arm64/9.0.0-rc.2.24473.5/runtimes/linux-bionic-arm64/native/libSystem.IO.Compression.Native.a /tmp/.nuget/microsoft.netcore.app.runtime.nativeaot.linux-bionic-arm64/9.0.0-rc.2.24473.5/runtimes/linux-bionic-arm64/native/libSystem.Globalization.Native.a /tmp/.nuget/microsoft.netcore.app.runtime.nativeaot.linux-bionic-arm64/9.0.0-rc.2.24473.5/runtimes/linux-bionic-arm64/native/libSystem.Native.a -Wl,--no-undefined -Wl,-z,noexecstack -landroid -z text -L/tmp/job17052320106867802331/build  -L/dynamo_home//lib/arm64-android -L/dynamo_home//ext/lib/arm64-android -L/tmp/.nuget/microsoft.netcore.app.runtime.nativeaot.linux-bionic-arm64/9.0.0-rc.2.24473.5/runtimes/linux-bionic-arm64/native  -Wl,-Bstatic -Wl,--start-group -lc++_static -lengine_release -lengine_service_null -lprofile_null -lremotery_null -lprofilerext_null  -lExtensionCSharp_3 -lExtensionCPP_2  -lmbedtls -lzip -lrecord_null -lgameobject -lddf -lresource -lgamesys -lgamesys_model -lgamesys_rig -lscript_box2d -lgraphics -lgraphics_transcoder_basisu -lbasis_transcoder -lphysics -lBulletDynamics -lBulletCollision -lplatform -lLinearMath -lBox2D -lrender -lscript -lluajit-5.1 -lextension -lhid -linput -lparticle -lrig -ldlib -limage -ldmglfw -lgui -lcrashext -lsound -ltremolo -lliveupdate -lunwind  -Wl,--end-group -Wl,-Bdynamic  -lEGL -lGLESv1_CM -lGLESv2 -lOpenSLES -lm -llog -lc  -Wl,--no-undefined -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now -shared

Expected behavior

I expect to be able to link a shared executable (.so) for the Android poatform.

Actual behavior

I get a linker error complaining about the library not being built with -fPIC (/tmp/.nuget/microsoft.netcore.app.runtime.nativeaot.linux-bionic-arm64/9.0.0-rc.2.24473.5/runtimes/linux-bionic-arm64/native/libRuntime.WorkstationGC.a)

Regression?

No response

Known Workarounds

I don't know of a workaround unfortunately, so this is a blocker.

Configuration

No response

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to 'arch-android': @vitek-karas, @simonrozsival, @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

[MichalStrehovsky]

Do you have more detailed repro steps? I tried:

dotnet new classlib -o droidlib 
cd droidlib
set PATH=C:\Programs\android-ndk-r26\toolchains\llvm\prebuilt\windows-x86_64\bin;%PATH% 
dotnet publish -r linux-bionic-arm64 -p:PublishAot=true -p:DisableUnsupportedError=true -p:PublishAotUsingRuntimePack=true

This linked fine. I also tried adding an UnmanagedCallersOnly method and it still linked fine.

[JCash]

Well, you are using dotnet all the way, whereas I only use Dotnet NativeAOT to produce static libraries.
Then I link them using android's clang. For many platforms, the -fPIC is required.

I'll see if I can get a 100% small example later, but these are the steps I take:

1. Create static library for Android using NativeAOT (I don't think this libr is the problematic one, but is it generated using fPIC?)
2. Link the (C++) engine with clang: android-clang++ -lMyNativeAOT.a -lRuntime.WorkstationGC.a -shared (simplified ofc)

The error message suggests that the libRuntime.WorkstationGC.a is not compiled using -fPIC, can you verify this?

[MichalStrehovsky]

As far as I know, we compile everything with fPIC in this repo:

runtime/eng/native/configureplatform.cmake

Line 494 in 1557f12

set(CMAKE_POSITION_INDEPENDENT_CODE ON)

You should be able to double check with something like readelf: https://stackoverflow.com/questions/1340402/how-can-i-tell-with-something-like-objdump-if-an-object-file-has-been-built-wi

The (unsupported) instructions for building with static libraries suggest to replicate the entire command line that is used by native AOT E2E. So my recommendation would be to capture the command line that is used in the workflow from #109341 (comment) and make sure the switches match when linking the static library.

[jkotas]

The error message suggests that the libRuntime.WorkstationGC.a is not compiled using -fPIC, can you verify this?

The error message is about this line in the handwritten asm code:

runtime/src/coreclr/nativeaot/Runtime/arm64/StubDispatch.S

Line 114 in 1557f12

PREPARE_EXTERNAL_VAR RhpCidResolve, xip0

. The problem is likely caused by combination of how this asm code is written and how you are linking your library.

[JCash]

Thanks for the info.

The (unsupported) instructions for building with static libraries suggest to replicate the entire command line that is used by native AOT E2E.

Well, I work under the assumption that this will work (and be supported) in the end, as per discussions in our private chat (ping @steveisok @jbevain).

As for the technical part, the linking requirements of the final shared library for the Android platform, should really be the deciding factor, not the library per se.

I'll try to make a small repro case tomorrow.

. The problem is likely caused by combination of how this asm code is written and how you are linking your library.

I posted the linking step in the OP, the main factor seem to be if I pass "-shared" or not to the linker.
No "shared", then it links, with "shared" then it fails with the error I posted in the OP.

[MichalStrehovsky]

I posted the linking step in the OP, the main factor seem to be if I pass "-shared" or not to the linker.

We do pass -shared in the shared library configuration as well:

"clang" "obj\Release\net9.0\linux-bionic-arm64\native\droidlib.o" -o "bin\Release\net9.0\linux-bionic-arm64\native\droidlib.so" -Wl,--version-script=obj\Release\net9.0\linux-bionic-arm64\native\droidlib.exports -Wl,--export-dynamic -gz=zlib -fuse-ld=lld C:\Users\michals\.nuget\packages\microsoft.netcore.app.runtime.nativeaot.linux-bionic-arm64\9.0.0-rc.2.24473.5/runtimes/linux-bionic-arm64/\native\libbootstrapperdll.o C:\Users\michals\.nuget\packages\microsoft.netcore.app.runtime.nativeaot.linux-bionic-arm64\9.0.0-rc.2.24473.5/runtimes/linux-bionic-arm64/\native\libRuntime.WorkstationGC.a C:\Users\michals\.nuget\packages\microsoft.netcore.app.runtime.nativeaot.linux-bionic-arm64\9.0.0-rc.2.24473.5/runtimes/linux-bionic-arm64/\native\libeventpipe-disabled.a C:\Users\michals\.nuget\packages\microsoft.netcore.app.runtime.nativeaot.linux-bionic-arm64\9.0.0-rc.2.24473.5/runtimes/linux-bionic-arm64/\native\libstandalonegc-disabled.a C:\Users\michals\.nuget\packages\microsoft.netcore.app.runtime.nativeaot.linux-bionic-arm64\9.0.0-rc.2.24473.5/runtimes/linux-bionic-arm64/\native\libstdc++compat.a C:\Users\michals\.nuget\packages\microsoft.netcore.app.runtime.nativeaot.linux-bionic-arm64\9.0.0-rc.2.24473.5/runtimes/linux-bionic-arm64/\native\libSystem.Native.a C:\Users\michals\.nuget\packages\microsoft.netcore.app.runtime.nativeaot.linux-bionic-arm64\9.0.0-rc.2.24473.5/runtimes/linux-bionic-arm64/\native\libSystem.Globalization.Native.a C:\Users\michals\.nuget\packages\microsoft.netcore.app.runtime.nativeaot.linux-bionic-arm64\9.0.0-rc.2.24473.5/runtimes/linux-bionic-arm64/\native\libSystem.IO.Compression.Native.a C:\Users\michals\.nuget\packages\microsoft.netcore.app.runtime.nativeaot.linux-bionic-arm64\9.0.0-rc.2.24473.5/runtimes/linux-bionic-arm64/\native\libSystem.Security.Cryptography.Native.OpenSsl.a --target=aarch64-linux-android21 -g -Wl,-rpath,"$ORIGIN" -Wl,--build-id=sha1 -Wl,--as-needed -Wl,-e,0x0 -pthread -ldl -lz -llog -lm -shared -Wl,-z,relro -Wl,-z,now -Wl,--eh-frame-hdr -Wl,-z,max-page-size=16384 -Wl,--discard-all -Wl,--gc-sections -Wl,-T,"obj\Release\net9.0\linux-bionic-arm64\native\sections.ld"

[JCash]

Thanks for the info.

I've boiled it down to a minimal repro case here.

1. I create a static library using NativeAOT (libNativeLibrary.a)
2. I link this library with a C++ main.cpp into a shared exe, using -shared and -fPIC

Prerequisites:

• Set the DOTNET_CLI_HOME
• Install android sdk+ndk

I build under x86_64 Linux (an Ubuntu 20.04 docker container)

For full details, check the scripts/compile.sh

To compile:

$ SDK=/opt/platformsdk/android/android-sdk-linux/ NDK=/opt/platformsdk/android/android-ndk-r25b/ ./scripts/compile.sh android

In the zip I included the full source and script, as well as the build log and the aot.binlog.

android_repro.zip

GIven the error:

ld: error: relocation R_AARCH64_ADD_ABS_LO12_NC cannot be used against symbol 'RhpCidResolve'; recompile with -fPIC
>>> defined in ./build/android/libNativeLibrary.a(libNativeLibrary.o)

I suspect that the libNativeLibrary.a that dotnet generates isn't actually built using the -fPIC flag?

[MichalStrehovsky]

We don't pass -fPIC when linking the shared library. Is there something broken at runtime if you don't pass -fPIC to the linker invocation?

I tried to modify my repro steps (#109341 (comment)) and added:

  <ItemGroup>
    <LinkerArg Include="-fPIC" />
  </ItemGroup>

to the project. I confirmed -fPIC is passed to the clang invocation by inspecting msbuild binlog. It still built without any problems. This looks specific to your build environment/command line.