[armel] CoreRT Linux/armel progress

[alpencolt]

This task is for reporting progress of CoreRT Linux/armel support.
People are working on it @Dmitri-Botcharnikov @sergign60 @alpencolt @BredPet

Applications

On the latest master we are able to compile and execute Binary-trees, Spectral-norm, Mandelbrot and Fannkuch-redux(without threading version) benchmarks from CoreCLR. ElmSharp Hello Tizen application as well. All apps launched on Samsung Z3 phone

armel cross build on x86 host

It's possible to cross compile application for armel on x86 host (or on x64 using rootfs) now.
Compilation time for HelloWorld application is reduced from 25 minutes (on Samsung Z3) to 3 minutes on PC. So for compiling all BringUpTests from CoreCLR it's needed about 6 hours (or about 3 days on mobile phone:scream:).
Documentation will be added soon.

BringUpTests from CoreCLR

1 Failed on compilation
11 FAIL
146 SUCCESS

1. Failed on compilation on CoreRT:

RecursiveTailCall.exe
Process is terminating due to StackOverflowException.

2. Assertion:

LocallocLarge.exe: corert/src/Native/Runtime/unix/UnixContext.cpp:538: bool FindProcInfo(UIntNative, UIntNative *, UIntNative *): Assertion `(procInfo.start_ip <= controlPC) && (controlPC < procInfo.end_ip)' failed.
FAIL LocallocLarge.exe

3. Bus error:

ArrayExc.exe
DivConst.exe
Localloc.exe
ModConst.exe
UDivConst.exe
UModConst.exe
div2.exet

4. Segmentation fault:

DblRoots.exe
Gcd.exe
Swap.exe

Bug fixing

There are a lot of remaining task and bugs. They will refer to the current task as far as possible.

cc @jkotas @MichalStrehovsky

[alpencolt]

@jkotas is it possible to create group like dotnet/arm32-contrib for involved people?

[jkotas]

Compilation time for HelloWorld application is reduced from 25 minutes (on Samsung Z3) to 3 minutes on PC.

Are you using release (or at least checked) build of RyuJIT? I would not expect it to take this long.

[jkotas]

This is awesome progress! Thank you for the update.

I have created @dotnet/arm32-corert-contrib . All of you are members except @BredPet who I am not able to add for some reason. I will work on it...

[alpencolt]

Thank you!
We use debug version because checked and release were unstable before. I will try it with checked version (or checked CoreCLR + cross RyuJIT with disabled optimizations).
It looks @BredPet isn't member of .Net Foundation.

[jkotas]

It looks @BredPet isn't member of .Net Foundation.

Fixed. @BredPet - I will be able to add you to the team once you accept the invitation.

[BredPet]

I joined. And requested membership in @dotnet/arm32-corert-contrib.
@jkotas thanks

[alpencolt]

@jkotas Release version still doesn't work, -O of ILC compiler as well (but with different errors). But if I run ILC on checked CoreCLR compilation of HelloWorld takes 35 sec on x86 host. And it's needed about 2 hours for compiling all BringUpTests.
Results of the tests are the same:
1 Failed on compilation
11 FAIL
146 SUCCESS

[sergign60]

#4880

[alpencolt]

I've looked for API calls in Xamarine which could be problematic for CoreRT and have got results below. Checked Xamarin.Forms.Core.dll, Xamarin.Forms.Platform.dll, Xamarin.Forms.Platform.Tizen.dll, Xamarin.Forms.Xaml.dll packages.
API calls: System.Reflection.*, System.Array::CreateInstance, System.Type::ReflectionOnlyGetType, System.Type::MakeGenericMethod, System.Type::MakeTypeArray, System.Type::GetType.

@jkotas what calls cannot be implemented in CoreRT (if there are such)? Is there any detailed list of API which cannot be implemented? I've looked to https://docs.microsoft.com/en-us/dotnet/framework/net-native/apis-that-rely-on-reflection is it complete?

Results:

# API from System.Reflection
51 call       class [System.Reflection]System.Reflection.TypeInfo [System.Reflection]System.Reflection.IntrospectionExtensions::GetTypeInfo()
34 callvirt   instance string [System.Reflection]System.Reflection.MemberInfo::get_Name()
27 callvirt   instance object [System.Reflection]System.Reflection.MethodBase::Invoke()
22 callvirt   instance class [System.Reflection]System.Reflection.ParameterInfo[] [System.Reflection]System.Reflection.MethodBase::GetParameters()
19 callvirt   instance class [System.Reflection]System.Reflection.MethodInfo [System.Reflection]System.Reflection.PropertyInfo::get_GetMethod()
14 callvirt   instance bool [System.Reflection]System.Reflection.MethodBase::get_IsPublic()
11 callvirt   instance class [System.Reflection]System.Reflection.Assembly [System.Reflection]System.Reflection.TypeInfo::get_Assembly()
 9 callvirt   instance class [System.Runtime]System.Type [System.Reflection]System.Reflection.ParameterInfo::get_ParameterType()
 9 callvirt   instance class [System.Runtime]System.Collections.Generic.IList`1<valuetype [System.Reflection]System.Reflection.CustomAttributeTypedArgument> [System.Reflection]System.Reflection.CustomAttributeData::get_ConstructorArguments()
 8 callvirt   instance object [System.Reflection]System.Reflection.FieldInfo::GetValue()
 8 callvirt   instance class [System.Runtime]System.Type [System.Reflection]System.Reflection.TypeInfo::get_BaseType()
 8 callvirt   instance class [System.Runtime]System.Type [System.Reflection]System.Reflection.MemberInfo::get_DeclaringType()
 8 callvirt   instance bool [System.Reflection]System.Reflection.PropertyInfo::get_CanRead()
 8 callvirt   instance bool [System.Reflection]System.Reflection.MethodBase::get_IsStatic()
 8 call       class [System.Reflection]System.Reflection.FieldInfo [System.Reflection]System.Reflection.FieldInfo::GetFieldFromHandle()
 7 callvirt   instance class [System.Runtime]System.Type [System.Reflection]System.Reflection.CustomAttributeData::get_AttributeType()
 7 callvirt   instance class [System.Reflection]System.Reflection.MethodInfo [System.Reflection]System.Reflection.PropertyInfo::get_SetMethod()
 6 callvirt   instance class [System.Runtime]System.Type [System.Reflection]System.Reflection.PropertyInfo::get_PropertyType()
 6 callvirt   instance class [System.Runtime]System.Type [System.Reflection]System.Reflection.MethodInfo::get_ReturnType()
 6 callvirt   instance class [System.Runtime]System.Collections.Generic.IEnumerable`1<class [System.Reflection]System.Reflection.CustomAttributeData> [System.Reflection]System.Reflection.ParameterInfo::get_CustomAttributes()
 6 callvirt   instance bool [System.Reflection]System.Reflection.TypeInfo::IsAssignableFrom()
 6 callvirt   instance bool [System.Reflection]System.Reflection.FieldInfo::get_IsStatic()
 5 callvirt   instance bool [System.Reflection]System.Reflection.TypeInfo::get_IsValueType()
 5 call       instance object [System.Reflection]System.Reflection.CustomAttributeTypedArgument::get_Value()
 4 callvirt   instance string [System.Reflection]System.Reflection.AssemblyName::get_Name()
 4 callvirt   instance class [System.Runtime]System.Collections.Generic.IEnumerable`1<class [System.Reflection]System.Reflection.ConstructorInfo> [System.Reflection]System.Reflection.TypeInfo::get_DeclaredConstructors()
 4 callvirt   instance bool [System.Reflection]System.Reflection.TypeInfo::get_IsGenericType()
 3 callvirt   instance void [System.Reflection]System.Reflection.EventInfo::AddEventHandler()
 3 callvirt   instance string [System.Reflection]System.Reflection.Assembly::get_FullName()
 3 callvirt   instance object [System.Reflection]System.Reflection.PropertyInfo::GetValue()
 3 callvirt   instance class [System.Runtime]System.Type [System.Reflection]System.Reflection.EventInfo::get_EventHandlerType()
 3 callvirt   instance class [System.Runtime]System.Delegate [System.Reflection]System.Reflection.MethodInfo::CreateDelegate()
 3 callvirt   instance class [System.Runtime]System.Collections.Generic.IEnumerable`1<class [System.Reflection]System.Reflection.CustomAttributeData> [System.Reflection]System.Reflection.MemberInfo::get_CustomAttributes()
 3 callvirt   instance class [System.Reflection]System.Reflection.PropertyInfo [System.Reflection]System.Reflection.TypeInfo::GetDeclaredProperty()
 3 callvirt   instance class [System.Reflection]System.Reflection.MethodInfo [System.Reflection]System.Reflection.TypeInfo::GetDeclaredMethod()
 3 call       instance class [System.Runtime]System.Type [System.Reflection]System.Reflection.CustomAttributeTypedArgument::get_ArgumentType()
 2 callvirt   instance string [System.Reflection]System.Reflection.ManifestResourceInfo::get_FileName()
 2 callvirt   instance class [System.Runtime]System.Type [System.Reflection]System.Reflection.TypeInfo::GetGenericTypeDefinition()
 2 callvirt   instance class [System.Runtime]System.Type [System.Reflection]System.Reflection.FieldInfo::get_FieldType()
 2 callvirt   instance class [System.IO]System.IO.Stream [System.Reflection]System.Reflection.Assembly::GetManifestResourceStream()
 2 callvirt   instance bool [System.Reflection]System.Reflection.PropertyInfo::get_CanWrite()
 2 callvirt   instance bool [System.Reflection]System.Reflection.MethodBase::get_IsSpecialName()
 2 callvirt   instance bool [System.Reflection]System.Reflection.FieldInfo::get_IsPublic()
 2 call       class [System.Reflection]System.Reflection.MethodBase [System.Reflection]System.Reflection.MethodBase::GetMethodFromHandle()
 1 callvirt   instance void [System.Reflection]System.Reflection.EventInfo::RemoveEventHandler()
 1 callvirt   instance string [System.Reflection]System.Reflection.AssemblyName::get_FullName()
 1 callvirt   instance string[] [System.Reflection]System.Reflection.Assembly::GetManifestResourceNames()
 1 callvirt   instance object [System.Reflection]System.Reflection.ConstructorInfo::Invoke()
 1 callvirt   instance class [System.Runtime]System.Type [System.Reflection]System.Reflection.TypeInfo::GetElementType()
 1 callvirt   instance class [System.Runtime]System.Type [System.Reflection]System.Reflection.TypeInfo::AsType()
 1 callvirt   instance class [System.Runtime]System.Type [System.Reflection]System.Reflection.Assembly::GetType()
 1 callvirt   instance class [System.Runtime]System.Collections.Generic.IEnumerable`1<class [System.Runtime]System.Type> [System.Reflection]System.Reflection.TypeInfo::get_ImplementedInterfaces()
 1 callvirt   instance class [System.Runtime]System.Collections.Generic.IEnumerable`1<class [System.Reflection]System.Reflection.PropertyInfo> [System.Reflection]System.Reflection.TypeInfo::get_DeclaredProperties()
 1 callvirt   instance class [System.Runtime]System.Collections.Generic.IEnumerable`1<class [System.Reflection]System.Reflection.FieldInfo> [System.Reflection]System.Reflection.TypeInfo::get_DeclaredFields()
 1 callvirt   instance class [System.Reflection]System.Reflection.TypeInfo [System.Reflection]System.Reflection.IReflectableType::GetTypeInfo()
 1 callvirt   instance class [System.Reflection]System.Reflection.ParameterInfo[] [System.Reflection]System.Reflection.PropertyInfo::GetIndexParameters()
 1 callvirt   instance class [System.Reflection]System.Reflection.Module [System.Reflection]System.Reflection.MemberInfo::get_Module()
 1 callvirt   instance class [System.Reflection]System.Reflection.ManifestResourceInfo [System.Reflection]System.Reflection.Assembly::GetManifestResourceInfo()
 1 callvirt   instance class [System.Reflection]System.Reflection.FieldInfo [System.Reflection]System.Reflection.TypeInfo::GetDeclaredField()
 1 callvirt   instance class [System.Reflection]System.Reflection.AssemblyName[] [System.Reflection]System.Reflection.Assembly::GetReferencedAssemblies()
 1 callvirt   instance bool [System.Reflection]System.Reflection.TypeInfo::get_IsEnum()
 1 callvirt   instance bool [System.Reflection]System.Reflection.TypeInfo::get_IsArray()
 1 callvirt   instance bool [System.Reflection]System.Reflection.MethodBase::get_IsPrivate()
 1 callvirt   instance bool [System.Reflection]System.Reflection.MethodBase::get_IsFamilyOrAssembly()
 1 callvirt   instance bool [System.Reflection]System.Reflection.MethodBase::get_IsFamily()
 1 callvirt   instance bool [System.Reflection]System.Reflection.MethodBase::get_IsAssembly()
 1 call       class [System.Reflection]System.Reflection.Assembly [System.Reflection]System.Reflection.Assembly::Load()

# All others calls
 4 call       class [System.Runtime]System.Type [System.Runtime]System.Type::GetType(string)
 1 call       class [System.Runtime]System.Array [System.Runtime]System.Array::CreateInstance()

[jkotas]

I've looked to https://docs.microsoft.com/en-us/dotnet/framework/net-native/apis-that-rely-on-reflection is it complete?

This article explains the main problem with reflection in full AOT tree-shaked environment.

A more fine grained list is in https://github.com/dotnet/corefx/wiki/UWP-Compat#systemreflection , but the differences are corner cases that do not cause problems in practice.

@jkotas what calls cannot be implemented in CoreRT (if there are such)?

The trick with making reflection work is to make sure that whatever the program reflects on gets compiled into the final image. Currently, the CoreRT compiler has a simple logic to keep everything that is statically reachable from the Main method. It can be further customized by rdxml files (look for RdXmlRootProvider), or it is sometimes easier to add a dummy references to parts of the program that should be compiled in.

I know Xamarin stack uses custom attributes to opt-in things into reflection. We do not recognize these attributes in CoreRT yet. Check https://developer.xamarin.com/guides/ios/advanced_topics/linker/#Preserving_Code for details. If you run into cases like this, we can work together on figuring out the best way to recognize the Xamarin custom attributes in CoreRT compiler.

[alpencolt]

Thank you!
I'm looking for functionality which cannot work in CoreRT by design. As I understand some of these API could be implemented. Others could be handled by compiler or config files.
Thing I want to understand is there something which can prevent using CoreRT in real huge applications (which use Xamarin or other frameworks)?

[jkotas]

The large missing features in CoreRT are dynamic loading (loading of binaries that were not seen by the AOT compiler), and Reflection.Emit (it is really just a special case of dynamic loading).

There are ways how to make these features work in CoreRT - either by having interpreter, or by using JIT - but it would be a lot of work.

[alpencolt]

It looks Reflection.Emit isn't used in Xamarin.
Another option for dynamic loading I think is distributing as precompiled binaries and open it with API like dlopen(). It should be easier than JIT.

[jkotas]

Another option for dynamic loading I think is distributing as precompiled binaries and open it with API like dlopen(). It should be easier than JIT.

The hard part with this model is versioning (what builds are ok to mix). You typically want to allow mixing and matching of different builds, and doing that for managed code with full AOT is pretty hard because of fragile base class problem.

[alpencolt]

OK thank you. It's the same issues like CoreCLR's crossgen has, right?

[jkotas]

Right, the FragileNonVersionable flavor of NGen images.

[alpencolt]

@jkotas we've reviewed opened issues. So I want to share tasks which are needed for stable work of CoreRT on ARM. Could you add another tasks if think that it's needed?

1. Critical (80: part 1/part 2)
   • C# logic
   • Attributes
   • Interoperability
   • Exceptions
   • Cross compilation
   • PAL
2. Debug[17]
   • ILVerify
   • DWARF debuginfo(types, vars)
   • Interpreter question
3. Reflection[16]
   • Metadata
   • Attributes
   • RD.XML behavior

Optimizations [25]:

• Size on disk
• Multi-threading compiler
• Porting CLR features and optimizations(threading, enums parsing/formatting, etc...)
• Jit-Helpers
• Virtual slots

Also there are a lot of other remaining task like testing or build infrastructure. I'm interesting in unimplemented features, API and bugs first.

[jkotas]

share tasks which are needed for stable work of CoreRT on ARM

Getting CoreRT to work great for everything is a ton of work - I do not think we have even issues opened for all of it. I would help to define the scenario you want to go after first. I assume you want to target relatively simple Xamarin-forms based apps on Tizen. Is this correct? Once there is agreement on the scenarios, we can help you to come up with prioritized list of issues to work on to make it work great.

[alpencolt]

@jkotas right, we're focusing on Xamarin based applications from tutorials first.

[jkotas]

For this, I think you should primarily work on these:

• Cross-compilation
• Debug info (DWARF): Dwarf DebugInfo for Vars #783, Dwarf DebugInfo for Types #785
• Reflection Finalize story for reflection roots #5001. In particular, recognize the Mono AOT/reflection attributes if Xamarin forms are using them.
• Testing, triaging failures, bug fixing:
  • CoreCLR tests first (https://github.com/dotnet/corert/blob/master/tests/runtest.sh#L9)
  • CoreFX tests once CoreCLR tests are stable enough. Opened Run CoreFX tests on CoreRT #5059 on that.

[alpencolt]

@jkotas thank you!
We will update plans.