Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Discussion: Ways to *actually* fix trimmer warnings around Type.GetType()? #1165

Closed
jonpryor opened this issue Nov 17, 2023 · 1 comment · Fixed by #1168
Closed

Discussion: Ways to *actually* fix trimmer warnings around Type.GetType()? #1165

jonpryor opened this issue Nov 17, 2023 · 1 comment · Fixed by #1168

Comments

@jonpryor
Copy link
Member

Context: #1153
Context: #1157

From the PR #1153 README.md:

Known Unknowns

With this sample "done" (-ish), there are several "future research directions" to
make NativeAOT + Java viable.

Type.GetType()

Next, Java.Interop and .NET Android make extensive use of Type.GetType(), which doesn't quite work "the same" in NativeAOT. It works when using a string constant:

var type = Type.GetType ("System.Int32, System.Runtime");

It fails if the string comes from "elsewhere", even if it's a type that exists.

Unfortunately, we do this in key places within Java.Interop. Consider this more complete Java Callable Wrapper fragment:

public class ManagedType
	extends java.lang.Object
	implements
		com.xamarin.java_interop.GCUserPeerable
{
/** @hide */
	public static final String __md_methods;
	static {
		__md_methods = 
			"getString:()Ljava/lang/String;:__export__\n" +
			"";
		com.xamarin.java_interop.ManagedPeer.registerNativeMembers (
				ManagedType.class,
				"Example.ManagedType, Hello-NativeAOTFromJNI",
				__md_methods);
	}


	public ManagedType (int p0)
	{
		super ();
		if (getClass () == ManagedType.class) {
			com.xamarin.java_interop.ManagedPeer.construct (
					this,
					"Example.ManagedType, Hello-NativeAOTFromJNI",
					"System.Int32, System.Runtime",
					new java.lang.Object[] { p0 });
		}
	}


	public native java.lang.String getString ();
}

There are two places that assembly-qualified names are used, both of which normally wind up at Type.GetType():

  • ManagedPeer.RegisterNativeMembers() is given an assembly-qualified name to register the native methods.
  • ManagedPeer.Construct() is given a :-separated list of assembly-qualified names for each parameter type. This is done to lookup a ConstructorInfo.

This sample "fixes" things by adding JniRuntime.JniTypeManager.GetTypeFromAssemblyQualifiedName(), which allows NativeAotTypeManager to override it and support the various assembly-qualified name values which the sample requires.

An alternate idea to avoid some of the new GetTypeFromAssemblyQualifiedName() invocations would be to declare native methods for each constructor overload, but fixing this gets increasingly difficult.

Which brings us to #1157 (comment):

How do we fix these? These are Type.GetType() invocations for string values which come from Java.

Can they be fixed?

We can just add UnconditionalSuppressMessageAttribute to entirely ignore the warning. This will cause things to break on NativeAOT, though, and the whole point to fixing things is to make things linker friendly and usable on NativeAOT.

PR #1153 implements a "punt" solution to the problem: instead of Type.GetType(), introduce a JniRuntime.JniTypeManager.GetTypeFromAssemblyQualifiedName() method which optionally does Type.GetType(). This would likely still produce IL2057, but we could also [UncondionalSuppressMessage] to silence the warning or somehow require that it be overridden within a NativeAOT app. This permits a path which can work with NativeAOT, but it wouldn't work by default, and would require "extra opt-in logic" in the form of a new method override.

A "proper by default" solution will require separately considering the Type.GetType() calls within ManagedPeer.cs.

Method Registration

Method Registration is done in ManagedPeer.RegisterNativeMembers(), which is invoked from Java:

		__md_methods = 
			"getString:()Ljava/lang/String;:__export__\n" +
			"";
		com.xamarin.java_interop.ManagedPeer.registerNativeMembers (
				ManagedType.class,
				"Example.ManagedType, Hello-NativeAOTFromJNI",
				__md_methods);

Note that ManagedPeer.registerNativeMembers() is given three values:

  1. The java.lang.Class instance of the type to register methods for
  2. The assembly qualified name of the .NET type which corresponds to (1)
  3. The methods to register (which involve Reflection on (2), but Reflection works, so…)

We could have ManagedPeer.RegisterNativeMembers() use JniRuntime.JniTypeManager.GetType(JniTypeSignature) instead of Type.GetType(), a'la:

var type  = JniEnvironment.Runtime.TypeManager.GetType (new JniTypeSignature (r_nativeClass.Name));

This also "punts" on the question to JniRuntime.JniTypeManager.GetType(JniTypeSignature), but that's a pre-existing (and required) extension point:

https://github.com/xamarin/java.interop/blob/320636df084d377e1fdabef6fa6d28feb9ede6f2/src/Java.Interop/Java.Interop/JniRuntime.JniTypeManager.cs#L270-L275

The problem with this approach is that, at present, performance would be "not great", given that JniTypeManager.GetType(JniTypeSignature) uses LINQ.

Constructor Invocation

Constructor Invocation is done in ManagedPeer.Construct(), which is invoked from Java:

	public ManagedType (int p0)
	{
		super ();
		if (getClass () == ManagedType.class) {
			com.xamarin.java_interop.ManagedPeer.construct (
					this,
					"Example.ManagedType, Hello-NativeAOTFromJNI",
					"System.Int32, System.Runtime",
					new java.lang.Object[] { p0 });
		}
	}

ManagedPeer.construct() is given four separate values:

  1. The java.lang.Object instance that we need to construct the .NET side for
  2. The assembly qualified type name of the C# type which corresponds to (1)
  3. A :-separated sequence of assembly qualified type names for the constructor signature to invoke
  4. The values to pass to the constructor.

(2) and (3) both involve Type.GetType() invocations.

As with ManagedPeer.RegisterNativeMembers(), (2) could be replaced with JniEnvironment.Runtime.TypeManager.GetType(), using new JniTypeSignature(JniEnvironment.Types.GetJniTypeNameFromInstance()).

(3) is more problematic.

We could resolve this through one of two mechanisms:

a. Use a JNI method signature, and parse that into System.Type instances at runtime. We would thus invoke ManagedPeer.construct(this, "…now ignored…", "(I)V", …), and turn (I)V into new Type[]{typeof(int)}, again via JniEnvironment.Runtime.TypeManager.GetType().
b. Stop using ManagedPeer.construct() entirely, and instead use a native method declaration.

(b) would result in a Java Callable Wrapper akin to:

/* partial */ class ManagedType {
    public ManagedType (int p0) {
        super ();
        if (getClass () == ManagedType.class) {
            __ctor (p0);
        }
    }
    private native void __ctor (int p0);
}

(a) has the benefit of being easier to implement: jcw-gen already has the JNI method signature, and could emit it. (a) also "feels like" it would have more runtime overhead.

(b) has the benefit of (probably) being faster at runtime, but is more complex. It would requires changes to:

  1. jcw-gen (to declare the new native method)
  2. All JniRuntime.JniTypeManager.RegisterNativeMembers() overrides/implementations would need to figure out how to handle these new native methods.
    .NET Android relies on generator-emitted methods as part of RegisterNativeMembers(), but I don't see how that approach could actually work here. We could pull in System.Reflection.Emit/DynamicMethod, but we're trying to get away from that.
    JavaInterop1 could have jnimarshalmethod-gen implement it, which has the added benefit of increased efficiency, but that's not a solution for .NET Android.
    There be complications here.
  3. jnimarshalmethod-gen (to emit & register these new methods), but only for Desktop Java.Base usage.
  4. Java.Interop.Export
  5. others…?
@jonpryor
Copy link
Member Author

It should be explicitly noted that at this time ManagedPeer.cs is not used by .NET Android. We can thus fix it any way we wish.

That said, the "core" architecture within Java.Interop.dll and .NET Android rhyme here; .NET Android doesn't use ManagedPeer.registerNativeMembers(); it instead uses Runtime.register(), which has the same semantics (just a different parameter ordering). .NET Android doesn't use ManagedPeer.construct(); it instead uses TypeManager.Activate(), which has the same semantics (just a different parameter ordering).

The benefit to there being two similar-yet-distinct implementations is that we can prototype things in "core" -- e.g. rethink the ManagedPeer.java methods and semantics -- without breaking .NET Android, while keeping the "rhyming" concerns in mind (e.g. anything that requires jnimarshalmethod-gen or LLVM Marshal Methods is a non-starter, which in turn means that native methods for constructors, while appealing, is likely not viable in the short term).

jonpryor added a commit that referenced this issue Nov 24, 2023
Fixes: #1165
Context: #1157

If we more strongly rely on JNI signatures, we can remove the need
for Java Callable Wrappers to contain assembly-qualified type names,
thus removing the need for `ManagedPeer` to use `Type.GetType()`
entirely, removing the [IL2057][0] warnings.

Furthermore, if we add `[DynamicallyAccessedMembers]` to
`JniRuntime.JniTypeManager.GetType()`, we can fix some [IL2075][1]
warnings which appeared after fixing the IL2057 warnings.

Aside: Excising assembly-qualified type names from Java Callable
Wrappers had some "interesting" knock-on effects in the unit tests,
requiring that more typemap information be explicitly provided.
(This same information was *implicitly* provided before, via the
provision of assembly-qualified type names everywhere…)

[0]: https://learn.microsoft.com/en-us/dotnet/core/deploying/trimming/trim-warnings/IL2057
[1]: https://learn.microsoft.com/en-us/dotnet/core/deploying/trimming/trim-warnings/il2075
jonpryor added a commit that referenced this issue Nov 25, 2023
Fixes: #1165

Context: #1153
Context: #1157

When building for NativeAOT (#1153) or when building .NET Android
apps with `-p:IsAotcompatible=true` (#1157), we get [IL2057][0]
warnings from `ManagedPeer.cs`:

        ManagedPeer.cs(93,19,93,112): warning IL2057: Unrecognized value passed to the parameter 'typeName' of method 'System.Type.GetType(String, Boolean)'. It's not possible to guarantee the availability of the target type.
        ManagedPeer.cs(156,18,156,65): warning IL2057: Unrecognized value passed to the parameter 'typeName' of method 'System.Type.GetType(String, Boolean)'. It's not possible to guarantee the availability of the target type.
        ManagedPeer.cs(198,35,198,92): warning IL2057: Unrecognized value passed to the parameter 'typeName' of method 'System.Type.GetType(String, Boolean)'. It's not possible to guarantee the availability of the target type.

These warnings are because `ManagedPeer.Construct()` and
`ManagedPeer.RegisterNativeMembers()` use `Type.GetType()` on string
values provided *from Java code*, and thus the IL trimmer does not
have visibility into those strings, and thus cannot reliably
determine which types need to be preserved:

	// Java Callable Wrapper
	/* partial */ class ManagedType
	{
	  public static final String __md_methods;
	  static {
	    __md_methods =
	      "n_GetString:()Ljava/lang/String;:__export__\n" +
	      "";
	    net.dot.jni.ManagedPeer.registerNativeMembers (
	        /* nativeClass */             ManagedType.class,
	        /* assemblyQualifiedName */   "Example.ManagedType, Hello-NativeAOTFromJNI",
	        /* methods */                 __md_methods);
	  }

	  public ManagedType (int p0)
	  {
	    super ();
	    if (getClass () == ManagedType.class) {
	      net.dot.jni.ManagedPeer.construct (
	          /* self */                  this,
	          /* assemblyQualifiedName */ "Example.ManagedType, Hello-NativeAOTFromJNI",
	          /* constructorSignature */  "System.Int32, System.Runtime",
	          /* arguments */             new java.lang.Object[] { p0 });
	    }
	  }
	}

`ManagedPeer.construct()` passes *two* sets of assembly-qualified
type names: `assemblyQualifiedName` contains the type to construct,
while `constructorSignature` contains a `:`-separated list of
assembly-qualified type names for the constructor parameters.
Each of these are passed to `Type.GetType()`.

`ManagedPeer.registerNativeMembers()` passes an assembly-qualified
type name to `ManagedPeer.RegisterNativeMembers()`, which passes the
assembly-qualified type name to `Type.GetType()` to find the type
to register native methods for.

If we more strongly rely on JNI signatures, we can remove the need
for Java Callable Wrappers to contain assembly-qualified type names
entirely, thus removing the need for `ManagedPeer` to use
`Type.GetType()`, removing the IL2057 warnings.

For `ManagedPeer.construct()`, `assemblyQualifiedName` can be
replaced with getting the JNI type signature from `self.getClass()`,
and `constructorSignature` can be replaced with a
*JNI method signature* of the calling constructor.

For `ManagedPeer.registerNativeMembers()`, `assemblyQualifiedName`
can be replaced with getting the JNI type signature from `nativeClass`.

	// Java Callable Wrapper
	/* partial */ class ManagedType
	{
	  public static final String __md_methods;
	  static {
	    __md_methods =
	      "n_GetString:()Ljava/lang/String;:__export__\n" +
	      "";
	    net.dot.jni.ManagedPeer.registerNativeMembers (
	        /* nativeClass */             ManagedType.class,
	        /* methods */                 __md_methods);
	  }

	  public ManagedType (int p0)
	  {
	    super ();
	    if (getClass () == ManagedType.class) {
	      net.dot.jni.ManagedPeer.construct (
	          /* self */                  this,
	          /* constructorSignature */  "(I)V",
	          /* arguments */             new java.lang.Object[] { p0 });
	    }
	  }
	}

Furthermore, if we add `[DynamicallyAccessedMembers]` to
`JniRuntime.JniTypeManager.GetType()`, we can fix some [IL2075][1]
warnings which appeared after fixing the IL2057 warnings.

Aside: Excising assembly-qualified type names from Java Callable
Wrappers had some "interesting" knock-on effects in the unit tests,
requiring that more typemap information be explicitly provided.
(This same information was *implicitly* provided before, via the
provision of assembly-qualified type names everywhere…)

[0]: https://learn.microsoft.com/en-us/dotnet/core/deploying/trimming/trim-warnings/IL2057
[1]: https://learn.microsoft.com/en-us/dotnet/core/deploying/trimming/trim-warnings/il2075
jonpryor added a commit that referenced this issue Dec 2, 2023
Fixes: #1165

Context: #1153
Context: #1157
Context: f60906c

When building for NativeAOT (#1153) or when building .NET Android
apps with `-p:IsAotcompatible=true` (#1157), we get [IL2057][0]
warnings from `ManagedPeer.cs`:

	ManagedPeer.cs(93,19,93,112): warning IL2057: Unrecognized value passed to the parameter 'typeName' of method 'System.Type.GetType(String, Boolean)'. It's not possible to guarantee the availability of the target type.
	ManagedPeer.cs(156,18,156,65): warning IL2057: Unrecognized value passed to the parameter 'typeName' of method 'System.Type.GetType(String, Boolean)'. It's not possible to guarantee the availability of the target type.
	ManagedPeer.cs(198,35,198,92): warning IL2057: Unrecognized value passed to the parameter 'typeName' of method 'System.Type.GetType(String, Boolean)'. It's not possible to guarantee the availability of the target type.

These warnings are because `ManagedPeer.Construct()` and
`ManagedPeer.RegisterNativeMembers()` use `Type.GetType()` on string
values provided *from Java code*, and thus the IL trimmer does not
have visibility into those strings, and thus cannot reliably
determine which types need to be preserved:

	// Java Callable Wrapper
	/* partial */ class ManagedType
	{
	  public static final String __md_methods;
	  static {
	    __md_methods =
	      "n_GetString:()Ljava/lang/String;:__export__\n" +
	      "";
	    net.dot.jni.ManagedPeer.registerNativeMembers (
	        /* nativeClass */             ManagedType.class,
	        /* assemblyQualifiedName */   "Example.ManagedType, Hello-NativeAOTFromJNI",
	        /* methods */                 __md_methods);
	  }

	  public ManagedType (int p0)
	  {
	    super ();
	    if (getClass () == ManagedType.class) {
	      net.dot.jni.ManagedPeer.construct (
	          /* self */                  this,
	          /* assemblyQualifiedName */ "Example.ManagedType, Hello-NativeAOTFromJNI",
	          /* constructorSignature */  "System.Int32, System.Runtime",
	          /* arguments */             new java.lang.Object[] { p0 });
	    }
	  }
	}

`ManagedPeer.construct()` passes *two* sets of assembly-qualified
type names: `assemblyQualifiedName` contains the type to construct,
while `constructorSignature` contains a `:`-separated list of
assembly-qualified type names for the constructor parameters.
Each of these are passed to `Type.GetType()`.

`ManagedPeer.registerNativeMembers()` passes an assembly-qualified
type name to `ManagedPeer.RegisterNativeMembers()`, which passes the
assembly-qualified type name to `Type.GetType()` to find the type
to register native methods for.

If we more strongly rely on JNI signatures, we can remove the need
for Java Callable Wrappers to contain assembly-qualified type names
entirely, thus removing the need for `ManagedPeer` to use
`Type.GetType()`, removing the IL2057 warnings.

For `ManagedPeer.construct()`, `assemblyQualifiedName` can be
replaced with getting the JNI type signature from `self.getClass()`,
and `constructorSignature` can be replaced with a
*JNI method signature* of the calling constructor.

For `ManagedPeer.registerNativeMembers()`, `assemblyQualifiedName`
can be replaced with getting the JNI type signature from `nativeClass`.
`jcw-gen --codegen-target=JavaInterop1` output becomes:

	// New JavaInterop1 Java Callable Wrapper
	/* partial */ class ManagedType
	{
	  public static final String __md_methods;
	  static {
	    __md_methods =
	      "n_GetString:()Ljava/lang/String;:__export__\n" +
	      "";
	    net.dot.jni.ManagedPeer.registerNativeMembers (
	        /* nativeClass */             ManagedType.class,
	        /* methods */                 __md_methods);
	  }

	  public ManagedType (int p0)
	  {
	    super ();
	    if (getClass () == ManagedType.class) {
	      net.dot.jni.ManagedPeer.construct (
	          /* self */                  this,
	          /* constructorSignature */  "(I)V",
	          /* arguments */             new java.lang.Object[] { p0 });
	    }
	  }
	}

This does not alter `jcw-gen --codegen-target=XAJavaInterop1` output;
.NET Android will continue to require `Type.GetType()` calls within
xamarin/xamarin-android, e.g.
[`AndroidTypeManager.RegisterNativeMembers()`][2].

Furthermore, if we add `[DynamicallyAccessedMembers]` to
`JniRuntime.JniTypeManager.GetType()`, we can fix some [IL2075][1]
warnings which appeared after fixing the IL2057 warnings.

Aside: Excising assembly-qualified type names from Java Callable
Wrappers had some "interesting" knock-on effects in the unit tests,
requiring that more typemap information be explicitly provided.
(This same information was *implicitly* provided before, via the
provision of assembly-qualified type names everywhere…)

One problem with the approach of using JNI signatures instead of
using assembly-qualified names is *ambiguity*: there can be multiple
managed types which correspond to a given JNI signature.  Consider
the JNI signature `[I`, which is a Java `int[]`.  This is bound as:

  * C# `int[]`
  * `JavaArray<int>`
  * `JavaPrimitiveArray<int>`
  * `JavaInt32Array`

How do we know which to use?  Using assembly-qualified type names
for constructor parameters nicely solved this issue, but if we're not
using them anymore…

Update `JavaCallableExample` to demonstrate this:

	partial class JavaCallableExample {
	    [JavaCallableConstructor(SuperConstructorExpression="")]
	    public JavaCallableExample (int[] a, JavaInt32Array b);
	}

The intention is twofold:

 1. This should result in a Java Callable Wrapper constructor with
    signature `JavaCallableExample(int[] p0, int[] p1)`, and

 2. Java code should be able to invoke this constructor.

Turns out, neither of these worked when `Type.GetType()` is not used
for constructor argument lookup: `JavaCallableWrapperGenerator`
didn't fully support e.g. `[JniTypeSignature("I", ArrayRank=1)]`
(present on `JavaInt32Array`), so it didn't know what to do with
the `JavaInt32Array` parameter.

Once (1) was fixed, (2) would fail because
`JniRuntime.JniTypeManager.GetType(JniTypeSignature.Parse("[I"))`
would return `JavaPrimitiveArray<int>`, which wasn't used in
`JavaCallableExample`, resulting in:

	System.NotSupportedException : Unable to find constructor
	  Java.InteropTests.JavaCallableExample(Java.Interop.JavaPrimitiveArray`1[[System.Int32, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], Java.Interop.JavaPrimitiveArray`1[[System.Int32, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]).
	  Please provide the missing constructor.
	  ----> Java.Interop.JniLocationException : Exception of type 'Java.Interop.JniLocationException' was thrown.
	  Stack Trace:
	     at Java.Interop.ManagedPeer.GetConstructor(JniTypeManager typeManager, Type type, String signature, Type[]& parameterTypes)
	   at Java.Interop.ManagedPeer.Construct(IntPtr jnienv, IntPtr klass, IntPtr n_self, IntPtr n_constructorSignature, IntPtr n_constructorArguments)
	…
	  --- End of managed Java.Interop.JavaException stack trace ---
	java.lang.Throwable
		at net.dot.jni.ManagedPeer.construct(Native Method)
		at net.dot.jni.test.JavaCallableExample.<init>(JavaCallableExample.java:32)
		at net.dot.jni.test.UseJavaCallableExample.test(UseJavaCallableExample.java:8)

The constructor couldn't be found because
`JniRuntime.JniTypeManager.GetTypes()` was incomplete, which is
a longstanding limitation from f60906c: for `[I`, it would only
return `JavaPrimitiveArray<int>` and `int[]`, in that order.

Fix both of these.
`JniRuntime.JniTypeManager.GetTypes(JniTypeSignature.Parse("[I"))`
will now include:

  * `JavaArray<int>`
  * `JavaPrimitiveArray<int>`
  * `JavaInt32Array`
  * `int[]`

This now allows the `JavaCallableExample` constructor to be invoked
from Java.

Because `ManagedPeer.Construct()` is now doing so much extra work
in order to find the `ConstructorInfo` to invoke, cache the lookups.
(Technically this is a "memory leak," as cache entries are never
removed.)

Finally, update `CecilCompilerExpressionVisitor` to emit `newobj`
in certain `VisitNew()` invocations.  This was needed while trying:

	partial class JavaCallableExample {
	    [JavaCallable ("getA")]
	    public int[] GetA() => this.a;
	}

in order to fix the IL error:

	% $HOME/.dotnet/tools/ilverify bin/TestDebug-net7.0/Java.Interop.Export-Tests.dll \
	    --tokens --system-module System.Private.CoreLib \
	    -r 'bin/TestDebug-net7.0/*.dll' \
	    -r '/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.10/*.dll'
	[IL]: Error [StackUnderflow]: […/bin/TestDebug-net7.0/Java.Interop.Export-Tests.dll : .__<$>_jni_marshal_methods::n_GetA(native int, native int)][offset 0x0000002F] Stack underflow.

Unfortunately, even after the above fix invalid IL was generated during
`jnimarshalmethod-gen` processing, which will be investigated later.

[0]: https://learn.microsoft.com/en-us/dotnet/core/deploying/trimming/trim-warnings/IL2057
[1]: https://learn.microsoft.com/en-us/dotnet/core/deploying/trimming/trim-warnings/il2075
[2]: https://github.com/xamarin/xamarin-android/blob/main/src/Mono.Android/Android.Runtime/AndroidRuntime.cs#L481-L577
jonpryor added a commit to dotnet/android that referenced this issue Feb 2, 2024
Context: dotnet/java-interop#1165
Context: dotnet/java-interop@005c914
Context: #8543
Context: dotnet/java-interop@07c7300
Context: #8625
Context: xamarin/monodroid@e3e4f12
Context: xamarin/monodroid@a04b73b
Context: efbec22

Changes: dotnet/java-interop@8b85462...07c7300

  * dotnet/java-interop@07c73009: [Java.Interop] Typemap support for JavaObject & `[JniTypeSignature]` (dotnet/java-interop#1181)
  * dotnet/java-interop@d529f3be: Bump to xamarin/xamarin-android-tools/main@ed102fc (dotnet/java-interop#1182)
  * dotnet/java-interop@def5bc0d: [ci] Add API Scan job (dotnet/java-interop#1178)
  * dotnet/java-interop@d5afa0af: [invocation-overhead] Add generated source files (dotnet/java-interop#1175)
  * dotnet/java-interop@473ef74c: Bump to xamarin/xamarin-android-tools/main@4889bf0 (dotnet/java-interop#1172)
  * dotnet/java-interop@005c9141: [Java.Interop] Avoid `Type.GetType()` in `ManagedPeer` (dotnet/java-interop#1168)
  * dotnet/java-interop@0f1efebd: [Java.Interop] Use PublicApiAnalyzers to ensure we do not break API (dotnet/java-interop#1170)

(From the "infinite scream" department…)

It started with a desire to remove some linker warnings
(dotnet/java-interop#1165):

	external/Java.Interop/src/Java.Interop/Java.Interop/ManagedPeer.cs(93,19,93,112):
	warning IL2057: Unrecognized value passed to the parameter 'typeName' of method 'System.Type.GetType(String, Boolean)'.
	It's not possible to guarantee the availability of the target type.

dotnet/java-interop@005c9141 attempted to fix this by requiring the
use of "typemaps" mapping Java type signatures to managed types,
replacing e.g.:

	Type            type            = Type.GetType ("Example.Type, AssemblyName", throwOnError: true)!;
	Type[]          parameterTypes  = GetParameterTypes ("System.Int32:System.Int32");
	ConstructorInfo ctor            = type.GetConstructor (ptypes);
	// ctor=Example.Type(int, int) constructor

with (not exactly, but for expository purposes):

	Type            type            = GetTypeFromSignature("crc64…/Type");
	Type[]          parameterTypes  = GetConstructorCandidateParameterTypes ("(II)V");
	ConstructorInfo ctor            = type.GetConstructor (ptypes);
	// ctor=Example.Type(int, int) constructor
	
among other changes.

This was a *significant* change that would alter *Java.Interop*
semantics but *not* .NET Android semantics -- .NET Android uses
`Java.Interop.TypeManager.n_Activate()` (in this repo) for Java-side
"activation" scenarios, not `Java.Interop.ManagedPeer` -- so in an
abundance of caution we did a manual integration test in
#8543 to make sure nothing broke before
merging it.

Something was apparently "off" in that integration.  (We're still not
sure what was off, or why it was completely green.)

Ever since dotnet/java-interop@005c9141 was merged, every attempt to
bump xamarin/Java.Interop has failed, in a number of ways described
below.  However, instead of reverting dotnet/java-interop@005c9141
we took this as an opportunity to understand *how and why* things
were failing, as apparently we had encountered some *long-standing*
corner cases in How Things Work.

The oversights and failures include:

 1. In order to make the Java.Interop unit tests work in .NET Android,
    the (largely hand-written) Java.Interop test types *also* need to
    participate with .NET Android typemap support, so that there is a
    typemap entry mapping `net/dot/jni/test/GenericHolder` to
    `Java.InteropTests.GenericHolder<T>` and vice-versa.

    dotnet/java-interop@07c73009 updates
    `Java.Interop.Tools.JavaCallableWrappers` to support creating
    typemap entries for `Java.Interop.JavaObject` subclasses,
    introducing a new `TypeDefinition.HasJavaPeer()` extension method.

 2. (1) meant that, for the first time ever, types in
    `Java.Interop-Tests` participated in .NET Android type mapping.
    This *sounds* fine, except that `Java.Interop-Tests` contains
    "competing bindings" for `java.lang.Object`:

        [JniTypeSignature ("java/lang/Object", GenerateJavaPeer=false)]
        partial class JavaLangRemappingTestObject : JavaObject {
        }

 3. (2) means that, for the first time ever, we *could* have the
    typemap entry for `java/lang/Object` map to
    `Java.InteropTests.JavaLangRemappingTestObject, Java.Interop-Tests`,
    *not* `Java.Lang.Object, Mono.Android`.

    Arguably a bug, arguably "meh", but this setup triggered some
    never previously encountered error conditions:

 4. `EmbeddedAssemblies::typemap_java_to_managed()` within
    `libmonodroid.so` returns a `System.Type` that corresponds to a
    JNI type.  `typemap_java_to_managed()` has a bug/corner case
    wherein it will only provide `Type` instances from assemblies
    which have already been loaded.

    Early in startup, `Java.Interop-Tests` hasn't been loaded yet, so
    when `java/lang/Object` was mapped to
    `Java.InteropTests.JavaLangRemappingTestObject, Java.Interop-Tests`,
    `typemap_java_to_managed()` would return `null`.

    This is a bug/corner case, which is being investigated in
    #8625.

 5. Calls to `Java.Lang.Object.GetObject<T>()` call
    `Java.Interop.TypeManager.CreateInstance()`, which loops through
    the type and all base types to find a known binding/wrapper.
    Because of (3)+(4), if (when) we try to find the wrapper for
    `java/lang/Object`, we would find *no* mapping.

    This would cause an `JNI DETECTED ERROR IN APPLICATION` *crash*.

    This was due to a "use after free" bug.

    See the "TypeManager.CreateInstance() Use After Free Bug" section.

 6. Once (5) is fixed we encounter our next issue: the
    `Java.InteropTests.JnienvTest.NewOpenGenericTypeThrows()` unit
    test started failing because
    `crc641855b07eca6dcc03.GenericHolder_1` couldn't be found.

    This was caused by a bug in `acw-map.txt` parsing within `<R8/>`.

    See the "`<R8/>` and `acw-map.txt` parsing.`" section.

 7. Once (6) was fixed, (3) caused a *new* set of failures:
    multiple tests started failing because `java/lang/Object` was
    being mapped to the wrong managed type.

    (3) becomes less "meh" and more "definitely a bug".

    See the "Correct `java/lang/Object` mappings" section.

*Now* things should work reliably.


~~ TypeManager.CreateInstance() Use After Free Bug ~~

On 2011-Oct-19, xamarin/monodroid@e3e4f123d8 introduced a
use-after-free bug within `TypeManager.CreateInstance()`:

	JNIEnv.DeleteRef (handle, transfer);
	throw new NotSupportedException (
	        FormattableString.Invariant ($"Internal error finding wrapper class for '{JNIEnv.GetClassNameFromInstance (handle)}'. (Where is the Java.Lang.Object wrapper?!)"),
	        CreateJavaLocationException ());

`handle` *cannot be used* after `JNIEnv.DeleteRef(handle)`.
Failure to do so results in a `JNI DETECTED ERROR IN APPLICATION`
crash; with `adb shell setprop debug.mono.log lref+` set, we see:

	I monodroid-lref: +l+ lrefc 1 handle 0x71/L from thread '(null)'(1)
	D monodroid-gref:    at Android.Runtime.AndroidObjectReferenceManager.CreatedLocalReference(JniObjectReference , Int32& )
	D monodroid-gref:    at Java.Interop.JniRuntime.JniObjectReferenceManager.CreatedLocalReference(JniEnvironmentInfo , JniObjectReference )
	D monodroid-gref:    at Java.Interop.JniEnvironment.LogCreateLocalRef(JniObjectReference )
	D monodroid-gref:    at Java.Interop.JniEnvironment.LogCreateLocalRef(IntPtr )
	D monodroid-gref:    at Java.Interop.JniEnvironment.InstanceMethods.CallObjectMethod(JniObjectReference , JniMethodInfo )
	D monodroid-gref:    …
	…
	I monodroid-lref: -l- lrefc 0 handle 0x71/L from thread '(null)'(1)
	D monodroid-gref:    at Android.Runtime.AndroidObjectReferenceManager.DeleteLocalReference(JniObjectReference& , Int32& )
	D monodroid-gref:    at Java.Interop.JniRuntime.JniObjectReferenceManager.DeleteLocalReference(JniEnvironmentInfo , JniObjectReference& )
	D monodroid-gref:    at Java.Interop.JniObjectReference.Dispose(JniObjectReference& reference)
	D monodroid-gref:    at Android.Runtime.JNIEnv.DeleteLocalRef(IntPtr )
	D monodroid-gref:    at Android.Runtime.JNIEnv.DeleteRef(IntPtr , JniHandleOwnership )
	D monodroid-gref:    at Java.Interop.TypeManager.CreateInstance(IntPtr , JniHandleOwnership , Type )
	D monodroid-gref:    at Java.Lang.Object.GetObject(IntPtr , JniHandleOwnership , Type )
	D monodroid-gref:    at Java.Lang.Object._GetObject[IIterator](IntPtr , JniHandleOwnership )
	D monodroid-gref:    at Java.Lang.Object.GetObject[IIterator](IntPtr handle, JniHandleOwnership transfer)
	D monodroid-gref:    …
	D monodroid-gref:
	E droid.NET_Test: JNI ERROR (app bug): accessed stale Local 0x71  (index 7 in a table of size 7)
	F droid.NET_Test: java_vm_ext.cc:570] JNI DETECTED ERROR IN APPLICATION: use of deleted local reference 0x71
	…
	F droid.NET_Test: runtime.cc:630]   native: #13 pc 00000000003ce865  /apex/com.android.runtime/lib64/libart.so (art::(anonymous namespace)::CheckJNI::GetObjectClass(_JNIEnv*, _jobject*)+837)

The immediate fix is Don't Do That™; use a temporary:

	class_name = JNIEnv.GetClassNameFromInstance (handle);
	JNIEnv.DeleteRef (handle, transfer);
	throw new NotSupportedException (
	        FormattableString.Invariant ($"Internal error finding wrapper class for '{class_name}'. (Where is the Java.Lang.Object wrapper?!)"),
	        CreateJavaLocationException ());

Unfortunately, *just* fixing the "use-after-free" bug is insufficient;
if we throw that `NotSupportedException`, things *will* break
elsewhere.  We'll just have an "elegant unhandled exception" app crash
instead of a "THE WORLD IS ENDING" failed assertion crash.

We could go with the simple fix for the crash, but this means that in
order to integrate dotnet/java-interop@005c9141 &
dotnet/java-interop@07c73009 we'd have to figure out how to *ensure*
that `java/lang/Object` is bound as `Java.Lang.Object, Mono.Android`,
not `Java.InteropTests.JavaLangRemappingTestObject, Java.Interop-Tests`.
(We actually need to do this *anyway*; see the
"Correct `java/lang/Object` mappings" section.  At the time we I was
trying to *avoid* special-casing `Mono.Android.dll`…)

There is a*slightly* more complicated approach which fixes (5)
while supporting (4) `typemap_java_to_managed()` returning null;
consider the `-l-` callstack:

	at Android.Runtime.JNIEnv.DeleteRef(IntPtr , JniHandleOwnership )
	at Java.Interop.TypeManager.CreateInstance(IntPtr , JniHandleOwnership , Type )
	at Java.Lang.Object.GetObject(IntPtr , JniHandleOwnership , Type )
	at Java.Lang.Object._GetObject[IIterator](IntPtr , JniHandleOwnership )
	at Java.Lang.Object.GetObject[IIterator](IntPtr handle, JniHandleOwnership transfer)
	at Android.Runtime.JavaSet.Iterator()

This is part of a generic `Object.GetObject<IIterator>()` invocation!
Additionally, because `IIterator` is an interface, in *normal* use
the `type` variable within `TypeManager.CreateInstance()` would be
`Java.Lang.Object, Mono.Android` and then *immediately discarded*
because `Java.Lang.Object` cannot be assigned to `IIterator`.

Moving the type compatibility check to *before* the
`type == null` check fixes *an* issue with `typemap_java_to_managed()`
returning null.


~~ `<R8/>` and `acw-map.txt` parsing.` ~~

There are many ways for Android+Java code to refer to managed types.

For example, consider the following View subclass:

	namespace Example {
	  partial class MyCoolView : Android.Views.View {
	    // …
	  }
	}

Within layout `.axml` files, you can mention an `Android.Views.View`
subclass by:

  * Using the .NET Full Class Name as an element name.

        <Example.MyCoolView />

  * Using the .NET Full Class Name with a *lowercased* namespace
    name as the element name.

        <example.MyCoolView />

  * Use the Java-side name directly.

        <crc64….NiftyView />

Within Fragments, you can also use the *assembly-qualified name*:

	<fragment class="Example.MyCoolView, AssemblyName" />

At build time, all instances of the .NET type names will be
*replaced* with the Java type names before the Android toolchain
processes the files.

The association between .NET type names and Java names is stored
within `$(IntermediateOutputPath)acw-map.txt`, which was introduced
in xamarin/monodroid@a04b73b3.

*Normally* `acw-map.txt` contains three entries:

 1. The fully-qualified .NET type name
 2. The .NET type name, no assembly
 3. (2) with a lowercased namespace name, *or* the `[Register]`
    value, if provided.

For example:

	Mono.Android_Test.Library.CustomTextView, Mono.Android-Test.Library.NET;crc6456ab8145c81c4100.CustomTextView
	Mono.Android_Test.Library.CustomTextView;crc6456ab8145c81c4100.CustomTextView   
	mono.android_test.library.CustomTextView;crc6456ab8145c81c4100.CustomTextView   
	Java.InteropTests.GenericHolder`1, Java.Interop-Tests;net.dot.jni.test.tests.GenericHolder
	Java.InteropTests.GenericHolder`1;net.dot.jni.test.tests.GenericHolder          
	net.dot.jni.test.tests.GenericHolder;net.dot.jni.test.tests.GenericHolder    

However, when warning XA4214 is emitted (efbec22), there is a
"collision" on the .NET side (but *not* the Java side); (2) and (3)
are potentially *ambiguous*, so one .NET type is arbitrarily chosen.
(Collisions on the Java side result in XA4215 *errors*.)

The first line is still possible, because of assembly qualification.

Enter ``Java.InteropTests.GenericHolder`1``: this type is present in
*both* `Java.Interop-Tests.dll` *and* `Mono.Android-Tests.dll`.
dotnet/java-interop@07c73009, this was "fine" because the
`GenericHolder<T>` within `Java.Interop-Tests.dll` did not participate
in typemap generation.  Now it does, resulting in the XA4214 warning.
XA4214 *also* means that instead of three lines, it's *one* line:

	Java.InteropTests.GenericHolder`1, Mono.Android.NET-Tests;crc641855b07eca6dcc03.GenericHolder_1

Enter `<R8/>`, which parses `acw-map.txt` to create a
`proguard_project_primary.cfg` file.  `<R8/>` did it's *own* parsing
of `acw-map.txt`, parsing only *one of every three lines*, on the
assumption that *all* entries took three lines.

This breaks in the presence of XA4214, because some entries only take
one line, not three lines.  This in turn meant that
`proguard_project_primary.cfg` could *miss* types, which could mean
that `r8` would *remove* the unspecified types, resulting in
`ClassNotFoundException` at runtime:

	Java.Lang.ClassNotFoundException : crc641855b07eca6dcc03.GenericHolder_1
	----> Java.Lang.ClassNotFoundException : Didn't find class "crc641855b07eca6dcc03.GenericHolder_1" on path: DexPathList[[zip file "/data/app/Mono.Android.NET_Tests-2stBqO43ov5F6bHfYemJHQ==/base.apk", zip file "/data/app/Mono.Android.NET_Tests-2stBqO43ov5F6bHfYemJHQ==/split_config.x86_64.apk", zip file "/data/app/Mono.Android.NET_Tests-2stBqO43ov5F6bHfYemJHQ==/split_config.xxhdpi.apk"],nativeLibraryDirectories=[/data/app/Mono.Android.NET_Tests-2stBqO43ov5F6bHfYemJHQ==/lib/x86_64, /system/fake-libs64, /data/app/Mono.Android.NET_Tests-2stBqO43ov5F6bHfYemJHQ==/base.apk!/lib/x86_64, /data/app/Mono.Android.NET_Tests-2stBqO43ov5F6bHfYemJHQ==/split_config.x86_64.apk!/lib/x86_64, /data/app/Mono.Android.NET_Tests-2stBqO43ov5F6bHfYemJHQ==/split_config.xxhdpi.apk!/lib/x86_64, /system/lib64, /system/product/lib64]]
	   at Java.Interop.JniEnvironment.StaticMethods.CallStaticObjectMethod(JniObjectReference , JniMethodInfo , JniArgumentValue* )
	   at Android.Runtime.JNIEnv.FindClass(String )

Update `<R8/>` to instead use `MonoAndroidHelper.LoadMapFile()`,
which reads all lines within `acw-map.txt`.  This results in a
`proguard_project_primary.cfg` file which properly contains a `-keep`
entry for XA4214-related types, such as
`crc641855b07eca6dcc03.GenericHolder_1`.


~~ Correct `java/lang/Object` mappings ~~`

Previous valiant efforts to allow `java/lang/Object` to be mapped to
"anything", not just `Java.Lang.Object, Mono.Android`, eventually
resulted in lots of unit test failures, e.g.:

`Android.RuntimeTests.XmlReaderPullParserTest.ToLocalJniHandle()`:

	System.NotSupportedException : Unable to activate instance of type Java.InteropTests.JavaLangRemappingTestObject from native handle 0x19 (key_handle 0x2408476).
	----> System.MissingMethodException : No constructor found for Java.InteropTests.JavaLangRemappingTestObject::.ctor(System.IntPtr, Android.Runtime.JniHandleOwnership)
	----> Java.Interop.JavaLocationException : Exception_WasThrown, Java.Interop.JavaLocationException
	   at Java.Interop.TypeManager.CreateInstance(IntPtr , JniHandleOwnership , Type )
	   at Java.Interop.TypeManager.CreateInstance(IntPtr , JniHandleOwnership )
	   at Android.Runtime.XmlResourceParserReader.FromNative(IntPtr , JniHandleOwnership )
	   at Android.Runtime.XmlResourceParserReader.FromJniHandle(IntPtr handle, JniHandleOwnership transfer)
	   at Android.Content.Res.Resources.GetXml(Int32 )
	   at Android.RuntimeTests.XmlReaderPullParserTest.ToLocalJniHandle()
	   at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
	   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object , BindingFlags )
	--MissingMethodException
	   at Java.Interop.TypeManager.CreateProxy(Type , IntPtr , JniHandleOwnership )
	   at Java.Interop.TypeManager.CreateInstance(IntPtr , JniHandleOwnership , Type )

With a partially heavy heart, we need to special-case typemap entries
by processing `Mono.Android.dll` *first*, so that it gets first dibs
at bindings for `java/lang/Object` and other types.

Update `NativeTypeMappingData` to process types from `Mono.Android`
before processing any other module.

Note that the special-casing needs to happen in `NativeTypeMappingData`
because typemaps were formerly processed in *sorted module order*, in
which the sort order is based on the *byte representation* of the
module's MVID (a GUID).  Additionally, *linking changes the MVID*,
which means module order is *effectively random*.  Consequently,
trying to special case typemap ordering anywhere else is ineffective.


~~ Other ~~

Update `JavaCompileToolTask` to log the contents of its response file.

Update LLVM-IR -related types within
`src/Xamarin.Android.Build.Tasks/Utilities` to use `TaskLoggingHelper`
for logging purposes, *not* `Action<string>`.  Update related types
to accept `TaskLoggingHelper`, so that we can more easily add
diagnostic messages to these types in the future.
jonpryor pushed a commit to dotnet/android that referenced this issue Mar 1, 2024
Context: #5652
Context: #8724
Context: dotnet/java-interop#1165
Context: dotnet/java-interop@b8f6f88
Context: dc3dc3ccf28cdbe9f8c0a705400b83c11a85c81a980ccf2

Fix another set of trimmer warnings found via:

	<IsTrimmable>true</IsTrimmable>
	<EnableAotAnalyzer>true</EnableAotAnalyzer>

~~ JavaObjectExtensions ~~

`Extensions.JavaCast<T>()` now requires `PublicConstructors` and
`NonPublicConstructors` because `TypeManager.CreateProxy()` uses
`ConstructorInfo.Invoke()`.  This change bubbles up to various other
types that have a `Find*ById<T>()` method:

  * `Activity`
  * `Dialog`
  * `FragmentManager`
  * `View`
  * `Window`

`JavaObjectExtensions.GetInvokerType()` also has suppressions around
`Assembly.GetType()` and `Type.MakeGenericType()`.  We track this for
the future at #8724.


~~ AndroidRuntime ~~

Update `[DynamicallyAccessedMembers]` based on changes to
`RegisterNativeMembers` in dotnet/java-interop@b8f6f888.


~~ JNINativeWrapper ~~

`$(EnableAotAnalyzer)` found usage of `DynamicMethod`.  Suppress for
now, as we track this for the future at #8724.


~~ ResourceIdManager ~~

Usage of `Type.GetMethod ("UpdateIdValues")` leads to decoration of
`[ResourceDesignerAttribute]` with:

	[DynamicallyAccessedMembers (DynamicallyAccessedMemberTypes.PublicMethods)]
	public string FullName { get; set; }

I also had to suppress warnings around `Assembly.GetType()`.
This *should* be OK, as `Resource.designer.cs` is always in the
"root assembly" of Android application projects.

Additionally, this code should no longer be used in .NET 8+ apps;
see dc3ccf2.


~~ JavaProxyThrowable ~~

Suppress warning around `StackFrame.GetMethod()`; we already handle
`null` return values and exceptions.  The existing code appears to be
"best effort" to provide additional stack trace information.


~~ TypeManager ~~

Suppress warning around a call to `Type.GetType()` with a string
passed in from Java.  There is not much we can really do yet, except
rely on the `MarkJavaObjects` trimmer step.

Likely also a problem for the future:

  * dotnet/java-interop#1165
  * #8724


~~ Impact on `.apk` size ~~

`BuildReleaseArm64XFormsDotNet.apkdesc` shows a ~33KB size increase
in the `.apk`.  Much of this is attributable to changes from
dotnet/runtime (`System.Private.CoreLib.dll` is ~20KB larger).

Some of this is due to increases in the size of `classes*.dex`.
These changes are because more managed constructors are now preserved
by the trimmer, which causes more constructors to be emitted into the
Java Callable Wrappers.
@github-actions github-actions bot locked and limited conversation to collaborators Apr 12, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant