From e0936b2b9549060a6ed943253251b931fb4f1900 Mon Sep 17 00:00:00 2001 From: Marek Habersack Date: Wed, 6 May 2020 14:48:22 +0200 Subject: [PATCH] [typemaps] Handle duplicate java to managed type maps properly MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fixes: https://github.com/xamarin/xamarin-android/issues/4596 Context: https://github.com/xamarin/monodroid/commit/192b1f1b71c98650d04bf0a4e52ee337a054ff4c Context: https://github.com/xamarin/java.interop/blob/fc18c54b2ccf14f444fadac0a1e7e81f837cc470/src/Java.Interop.Tools.JavaCallableWrappers/Java.Interop.Tools.JavaCallableWrappers/TypeNameMapGenerator.cs#L149-L186 In some environments, `BundleTest.TestBundleIntegerArrayList2()` fails when using the commercial shared runtime: Test 'Xamarin.Android.RuntimeTests.BundleTest.TestBundleIntegerArrayList2' failed: System.MemberAccessException : Cannot create an instance of Android.Runtime.JavaList`1[T] because Type.ContainsGenericParameters is true. at System.Reflection.RuntimeConstructorInfo.DoInvoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) at System.Reflection.RuntimeConstructorInfo.Invoke (System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) at System.Reflection.ConstructorInfo.Invoke (System.Object[] parameters) at Java.Interop.TypeManager.CreateProxy (System.Type type, System.IntPtr handle, Android.Runtime.JniHandleOwnership transfer) at Java.Interop.TypeManager.CreateInstance (System.IntPtr handle, Android.Runtime.JniHandleOwnership transfer, System.Type targetType) at Java.Lang.Object.GetObject (System.IntPtr handle, Android.Runtime.JniHandleOwnership transfer, System.Type type) at Java.Lang.Object._GetObject[T] (System.IntPtr handle, Android.Runtime.JniHandleOwnership transfer) at Java.Lang.Object.GetObject[T] (System.IntPtr handle, Android.Runtime.JniHandleOwnership transfer) at Android.OS.Bundle.Get (System.String key) at Xamarin.Android.RuntimeTests.BundleTest.TestBundleIntegerArrayList2 () at (wrapper managed-to-native) System.Reflection.RuntimeMethodInfo.InternalInvoke(System.Reflection.RuntimeMethodInfo,object,object[],System.Exception&) at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) The problem appears to be rooted in ce2bc689. In a pre-ce2bc689 world, there were two separate sets of "typemap" files that the application could use: * Java-to-managed mappings, in `typemap.jm` * Managed-to-Java mappings, in `typemap.mj`. These files did *not* need to have the same number of entries! % unzip /Library/Frameworks/Xamarin.Android.framework/Versions/10.2.0.100/lib/xamarin.android/xbuild/Xamarin/Android/Mono.Android.DebugRuntime-debug.apk typemap.jm % unzip /Library/Frameworks/Xamarin.Android.framework/Versions/10.2.0.100/lib/xamarin.android/xbuild/Xamarin/Android/Mono.Android.DebugRuntime-debug.apk typemap.mj % strings typemap.jm | wc -l 10318 % strings typemap.mj | wc -l 11956 The reason why they have a different number of entries is *aliasing*: there can be *multiple* bindings for a given Java type. The managed-to-Java mappings can contain *all* of them; the Java-to-managed mapping *must* pick Only One. For example, [`java.util.ArrayList`][0] contains (at least?) *three* bindings: * [`Android.Runtime.JavaList`][1] * [`Android.Runtime.JavaList`][2] * `Java.Util.ArrayList` (generated at build time) In a pre-ce2bc689 world, this was "fine": we'd binary search each file *separately* to find type mappings. This changed in ce2bc689, as the typemap information was merged into a *single* array in order to de-duplicate information. This reduced `.apk` size. An unanticipated result of this is that the Java-to-managed mappings can now contain *duplicate* keys, "as if" the original `typemap.jm` had the contents: java/util/ArrayList Android.Runtime.JavaList, Mono.Android java/util/ArrayList Android.Runtime.JavaList`1, Mono.Android java/util/ArrayList Java.Util.ArrayLlist, Mono.Android Whereas pre-ce2bc689 there would have only been the first entry. "Sometimes" this duplication is "fine": the typemap search "Just happens" to return the first entry, or the 3rd entry, and apps continue to work as intended. "Sometimes" it isn't: the typemap binary search finds the 2nd entry, which is a generic type. This results in attempting to instantiate a generic type *without appropriate type parameters*, which results in the aforementioned `MemberAccessException`. Update typemap generation code in `Xamarin.Android.Build.Tasks` so that all the duplicate Java type names will point to the same managed type name. Additionally, make sure we select the managed type in the same fashion the old typemap generator in `Java.Interop` worked - prefer base types to the derived ones if the type is an interface or an abstract class. The effect of this change is that no matter which entry `binary_search` ends up selecting it will always return the same managed type name for all the aliased Java types. Additionally, update `TestBundleIntegerArrayList2()` so that it asserts the runtime *type* of `obj` returned, asserting that it is in fact a `JavaList` and not e.g. `Java.Util.ArrayList` or something. (The linker could otherwise "change" things, and this assertion will ensure that `JavaList` won't be linked away…) [0]: https://developer.android.com/reference/java/util/ArrayList [1]: https://github.com/xamarin/xamarin-android/blob/61f09bf2ef0c8f09550e2d77eb97f417432b315b/src/Mono.Android/Android.Runtime/JavaList.cs#L11-L13 [2]: https://github.com/xamarin/xamarin-android/blob/61f09bf2ef0c8f09550e2d77eb97f417432b315b/src/Mono.Android/Android.Runtime/JavaList.cs#L667-L668 --- .../Test/Android.OS/BundleTest.cs | 1 + .../Tasks/GenerateJavaStubs.cs | 6 +- .../Utilities/TypeMapGenerator.cs | 71 +++++++++++++++---- 3 files changed, 63 insertions(+), 15 deletions(-) diff --git a/src/Mono.Android/Test/Android.OS/BundleTest.cs b/src/Mono.Android/Test/Android.OS/BundleTest.cs index cb456af7b1f..f2e96399c5e 100644 --- a/src/Mono.Android/Test/Android.OS/BundleTest.cs +++ b/src/Mono.Android/Test/Android.OS/BundleTest.cs @@ -19,6 +19,7 @@ public void TestBundleIntegerArrayList() Assert.NotNull (list, "'key' doesn't refer to a list of integers"); var obj = b.Get ("key"); Assert.NotNull (obj, "Missing 'key' in bundle"); + Assert.IsTrue (obj is global::Android.Runtime.JavaList, "`obj` should be a JavaList!"); try { list = b.GetIntegerArrayList ("key"); Assert.NotNull (list, "'key' doesn't refer to a list of integers after non-generic call"); diff --git a/src/Xamarin.Android.Build.Tasks/Tasks/GenerateJavaStubs.cs b/src/Xamarin.Android.Build.Tasks/Tasks/GenerateJavaStubs.cs index a738f99f28e..74b26a7e49f 100644 --- a/src/Xamarin.Android.Build.Tasks/Tasks/GenerateJavaStubs.cs +++ b/src/Xamarin.Android.Build.Tasks/Tasks/GenerateJavaStubs.cs @@ -174,7 +174,7 @@ void Run (DirectoryAssemblyResolver res) // Step 2 - Generate type maps // Type mappings need to use all the assemblies, always. - WriteTypeMappings (allJavaTypes); + WriteTypeMappings (allJavaTypes, cache); var javaTypes = new List (); foreach (TypeDefinition td in allJavaTypes) { @@ -399,10 +399,10 @@ void SaveResource (string resource, string filename, string destDir, Func types) + void WriteTypeMappings (List types, TypeDefinitionCache cache) { var tmg = new TypeMapGenerator ((string message) => Log.LogDebugMessage (message), SupportedAbis); - if (!tmg.Generate (Debug, SkipJniAddNativeMethodRegistrationAttributeScan, types, TypemapOutputDirectory, GenerateNativeAssembly, out ApplicationConfigTaskState appConfState)) + if (!tmg.Generate (Debug, SkipJniAddNativeMethodRegistrationAttributeScan, types, cache, TypemapOutputDirectory, GenerateNativeAssembly, out ApplicationConfigTaskState appConfState)) throw new XamarinAndroidException (4308, Properties.Resources.XA4308); GeneratedBinaryTypeMaps = tmg.GeneratedBinaryTypeMaps.ToArray (); BuildEngine4.RegisterTaskObject (ApplicationConfigTaskState.RegisterTaskObjectKey, appConfState, RegisteredTaskObjectLifetime.Build, allowEarlyCollection: false); diff --git a/src/Xamarin.Android.Build.Tasks/Utilities/TypeMapGenerator.cs b/src/Xamarin.Android.Build.Tasks/Utilities/TypeMapGenerator.cs index 25d2ff1cd2e..6937b2e6e46 100644 --- a/src/Xamarin.Android.Build.Tasks/Utilities/TypeMapGenerator.cs +++ b/src/Xamarin.Android.Build.Tasks/Utilities/TypeMapGenerator.cs @@ -4,6 +4,7 @@ using System.Linq; using System.Text; +using Java.Interop.Tools.Cecil; using Mono.Cecil; namespace Xamarin.Android.Tasks @@ -76,6 +77,7 @@ internal sealed class TypeMapDebugEntry public string ManagedLabel; public int JavaIndex; public int ManagedIndex; + public TypeDefinition TypeDefinition; } // Widths include the terminating nul character but not the padding! @@ -125,7 +127,7 @@ void UpdateApplicationConfig (TypeDefinition javaType, ApplicationConfigTaskStat } } - public bool Generate (bool debugBuild, bool skipJniAddNativeMethodRegistrationAttributeScan, List javaTypes, string outputDirectory, bool generateNativeAssembly, out ApplicationConfigTaskState appConfState) + public bool Generate (bool debugBuild, bool skipJniAddNativeMethodRegistrationAttributeScan, List javaTypes, TypeDefinitionCache cache, string outputDirectory, bool generateNativeAssembly, out ApplicationConfigTaskState appConfState) { if (String.IsNullOrEmpty (outputDirectory)) throw new ArgumentException ("must not be null or empty", nameof (outputDirectory)); @@ -140,25 +142,26 @@ public bool Generate (bool debugBuild, bool skipJniAddNativeMethodRegistrationAt string typemapsOutputDirectory = Path.Combine (outputDirectory, "typemaps"); if (debugBuild) { - return GenerateDebug (skipJniAddNativeMethodRegistrationAttributeScan, javaTypes, typemapsOutputDirectory, generateNativeAssembly, appConfState); + return GenerateDebug (skipJniAddNativeMethodRegistrationAttributeScan, javaTypes, cache, typemapsOutputDirectory, generateNativeAssembly, appConfState); } return GenerateRelease (skipJniAddNativeMethodRegistrationAttributeScan, javaTypes, typemapsOutputDirectory, appConfState); } - bool GenerateDebug (bool skipJniAddNativeMethodRegistrationAttributeScan, List javaTypes, string outputDirectory, bool generateNativeAssembly, ApplicationConfigTaskState appConfState) + bool GenerateDebug (bool skipJniAddNativeMethodRegistrationAttributeScan, List javaTypes, TypeDefinitionCache cache, string outputDirectory, bool generateNativeAssembly, ApplicationConfigTaskState appConfState) { if (generateNativeAssembly) - return GenerateDebugNativeAssembly (skipJniAddNativeMethodRegistrationAttributeScan, javaTypes, outputDirectory, appConfState); - return GenerateDebugFiles (skipJniAddNativeMethodRegistrationAttributeScan, javaTypes, outputDirectory, appConfState); + return GenerateDebugNativeAssembly (skipJniAddNativeMethodRegistrationAttributeScan, javaTypes, cache, outputDirectory, appConfState); + return GenerateDebugFiles (skipJniAddNativeMethodRegistrationAttributeScan, javaTypes, cache, outputDirectory, appConfState); } - bool GenerateDebugFiles (bool skipJniAddNativeMethodRegistrationAttributeScan, List javaTypes, string outputDirectory, ApplicationConfigTaskState appConfState) + bool GenerateDebugFiles (bool skipJniAddNativeMethodRegistrationAttributeScan, List javaTypes, TypeDefinitionCache cache, string outputDirectory, ApplicationConfigTaskState appConfState) { var modules = new Dictionary (StringComparer.Ordinal); int maxModuleFileNameWidth = 0; int maxModuleNameWidth = 0; + var javaDuplicates = new Dictionary> (StringComparer.Ordinal); foreach (TypeDefinition td in javaTypes) { UpdateApplicationConfig (td, appConfState); string moduleName = td.Module.Assembly.Name.Name; @@ -186,6 +189,7 @@ bool GenerateDebugFiles (bool skipJniAddNativeMethodRegistrationAttributeScan, L } TypeMapDebugEntry entry = GetDebugEntry (td); + HandleDebugDuplicates (javaDuplicates, entry, td, cache); if (entry.JavaName.Length > module.JavaNameWidth) module.JavaNameWidth = (uint)entry.JavaName.Length + 1; @@ -195,6 +199,7 @@ bool GenerateDebugFiles (bool skipJniAddNativeMethodRegistrationAttributeScan, L module.JavaToManagedMap.Add (entry); module.ManagedToJavaMap.Add (entry); } + SyncDebugDuplicates (javaDuplicates); foreach (ModuleDebugData module in modules.Values) { PrepareDebugMaps (module); @@ -217,18 +222,22 @@ bool GenerateDebugFiles (bool skipJniAddNativeMethodRegistrationAttributeScan, L return true; } - bool GenerateDebugNativeAssembly (bool skipJniAddNativeMethodRegistrationAttributeScan, List javaTypes, string outputDirectory, ApplicationConfigTaskState appConfState) + bool GenerateDebugNativeAssembly (bool skipJniAddNativeMethodRegistrationAttributeScan, List javaTypes, TypeDefinitionCache cache, string outputDirectory, ApplicationConfigTaskState appConfState) { var javaToManaged = new List (); var managedToJava = new List (); + var javaDuplicates = new Dictionary> (StringComparer.Ordinal); foreach (TypeDefinition td in javaTypes) { UpdateApplicationConfig (td, appConfState); TypeMapDebugEntry entry = GetDebugEntry (td); + HandleDebugDuplicates (javaDuplicates, entry, td, cache); + javaToManaged.Add (entry); managedToJava.Add (entry); } + SyncDebugDuplicates (javaDuplicates); var data = new ModuleDebugData { EntryCount = (uint)javaToManaged.Count, @@ -246,6 +255,39 @@ bool GenerateDebugNativeAssembly (bool skipJniAddNativeMethodRegistrationAttribu return true; } + void SyncDebugDuplicates (Dictionary> javaDuplicates) + { + foreach (List duplicates in javaDuplicates.Values) { + if (duplicates.Count < 2) { + continue; + } + + TypeMapDebugEntry template = duplicates [0]; + for (int i = 1; i < duplicates.Count; i++) { + duplicates[i].TypeDefinition = template.TypeDefinition; + duplicates[i].ManagedName = template.ManagedName; + } + } + } + + void HandleDebugDuplicates (Dictionary> javaDuplicates, TypeMapDebugEntry entry, TypeDefinition td, TypeDefinitionCache cache) + { + List duplicates; + + if (!javaDuplicates.TryGetValue (entry.JavaName, out duplicates)) { + javaDuplicates.Add (entry.JavaName, new List { entry }); + } else { + duplicates.Add (entry); + TypeMapDebugEntry oldEntry = duplicates[0]; + if (td.IsAbstract || td.IsInterface || oldEntry.TypeDefinition.IsAbstract || oldEntry.TypeDefinition.IsInterface) { + if (td.IsAssignableFrom (oldEntry.TypeDefinition, cache)) { + oldEntry.TypeDefinition = td; + oldEntry.ManagedName = GetManagedTypeName (td); + } + } + } + } + void PrepareDebugMaps (ModuleDebugData module) { module.JavaToManagedMap.Sort ((TypeMapDebugEntry a, TypeMapDebugEntry b) => String.Compare (a.JavaName, b.JavaName, StringComparison.Ordinal)); @@ -261,6 +303,15 @@ void PrepareDebugMaps (ModuleDebugData module) } TypeMapDebugEntry GetDebugEntry (TypeDefinition td) + { + return new TypeMapDebugEntry { + JavaName = Java.Interop.Tools.TypeNameMappings.JavaNativeTypeManager.ToJniName (td), + ManagedName = GetManagedTypeName (td), + TypeDefinition = td, + }; + } + + string GetManagedTypeName (TypeDefinition td) { // This is necessary because Mono runtime will return to us type name with a `.` for nested types (not a // `/` or a `+`. So, for instance, a type named `DefaultRenderer` found in the @@ -277,17 +328,13 @@ TypeMapDebugEntry GetDebugEntry (TypeDefinition td) // string managedTypeName = td.FullName.Replace ('/', '+'); - return new TypeMapDebugEntry { - JavaName = Java.Interop.Tools.TypeNameMappings.JavaNativeTypeManager.ToJniName (td), - ManagedName = $"{managedTypeName}, {td.Module.Assembly.Name.Name}", - }; + return $"{managedTypeName}, {td.Module.Assembly.Name.Name}"; } bool GenerateRelease (bool skipJniAddNativeMethodRegistrationAttributeScan, List javaTypes, string outputDirectory, ApplicationConfigTaskState appConfState) { int assemblyId = 0; int maxJavaNameLength = 0; - int maxModuleFileNameLength = 0; var knownAssemblies = new Dictionary (StringComparer.Ordinal); var tempModules = new Dictionary (); Dictionary moduleCounter = null;