[Xamarin.Android.Build.Tasks] generate R.java like Android Studio

Context: https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/
Fixes: #2680
Fixes: #2836

The current behavior in the `_GenerateJavaDesignerForComponent`
MSBuild target does the following:

* For each library that has Android resources... (in parallel)
* Run an instance of aapt/aapt2 to generate the `R.java` file for each
  library.
* This actually creates an `R.java` file that contains *every*
  resource id for *every* library. These libraries are not using most
  of these ids.

This has a few problems:

* #2680 notes a problem where a file is locked on Windows during
  `_GenerateJavaDesignerForComponent`.

    Xamarin.Android.Common.targets(1541,2): The process cannot access the file 'C:\repos\msalnet\tests\devapps\XForms\XForms.Android\obj\Debug\90\lp\26\jl\manifest\AndroidManifest.xml' because it is being used by another process.
        at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
        at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
        at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
        at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost)
        at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost)
        at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding)
        at Xamarin.Android.Tasks.ManifestDocument.Save(String filename)
        at Xamarin.Android.Tasks.Aapt.GenerateCommandLineCommands(String ManifestFile, String currentAbi, String currentResourceOutputFile)
        at Xamarin.Android.Tasks.Aapt.ProcessManifest(ITaskItem manifestFile)
        at System.Threading.Tasks.Parallel.<>c__DisplayClass30_0`2.<ForEachWorker>b__0(Int32 i)
        at System.Threading.Tasks.Parallel.<>c__DisplayClass17_0`1.<ForWorker>b__1()
        at System.Threading.Tasks.Task.InnerInvoke()
        at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
        at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object ) [C:\repos\msalnet\tests\devapps\XForms\XForms.Android\XForms.Android.csproj]

* We are hugely contributing to the dex limit for fields. Apps contain
  exponentially more fields for each library with resources.

An example from @PureWeen:

    1>  trouble writing output: Too many field references to fit in one dex file: 70468; max is 65536.

* Quite a few instances of aapt/aapt2 startup on developer's machines:
  this pegs the CPU. We have had a few general complaints about it.

Reviewing the source code for the Android gradle plugin, here is what
they do:

* Build the main app's "full" `R.txt` file.
* For each library, load its `R.txt` file.
* Map each resource in the library's `R.txt` back to the main app
* Write a small `R.java` file for each library: containing *only* the
  lines from the `R.txt` and updated integer values from the main app
  `R.txt` file.

Looking into this, we can do the exact same thing? We have the `R.txt`
file one directory above where we extract resources for each library.
We already had code parsing `R.txt` files I could repurpose, the only
thing *new* is a `R.java` writer: a pretty simple port from java.

The results are great!

    Before:
    3173 ms  _GenerateJavaDesignerForComponentAapt2     1 calls
    After:
      20 ms  GenerateLibraryResources                   1 calls

`_GenerateJavaDesignerForComponent` is now completely gone. This is a
total savings of ~3 seconds on first build and incremental builds
with library changes.

To compare APKs, I used:

    $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-Before.apk | grep ^F

Which omits a line for each field such as:

    F d 0	0	16	xamarin.forms_performance_integration.R$color int abc_background_cache_hint_selector_material_dark

So then, before these changes:

    $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-Before.apk | grep ^F | wc -l
    29681

After:

    $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-After.apk | grep ^F | wc -l
    17210

12K less fields in a "Hello World" Xamarin.Forms app!

Comparing file sizes seems good, too:

    $ zipinfo Xamarin.Forms_Performance_Integration-Before.apk | grep classes.dex
    -rw-rw-r--  6.3 unx  3657872 b- defX 19-Mar-28 16:37 classes.dex
    $ zipinfo Xamarin.Forms_Performance_Integration-After.apk | grep classes.dex
    -rw-rw-r--  6.3 unx  3533120 b- defX 19-Mar-28 16:20 classes.dex

Dex file in the APK is ~120KB smaller.

~~ What if R.txt is missing? ~~

I found this was the case when the
`<GetAdditionalResourcesFromAssemblies/>` MSBuild task runs. This is
an old codepath that allowed old support libraries to work.

In this case, a directory is created such as:

* `obj\Debug\resourcecache\CF390EBB0064FDA00BB090E733D37E89`
  * `adil`
  * `assets`
  * `libs`
  * `res`
  * `AndroidManifest.xml`
  * `classes.jar`

No `R.txt` file?

Checking the zip files we download:

    $ for z in ~/.local/share/Xamarin/zips/*.zip; do zipinfo $z; done | grep R.txt
    # no results

This actually makes sense, since the zip file contains the *actual
resources*.

To make this case work properly, we should just process the main app's
`R.txt` file when no library `R.txt` file is found. This will still be
faster than invoking `aapt`, even though we have more fields than
needed.

~~ Tests ~~

I added a set of unit tests for the `<GenerateLibraryResources/>`
MSBuild task. I also had to remove a few assertions that looked for
the `_GenerateJavaDesignerForComponent` MSBuild target.

Lastly, I added some assertions to a test that uses an old support
library to verify it's main `R.java` reasonably matches the library
`R.java` we generate.

src/Xamarin.Android.Build.Tasks/MSBuild/Xamarin/Android/Xamarin.Android.Aapt2.targets

@@ -138,32 +138,4 @@ Copyright (C) 2011-2012 Xamarin. All rights reserved.
   </ItemGroup>
 </Target>
 
-<Target Name="_GenerateJavaDesignerForComponentAapt2"
-    Condition=" '$(_AndroidUseAapt2)' == 'True' "
-    Inputs="@(_AdditonalAndroidResourceCacheFiles);@(_LibraryResourceDirectoryStamps);$(_AndroidResgenFlagFile)"
-    Outputs="$(_AndroidComponentResgenFlagFile)">
-  <!-- Run aapt to generate R.java for additional Android resources-->
-  <Aapt2Link
-      Condition=" '$(_AndroidUseAapt2)' == 'True' "
-      ContinueOnError="$(DesignTimeBuild)"
-      AdditionalAndroidResourcePaths="@(_LibraryResourceHashDirectories)"
-      AdditionalResourceArchives="@(_LibraryResourceHashDirectories->'$(_AndroidLibraryFlatArchivesDirectory)%(Hash).flata')"
-      ApplicationName="$(_AndroidPackage)"
-      AssemblyIdentityMapFile="$(_AndroidLibrayProjectAssemblyMapFile)"
-      CompiledResourceFlatArchive="$(_AndroidLibraryFlatArchivesDirectory)\compiled.flata"
-      ExtraArgs="$(AndroidAapt2LinkExtraArgs)"
-      ImportsDirectory="$(_LibraryProjectImportsDirectoryName)"
-      JavaDesignerOutputDirectory="$(IntermediateOutputPath)android\src"
-      JavaPlatformJarPath="$(JavaPlatformJarPath)"
-      ManifestFiles="@(_AdditonalAndroidResourceCachePaths->'%(Identity)\AndroidManifest.xml');@(LibraryResourceDirectories->'%(Identity)\..\AndroidManifest.xml')"
-      OutputImportDirectory="$(_AndroidLibrayProjectIntermediatePath)"
-      ResourceNameCaseMap="$(_AndroidResourceNameCaseMap)"
-      ResourceDirectories="$(MonoAndroidResDirIntermediate)"
-      ToolPath="$(Aapt2ToolPath)"
-      ToolExe="$(Aapt2ToolExe)"
-      UseShortFileNames="$(UseShortFileNames)"
-      YieldDuringToolExecution="$(YieldDuringToolExecution)"
-  />
-  <Touch Files="$(_AndroidComponentResgenFlagFile)" AlwaysCreate="True" />
-</Target>
 </Project>

src/Xamarin.Android.Build.Tasks/Tasks/GenerateLibraryResources.cs

@@ -0,0 +1,234 @@
+using Microsoft.Build.Framework;
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Text;
+using Xamarin.Android.Tools;
+using Xamarin.Build;
+
+namespace Xamarin.Android.Tasks
+{
+	/// <summary>
+	/// We used to invoke aapt/aapt2 per library (many times!), this task does the work to generate R.java for libraries without calling aapt/aapt2.
+	/// </summary>
+	public class GenerateLibraryResources : AsyncTask
+	{
+		/// <summary>
+		/// The main R.txt for the app
+		/// </summary>
+		[Required]
+		public string ResourceSymbolsTextFile { get; set; }
+
+		/// <summary>
+		/// The output directory for Java source code, such as: $(IntermediateOutputPath)android\src
+		/// </summary>
+		[Required]
+		public string OutputDirectory { get; set; }
+
+		/// <summary>
+		/// The list of R.txt files for each library
+		/// </summary>
+		public string [] LibraryTextFiles { get; set; }
+
+		/// <summary>
+		/// The accompanying manifest file for each library
+		/// </summary>
+		public string [] ManifestFiles { get; set; }
+
+		public override bool Execute ()
+		{
+			Yield ();
+			try {
+				this.RunTask (DoExecute).ContinueWith (Complete);
+
+				base.Execute ();
+			} finally {
+				Reacquire ();
+			}
+
+			return !Log.HasLoggedErrors;
+		}
+
+		string main_r_txt;
+		string output_directory;
+		Dictionary<Tuple<string, string>, string> r_txt_mapping;
+
+		void DoExecute ()
+		{
+			if (LibraryTextFiles == null || LibraryTextFiles.Length == 0)
+				return;
+
+			// Load the "main" R.txt file into a dictionary
+			main_r_txt = Path.GetFullPath (ResourceSymbolsTextFile);
+			r_txt_mapping = new Dictionary<Tuple<string, string>, string> ();
+			using (var reader = File.OpenText (main_r_txt)) {
+				foreach (var line in ParseFile (reader)) {
+					var key = new Tuple<string, string> (line [Index.Class], line [Index.Name]);
+					r_txt_mapping [key] = line [Index.Value];
+				}
+			}
+
+			Directory.CreateDirectory (OutputDirectory);
+			output_directory = Path.GetFullPath (OutputDirectory);
+
+			var libraries = LibraryTextFiles.Zip (ManifestFiles, (textFile, manifestFile) => new Library (textFile, manifestFile));
+			this.ParallelForEach (libraries, GenerateJava);
+		}
+
+		/// <summary>
+		/// A quick class to combine the paths to R.txt and Manifest
+		/// </summary>
+		class Library
+		{
+			public Library (string textFile, string manifestFile)
+			{
+				TextFile = Path.GetFullPath (textFile);
+				ManifestFile = Path.GetFullPath (manifestFile);
+			}
+
+			/// <summary>
+			/// A full path to the R.txt file
+			/// </summary>
+			public string TextFile { get; }
+
+			/// <summary>
+			/// A full path to the AndroidManifest.xml file
+			/// </summary>
+			public string ManifestFile { get; }
+		}
+
+		/// <summary>
+		/// NOTE: all file paths used in this method should be full paths. (Or use AsyncTask.WorkingDirectory)
+		/// </summary>
+		void GenerateJava (Library library)
+		{
+			// In some cases (such as ancient support libraries), R.txt does not exist.
+			// We can just use the main app's R.txt file and write *all fields* in this case.
+			bool using_main_r_txt = false;
+			var r_txt = library.TextFile;
+			if (!File.Exists (r_txt)) {
+				LogDebugMessage ($"Using main R.txt, R.txt does not exist: {r_txt}");
+				using_main_r_txt = true;
+				r_txt = main_r_txt;
+			}
+
+			var manifestFile = library.ManifestFile;
+			if (!File.Exists (manifestFile)) {
+				LogDebugMessage ($"Skipping, AndroidManifest.xml does not exist: {manifestFile}");
+				return;
+			}
+
+			var manifest = AndroidAppManifest.Load (manifestFile, MonoAndroidHelper.SupportedVersions);
+
+			using (var memory = new MemoryStream ())
+			using (var writer = new StreamWriter (memory, Encoding)) {
+				// This code is based on the Android gradle plugin
+				// https://android.googlesource.com/platform/tools/base/+/908b391a9c006af569dfaff08b37f8fdd6c4da89/build-system/builder/src/main/java/com/android/builder/internal/SymbolWriter.java
+
+				writer.WriteLine ("/* AUTO-GENERATED FILE. DO NOT MODIFY.");
+				writer.WriteLine (" *");
+				writer.WriteLine (" * This class was automatically generated by");
+				writer.WriteLine (" * Xamarin.Android from the resource data it found.");
+				writer.WriteLine (" * It should not be modified by hand.");
+				writer.WriteLine (" */");
+
+				writer.Write ("package ");
+				writer.Write (manifest.PackageName);
+				writer.WriteLine (';');
+				writer.WriteLine ();
+				writer.WriteLine ("public final class R {");
+
+				using (var reader = File.OpenText (r_txt)) {
+					string currentClass = null;
+					foreach (var line in ParseFile (reader)) {
+						var type  = line [Index.Type];
+						var clazz = line [Index.Class];
+						var name  = line [Index.Name];
+						if (GetValue (clazz, name, line, using_main_r_txt, out string value)) {
+							if (clazz != currentClass) {
+								// If not the first inner class
+								if (currentClass != null) {
+									writer.WriteLine ("\t}");
+								}
+
+								currentClass = clazz;
+								writer.Write ("\tpublic static final class ");
+								writer.Write (currentClass);
+								writer.WriteLine (" {");
+							}
+
+							writer.Write ("\t\tpublic static final ");
+							writer.Write (type);
+							writer.Write (' ');
+							writer.Write (name);
+							writer.Write (" = ");
+							// It may be an int[]
+							if (value.StartsWith ("{", StringComparison.Ordinal)) {
+								writer.Write ("new ");
+								writer.Write (type);
+								writer.Write (' ');
+							}
+							writer.Write (value);
+							writer.WriteLine (';');
+						} else {
+							LogDebugMessage ($"{r_txt}: `{type} {clazz} {name}` value not found");
+						}
+					}
+
+					// If we wrote at least one inner class
+					if (currentClass != null) {
+						writer.WriteLine ("\t}");
+					}
+					writer.WriteLine ('}');
+				}
+
+				writer.Flush ();
+				var r_java = Path.Combine (output_directory, manifest.PackageName.Replace ('.', Path.DirectorySeparatorChar), "R.java");
+				if (MonoAndroidHelper.CopyIfStreamChanged (memory, r_java)) {
+					LogDebugMessage ($"Writing: {r_java}");
+				} else {
+					LogDebugMessage ($"Up to date: {r_java}");
+				}
+			}
+		}
+
+		bool GetValue (string clazz, string name, string[] line, bool using_main_r_txt, out string value)
+		{
+			// If this is the main R.txt file, we don't need to do a lookup
+			if (using_main_r_txt) {
+				value = line [Index.Value];
+				return true;
+			}
+
+			var key = new Tuple<string, string> (clazz, name);
+			return r_txt_mapping.TryGetValue (key, out value);
+		}
+
+		static readonly Encoding Encoding = new UTF8Encoding (encoderShouldEmitUTF8Identifier: false);
+		static readonly char [] Delimiter = new [] { ' ' };
+
+		class Index
+		{
+			public const int Type  = 0;
+			public const int Class = 1;
+			public const int Name  = 2;
+			public const int Value = 3;
+		}
+
+		/// <summary>
+		/// R.txt is of the format:
+		///    int id icon 0x7f0c000a
+		///    int[] styleable ViewStubCompat { 0x010100d0, 0x010100f2, 0x010100f3 }
+		/// This returns a 4-length string[] of the parts.
+		/// </summary>
+		IEnumerable<string []> ParseFile (StreamReader reader)
+		{
+			while (!reader.EndOfStream) {
+				var line = reader.ReadLine ();
+				var items = line.Split (Delimiter, 4);
+				yield return items;
+			}
+		}
+	}
+}

src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/AndroidUpdateResourcesTest.cs

@@ -638,36 +638,6 @@ public void CheckOldResourceDesignerWithWrongCasingIsRemoved (bool isRelease, Pr
 			}
 		}
 
-		[Test]
-		public void TargetGenerateJavaDesignerForComponentIsSkipped ([Values(false, true)] bool isRelease)
-		{ 
-			// build with packages... then tweak a package..
-			var proj = new XamarinAndroidApplicationProject () {
-				IsRelease = isRelease,
-			};
-			proj.PackageReferences.Add (KnownPackages.AndroidSupportV4_21_0_3_0);
-			proj.PackageReferences.Add (KnownPackages.SupportV7AppCompat_21_0_3_0);
-			proj.SetProperty ("TargetFrameworkVersion", "v5.0");
-			using (var b = CreateApkBuilder (Path.Combine ("temp", TestContext.CurrentContext.Test.Name))) {
-				b.Verbosity = LoggerVerbosity.Diagnostic;
-				Assert.IsTrue (b.Build (proj), "Build should have succeeded.");
-				StringAssertEx.DoesNotContain ("Skipping target \"_GenerateJavaDesignerForComponent\" because",
-					b.LastBuildOutput, "Target _GenerateJavaDesignerForComponent should not have been skipped");
-				Assert.IsTrue (b.Build (proj), "Build should have succeeded.");
-				StringAssertEx.Contains ("Skipping target \"_GenerateJavaDesignerForComponent\" because",
-					b.LastBuildOutput, "Target _GenerateJavaDesignerForComponent should have been skipped");
-				var files = Directory.EnumerateFiles (Path.Combine (Root, b.ProjectDirectory, proj.IntermediateOutputPath, "resourcecache")
-					, "abc_fade_in.xml", SearchOption.AllDirectories);
-				Assert.AreEqual (1, files.Count (), "There should only be one abc_fade_in.xml in the resourcecache");
-				var resFile = files.First ();
-				Assert.IsTrue (File.Exists (resFile), "{0} should exist", resFile);
-				File.SetLastWriteTime (resFile, DateTime.UtcNow);
-				Assert.IsTrue (b.Build (proj), "Build should have succeeded.");
-				StringAssertEx.DoesNotContain ("Skipping target \"_GenerateJavaDesignerForComponent\" because",
-					b.LastBuildOutput, "Target _GenerateJavaDesignerForComponent should not have been skipped");
-			}
-		}
-
 		[Test]
 		public void CheckAaptErrorRaisedForMissingResource ()
 		{

src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/BuildTest.cs

@@ -350,7 +350,6 @@ public CustomTextView(Context context, IAttributeSet attributes) : base(context,
 					//      And so the built assembly changes between DTB and regular build, triggering `_LinkAssembliesNoShrink`
 					//"_LinkAssembliesNoShrink",
 					"_UpdateAndroidResgen",
-					"_GenerateJavaDesignerForComponentAapt2",
 					"_BuildLibraryImportsCache",
 					"_CompileJava",
 				};
@@ -455,7 +454,6 @@ public void CheckTimestamps ([Values (true, false)] bool isRelease)
 				var targetsToBeSkipped = new [] {
 					isRelease ? "_LinkAssembliesShrink" : "_LinkAssembliesNoShrink",
 					"_UpdateAndroidResgen",
-					"_GenerateJavaDesignerForComponentAapt2",
 					"_BuildLibraryImportsCache",
 					"_CompileJava",
 				};
@@ -2443,6 +2441,37 @@ public void BuildReleaseApplicationWithNugetPackages ()
 				var assets = b.Output.GetIntermediaryAsText (Path.Combine ("..", "project.assets.json"));
 				StringAssert.Contains ("Xamarin.Android.Support.v4", assets,
 					"Nuget Package Xamarin.Android.Support.v4.21.0.3.0 should have been restored.");
+
+				//Since this is using an old support library, its main R.java should "match" the library one
+				var src = Path.Combine (Root, b.ProjectDirectory, proj.IntermediateOutputPath, "android", "src");
+				var main_r_java = Path.Combine (src, "unnamedproject", "unnamedproject", "R.java");
+				FileAssert.Exists (main_r_java);
+				var lib_r_java = Path.Combine (src, "android", "support", "v4", "R.java");
+				FileAssert.Exists (lib_r_java);
+
+				void TrimHeader (List<string> lines)
+				{
+					for (int i = 0; i < lines.Count; i++) {
+						if (lines [i].StartsWith ("package ", StringComparison.Ordinal)) {
+							lines.RemoveRange (0, i + 1);
+							break;
+						}
+					}
+				}
+
+				//Beyond the `package com.foo;` line, each line should match: ignoring whitespace
+				var main_r_contents = File.ReadAllLines (main_r_java).ToList ();
+				TrimHeader (main_r_contents);
+				var lib_r_contents = File.ReadAllLines (lib_r_java).ToList ();
+				TrimHeader (lib_r_contents);
+				var regex = new Regex (@"\s", RegexOptions.Compiled);
+				for (int i = 0; i < main_r_contents.Count && i < lib_r_contents.Count; i++) {
+					var main = main_r_contents [i];
+					var lib = lib_r_contents [i];
+					var expected = regex.Replace (main, "");
+					var actual = regex.Replace (lib, "");
+					Assert.AreEqual (expected, actual, $"Main R.java `{main}` does not match library R.java `{lib}");
+				}
 			}
 		}