Rework _GenerateJavaDesignerForComponent
#2836
Comments
|
So if I look at the It looks like this switch is putting If we built a single I think there can be Java code that depends on |
|
We use the list of |
|
Ok, I see:
Since we are already doing things this way, we can reduce this to a single The only other diffs I see are in the |
|
Ok, I think our current behavior actually might be a problem:
I have not had time to actually investigate and see if this is the case--and see by how much. For a "Hello World" Xamarin.Forms app, if I look at Does this mean we have potentially some amount of extra fields? Could be some percentage of 18,136 fields (8 x 2,267)? I see the same number of fields with |
|
Ok so if I just look at a random class:
Here is dexdump output of each: dexdump.zip |
|
The problem we have is there are java classes in the support libs which reference So I believe that the whole point of |
jonathanpeppers
added a commit
to jonathanpeppers/xamarin-android
that referenced
this issue
Mar 26, 2019
Context: dotnet#2836 (comment) We have noticed apps using Xamarin.Forms.Maps might be close to hitting the dex limit for fields. An example from @PureWeen: 1> trouble writing output: Too many field references to fit in one dex file: 70468; max is 65536. We think the way Xamarin.Android invokes `aapt` (or `aapt2`) might be creating `R.java` files with more fields than are needed. Step 1 is to add a test project that uses Xamarin.Forms.Maps: we did not have one.
jonpryor
pushed a commit
that referenced
this issue
Mar 27, 2019
Context: #2836 (comment) We have noticed apps using Xamarin.Forms.Maps might be close to hitting the dex limit for fields. An example from @PureWeen: 1> trouble writing output: Too many field references to fit in one dex file: 70468; max is 65536. We think the way Xamarin.Android invokes `aapt` (or `aapt2`) might be creating `R.java` files with more fields than are needed. Step 1 is to add a test project that uses Xamarin.Forms.Maps: we did not have one.
|
Ok, so I think I understand now what Java/Android Studio does. At the top level, the Android gradle plugin has a It runs Then the This Java code writes out the individual So we would have to port this code to C#? Or consider integrating with |
|
lets not integrate with gradle if we can help it. We already produce the R.txt with our calls to |
|
We have a basic my App Looking at the code you posted it seems the Luckily we do include that in our extraction process so we have So I guess they do some kind of merge process. between that main one and the one the library has. |
For the record, the other side acting the same way with Since I have OCD against junk IL code, I try to move resource access out from XA Libraries, to the root App via XML + IDs (this way the |
|
@erikpowa if you enable the linker (link all assemblies) are these removed? You can also add You can also prevent |
Not really... especially from the
I never heard of it actually. Forgot to mention that my context is with
Well if I omit it, how would that compile if I have resource that needs to be accessed? or how the resource id would point to the correct resource ? since the Btw the |
|
@erikpowa so I think we should revisit the managed side after fixing the java side. Either a new linker step that strips unused fields--or maybe we don't need to generate as much C# code in the first place? Although, for Intellisense to work, it would need fields that aren't in use yet... If you have a linker step fixing this up, is it compatible with our stuff? would you be willing to open a PR? |
|
@jonathanpeppers I haven't dived myself into the designtime resource-generator, but the only problem is the "strong references" to the fields. As I see as for solution: If the res designer already knows what resource will be set for which static field, then why isn't this knowledge/process moved out?
For resource cleaning, I have just temporary solution, unsafe and not optimized at all. |
jonathanpeppers
added a commit
to jonathanpeppers/xamarin-android
that referenced
this issue
Mar 29, 2019
Context: https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/ Fixes: dotnet#2680 Fixes: dotnet#2836 The current behavior in the `_GenerateJavaDesignerForComponent` MSBuild target does the following: * For each library that has Android resources... (in parallel) * Run an instance of aapt/aapt2 to generate the `R.java` file for each library. * This actually creates an `R.java` file that contains *every* resource id for *every* library. These libraries are not using most of these ids. This has a few problems: * dotnet#2680 notes a problem where a file is locked on Windows during `_GenerateJavaDesignerForComponent`. Xamarin.Android.Common.targets(1541,2): The process cannot access the file 'C:\repos\msalnet\tests\devapps\XForms\XForms.Android\obj\Debug\90\lp\26\jl\manifest\AndroidManifest.xml' because it is being used by another process. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost) at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost) at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding) at Xamarin.Android.Tasks.ManifestDocument.Save(String filename) at Xamarin.Android.Tasks.Aapt.GenerateCommandLineCommands(String ManifestFile, String currentAbi, String currentResourceOutputFile) at Xamarin.Android.Tasks.Aapt.ProcessManifest(ITaskItem manifestFile) at System.Threading.Tasks.Parallel.<>c__DisplayClass30_0`2.<ForEachWorker>b__0(Int32 i) at System.Threading.Tasks.Parallel.<>c__DisplayClass17_0`1.<ForWorker>b__1() at System.Threading.Tasks.Task.InnerInvoke() at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask) at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object ) [C:\repos\msalnet\tests\devapps\XForms\XForms.Android\XForms.Android.csproj] * We are hugely contributing to the dex limit for fields. Apps contain exponentially more fields for each library with resources. An example from @PureWeen: 1> trouble writing output: Too many field references to fit in one dex file: 70468; max is 65536. * Quite a few instances of aapt/aapt2 startup on developer's machines: this pegs the CPU. We have had a few general complaints about it. Reviewing the source code for the Android gradle plugin, here is what they do: * Build the main app's "full" `R.txt` file. * For each library, load its `R.txt` file. * Map each resource in the library's `R.txt` back to the main app * Write a small `R.java` file for each library: containing *only* the lines from the `R.txt` and updated integer values from the main app `R.txt` file. Looking into this, we can do the exact same thing? We have the `R.txt` file one directory above where we extract resources for each library. We already had code parsing `R.txt` files I could repurpose, the only thing *new* is a `R.java` writer: a pretty simple port from java. The results are great! Before: 3173 ms _GenerateJavaDesignerForComponentAapt2 1 calls After: 20 ms GenerateLibraryResources 1 calls `_GenerateJavaDesignerForComponent` is now completely gone. This is a total savings of ~3 seconds on first build and incremental builds with library changes. To compare APKs, I used: $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-Before.apk | grep ^F Which omits a line for each field such as: F d 0 0 16 xamarin.forms_performance_integration.R$color int abc_background_cache_hint_selector_material_dark So then, before these changes: $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-Before.apk | grep ^F | wc -l 29681 After: $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-After.apk | grep ^F | wc -l 17210 12K less fields in a "Hello World" Xamarin.Forms app! Comparing file sizes seems good, too: $ zipinfo Xamarin.Forms_Performance_Integration-Before.apk | grep classes.dex -rw-rw-r-- 6.3 unx 3657872 b- defX 19-Mar-28 16:37 classes.dex $ zipinfo Xamarin.Forms_Performance_Integration-After.apk | grep classes.dex -rw-rw-r-- 6.3 unx 3533120 b- defX 19-Mar-28 16:20 classes.dex Dex file in the APK is ~120KB smaller.
|
@erikpowa to add a new linker step, you would probably have to add a new linker step to I'll file a new issue about this--it is kind of unrelated to the |
jonathanpeppers
added a commit
to jonathanpeppers/xamarin-android
that referenced
this issue
Mar 29, 2019
Context: https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/ Fixes: dotnet#2680 Fixes: dotnet#2836 The current behavior in the `_GenerateJavaDesignerForComponent` MSBuild target does the following: * For each library that has Android resources... (in parallel) * Run an instance of aapt/aapt2 to generate the `R.java` file for each library. * This actually creates an `R.java` file that contains *every* resource id for *every* library. These libraries are not using most of these ids. This has a few problems: * dotnet#2680 notes a problem where a file is locked on Windows during `_GenerateJavaDesignerForComponent`. Xamarin.Android.Common.targets(1541,2): The process cannot access the file 'C:\repos\msalnet\tests\devapps\XForms\XForms.Android\obj\Debug\90\lp\26\jl\manifest\AndroidManifest.xml' because it is being used by another process. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost) at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost) at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding) at Xamarin.Android.Tasks.ManifestDocument.Save(String filename) at Xamarin.Android.Tasks.Aapt.GenerateCommandLineCommands(String ManifestFile, String currentAbi, String currentResourceOutputFile) at Xamarin.Android.Tasks.Aapt.ProcessManifest(ITaskItem manifestFile) at System.Threading.Tasks.Parallel.<>c__DisplayClass30_0`2.<ForEachWorker>b__0(Int32 i) at System.Threading.Tasks.Parallel.<>c__DisplayClass17_0`1.<ForWorker>b__1() at System.Threading.Tasks.Task.InnerInvoke() at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask) at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object ) [C:\repos\msalnet\tests\devapps\XForms\XForms.Android\XForms.Android.csproj] * We are hugely contributing to the dex limit for fields. Apps contain exponentially more fields for each library with resources. An example from @PureWeen: 1> trouble writing output: Too many field references to fit in one dex file: 70468; max is 65536. * Quite a few instances of aapt/aapt2 startup on developer's machines: this pegs the CPU. We have had a few general complaints about it. Reviewing the source code for the Android gradle plugin, here is what they do: * Build the main app's "full" `R.txt` file. * For each library, load its `R.txt` file. * Map each resource in the library's `R.txt` back to the main app * Write a small `R.java` file for each library: containing *only* the lines from the `R.txt` and updated integer values from the main app `R.txt` file. Looking into this, we can do the exact same thing? We have the `R.txt` file one directory above where we extract resources for each library. We already had code parsing `R.txt` files I could repurpose, the only thing *new* is a `R.java` writer: a pretty simple port from java. The results are great! Before: 3173 ms _GenerateJavaDesignerForComponentAapt2 1 calls After: 20 ms GenerateLibraryResources 1 calls `_GenerateJavaDesignerForComponent` is now completely gone. This is a total savings of ~3 seconds on first build and incremental builds with library changes. To compare APKs, I used: $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-Before.apk | grep ^F Which omits a line for each field such as: F d 0 0 16 xamarin.forms_performance_integration.R$color int abc_background_cache_hint_selector_material_dark So then, before these changes: $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-Before.apk | grep ^F | wc -l 29681 After: $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-After.apk | grep ^F | wc -l 17210 12K less fields in a "Hello World" Xamarin.Forms app! Comparing file sizes seems good, too: $ zipinfo Xamarin.Forms_Performance_Integration-Before.apk | grep classes.dex -rw-rw-r-- 6.3 unx 3657872 b- defX 19-Mar-28 16:37 classes.dex $ zipinfo Xamarin.Forms_Performance_Integration-After.apk | grep classes.dex -rw-rw-r-- 6.3 unx 3533120 b- defX 19-Mar-28 16:20 classes.dex Dex file in the APK is ~120KB smaller. ~~ What if R.txt is missing? ~~ I found this was the case when the `<GetAdditionalResourcesFromAssemblies/>` MSBuild task runs. This is an old codepath that allowed old support libraries to work. In this case, a directory is created such as: * `obj\Debug\resourcecache\CF390EBB0064FDA00BB090E733D37E89` * `adil` * `assets` * `libs` * `res` * `AndroidManifest.xml` * `classes.jar` No `R.txt` file? Checking the zip files we download: $ for z in ~/.local/share/Xamarin/zips/*.zip; do zipinfo $z; done | grep R.txt # no results This actually makes sense, since the zip file contains the *actual resources*. To make this case work properly, we should just process the main app's `R.txt` file when no library `R.txt` file is found. This will still be faster than invoking `aapt`, even though we have more fields than needed. ~~ Tests ~~ I added a set of unit tests for the `<GenerateLibraryResources/>` MSBuild task. I also had to remove a few assertions that looked for the `_GenerateJavaDesignerForComponent` MSBuild target. Lastly, I added some assertions to a test that uses an old support library to verify it's main `R.java` reasonably matches the library `R.java` we generate.
jonathanpeppers
added a commit
to jonathanpeppers/xamarin-android
that referenced
this issue
Apr 1, 2019
Context: https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/ Fixes: dotnet#2680 Fixes: dotnet#2836 The current behavior in the `_GenerateJavaDesignerForComponent` MSBuild target does the following: * For each library that has Android resources... (in parallel) * Run an instance of aapt/aapt2 to generate the `R.java` file for each library. * This actually creates an `R.java` file that contains *every* resource id for *every* library. These libraries are not using most of these ids. This has a few problems: * dotnet#2680 notes a problem where a file is locked on Windows during `_GenerateJavaDesignerForComponent`. Xamarin.Android.Common.targets(1541,2): The process cannot access the file 'C:\repos\msalnet\tests\devapps\XForms\XForms.Android\obj\Debug\90\lp\26\jl\manifest\AndroidManifest.xml' because it is being used by another process. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost) at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost) at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding) at Xamarin.Android.Tasks.ManifestDocument.Save(String filename) at Xamarin.Android.Tasks.Aapt.GenerateCommandLineCommands(String ManifestFile, String currentAbi, String currentResourceOutputFile) at Xamarin.Android.Tasks.Aapt.ProcessManifest(ITaskItem manifestFile) at System.Threading.Tasks.Parallel.<>c__DisplayClass30_0`2.<ForEachWorker>b__0(Int32 i) at System.Threading.Tasks.Parallel.<>c__DisplayClass17_0`1.<ForWorker>b__1() at System.Threading.Tasks.Task.InnerInvoke() at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask) at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object ) [C:\repos\msalnet\tests\devapps\XForms\XForms.Android\XForms.Android.csproj] * We are hugely contributing to the dex limit for fields. Apps contain exponentially more fields for each library with resources. An example from @PureWeen: 1> trouble writing output: Too many field references to fit in one dex file: 70468; max is 65536. * Quite a few instances of aapt/aapt2 startup on developer's machines: this pegs the CPU. We have had a few general complaints about it. Reviewing the source code for the Android gradle plugin, here is what they do: * Build the main app's "full" `R.txt` file. * For each library, load its `R.txt` file. * Map each resource in the library's `R.txt` back to the main app * Write a small `R.java` file for each library: containing *only* the lines from the `R.txt` and updated integer values from the main app `R.txt` file. Looking into this, we can do the exact same thing? We have the `R.txt` file one directory above where we extract resources for each library. We already had code parsing `R.txt` files I could repurpose, the only thing *new* is a `R.java` writer: a pretty simple port from java. The results are great! Before: 3173 ms _GenerateJavaDesignerForComponentAapt2 1 calls After: 20 ms GenerateLibraryResources 1 calls `_GenerateJavaDesignerForComponent` is now completely gone. This is a total savings of ~3 seconds on first build and incremental builds with library changes. To compare APKs, I used: $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-Before.apk | grep ^F Which omits a line for each field such as: F d 0 0 16 xamarin.forms_performance_integration.R$color int abc_background_cache_hint_selector_material_dark So then, before these changes: $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-Before.apk | grep ^F | wc -l 29681 After: $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-After.apk | grep ^F | wc -l 17210 12K less fields in a "Hello World" Xamarin.Forms app! Comparing file sizes seems good, too: $ zipinfo Xamarin.Forms_Performance_Integration-Before.apk | grep classes.dex -rw-rw-r-- 6.3 unx 3657872 b- defX 19-Mar-28 16:37 classes.dex $ zipinfo Xamarin.Forms_Performance_Integration-After.apk | grep classes.dex -rw-rw-r-- 6.3 unx 3533120 b- defX 19-Mar-28 16:20 classes.dex Dex file in the APK is ~120KB smaller. ~~ What if R.txt is missing? ~~ I found this was the case when the `<GetAdditionalResourcesFromAssemblies/>` MSBuild task runs. This is an old codepath that allowed old support libraries to work. In this case, a directory is created such as: * `obj\Debug\resourcecache\CF390EBB0064FDA00BB090E733D37E89` * `adil` * `assets` * `libs` * `res` * `AndroidManifest.xml` * `classes.jar` No `R.txt` file? Checking the zip files we download: $ for z in ~/.local/share/Xamarin/zips/*.zip; do zipinfo $z; done | grep R.txt # no results This actually makes sense, since the zip file contains the *actual resources*. To make this case work properly, we should just process the main app's `R.txt` file when no library `R.txt` file is found. This will still be faster than invoking `aapt`, even though we have more fields than needed. ~~ Tests ~~ I added a set of unit tests for the `<GenerateLibraryResources/>` MSBuild task. I also had to remove a few assertions that looked for the `_GenerateJavaDesignerForComponent` MSBuild target. Lastly, I added some assertions to a test that uses an old support library to verify it's main `R.java` reasonably matches the library `R.java` we generate.
jonathanpeppers
added a commit
to jonathanpeppers/xamarin-android
that referenced
this issue
Apr 1, 2019
Context: https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/ Fixes: dotnet#2680 Fixes: dotnet#2836 The current behavior in the `_GenerateJavaDesignerForComponent` MSBuild target does the following: * For each library that has Android resources... (in parallel) * Run an instance of aapt/aapt2 to generate the `R.java` file for each library. * This actually creates an `R.java` file that contains *every* resource id for *every* library. These libraries are not using most of these ids. This has a few problems: * dotnet#2680 notes a problem where a file is locked on Windows during `_GenerateJavaDesignerForComponent`. Xamarin.Android.Common.targets(1541,2): The process cannot access the file 'C:\repos\msalnet\tests\devapps\XForms\XForms.Android\obj\Debug\90\lp\26\jl\manifest\AndroidManifest.xml' because it is being used by another process. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost) at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost) at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding) at Xamarin.Android.Tasks.ManifestDocument.Save(String filename) at Xamarin.Android.Tasks.Aapt.GenerateCommandLineCommands(String ManifestFile, String currentAbi, String currentResourceOutputFile) at Xamarin.Android.Tasks.Aapt.ProcessManifest(ITaskItem manifestFile) at System.Threading.Tasks.Parallel.<>c__DisplayClass30_0`2.<ForEachWorker>b__0(Int32 i) at System.Threading.Tasks.Parallel.<>c__DisplayClass17_0`1.<ForWorker>b__1() at System.Threading.Tasks.Task.InnerInvoke() at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask) at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object ) [C:\repos\msalnet\tests\devapps\XForms\XForms.Android\XForms.Android.csproj] * We are hugely contributing to the dex limit for fields. Apps contain exponentially more fields for each library with resources. An example from @PureWeen: 1> trouble writing output: Too many field references to fit in one dex file: 70468; max is 65536. * Quite a few instances of aapt/aapt2 startup on developer's machines: this pegs the CPU. We have had a few general complaints about it. Reviewing the source code for the Android gradle plugin, here is what they do: * Build the main app's "full" `R.txt` file. * For each library, load its `R.txt` file. * Map each resource in the library's `R.txt` back to the main app * Write a small `R.java` file for each library: containing *only* the lines from the `R.txt` and updated integer values from the main app `R.txt` file. Looking into this, we can do the exact same thing? We have the `R.txt` file one directory above where we extract resources for each library. We already had code parsing `R.txt` files I could repurpose, the only thing *new* is a `R.java` writer: a pretty simple port from java. The results are great! Before: 3173 ms _GenerateJavaDesignerForComponentAapt2 1 calls After: 20 ms GenerateLibraryResources 1 calls `_GenerateJavaDesignerForComponent` is now completely gone. This is a total savings of ~3 seconds on first build and incremental builds with library changes. To compare APKs, I used: $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-Before.apk | grep ^F Which omits a line for each field such as: F d 0 0 16 xamarin.forms_performance_integration.R$color int abc_background_cache_hint_selector_material_dark So then, before these changes: $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-Before.apk | grep ^F | wc -l 29681 After: $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-After.apk | grep ^F | wc -l 17210 12K less fields in a "Hello World" Xamarin.Forms app! Comparing file sizes seems good, too: $ zipinfo Xamarin.Forms_Performance_Integration-Before.apk | grep classes.dex -rw-rw-r-- 6.3 unx 3657872 b- defX 19-Mar-28 16:37 classes.dex $ zipinfo Xamarin.Forms_Performance_Integration-After.apk | grep classes.dex -rw-rw-r-- 6.3 unx 3533120 b- defX 19-Mar-28 16:20 classes.dex Dex file in the APK is ~120KB smaller. ~~ What if R.txt is missing? ~~ I found this was the case when the `<GetAdditionalResourcesFromAssemblies/>` MSBuild task runs. This is an old codepath that allowed old support libraries to work. In this case, a directory is created such as: * `obj\Debug\resourcecache\CF390EBB0064FDA00BB090E733D37E89` * `adil` * `assets` * `libs` * `res` * `AndroidManifest.xml` * `classes.jar` No `R.txt` file? Checking the zip files we download: $ for z in ~/.local/share/Xamarin/zips/*.zip; do zipinfo $z; done | grep R.txt # no results This actually makes sense, since the zip file contains the *actual resources*. To make this case work properly, we should just process the main app's `R.txt` file when no library `R.txt` file is found. This will still be faster than invoking `aapt`, even though we have more fields than needed. ~~ Tests ~~ I added a set of unit tests for the `<GenerateLibraryResources/>` MSBuild task. I also had to remove a few assertions that looked for the `_GenerateJavaDesignerForComponent` MSBuild target. Lastly, I added some assertions to a test that uses an old support library to verify it's main `R.java` reasonably matches the library `R.java` we generate.
jonpryor
pushed a commit
that referenced
this issue
Apr 1, 2019
) Context: https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/ Fixes: #2680 Fixes: #2836 The current behavior in the `_GenerateJavaDesignerForComponent` MSBuild target does the following: * For each library that has Android resources... (in parallel) * Run an instance of `aapt`/`aapt2` to generate the `R.java` file for each library. * This actually creates an `R.java` file that contains *every* resource id for *every* library. These libraries are not using most of these ids. This has a few problems: * Issue #2680 notes a problem where a file is locked on Windows during `_GenerateJavaDesignerForComponent`: Xamarin.Android.Common.targets(1541,2): The process cannot access the file 'C:\repos\msalnet\tests\devapps\XForms\XForms.Android\obj\Debug\90\lp\26\jl\manifest\AndroidManifest.xml' because it is being used by another process. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost) at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost) at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding) at Xamarin.Android.Tasks.ManifestDocument.Save(String filename) at Xamarin.Android.Tasks.Aapt.GenerateCommandLineCommands(String ManifestFile, String currentAbi, String currentResourceOutputFile) at Xamarin.Android.Tasks.Aapt.ProcessManifest(ITaskItem manifestFile) at System.Threading.Tasks.Parallel.<>c__DisplayClass30_0`2.<ForEachWorker>b__0(Int32 i) at System.Threading.Tasks.Parallel.<>c__DisplayClass17_0`1.<ForWorker>b__1() at System.Threading.Tasks.Task.InnerInvoke() at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask) at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object ) [C:\repos\msalnet\tests\devapps\XForms\XForms.Android\XForms.Android.csproj] * We are hugely contributing to the dex limit for fields. Apps contain exponentially more fields for each library with resources. An example from @PureWeen: 1> trouble writing output: Too many field references to fit in one dex file: 70468; max is 65536. * Quite a few instances of `aapt`/`aapt2` startup on developer's machines: this pegs the CPU. We have had a few general complaints about it. Reviewing the source code for the Android gradle plugin, here is what they do: 1. Build the main app's "full" `R.txt` file. 2. For each library, load its `R.txt` file. 3. Map each resource in the library's `R.txt` back to the main app 4. Write a small `R.java` file for each library containing *only* the lines from the `R.txt` and updated integer values from the main app `R.txt` file. Looking into this, can we do the exact same thing? We have the `R.txt` file one directory above where we extract resources for each library. We already had code parsing `R.txt` files we could repurpose. The only thing *new* is a `R.java` writer: a pretty simple port from java. The results are great! Before: 3173 ms _GenerateJavaDesignerForComponentAapt2 1 calls After: 20 ms GenerateLibraryResources 1 calls `_GenerateJavaDesignerForComponent` is now completely gone. This is a total savings of ~3 seconds on first build and incremental builds with library changes. To compare APKs, I used: $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-Before.apk | grep ^F Which omits a line for each field such as: F d 0 0 16 xamarin.forms_performance_integration.R$color int abc_background_cache_hint_selector_material_dark So then, before these changes there were ~30000 fields: $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-Before.apk | grep ^F | wc -l 29681 After, there are less than 18000 (58%!): $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-After.apk | grep ^F | wc -l 17210 12K less fields in a "Hello World" Xamarin.Forms app! Comparing file sizes seems good, too: $ zipinfo Xamarin.Forms_Performance_Integration-Before.apk | grep classes.dex -rw-rw-r-- 6.3 unx 3657872 b- defX 19-Mar-28 16:37 classes.dex $ zipinfo Xamarin.Forms_Performance_Integration-After.apk | grep classes.dex -rw-rw-r-- 6.3 unx 3533120 b- defX 19-Mar-28 16:20 classes.dex The `.dex` file in the `.apk` is ~120KB smaller. ~~ What if R.txt is missing? ~~ I found this was the case when the `<GetAdditionalResourcesFromAssemblies/>` MSBuild task runs. This is an old codepath that allowed old support libraries to work. In this case, a directory is created such as: * `obj\Debug\resourcecache\CF390EBB0064FDA00BB090E733D37E89` * `aidl` * `AndroidManifest.xml` * `assets` * `classes.jar` * `libs` * `res` No `R.txt` file? Checking the zip files we download: $ for z in ~/.local/share/Xamarin/zips/*.zip; do zipinfo $z; done | grep R.txt # no results This actually makes sense, since the zip file contains the *actual resources*. To make this case work properly, we should just process the main app's `R.txt` file when no library `R.txt` file is found. This will still be faster than invoking `aapt`, even though we have more fields than needed. ~~ Tests ~~ I added a set of unit tests for the `<GenerateLibraryResources/>` MSBuild task. I also had to remove a few assertions that looked for the `_GenerateJavaDesignerForComponent` MSBuild target. Lastly, I added some assertions to a test that uses an old support library to verify it's main `R.java` reasonably matches the library `R.java` we generate.
jonathanpeppers
added a commit
that referenced
this issue
Apr 1, 2019
) Context: https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/ Fixes: #2680 Fixes: #2836 The current behavior in the `_GenerateJavaDesignerForComponent` MSBuild target does the following: * For each library that has Android resources... (in parallel) * Run an instance of `aapt`/`aapt2` to generate the `R.java` file for each library. * This actually creates an `R.java` file that contains *every* resource id for *every* library. These libraries are not using most of these ids. This has a few problems: * Issue #2680 notes a problem where a file is locked on Windows during `_GenerateJavaDesignerForComponent`: Xamarin.Android.Common.targets(1541,2): The process cannot access the file 'C:\repos\msalnet\tests\devapps\XForms\XForms.Android\obj\Debug\90\lp\26\jl\manifest\AndroidManifest.xml' because it is being used by another process. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost) at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost) at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding) at Xamarin.Android.Tasks.ManifestDocument.Save(String filename) at Xamarin.Android.Tasks.Aapt.GenerateCommandLineCommands(String ManifestFile, String currentAbi, String currentResourceOutputFile) at Xamarin.Android.Tasks.Aapt.ProcessManifest(ITaskItem manifestFile) at System.Threading.Tasks.Parallel.<>c__DisplayClass30_0`2.<ForEachWorker>b__0(Int32 i) at System.Threading.Tasks.Parallel.<>c__DisplayClass17_0`1.<ForWorker>b__1() at System.Threading.Tasks.Task.InnerInvoke() at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask) at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object ) [C:\repos\msalnet\tests\devapps\XForms\XForms.Android\XForms.Android.csproj] * We are hugely contributing to the dex limit for fields. Apps contain exponentially more fields for each library with resources. An example from @PureWeen: 1> trouble writing output: Too many field references to fit in one dex file: 70468; max is 65536. * Quite a few instances of `aapt`/`aapt2` startup on developer's machines: this pegs the CPU. We have had a few general complaints about it. Reviewing the source code for the Android gradle plugin, here is what they do: 1. Build the main app's "full" `R.txt` file. 2. For each library, load its `R.txt` file. 3. Map each resource in the library's `R.txt` back to the main app 4. Write a small `R.java` file for each library containing *only* the lines from the `R.txt` and updated integer values from the main app `R.txt` file. Looking into this, can we do the exact same thing? We have the `R.txt` file one directory above where we extract resources for each library. We already had code parsing `R.txt` files we could repurpose. The only thing *new* is a `R.java` writer: a pretty simple port from java. The results are great! Before: 3173 ms _GenerateJavaDesignerForComponentAapt2 1 calls After: 20 ms GenerateLibraryResources 1 calls `_GenerateJavaDesignerForComponent` is now completely gone. This is a total savings of ~3 seconds on first build and incremental builds with library changes. To compare APKs, I used: $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-Before.apk | grep ^F Which omits a line for each field such as: F d 0 0 16 xamarin.forms_performance_integration.R$color int abc_background_cache_hint_selector_material_dark So then, before these changes there were ~30000 fields: $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-Before.apk | grep ^F | wc -l 29681 After, there are less than 18000 (58%!): $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-After.apk | grep ^F | wc -l 17210 12K less fields in a "Hello World" Xamarin.Forms app! Comparing file sizes seems good, too: $ zipinfo Xamarin.Forms_Performance_Integration-Before.apk | grep classes.dex -rw-rw-r-- 6.3 unx 3657872 b- defX 19-Mar-28 16:37 classes.dex $ zipinfo Xamarin.Forms_Performance_Integration-After.apk | grep classes.dex -rw-rw-r-- 6.3 unx 3533120 b- defX 19-Mar-28 16:20 classes.dex The `.dex` file in the `.apk` is ~120KB smaller. ~~ What if R.txt is missing? ~~ I found this was the case when the `<GetAdditionalResourcesFromAssemblies/>` MSBuild task runs. This is an old codepath that allowed old support libraries to work. In this case, a directory is created such as: * `obj\Debug\resourcecache\CF390EBB0064FDA00BB090E733D37E89` * `aidl` * `AndroidManifest.xml` * `assets` * `classes.jar` * `libs` * `res` No `R.txt` file? Checking the zip files we download: $ for z in ~/.local/share/Xamarin/zips/*.zip; do zipinfo $z; done | grep R.txt # no results This actually makes sense, since the zip file contains the *actual resources*. To make this case work properly, we should just process the main app's `R.txt` file when no library `R.txt` file is found. This will still be faster than invoking `aapt`, even though we have more fields than needed. ~~ Tests ~~ I added a set of unit tests for the `<GenerateLibraryResources/>` MSBuild task. I also had to remove a few assertions that looked for the `_GenerateJavaDesignerForComponent` MSBuild target. Lastly, I added some assertions to a test that uses an old support library to verify it's main `R.java` reasonably matches the library `R.java` we generate.
ghost
locked as resolved and limited conversation to collaborators
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
The
_GenerateJavaDesignerForComponentis tasked with generating theR.javafiles for all the referenced libraries which have anAndroidManifest.xml. This is so any java code which use resource items such asandroid.support.transition.R.abc_fade_inwill compile.Currently we call
aaptoraapt2once per library to generate the file. We do this on the first build. But we also do it on subsequent builds if the library changes. For Nuget packages this means once its built it never gets built again unless the user does aClean.We do run these tasks in Parallel, which does improve things quite a bit. But it does put the CPU under allot of stress, since we have to launch a
aapt*process per library. So if you have 8 cores on your machine you get 7aapt*processes running.Is this the most efficient way of doing this? Looking at the output of the
R.javafile the ONLY code difference between ALL the files is the package declaration at the top.E.g
package android.support.compat;vspackage android.support.transition;So the next question is why don't we generate one
R.javafile and then do a search & replace for each library reading the requiredpackageNamefrom theAndroidManifest.xml.This would mean
A) We are not duplicating a whole TON of work
B) the initial build should be much quicker since we are not spawning up a whole ton of processes.
The text was updated successfully, but these errors were encountered: