Embedded assemblies blob #6311

grendello · 2021-09-20T12:04:50Z

Let's see what breaks

With the current version I was able to run tests using both Hello MAUI and plain XA apps. I ran only NET6 tests so far, however (shouldn't be any different for legacy, but YMMV).
The changes also need to be tested on APIs older than 21 (because of the MonoPackageManager changes which use fields introduced in API21)

In performance tests Before refers to the tip of the main branch (82fed829772145028cb494cafa10e0f8d080c7a7) and After refers to this PR.

Device

Model: Pixel 3 XL
Native architecture: arm64-v8a
SDK version: 31

MAUI startup performance (6.0.101-preview.10.1952):

Displayed time

Before	After	Δ	Notes
1016.800	892.600	-12.21% ✓	defaults; profiled AOT; 32-bit build
1016.100	894.700	-11.95% ✓	defaults; profiled AOT; 64-bit build
1104.200	922.000	-16.50% ✓	defaults; full AOT+LLVM; 64-bit build
1102.700	926.100	-16.02% ✓	defaults; full AOT; 32-bit build
1108.400	932.600	-15.86% ✓	defaults; full AOT; 64-bit build
1106.300	932.600	-15.70% ✓	defaults; full AOT+LLVM; 32-bit build
1292.000	1271.800	-1.56% ✓	defaults; 64-bit build
1307.000	1275.400	-2.42% ✓	defaults; 32-bit build

Total native init time (before OnCreate)

Before	After	Δ	Notes
96.727	88.921	-8.07% ✓	defaults; 32-bit build
97.236	89.693	-7.76% ✓	defaults; 64-bit build
169.315	108.845	-35.71% ✓	defaults; profiled AOT; 32-bit build
170.061	109.071	-35.86% ✓	defaults; profiled AOT; 64-bit build
363.864	208.949	-42.57% ✓	defaults; full AOT; 64-bit build
363.629	209.092	-42.50% ✓	defaults; full AOT; 32-bit build
373.203	218.289	-41.51% ✓	defaults; full AOT+LLVM; 64-bit build
372.783	219.003	-41.25% ✓	defaults; full AOT+LLVM; 32-bit build

It is interesting to note that Displayed time is nearly identical for the default settings case. It's most probably caused by the amount of JIT-ed code between OnCreate and the time when the application screen is presented, most likely the time is spent JIT-ing MAUI rendering code.

Plain XA startup performance:

Displayed time

Before	After	Δ	Notes
289.300	251.000	-13.24% ✓	defaults; full AOT+LLVM; 64-bit build
286.300	252.900	-11.67% ✓	defaults; full AOT; 64-bit build
285.700	255.300	-10.64% ✓	defaults; profiled AOT; 32-bit build
282.900	255.800	-9.58% ✓	defaults; full AOT+LLVM; 32-bit build
286.100	256.500	-10.35% ✓	defaults; full AOT; 32-bit build
286.100	258.000	-9.82% ✓	defaults; profiled AOT; 64-bit build
328.900	310.600	-5.56% ✓	defaults; 32-bit build
319.300	313.000	-1.97% ✓	defaults; 64-bit build

Total native init time (before OnCreate)

Before	After	Δ	Notes
59.768	42.694	-28.57% ✓	defaults; profiled AOT; 64-bit build
60.056	42.990	-28.42% ✓	defaults; profiled AOT; 32-bit build
65.829	48.684	-26.05% ✓	defaults; full AOT; 64-bit build
65.688	48.713	-25.84% ✓	defaults; full AOT; 32-bit build
67.159	49.938	-25.64% ✓	defaults; full AOT+LLVM; 64-bit build
67.514	50.465	-25.25% ✓	defaults; full AOT+LLVM; 32-bit build
66.758	62.531	-6.33% ✓	defaults; 32-bit build
67.252	62.829	-6.58% ✓	defaults; 64-bit build

src/Xamarin.Android.Build.Tasks/Tasks/BuildApk.cs

src/Xamarin.Android.Build.Tasks/Tasks/GeneratePackageManagerJava.cs

src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/AotTests.cs

jonpryor · 2021-10-07T00:49:23Z

src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/AotTests.cs

@@ -264,13 +280,12 @@ public void BuildBasicApplicationReleaseProfiledAotWithoutDefaultProfile ()
 					Assert.IsTrue (File.Exists (assemblies), "{0} libaot-UnnamedProject.dll.so does not exist", abi);
 					var apk = Path.Combine (Root, b.ProjectDirectory,
 						proj.OutputPath, $"{proj.PackageName}-Signed.apk");
+					var helper = new ArchiveAssemblyHelper (apk, usesAssemblyBlobs);
+					Assert.IsFalse (helper.Exists ("assemblies/UnnamedProject.dll"), $"UnnamedProject.dll should not be in the {proj.PackageName}-Signed.apk");


I feel like I'm missing something here, as this assertion isn't wrapped in an if(usesAssemblyBlobs) condition. Shouldn't this assert be triggered when usesAssemblyBlobs is false?

The difference is handled and hidden by ArchiveAssemblyHelper.

jonpryor · 2021-10-07T00:51:29Z

...rin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Utilities/ArchiveAssemblyHelper.cs

+		static readonly Dictionary<string, string> ArchToAbi = new Dictionary<string, string> (StringComparer.OrdinalIgnoreCase) {
+			{"x86", "x86"},
+			{"x86_64", "x86_64"},
+			{"armeabi_v7a", "armeabi-v7a"},


Why/how are we adding yet another set of "architecture" names? What happens under net6, which has a different set of architecture names?

These ABIs don't change with .NET6, it's a mapping between "our" ABI names and Android runtime ABI names.

@grendello wrote:

it's a mapping between "our" ABI names and Android runtime ABI names

Which still confuses me: where do "our" ABI names use _?

% git grep armeabi_v7 # no matches % git grep armeabi-v7 | wc -l 468

Our existing ABI names use -: armeabi-v7a, arm64-v8a, etc.

Our blobs use the same convention as Google, see here.

jonpryor · 2021-10-07T01:21:00Z

...Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Utilities/EnvironmentHelper.cs

@@ -158,37 +160,47 @@ static ApplicationConfig ReadApplicationConfig (string envFile)
 						ret.have_runtime_config_blob = ConvertFieldToBool ("have_runtime_config_blob", envFile, i, field [1]);
 						break;

-					case 8: // bound_stream_io_exception_type: byte / .byte
+					case 8: // have_assemblies_blob: bool / .byte


Given that every time we update libxamarin-app.so this switch block is updated, and some (many?) of the case labels changed… is there a more "dummy proof" data structure that can be used here? A List<Action> or something?

Why complicate/obfuscate matters if they can remain simple? The switch is self-documenting and clear in its purpose. Adding a level of indirection wouldn't give us any advantage IMO except for a smaller occasional diff.

jonpryor · 2021-10-07T01:24:31Z

src/Xamarin.Android.Build.Tasks/Utilities/AssemblyBlob.cs

+		const uint BlobVersion = 1; // Must match the BUNDLED_ASSEMBLIES_BLOB_VERSION native constant
+
+		// MUST be equal to the size of the BlobBundledAssembly struct in src/monodroid/jni/xamarin-app.hh
+		const uint BlobBundledAssemblyNativeStructSize = 6 * sizeof (uint);


(Insert screaming for create-native-map here…. Pity that's not packaged anymore.)

I suppose a more "in-tree" solution to ensuring that things are consistent would be to have Xamarin.Android.Build.Tasks.csproj have a @(ProjectReference) on src/monodroid/monodroid.csproj, and have monodroid.csproj write out a file containing the sizes of these structures? Or vice-versa, have monodroid.csproj reference Xamarin.Android.Build.Tasks.csproj, and have XABT emit a .h file as part of its build?

I considered implementing something like that many times, but it's a pain in the nether regions every time. I don't want to have to modify XABT.csproj each time I make a change to the C++ structures... The .h file is the source of truth and I want to keep it this way. I do, however, want to remove the need to manually keep things in sync eventually. The solution I'm currently mulling over is to write a small C++ program that would generate the managed description of the structures as part of monodroid.csproj build - similar to what is your first suggestion above. A variation of this is that xaprepare would compile and run that utility, putting the generated file in bin/Build$(Configuration)

jonpryor · 2021-10-07T01:31:07Z

src/monodroid/jni/shared-constants.hh

@@ -39,6 +43,18 @@ namespace xamarin::android::internal
 		static constexpr char MONO_SGEN_SO[]      = "monosgen-2.0";
 		static constexpr char MONO_SGEN_ARCH_SO[] = "monosgen-" __BITNESS__ "-2.0";
 #endif
+
+#if __arm__
+		static constexpr char android_abi[] = "armeabi_v7a";


Why can't this use -, so that armeabi-v7a is valid?

Because _ is what Android uses when generating split APKs: split_config.armeabi_v7a.apk is what they generate and we check that name against the list of APKs passed to runtime init

jonpryor · 2021-10-07T01:38:24Z

src/monodroid/jni/xamarin-app.hh

@@ -106,6 +108,86 @@ struct XamarinAndroidBundledAssembly final
 	char    *name;
 };

+//
+// Blob format


Format description could also use a "fleshed out" example, e.g. what structs, struct order, and struct values would one expect for App.dll & mscorlib.dll? Where do satellite assemblies fit in? Or do they just "fall out" because their "relative pathname" is hashed, and it's the hash that we care about?

Are there any checks for hash collisions, and an "appropriate error" when a collision is detected?

How's the hash mechanism cope with multiple different ABIs for the "same" assembly, e.g. System.Private.CoreLib.dll for (separately) x86 & arm64-v8a?

Format description could also use a "fleshed out" example, e.g. what structs, struct order, and struct values would one expect for App.dll & mscorlib.dll? Where do satellite assemblies fit in? Or do they just "fall out" because their "relative pathname" is hashed, and it's the hash that we care about?

Are there any checks for hash collisions, and an "appropriate error" when a collision is detected?

I was focused on implementing the thing first instead of writing and rewriting documents... Once I'm sure that the format is final, I can spend time on documentation

How's the hash mechanism cope with multiple different ABIs for the "same" assembly, e.g. System.Private.CoreLib.dll for (separately) x86 & arm64-v8a?

The hashes are identical across all the arch-specific blobs, after all they contain assemblies with exactly the same names and in exactly the same order. On the runtime always only one arch-specifc blob is loaded, even if there are 4 of them in the apk. Therefore there isn't any conflict.

Format description could also use a "fleshed out" example, e.g. what structs, struct order, and struct values would one expect for App.dll & mscorlib.dll? Where do satellite assemblies fit in? Or do they just "fall out" because their "relative pathname" is hashed, and it's the hash that we care about?

Are there any checks for hash collisions, and an "appropriate error" when a collision is detected?

Yes there are, here and here. This code generates the global index (contained only in a single blob, the blob with ID 0) spanning all of the blobs that are generated for the application

jonpryor · 2021-10-08T17:54:30Z

Documentation/project-docs/AssemblyBlobs.md

+# Blob format
+
+Each blob is a structured binary file, using little-endian byte order
+and aligned to a byte boundary.  Each blob consists of a header, an


A byte boundary? Really? Shouldn't we align to 64-bit boundaries?

Why? It's an on-disk format, there's no need to waste space. It will be aligned on the page boundary in the ZIP and when mmapped, that's sufficient.

jonpryor · 2021-10-08T18:50:35Z

Documentation/project-docs/AssemblyBlobs.md

+ - `version`: a value increased every time blob format changes.
+ - `local_entry_count`: number of assemblies stored in this blob (also
+   the number of entries in the assembly descriptor table, see below)
+ - `global_entry_count`: number of entries in the index blob's (see


I don't understand.

I think we need more nouns. :-/

Thus, a suggestion/clarification:

There are the following blob files:

App Architecture-Independent Assembly [& related] Blob (assemblies.blob in main app)

App Architecture-Specific Assembly Blobs (assemblies.[ARCH].blob in main app)

Feature Architecture-Independent Assembly [& related] Blob (zero or more) (assemblies.blob in feature)

Feature Architecture-Specific Assembly Blobs (zero or more) (assemblies.[ARCH].blob in feature)

Each of those blob files has an internal structure.

We thus return to BundledAssemblyBlobHeader::global_entry_count: what is it, and for where?

For the App Architecture-Independent Blob, is it the count of all of the above (all App + Feature blobs)? Just the "current" blob". Just the App blobs? Other?

The "all the other blobs store 0 in this field" sentence fragment (below) is not helping.

I imagine that it may be possible to update a Feature without updating the corresponding App -- and, if supported, would be beneficial, as it means less data needs to be sent to devices -- so I hope that the App Arch-Independent Blob's global_entry_count doesn't include counts for all related Feature blobs.

@dellis1972: is it/should it be possible to update a Feature without updating the correspond App? Say for a bugfix release? What are the constraints here?

I don't understand.

I think we need more nouns. :-/

Thus, a suggestion/clarification:

There are the following blob files:

* App Architecture-Independent Assembly [& related] Blob (`assemblies.blob` in main app) * App Architecture-Specific Assembly Blobs (`assemblies.[ARCH].blob` in main app) * Feature Architecture-Independent Assembly [& related] Blob (zero or more) (`assemblies.blob` in feature) * Feature Architecture-Specific Assembly Blobs (zero or more) (`assemblies.[ARCH].blob` in feature)

Each of those blob files has an internal structure.

Which is identical for all the blobs, with the only difference being the global hash index which is not part of the general blob header.

We thus return to BundledAssemblyBlobHeader::global_entry_count: what is it, and for where?

A count of all the entries in all the blobs in the application together - a global count of assemblies, versus a local count in a single blob file.

For the App Architecture-Independent Blob, is it the count of all of the above (all App + Feature blobs)? Just the "current" blob". Just the App blobs? Other?

It's a global count for the entire app - all the blobs summed up. It's essentially the size of the pre-generated table in libxamarin-app.so

The "all the other blobs store 0 in this field" sentence fragment (below) is not helping.

Not sure what you mean, it's a format specification and it clearly states that this field is 0 for all the blobs except the index one.

I imagine that it may be possible to update a Feature without updating the corresponding App -- and, if supported, would be beneficial, as it means less data needs to be sent to devices -- so I hope that the App Arch-Independent Blob's global_entry_count doesn't include counts for all related Feature blobs.

I doubt it will be possible to update a Feature without updating the app. With the AAB format being the required distribution format for the Play Store, I don't think it's possible to update the app piecemeal - how would it be done? By uploading a "mini-AAB" with just the feature update in it? I rather think the apps will be uploaded as a single AAB from which feature APKs will be created by Google, on the Play Store side. So the app will be rebuilt entirely and there's no issue with this global count.

jonpryor · 2021-10-08T18:52:57Z

Documentation/project-docs/AssemblyBlobs.md

+    beginning of the blob file. A value of `0` indicates there's no
+    debug data for this assembly.
+  - `debug_data_size`: number of bytes of debug data. Can be `0` only
+    if `debug_data_offset` is `0`


Raises an interesting "error reporting" question: what happens if this isn't the case? Do we error & exit at runtime?

Scenario is "someone creates their own assemblies.blob file and manually adds to an app". What kind of error reporting experience do they get?

If it isn't the case, then the runtime will register such debug data since it has no way of knowing if the numbers stored there are valid or not. And what kind of error reporting they get depends on how Mono treats invalid PDB/MDB data. I considered adding a runtime error for this case and abort the application, but it seems to be a bit of an overreaction since the app can run perfectly fine without valid debug data and this is a very improbable corner case, so it seemed to be a waste of time to check for this condition.

jonpryor · 2021-10-08T18:58:16Z

Documentation/project-docs/AssemblyBlobs.md

+Each application will contain exactly one blob with a global index -
+two tables with assembly name hashes.  All the other blobs **do not**
+contain these tables.  Two hash tables are necessary because hashes
+for 32-bit and 64-bit devices are different.


I think I've asked this before, but I forget the answer: what is the perf impact for doing a 64-bit comparison on a 32-bit device, or a 32-bit comparison on a 64-bit device? Is there no way to unify these tables?

…or for that matter, have one "table" with two different fields for comparison? "Just" remove the union in BlobHashEntry?

struct BlobHashEntry { uint64_t hash64; uint32_t hash32; uint32_t _alignment; // … };

I think I've asked this before, but I forget the answer: what is the perf impact for doing a 64-bit comparison on a 32-bit device, or a 32-bit comparison on a 64-bit device? Is there no way to unify these tables?

32-bit hash on 64-bit machine is more than 50% slower than the native 64-bit one. The 64-bit on 32-bit performance is described here.

…or for that matter, have one "table" with two different fields for comparison? "Just" remove the union in BlobHashEntry?

struct BlobHashEntry { uint64_t hash64; uint32_t hash32; uint32_t _alignment; // … };

Yes, it could be done. It would save 8 bytes per entry (the padding isn't necessary).

I've just realized it cannot be done after all. Each table has a different sort order, since the 32-bit and 64-bit hashes are different. We need the tables sorted in order to use binary search at the run time.

bad rename, bad

- remove some debug logging, too

For some reasons the following assemblies are now passed duplicated to `BuildApk`: - mscorlib - System - System.Xml - System.Private.Uri - System.Data - System.Collections.Concurrent There are as many duplicates as target ABIs

jonpryor · 2021-10-22T20:14:17Z

Configuration.props

@@ -25,6 +25,7 @@
    <!-- Should correspond to the first value from `$(API_LEVELS)` in `build-tools/api-xml-adjuster/Makefile` -->
    <AndroidFirstFrameworkVersion Condition="'$(AndroidFirstFrameworkVersion)' == ''">v4.4</AndroidFirstFrameworkVersion>
    <AndroidFirstApiLevel Condition="'$(AndroidFirstApiLevel)' == ''">19</AndroidFirstApiLevel>
+    <AndroidJavaRuntimeApiLevel Condition="'$(AndroidJavaRuntimeApiLevel)' == ''">21</AndroidJavaRuntimeApiLevel>


Doesn't "Classic"/"Xammie" Xamarin.Android still support API-19? Shouldn't this value likewise by 19, not 21?

Could also use a comment mentioning that this controls the android.jar used when building src/java-runtime.

This is necessary to build Java code that accesses properties added in 21. Access to those properties is protected with an API level check, so it will run fine on 19

It's used in src/java-runtime/java-runtime.targes like so:

<_AndroidJar>"$(AndroidToolchainDirectory)\sdk\platforms\android-$(AndroidJavaRuntimeApiLevel)\android.jar"</_AndroidJar>

jonpryor · 2021-10-22T21:26:51Z

in-progress commit message:

What do we want?  Faster (Release) App Startup!

How do we get that?  Assembly Stores!

"In the beginning", assemblies were stored in the `assemblies`
directory within the `.apk`.  App startup would open the `.apk`,
traverse all entries within the `.apk` looking for `assemblies/*.dll`,
`assemblies/*.dll.config`, and `assemblies/*.pdb` files.  When a
"supported" `assemblies/*` entry was encountered, the entry would be
**mmap**(2)'d so that it could be used; see also commit c1956835bd.

Of particular note is:

 1. The need to enumerate *all* entries within the `.apk`, as there
    is no guarantee of entry ordering, and

 2. The need for *N* `mmap()` invocations, one per assembly included
    in the app, *plus* additional `mmap()` invocations for the `.pdb`
    and `.dll.config` files, if present.

    Useful contextual note: a "modern" AndroidX-using app could pull
    in dozens to over 200 assemblies without really trying.

    There will be *lots* of `mmap()` invocations.

Instead of adding (compressed! d236af54) data for each assembly
separately, instead add a small set of "Assembly Store" files which
contain the assembly & related data to use within the app:

  * `assemblies/assemblies.blob`
  * `assemblies/assemblies.[ARCHITECTURE].blob`

`assemblies.[ARCHITECTURE].blob` contains architecture-specific
assemblies, e.g. `System.Private.CoreLib.dll` built for x86 would be
placed within `assemblies.x86.blob`.  `ARCHITECTURE` is one of `x86`,
`x86_64`, `armeabi_v7a`, or `arm64_v8a`; note use of `_` instead of
`-`, which is different from the `lib/ARCHITECTURE` convention within
`.apk` files.  This is done because this is apparently what Android
and `bundletool` do, e.g. creating `split_config.armeabi_v7a.apk`.

Once the architecture-neutral `assemblies.blob` and appropriate
(singular!) `assemblies.[ARCHITECTURE].blob` for the current
architecture is found and `mmap()`'d, `.apk` entry traversal can end.
There is no longer a need to parse the entire `.apk` during startup.

The reduction in the number of `mmap()` system calls required can
have a noticeable impact on process startup, particularly with
.NET SDK for Android & MAUI; see below for timing details.

The assembly store format uses the followings structures:

	struct AssemblyStoreHeader {
	    uint32_t magic, version;
	    uint32_t local_entry_count;                    // Number of AssemblyStoreAssemblyDescriptor entries
	    uint32_t global_entry_count;                   // Number of AssemblyStoreAssemblyDescriptor entries in entire app, across all *.blob files
	    uint32_t store_id;
	};
	struct AssemblyStoreAssemblyDescriptor {
	    uint32_t data_offset, data_size;                // Offset from beginning of file for .dll data
	    uint32_t debug_data_offset, debug_data_size;    // Offset from beginning of file for .pdb data
	    uint32_t config_data_offset, config_data_size;  // Offset from beginning of file for .dll.config data
	};
	struct AssemblyStoreHashEntry {
	    union {
	        uint64_t hash64;                            // 64-bit xxhash of assembly filename
	        uint32_t hash64;                            // 32-bit xxhash of assembly filename
	    };
	    uint32_t mapping_index, local_store_index, store_id;
	};

The assembly store format is roughly as follows:

	AssemblyStoreHeader                 header {…};
	AssemblyStoreAssemblyDescriptor     assemblies [header.local_entry_count];

	// The following two entries exist only when header.store_id == 0
	AssemblyStoreHashEntry              hashes32[header.global_entry_count];
	AssemblyStoreHashEntry              hashes64[header.global_entry_count];

	uint8_t data[];

Note that `AssemblyStoreFileFormat::hashes32` and
`AssemblyStoreFileFormat::hashes64` are *sorted by their hash*.
Further note that assembly *filenames* are not present.
`EmbeddedAssemblies::blob_assemblies_open_from_bundles()` will hash
the filename, then binary search the appropriate `hashes*` array to
get the appropriate assembly information.

As the assembly store format doesn't include assembly names, `.apk`
and `.aab` files will also contain an `assemblies.manifest` file,
which contains the assembly names and other information in a human-
readable format; it is also used by `assembly-store-reader`:

	Hash 32     Hash 64             Blob ID  Blob idx  Name
	0xa2e0939b  0x4288cfb749e4c631  000      0000      Xamarin.AndroidX.Activity
	…
	0xad6f1e8a  0x6b0ff375198b9c17  001      0000      System.Private.CoreLib

Add a new `tools/assembly-store-reader` utility which can read the
new `assemblies*.blob` files:

	% tools/scripts/read-assembly-store path/to/app.apk
	Store set 'base_assemblies':
	  Is complete set? yes
	  Number of stores in the set: 5

	Assemblies:
	  0:
	    Name: Xamarin.AndroidX.Activity
	    Store ID: 0 (shared)
	    Hashes: 32-bit == 0xa2e0939b; 64-bit == 0x4288cfb749e4c631
	    Assembly image: offset == 1084; size == 14493
	    Debug data: absent
	    Config file: absent
	  …
	  16:
	    Name: System.Private.CoreLib
	    Store ID: 1 (x86)
	    Hashes: 32-bit == 0xad6f1e8a; 64-bit == 0x6b0ff375198b9c17
	    Assembly image: offset == 44; size == 530029
	    Debug data: absent
	    Config file: absent
	  …

On a Pixel 3 XL (arm64-v8a) running Android 12 with MAUI
6.0.101-preview.10.1952, we observe:

~~ MAUI: Displayed Time ~~

| Before ms |  After ms |            Δ | Notes                                 |
| ---------:| --------: | -----------: | ------------------------------------- |
|  1016.800 |   892.600 |    -12.21% ✓ | defaults; profiled AOT; 32-bit build  |
|  1016.100 |   894.700 |    -11.95% ✓ | defaults; profiled AOT; 64-bit build  |
|  1104.200 |   922.000 |    -16.50% ✓ | defaults; full AOT+LLVM; 64-bit build |
|  1102.700 |   926.100 |    -16.02% ✓ | defaults; full AOT; 32-bit build      |
|  1108.400 |   932.600 |    -15.86% ✓ | defaults; full AOT; 64-bit build      |
|  1106.300 |   932.600 |    -15.70% ✓ | defaults; full AOT+LLVM; 32-bit build |
|  1292.000 |  1271.800 |     -1.56% ✓ | defaults; 64-bit build                |
|  1307.000 |  1275.400 |     -2.42% ✓ | defaults; 32-bit build                |

Displayed time reduces by ~12% when Profiled AOT is used.

It is interesting to note that **Displayed time** is nearly identical
for the default (JIT) settings case. It's most probably caused by the
amount of JIT-ed code between `OnCreate()` and the time when the
application screen is presented, most likely the time is spent JIT-ing
MAUI rendering code.


~~ MAUI: Total native init time (before `OnCreate()`) ~~

| Before ms |  After ms |            Δ | Notes                                 |
| --------: | --------: | -----------: | ------------------------------------- |
|    96.727 |    88.921 |     -8.07% ✓ | defaults; 32-bit build                |
|    97.236 |    89.693 |     -7.76% ✓ | defaults; 64-bit build                |
|   169.315 |   108.845 |    -35.71% ✓ | defaults; profiled AOT; 32-bit build  |
|   170.061 |   109.071 |    -35.86% ✓ | defaults; profiled AOT; 64-bit build  |
|   363.864 |   208.949 |    -42.57% ✓ | defaults; full AOT; 64-bit build      |
|   363.629 |   209.092 |    -42.50% ✓ | defaults; full AOT; 32-bit build      |
|   373.203 |   218.289 |    -41.51% ✓ | defaults; full AOT+LLVM; 64-bit build |
|   372.783 |   219.003 |    -41.25% ✓ | defaults; full AOT+LLVM; 32-bit build |

Note that "native init time" includes running `JNIEnv.Initialize()`,
which requires loading `Mono.Android.dll` + dependencies such as
`System.Private.CoreLib.dll`, which in turn means that the AOT DSOs
such as `libaot-System.Private.CoreLib.dll.so` must *also* be loaded.
The loading of the AOT DSOs is why JIT is fastest here (no AOT DSOs),
and why Profiled AOT is faster than Full AOT (smaller DSOs).


~~ Plain Xamarin.Android: Displayed Time ~~

| Before ms |  After ms |            Δ | Notes                                 |
| --------: | --------: | -----------: | ------------------------------------- |
|   289.300 |   251.000 |    -13.24% ✓ | defaults; full AOT+LLVM; 64-bit build |
|   286.300 |   252.900 |    -11.67% ✓ | defaults; full AOT; 64-bit build      |
|   285.700 |   255.300 |    -10.64% ✓ | defaults; profiled AOT; 32-bit build  |
|   282.900 |   255.800 |     -9.58% ✓ | defaults; full AOT+LLVM; 32-bit build |
|   286.100 |   256.500 |    -10.35% ✓ | defaults; full AOT; 32-bit build      |
|   286.100 |   258.000 |     -9.82% ✓ | defaults; profiled AOT; 64-bit build  |
|   328.900 |   310.600 |     -5.56% ✓ | defaults; 32-bit build                |
|   319.300 |   313.000 |     -1.97% ✓ | defaults; 64-bit build                |


~~ Plain Xamarin.Android: Total native init time (before `OnCreate()`) ~~

| Before ms |  After ms |            Δ | Notes                                 |
| --------: | --------: | -----------: | ------------------------------------- |
|    59.768 |    42.694 |    -28.57% ✓ | defaults; profiled AOT; 64-bit build  |
|    60.056 |    42.990 |    -28.42% ✓ | defaults; profiled AOT; 32-bit build  |
|    65.829 |    48.684 |    -26.05% ✓ | defaults; full AOT; 64-bit build      |
|    65.688 |    48.713 |    -25.84% ✓ | defaults; full AOT; 32-bit build      |
|    67.159 |    49.938 |    -25.64% ✓ | defaults; full AOT+LLVM; 64-bit build |
|    67.514 |    50.465 |    -25.25% ✓ | defaults; full AOT+LLVM; 32-bit build |
|    66.758 |    62.531 |     -6.33% ✓ | defaults; 32-bit build                |
|    67.252 |    62.829 |     -6.58% ✓ | defaults; 64-bit build                |

velocitysystems · 2022-06-13T16:18:26Z

@grendello For verifying DLLs inside the blob, are there any good CLI tools (already written) to unpack the blob and DLLs?

jonathanpeppers · 2022-06-13T16:21:14Z

This tool should be able to unpack/decompress them:

https://github.com/xamarin/xamarin-android/tree/main/tools/decompress-assemblies

velocitysystems · 2022-06-13T16:29:11Z

Awesome. Thanks @jonathanpeppers!
I see this has a net472 target. Would there be much work to port this to .NET 6?

jonathanpeppers · 2022-06-13T16:31:34Z

I would just try it and see if it builds.

velocitysystems · 2022-06-13T16:46:16Z

Thanks @jonathanpeppers. Did a quick port to a standalone .NET 6 console project and works perfectly!

grendello requested review from jonpryor, dellis1972 and jonathanpeppers September 20, 2021 12:04

grendello force-pushed the embedded-assemblies-blob branch 7 times, most recently from 8cd4e9f to e13e42e Compare September 30, 2021 08:35

grendello force-pushed the embedded-assemblies-blob branch 2 times, most recently from a6fe315 to 037fc5c Compare October 5, 2021 08:42

grendello marked this pull request as ready for review October 5, 2021 16:09

jonpryor reviewed Oct 6, 2021

View reviewed changes

src/Xamarin.Android.Build.Tasks/Tasks/BuildApk.cs Outdated Show resolved Hide resolved

jonpryor reviewed Oct 6, 2021

View reviewed changes

src/Xamarin.Android.Build.Tasks/Tasks/GeneratePackageManagerJava.cs Outdated Show resolved Hide resolved

jonpryor reviewed Oct 6, 2021

View reviewed changes

src/Xamarin.Android.Build.Tasks/Tasks/GeneratePackageManagerJava.cs Outdated Show resolved Hide resolved

jonpryor reviewed Oct 7, 2021

View reviewed changes

src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/AotTests.cs Outdated Show resolved Hide resolved

jonpryor reviewed Oct 7, 2021

View reviewed changes

src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/AotTests.cs Outdated Show resolved Hide resolved

jonpryor reviewed Oct 7, 2021

View reviewed changes

src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/AotTests.cs Show resolved Hide resolved

jonpryor reviewed Oct 7, 2021

View reviewed changes

grendello force-pushed the embedded-assemblies-blob branch from bc0216f to 15e80bd Compare October 7, 2021 12:22

jonpryor reviewed Oct 8, 2021

View reviewed changes

grendello added 18 commits October 22, 2021 09:05

[WIP] Replace all instances of 'blob' with 'store'

c8673b9

[WIP] The blob -> store renaming changed file extensions for blobs,

3759790

bad rename, bad

[WIP] Mind your extensions + reuse constants more

3e58097

[WIP] Design blobs for Design Time Build

d57ba72

[WIP] Try to fix designer tests again

2bc8d38

[WIP] Add a trailing comma

4de3779

[WIP] Force Linux builds to use g++/gcc 10, for full C++20 support

fd9a54a

[WIP] Attempt to work around a mingw bug on the CI bots

d930fef

[WIP] Another attempt to work around a mingw issue

df09449

[WIP] And another attempt to make Linux CI bots mingw work

f12f7aa

[WIP] Bundled apps should not use blobs

f0f8293

- remove some debug logging, too

[WIP] Fix single-rid builds

03be452

[WIP] Do not overwrite binlogs when using dotnet run in tests

a6bac6a

[WIP] Ignore duplicate assemblies when adding to sets

ed879e0

For some reasons the following assemblies are now passed duplicated to `BuildApk`: - mscorlib - System - System.Xml - System.Private.Uri - System.Data - System.Collections.Concurrent There are as many duplicates as target ABIs

[WIP] Fix blob assembly synthesis when only one rid is used

d667f81

[WIP] More test fixes

03124dd

[WIP] And another test fix

fd883c7

[WIP] Duplicate assemblies are treated as arch-specific ones

dcbaa06

grendello force-pushed the embedded-assemblies-blob branch from 34680e9 to dcbaa06 Compare October 22, 2021 07:24

PureWeen mentioned this pull request Oct 22, 2021

[Bug] Evaluate use of string comparisons inside Maui.Core dotnet/maui#3085

Closed

jonpryor reviewed Oct 22, 2021

View reviewed changes

jonpryor merged commit c927026 into dotnet:main Oct 25, 2021

grendello deleted the embedded-assemblies-blob branch October 25, 2021 14:01

github-actions bot locked and limited conversation to collaborators Jan 24, 2024

Embedded assemblies blob #6311

Embedded assemblies blob #6311

Conversation

grendello commented Sep 20, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grendello Oct 11, 2021 • edited Loading

Choose a reason for hiding this comment

grendello Oct 11, 2021 • edited Loading

Choose a reason for hiding this comment

grendello Oct 11, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonpryor commented Oct 22, 2021 • edited Loading

velocitysystems commented Jun 13, 2022

jonathanpeppers commented Jun 13, 2022

velocitysystems commented Jun 13, 2022

jonathanpeppers commented Jun 13, 2022

velocitysystems commented Jun 13, 2022

grendello commented Sep 20, 2021 •

edited

Loading

grendello Oct 11, 2021 •

edited

Loading

grendello Oct 11, 2021 •

edited

Loading

grendello Oct 11, 2021 •

edited

Loading

jonpryor commented Oct 22, 2021 •

edited

Loading