Singlefile: enabling compression for managed assemblies. #50817

VSadov · 2021-04-06T23:04:31Z

The second part of singlefile compression change which enables compression/decompression of managed assemblies.

=== Implementation details:

When creating a PE layout for a compressed file, we start with the same steps as before - with memory mapping the region of the singlefile app that contains the file we need. No changes here.

For obvious reasons we cannot use the direct mapping as-is since that contains compressed data. So we create another intermediate anonymous (memory-only) mapping of appropriate size and decompress the file into the mapping.
After that we can release the original mapping/view that were created for the compressed data.

The PE layout with the anonymous mapping will manage the life time of the decompressed data (same as before) and can be used when flat image is sufficient, or can be further transformed into "mapped" executable form if the file contains native executable sections (such as R2R).

Such conversion was already performed on Windows, while Unix could always get away with virtually mapping to the original data with appropriate alignment and access. Now the conversion will be used on Unix as well (when the original data is compressed).

ghost · 2021-04-06T23:04:33Z

Tagging subscribers to this area: @agocke, @vitek-karas, @VSadov
See info in area-owners.md if you want to be subscribed.

Issue Details

In progress.

Author:	VSadov
Assignees:	-
Labels:	`area-Single-File`
Milestone:	-

src/coreclr/vm/peimagelayout.cpp

VSadov · 2021-04-09T23:47:00Z

I think this PR is ready to be reviewed.

vitek-karas

I assume most of the test coverage is provided by the existing tests, right?

vitek-karas · 2021-04-12T12:14:00Z

src/coreclr/hosts/inc/coreclrhost.h

@@ -126,7 +126,7 @@ CORECLR_HOSTING_API(coreclr_execute_assembly,
 //
 // Callback types used by the hosts
 //
-typedef bool(CORECLR_CALLING_CONVENTION BundleProbeFn)(const char* path, int64_t* offset, int64_t* size);
+typedef bool(CORECLR_CALLING_CONVENTION BundleProbeFn)(const char* path, int64_t* offset, int64_t* sizes);


Why can't we pass two separate parameters?

I am concerned about framework-dependent scenarios where we may be calling the 5.0 version of the probe. Having the same signature allows the old clients to be forwards-compatible with new runtimes.
I.E. 5.0 probe will use only the first element of the array, which contains the file size, which is compatible with the new {size, uncompressedSize}.

Passing an extra parameter "may work", depending on calling conventions. Passing an array does not need to rely on that.

BTW - if we may need another piece of information from the probe in the future, we can continue adding optional array elements.

But framework dependent app should use hostpolicy.dll which is next to the runtime - we should not be statically linking hostpolicy anywhere but superhost. So far we've made the assumption that hostpolicy/runtime interface is not "public" and can be change at any time without versioning considerations, because we always ship hostpolicy and runtime together.

Hmm. I think I saw adding an extra parameter causing failures in framework-dependent 5.0 tests on OSX.
Although we also had uninitialized data problem as well and it could be just that. Let me think about this.

I thought we could have a case where 6.0 clr calls the probe and there is 5.0 hostpolicy on the other end. Perhaps this can't happen.

I think you are correct. The probe is always in sync with runtime. hostpolicy is either bundled together with runtime in superhost or picked up next to the runtime. Either way it will match the version/build.

That allows just using another parameter, so I have reverted the change that packs sizes in an array.

I have reverted the change that packs sizes into an array. The failures in framework-dependent 5.0 tests on OSX were indeed caused by uninitialized data (although they looked like a probe mismatch)

vitek-karas · 2021-04-12T12:17:00Z

src/coreclr/vm/bundle.cpp

+    if (sizes[1] != 0)
+    {
+        loc.Size = sizes[1];
+        loc.UncompresedSize = sizes[0];


It just occurred to me - technically there's no reason to store the uncompressed size in the bundle headers. We would really need just a boolean. I assume it makes the code more efficient to know how much to allocate upfront to uncompress.

But what if the size in the header doesn't match the actual uncompressed size - we should have a test for that at the very least.

Passing uncompressed allows to set up the storage for uncompressed data precisely, so it is slightly more convenient than a bool.

If we have corrupted data we will not overrun the buffer, since we provide the size limit to the zlib.
It does make sense to add a check though and failfast if uncompressed size is less than expected, or compressed data was not all consumed. Good point!

vitek-karas · 2021-04-12T12:21:59Z

src/coreclr/vm/peimagelayout.cpp

@@ -502,6 +519,7 @@ MappedImageLayout::MappedImageLayout(PEImage* pOwner)

    HANDLE hFile = pOwner->GetFileHandle();
    INT64 offset = pOwner->GetOffset();
+    _ASSERTE(!pOwner->GetUncompressedSize());


Nit: Personally I would prefer to have a full == 0 in this case. I hate the C++ tricks of treating numericals as booleans - it confuses me when reading the code.

vitek-karas · 2021-04-12T12:23:17Z

src/coreclr/vm/peimagelayout.cpp

+
+#else
+            _ASSERTE(!"Failure extracting contents of the application bundle. Compressed files used with a standalone (not singlefile) apphost.");
+            ThrowLastError();


Will this actually fail correctly - ThrowLastError presumably throws the last Win32 error code, but in this case there was no OS call which failed before this... so the last error code is effectively random value.

This is a failfast for "impossible" situation. Something is badly corrupted and we cannot recover, if we hit this.
I will check what we use in such cases.

agocke · 2021-04-12T17:02:53Z

src/coreclr/hosts/inc/coreclrhost.h

@@ -126,7 +126,7 @@ CORECLR_HOSTING_API(coreclr_execute_assembly,
 //
 // Callback types used by the hosts
 //
-typedef bool(CORECLR_CALLING_CONVENTION BundleProbeFn)(const char* path, int64_t* offset, int64_t* size);
+typedef bool(CORECLR_CALLING_CONVENTION BundleProbeFn)(const char* path, int64_t* offset, int64_t* sizes);


Does Windows not allow unicode characters in the path? Or is this really a byte stream?

If I remember correctly - all runtime/host interactions use UTF8 strings.

agocke · 2021-04-12T17:10:01Z

src/installer/tests/Assets/TestProjects/BundleProbeTester/Program.cs

@@ -13,16 +13,26 @@ public static class Program
        // The bundle-probe callback is only called from native code in the product
        // Therefore the type on this test is adjusted to circumvent the failure.
        [UnmanagedFunctionPointer(CallingConvention.StdCall)]
-        public delegate byte BundleProbeDelegate([MarshalAs(UnmanagedType.LPUTF8Str)] string path, IntPtr size, IntPtr offset);
+        public unsafe delegate byte BundleProbeDelegate([MarshalAs(UnmanagedType.LPUTF8Str)] string path, IntPtr offset, long* sizes);


~~You use IntPtr for offset here, but the function pointer at the top is uint64_t. Won't that be the wrong size on x86?~~

Never mind, forgot the pointer part.

src/coreclr/utilcode/pedecoder.cpp

janvorli · 2021-04-13T00:21:04Z

src/coreclr/vm/CMakeLists.txt

@@ -891,6 +895,11 @@ endif(CLR_CMAKE_TARGET_WIN32)
 set (VM_SOURCES_WKS_SPECIAL
    codeman.cpp
    ceemain.cpp
+    peimagelayout.cpp


Why couldn't it stay in the common list? I have originally structured the lists so that no file needed to be duplicated in multiple lists.

The common list is compiled by the build only once. That includes almost every file. And we reuse objects for singlefile and regular coreclr.

Very few files need to know about singlefile and have parts conditionally compiled for CORECLR_EMBEDDED. peimagelayout.cpp cannot and does not need to support decompression in regular coreclr mode, so it needs to use #ifdef CORECLR_EMBEDDED

janvorli · 2021-04-13T00:22:37Z

src/coreclr/vm/peimage.cpp

@@ -887,7 +887,9 @@ void PEImage::EnumMemoryRegions(CLRDataEnumMemoryFlags flags)


 PEImage::PEImage():
+    m_path(),


Are these additions really needed? Isn't the parameter-less constructor used by default?

Only if you do not have a constructor, compiler may generate one. Once you have a constructor, compiler just calls it.

new PEImage() would leave these fields uninitialized, but typically we were lucky except for OSX with optimizations (i.e. does not fail in Debug, but started failing in Checked/Release)

janvorli · 2021-04-13T00:27:51Z

src/coreclr/vm/peimagelayout.cpp

+#if defined(CORECLR_EMBEDDED)
+extern "C"
+{
+#include "../../../libraries/Native/AnyOS/zlib/pal_zlib.h"


I don't like long relative paths in general, they are a burden when structure of the source tree changes. It seems it would be nicer to add the ../../../libraries/Native/AnyOS/zlib to the include paths in cmake scripts and have just

#include "pal_zlib.h"

here. It would be also nice to define the libraries path at one place in the cmake files and use it as a basis for this and the path in src/coreclr/vm/CMakeLists.txt.

There was another long pall_zlib.h include in apphost/static. I have changed that too.

agocke · 2021-04-13T04:28:30Z

Any perf numbers for this change? Size changes? Startup perf changes? Working set?

VSadov · 2021-04-13T16:51:38Z

@agocke - no measurements yet. So far I was more concerned about making it work correctly. The size will definitely be smaller, but how much of a startup penalty it will require is yet to be seen.
It would be a lot easier to measure a range of scenarios once this change meets its SDK counterpart (dotnet/sdk#16838)

VSadov · 2021-04-13T17:04:17Z

@vitek-karas @janvorli - I think I have addressed all the concerns. Can you take another look?

janvorli

LGTM, thank you!

vitek-karas · 2021-04-14T21:36:20Z

src/coreclr/vm/peimagelayout.cpp

@@ -94,15 +101,21 @@ PEImageLayout* PEImageLayout::Map(PEImage* pOwner)
    }
    CONTRACT_END;

-    PEImageLayoutHolder pAlloc(new MappedImageLayout(pOwner));
+    PEImageLayoutHolder pAlloc = pOwner->GetUncompressedSize() ?


Similar as the other place - I would prefer pOwner->GetUncompressedSize() != 0 - using numbers as booleans is confusing.

src/coreclr/vm/peimagelayout.cpp

Co-authored-by: Vitek Karas <[email protected]>

ghost · 2021-04-14T22:21:50Z

Hello @VSadov!

Because this pull request has the auto-merge label, I will be glad to assist with helping to merge this pull request once all check-in policies pass.

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (`@msftbot`) and give me an instruction to get started! Learn more here.

VSadov added the area-Single-File label Apr 6, 2021

vitek-karas reviewed Apr 7, 2021

View reviewed changes

src/coreclr/vm/peimagelayout.cpp Outdated Show resolved Hide resolved

vitek-karas reviewed Apr 7, 2021

View reviewed changes

src/coreclr/vm/peimagelayout.cpp Show resolved Hide resolved

vitek-karas reviewed Apr 7, 2021

View reviewed changes

src/coreclr/vm/peimagelayout.cpp Show resolved Hide resolved

runfoapp bot mentioned this pull request Apr 7, 2021

Helix DNS failures causing work item failures in CI #50750

Closed

VSadov force-pushed the compression02 branch from 41b1f85 to 994b1d7 Compare April 8, 2021 23:58

vitek-karas reviewed Apr 9, 2021

View reviewed changes

src/coreclr/vm/peimagelayout.cpp Show resolved Hide resolved

This was referenced Apr 9, 2021

Runtime jobs being abandoned due to infra #50746

Closed

Runtime jobs being abandoned due to infra #50747

Closed

VSadov added 4 commits April 9, 2021 16:27

enable compression of assemblies

aba4619

fix Unix build

04a1791

map should use converted layout for compressed

7cf28fb

enable execution for R2R

937e10d

VSadov force-pushed the compression02 branch from 1d477de to fb4d141 Compare April 9, 2021 23:27

VSadov changed the title ~~[WIP] Singlefile: enabling compression for managed assemblies.~~ Singlefile: enabling compression for managed assemblies. Apr 9, 2021

VSadov marked this pull request as ready for review April 9, 2021 23:45

VSadov requested review from janvorli and vitek-karas April 9, 2021 23:45

vitek-karas reviewed Apr 12, 2021

View reviewed changes

agocke reviewed Apr 12, 2021

View reviewed changes

VSadov force-pushed the compression02 branch from fb4d141 to c32f4b6 Compare April 12, 2021 22:31

VSadov added 3 commits April 12, 2021 17:13

fixes for OSX

7b5d43e

PR feedback (comments)

f218f19

more PR feedback

f522313

VSadov force-pushed the compression02 branch from c32f4b6 to f522313 Compare April 13, 2021 00:13

janvorli reviewed Apr 13, 2021

View reviewed changes

shorter include path to pal_zlib.h

9b3f8d4

VSadov mentioned this pull request Apr 14, 2021

Enable compression in singlefile dotnet/sdk#16838

Merged

3 tasks

janvorli approved these changes Apr 14, 2021

View reviewed changes

vitek-karas approved these changes Apr 14, 2021

View reviewed changes

VSadov commented Apr 14, 2021

View reviewed changes

src/coreclr/vm/peimagelayout.cpp Outdated Show resolved Hide resolved

Apply suggestions from code review

113bd6b

Co-authored-by: Vitek Karas <[email protected]>

VSadov added the auto-merge label Apr 14, 2021

VSadov merged commit 7114248 into dotnet:main Apr 15, 2021

VSadov deleted the compression02 branch April 15, 2021 02:00

vitek-karas mentioned this pull request Apr 19, 2021

Support for single-file bundle from .NET 6. icsharpcode/ILSpy#2373

Merged

ghost locked as resolved and limited conversation to collaborators May 15, 2021

karelz added this to the 6.0.0 milestone May 20, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Singlefile: enabling compression for managed assemblies. #50817

Singlefile: enabling compression for managed assemblies. #50817

VSadov commented Apr 6, 2021 •

edited

Loading

ghost commented Apr 6, 2021

VSadov commented Apr 9, 2021

vitek-karas left a comment

vitek-karas Apr 12, 2021

VSadov Apr 12, 2021 •

edited

Loading

VSadov Apr 12, 2021

vitek-karas Apr 12, 2021

VSadov Apr 12, 2021

VSadov Apr 12, 2021

VSadov Apr 13, 2021

vitek-karas Apr 12, 2021

VSadov Apr 12, 2021

VSadov Apr 12, 2021 •

edited

Loading

vitek-karas Apr 12, 2021

vitek-karas Apr 12, 2021

VSadov Apr 12, 2021

agocke Apr 12, 2021

vitek-karas Apr 12, 2021

agocke Apr 12, 2021 •

edited

Loading

janvorli Apr 13, 2021

VSadov Apr 13, 2021 •

edited

Loading

janvorli Apr 13, 2021

VSadov Apr 13, 2021 •

edited

Loading

janvorli Apr 13, 2021

VSadov Apr 13, 2021

agocke commented Apr 13, 2021

VSadov commented Apr 13, 2021 •

edited

Loading

VSadov commented Apr 13, 2021

janvorli left a comment

vitek-karas Apr 14, 2021

ghost commented Apr 14, 2021

		@@ -887,7 +887,9 @@ void PEImage::EnumMemoryRegions(CLRDataEnumMemoryFlags flags)


		PEImage::PEImage():
		m_path(),

Singlefile: enabling compression for managed assemblies. #50817

Singlefile: enabling compression for managed assemblies. #50817

Conversation

VSadov commented Apr 6, 2021 • edited Loading

ghost commented Apr 6, 2021

VSadov commented Apr 9, 2021

vitek-karas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VSadov Apr 12, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VSadov Apr 12, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agocke Apr 12, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VSadov Apr 13, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VSadov Apr 13, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agocke commented Apr 13, 2021

VSadov commented Apr 13, 2021 • edited Loading

VSadov commented Apr 13, 2021

janvorli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost commented Apr 14, 2021

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (@msftbot) and give me an instruction to get started! Learn more here.

VSadov commented Apr 6, 2021 •

edited

Loading

VSadov Apr 12, 2021 •

edited

Loading

VSadov Apr 12, 2021 •

edited

Loading

agocke Apr 12, 2021 •

edited

Loading

VSadov Apr 13, 2021 •

edited

Loading

VSadov Apr 13, 2021 •

edited

Loading

VSadov commented Apr 13, 2021 •

edited

Loading

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (`@msftbot`) and give me an instruction to get started! Learn more here.