-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[mono][wasm] Bundle assemblies as WebCIL #79416
Conversation
Tagging subscribers to 'arch-wasm': @lewing Issue DetailsDefine a new container format for .NET assemblies that looks less like a Windows PE file. Use it for bundling assemblies in wasm projects. TODO:
|
Current status: it's working for basic samples, but failing with the libraries tests, for example, because various bits of code above the runtime assume that assemblies have .dll extensions. I was hoping to avoid "logical name"/"physical name" shenanigans, but I might need to adjust the wasm driver.c to consume .webcil files from the app bundle but expose them to the runtime as if they were ".dll". I'm going to press on without that for a bit longer and try to adjust the loader's search algorithm to just substitute .webcil when people ask for .dll, and if it doesn't pan out, I'll change the bundling logic instead. |
Would this be an interesting format to support on all platforms? Or just WebAssembly platforms? Should we define tooling to be able to emit to this format directly (a la APIs in System.Reflection.Metadata)? Also cc: @AaronRobinsonMSFT who is doing some work with metadata parsing in unmanaged code. |
Yea I was going to noodle around with making something like But to be honest, I'm somewhat undecided if this is just an internal implementation detail of the final publishing step for WASM or if it's something that we want to use more broadly. For example, we may not want to keep a strict 1-1 correlation between the assemblies that comprise an app and the .webcil files that comprise an app bundle - it would be cool to split CoreLib and the main app assembly into hot/cold parts that can be downloaded separately, for example. Also I don't think we're entirely settled on whether we will serve .webcil directly or if it will be further bundled (for example by baking webcil data into web assembly .wasm modules). Edit As a practical matter to make testing/debugging a little bit easier I made mono support loading .webcil data on all platforms, but I'm not sure it makes sense to ship that |
a62a504
to
f3fd473
Compare
09d9439
to
84136fe
Compare
The debugger tests are failing because |
Ok, next roadblock is that we actually need a |
This comment was marked as off-topic.
This comment was marked as off-topic.
Next set of failures is because the PE/COFF debug directory entries have a file pointer (ie a raw file offset) to locate the actual embedded PPDB or CodeView data. So I guess the WebcilWriter will need to fix those up when writing them out 🤮 Or maybe I can fudge the file format to have a header the same size as the original PE/COFF file and just fill the intervening space with zeroes? Not sure which is more of a hack |
This comment was marked as outdated.
This comment was marked as outdated.
b17d45f
to
385db1d
Compare
/azp run runtime-wasm |
Azure Pipelines successfully started running 1 pipeline(s). |
does this have any conflicts with #79284 |
Merge conflicts probably yes, theoretical conflicts I don't think so, once we still have information that we need to get the symbols from symbol server, like pdbGuid, it's just in another format. We also use the real dll name to find it on symbol server, not sure if this will not be |
#ifdef ENABLE_WEBCIL | ||
const char *p = strstr (assembly_name, ".webcil"); | ||
/* if assembly_name ends with .webcil, check if aname matches, with a .dll extension instead */ | ||
if (p && *(p + 7) == 0) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit
if (p && *(p + 7) == 0) { | |
if (p && *(p + strlen (".webcil")) == 0) { |
the compiler will do the right thing
var tmpWebcil = Path.GetTempFileName(); | ||
var webcilWriter = Microsoft.WebAssembly.Build.Tasks.WebcilConverter.FromPortableExecutable(inputPath: assembly, outputPath: tmpWebcil, logger: Log); | ||
webcilWriter.ConvertToWebcil(); | ||
var finalWebcil = Path.ChangeExtension(assembly, ".webcil"); | ||
FileCopyChecked(tmpWebcil, Path.Combine(asmRootPath, Path.GetFileName(finalWebcil)), "Assemblies"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is there a reason you aren't doing something like
var tmpWebcil = Path.GetTempFileName(); | |
var webcilWriter = Microsoft.WebAssembly.Build.Tasks.WebcilConverter.FromPortableExecutable(inputPath: assembly, outputPath: tmpWebcil, logger: Log); | |
webcilWriter.ConvertToWebcil(); | |
var finalWebcil = Path.ChangeExtension(assembly, ".webcil"); | |
FileCopyChecked(tmpWebcil, Path.Combine(asmRootPath, Path.GetFileName(finalWebcil)), "Assemblies"); | |
var finalWebcil = Path.Combine(asmRootPath, Path.GetFileName(Path.ChangeExtension(assembly, ".webcil")), "Assemblies"); | |
var webcilWriter = Microsoft.WebAssembly.Build.Tasks.WebcilConverter.FromPortableExecutable(inputPath: assembly, outputPath: finalWebcil, logger: Log); | |
webcilWriter.ConvertToWebcil(); | |
_fileWrites.Add(finalWebcil); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@lewing I think I misunderstood FileCopyChecked
. I thought it was avoiding writing to dst
if it already exists with the same content. My intention was to avoid messing with modification times of finalWebcil
unless something really changed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can use Utils.CopyIfDifferent(string src, string dst, bool useHash)
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
runtime/src/tasks/WasmAppBuilder/EmccCompile.cs
Lines 216 to 219 in 0e12347
if (!Utils.CopyIfDifferent(tmpObjFile, objFile, useHash: true)) | |
Log.LogMessage(MessageImportance.Low, $"Did not overwrite {objFile} as the contents are unchanged"); | |
else | |
Log.LogMessage(MessageImportance.Low, $"Copied {tmpObjFile} to {objFile}"); |
we'll need to update https://github.com/dotnet/sdk/blob/ea03dc7f05bfbaac843dc2ac09e26a484acfe8f8/src/BlazorWasmSdk/Targets/BlazorWasm.web.config#L13 as well and
|
/azp run runtime-wasm |
Azure Pipelines successfully started running 1 pipeline(s). |
@lewing Is it alright if I do your suggestions in a follow-up PR? |
temporarily disabling dedup and re-running |
/azp run runtime-wasm |
Azure Pipelines successfully started running 1 pipeline(s). |
This reverts commit 5983e65.
WebCIL is a new container format for .NET assemblies used when publishing WebAssembly apps (see dotnet/runtime#79416) Related to dotnet/runtime#80807
WebCIL is a new container format for .NET assemblies when publishing to webassembly. (See dotnet/runtime#79416) Related to dotnet/runtime#80807
WebCIL is a new container format for .NET assemblies used when publishing WebAssembly apps (see dotnet/runtime#79416) Related to dotnet/runtime#80807
Define a new container format for .NET assemblies that looks less like a Windows PE file. Use it for bundling assemblies in wasm projects. * Implement WebCIL loader It will try to look for WebCIL formatted images instread of normal .dll files * Checkpoint works on wasm sample; add design doc * Push .dll->.webcil probing lower in the bundle logic * Also convert satellite assemblies and implement satellite matching * [wasm] don't leak .webcil image names to the debugger In particular this will make source and breakpoint URLs look like `dotnet://foo.dll/Foo.cs` which means that grabbing PDBs via source link will work, etc. * Add PE DebugTableDirectory to webcil This is used to retrieve the PPDB data and/or the PDB checksum from an image. Refactor mono_has_pdb_checksum to support webcil in addition to PE images * Implement a WebcilReader for BorwserDebugProxy like PEReader This needs some improvements: - add support for reading CodeView and EmbeddedPDB data - copy/paste less from the WebcilWriter task - copy/paste less from PEReader (will require moving WebcilReader to SRM) * [debug] Match bundled pdbs if we're looking up .webcil files The pdbs are registered by wasm with a notional .dll filename. if the debugger does a lookup using a .webcil name instead, allow the match * Adjust debug directory entries when writing webcil files the PE COFF debug directory entries contain a 'pointer' field which is an offset from the start of the file. When writing the webcil file, the header is typically smaller than a PE file, so the offsets are wrong. Adjust the offsets by the size of the file. We assume (and assert) the debug directory entries actually point at some PE COFF sections in the PE file (as opposed to somewhere past the end of the known PE data). When writing, we initially just copy all the sections directly, then seek to where the debug directory entries are, and overwrite them with updated entries that have the correct 'pointer' * Fix bug in WebcilWriter Stream.CopyTo takes a buffer size, not the number of bytes to copy. * bugfix: the debug directory is at pe_debug_rva not at the CLI header * skip debug fixups if there's no debug directory * WebcilReader: implement CodeView and Emebedded PPDB support * [WBT] Add UseWebcil option (default to true) * rename WebcilWriter -> WebcilConverter [NFC] * fixup AssemblyLoadedEventTest * hack: no extension on assembly for breakpoint * pass normal .dll name for MainAssemblyName in config let the runtime deal with it - bundle matching will resolve it to the .webcil file * Wasm.Debugger.Tests: give CI 10 more minutes * Add Microsoft.NET.WebAssembly.Webcil assembly project Mark it as shipping, but not shipping a nuget package. The idea is that it will be shipped along with the WasmAppBuilder msbuild task, and with the BrowserDebugProxy tool. * Move WebcilConverter to Microsoft.NET.WebAssembly.Webcil * Move WebcilReader to Microsoft.NET.WebAssembly.Webcil delete the duplicated utility classes * make the webcil magic and version longer * Code style improvements from review * Improve some exception messages, when possible * Suggestings from code review * Add WasmEnableWebcil msbuild property. Off by default * Build non-wasm runtimes without .webcil support * Run WBT twice: with and without webcil This is a total of 4 runs: with and without workloads x with and without webcil * do the cartesian product correctly in msbuild * also add webcil to template projects * environment variable has to be non-null and "true" We set it to "false" sometimes * Fix wasm work items They should be the same whether or not webcil is used. Just the WorkloadItemPrefix should be used to change the name. * Update src/libraries/sendtohelix-wasm.targets * PInvokeTableGeneratorTests: don't try to use the net472 WasmAppBuilder Look for the default target framework subdirectory under the tasks directory in the runtime pack when trying to find the tasks dll. In particular don't try to load the net472 version on modern .NET * PInvokeTableGeneratorTests: Add more diagnostic output if tasksDir is not found * simplify prefix comparison in bundled_assembly_match * WasmAppBuilder improve logging Just emit a single Normal importance message about webcil; details as Low importance. * Add missing using Co-authored-by: Ankit Jain <[email protected]> Co-authored-by: Larry Ewing <[email protected]>
WebCIL is a new container format for .NET assemblies when publishing to webassembly. (See dotnet/runtime#79416) Related to dotnet/runtime#80807
WebCIL is a new container format for .NET assemblies when publishing to webassembly. (See dotnet/runtime#79416) Related to dotnet/runtime#80807
WebCIL is a new container format for .NET assemblies when publishing to webassembly. (See dotnet/runtime#79416) Related to dotnet/runtime#80807
Define a new container format for .NET assemblies that looks less like a Windows PE file. Use it for bundling assemblies in wasm projects.
TODO:
<WasmEnableWebcil>true</WasmEnableWebcil>
to try itNice to have:
Follow-up PR: