-
Notifications
You must be signed in to change notification settings - Fork 5.2k
[browser] build browserhost in host.native subset #120298
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Tagging subscribers to 'arch-wasm': @lewing, @pavelsavara |
32ecbfa
to
04232e2
Compare
3dfa996
to
2e8702e
Compare
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
For Linux and Windows, we were hitting issues with linker incompatibilities and a ton of complexity because of the flags we pass around and the fact that we shuffle pieces between different machines in the PR builds. For wasm, we already have an established path of shipping static libs, so I'm not as concerned that we'll hit the same problem. If you're able to pre-link everything with the new host like how we do the singlefilehost, I'd recommend that. If you can't, then splitting it is fine. |
f86a2dc
to
7070241
Compare
f1b86aa
to
a1543be
Compare
@jkoritzinsky I'm not fan either but I don't have better idea. Let's discuss options or invent something else A) ideally we would have only one CMake root for all native builds, on all targets.
B) singlefilehost style, static linking of host and libs within coreclr root:
C) what I'm doing in this PR. 3 cmake roots and loose dependency via msbuild
cc: @jkotas @AaronRobinsonMSFT @janvorli @akoeplinger thoughts ? I will take non-controversial changes from this PR into different PR in the meantime and focus this PR only on this topic. |
This comment was marked as outdated.
This comment was marked as outdated.
This is something I see as possible in the future, but due to the fact that the Mono CMake tree is designed fundamentally differently from the rest of the CMake trees makes unification difficult.
This is not a problem, as the Debug/Release versions of the native libs shouldn't differ in contract (so combining debug libs.native in the host with release managed libs should be fine, or we aren't sufficiently testing singlefilehost in CI).
Don't we need to do this anyway for linking the out-of-tree browserhost (built in our CMake in
My preference would be to build the For libs.native, how does Mono WASM get the native libs binaries?
Does Mono always link the .a files later (and never build a pre-linked-together browserhost equivalent)?
If the innerloop for making a runtime change would be
Generally, we've kept flags consistent by centralizing on
|
This comment was marked as outdated.
This comment was marked as outdated.
7b745e2
to
8c22189
Compare
@jkoritzinsky I rebased this PR on top of option B) that was already merged into main branch. Now it should be easier to discuss option C) I don't insist it needs to be done this way, but it's much closer to what customers would run if they link the .a files on their dev environment using workload (LLVM/emcc). It also makes coreCLR CMake innner dev loop significantly faster. Emcc linking is slow and this way we only link corerun but not corehost in there. Effectively in half of the time. I also updated the top description. |
# WASM-TODO set GEN_PINVOKE to 1 once we generate them. static entrypoint.c prevents C linker from trimming, when IL trimming trims PInvokes | ||
set(GEN_PINVOKE 0) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One of my concerns is that if you resolve this TODO, then the mechanism with which this PR references the libraries native bits will get the wrong libraries native bits (the ones for NativeAOT, not the ones that build with the GEN_PINVOKE=1
logic).
Maybe that's a product problem as well (and this PR is just exposing it), I'm not sure.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we are happy to use libs with entrypoint.c
when building coreRun
.
We should use generated pinvoke instead, when we produce runtime pack (so that for example TCP sockets are trimmed). That means that libraries built by libs.native
subset should not use entrypoint.c
(once we have the generator). This makes this PR important.
We would run the generator in the src\native\corehost\corehost.proj
MSBuild. We would compile the generated .c file as part of host.native
CMake and it would serve same purpose as entrypoint.c
, but just with DLLImports that are really used in the final application.
This maps to what we do for Mono in browser.proj
which is mono.wasmruntime
subset. It logically maps to host.native
for coreCLR.
- The "final application" for runtime pack is
dotnet.native.wasm
that has all features supported by libraries managed code on browser tartget. - The "final application" for customer app, would have full trimming, and so it would drop C code that is not being used by that app. This is why we ship workload and why we link on customer machine.
cc @radekdoulik
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is why we ship workload
Nit: I do not think this has anything to do with shipping as a workload. NativeAOT works the same way, links on customer machine, and it ships as a package.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How could we run a generator in corehost.proj that produces a .c
file that only has the entrypoints used by the final application?
Doesn't the generator have to run on the customer machine? Wouldn't a generator that runs in our build need to include everything (effectively reproducing the entrypoints.c files we have today)?
I don't see a reason for us to move the browserhost out of the CoreCLR CMake for the purpose of running a tool that will generate the same information we already have with less complicated infrastructure.
Sure, we need to run the tool for customer apps, but that's what tests are for. We should have tests that validate this experience instead of adding more complexity to the build for the purpose of adding test coverage (when we really should have dedicated tests).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This ties into my idea where the browserhost
target does not link against the browserhost.Static
target directly and just includes the same sources and flags (and stays within the CoreCLR build) as there's fundamental differences between the two.
I think that the browserhost target should build with CoreCLR and link against the libs.native builds that have entrypoints.c.
The static lib browserhost.Static should build within host.native. As it's a static lib, it doesn't need libs.native or clr.native to have been built beforehand to build.
Then, when constructing the package to ship the static assets (transport package/workload/runtime pack/whatever), the no-entrypoints.c libs.native assets can be pulled in (as well as the static CoreCLR assets) to go into the package (just like how NativeAOT pulls things together).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need to have some sort of pre-built entrypoints.c file for the
add_executable
target if that one is supposed to bundle the native libraries within itself and not require the user to link on their machine.Yes, but the
entrypoint.c
set as-is is too broad.
That makes sense.
Do you want to run the tooling after building the libs.sfx subset and running library trimming? Does that actually get us the result we want?
Yes, I think so.
Could we instead structure this as a "validation test" where we check in the result (so we can build browserhost in clr.runtime) and validate post-trimming that the tooling generates the correct results?
Yes, I think this is my B1) alternative. I don't love it, tho.
Would it be in constant flux in git or we assume it's very stable ?
I would assume that this would be very stable (likely only changing with fundamental feature rewrites or new APIs). Any change here would likely be reflected later in one of our size benchmarks.
If we want to trim QCalls that call into CoreCLR as well, this is the only solution that will work cleanly in our infrastructure.
qcallentrypoints.cpp
are already full of#ifdef
Could you please elaborate how we could validate this ? Do you mean that we somehow check the list of native symbols and make sure that all of them are used in IL trimmed corelib ? That would work.
For the WASM build, we could have a separate qcallentrypoints.cpp that is built with the same tooling as the entrypoints.c generator. Since QCalls are fundamentally just P/Invokes, the same tooling should work.
@jkotas brings up a good point about the interp thunks for the P/Invoke transitions.
I think we can do this as a first pass and revisit in the future. I really don't like that this introduces a dependency from the native builds on the managed builds (we already have the reverse for NativeAOT), but that's something we could address in a future PR (ie the B1 alternative above).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
build libs.sfx -> generate call stubs for everything used by libs.sfx -> link
dotnet.native.wasm
together with the unmanaged runtime bitsThe generation of these call stubs can take care of generating trimmed down equivalent of
entrypoints.c
too.
I think this describes the process pretty well. In mono the generator is a msbuild task, that scans the assemblies and generates the C source files.
I am in the process of porting the generator to the coreclr. I will have the interp to native thunks ready soon and then will take a look at the pinvoke and entrypoints generation.
For the question of whether to generate entrypoints files or not, I think it makes sense to generate them. The alternative would be to keep the ifdefs up to date. That seems fragile.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can do this as a first pass and revisit in the future. I really don't like that this introduces a dependency from the native builds on the managed builds (we already have the reverse for NativeAOT), but that's something we could address in a future PR (ie the B1 alternative above).
I see the pain point with the dependency direction. If the changes to generated entrypoints are rare, we can keep the generated entrypoints in the repo (B1) and add a step after native and managed are built? This step would compare generated ones with the repo ones, fail the build if they differ and request manual update to the sources in the repo.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we can keep the generated entrypoints in the repo (B1)
This would be a temporary solution. We are going to have this sort of dependency once partial R2R gets online anyway.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think long-term (.NET 12/13 or later) the solution will look as follows:
- We'll have one CMake tree for everything but the linked-together browserhost. This will run before building any managed code. (This is a general goal I have for the future).
- We'll build the managed code.
- We'll link together the prebuilt browserhost after having built the managed code (likely as part of some project in the src/installer/sfx tree so we can pull in the partial R2R).
As @jkotas says, the partial R2R dependency forces us to go down a different route. But longer term (not in this PR), we can avoid going back into our CMake tree to do the final link (and instead do something like what our NAOT targets do or a slimmed down version of what the workload targets do)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR restructures the browser (WASM) build for CoreCLR to separate the browserhost build into the host.native
subset, aligning it with the standard build pattern. Previously, browserhost was built within the CoreCLR runtime subset, which violated the architecture where host components should be built separately.
Key Changes:
- Restored
host.native
subset for CoreCLR browser builds - Removed browserhost from
clr.runtime
subset - Changed browserhost dependencies from CMake targets to static library file paths
- Introduced
CORERUN_LIBS_ONLY
flag to reduce native library build scope when building for CoreCLR
Reviewed Changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 5 comments.
Show a summary per file
File | Description |
---|---|
src/coreclr/CMakeLists.txt | Moved browser-specific build configuration from the runtime subset to the libs-native section and removed browserhost as a runtime dependency |
src/native/corehost/browserhost/CMakeLists.txt | Changed from CMake target dependencies to explicit .a file dependencies; updated paths to use SHARED_LIB_DESTINATION and added new install targets |
src/native/corehost/browserhost/sample/CMakeLists.txt | Updated sample paths from STATIC_LIB_DESTINATION to SHARED_LIB_DESTINATION |
src/native/libs/CMakeLists.txt | Added conditional compilation guards using CORERUN_LIBS_ONLY to exclude non-essential libraries |
src/native/libs/System.Native/CMakeLists.txt | Added dependency declarations for timezone data libraries |
src/native/libs/Common/JavaScript/CMakeLists.txt | Added installation of package.json to sharedFramework |
eng/native/configurepaths.cmake | Added CLR_ARTIFACTS_BIN_DIR variable definition |
eng/native.props | Added CMake arguments for version and configuration information |
eng/liveBuilds.targets | Updated artifact paths and file collections to reflect the new browserhost location in host.native subset |
eng/Subsets.props | Added host.native to default CoreCLR browser subsets and reorganized libs.native subset definition |
<!-- Libraries native set --> | ||
<ItemGroup Condition="$(_subset.Contains('+libs.native+'))"> | ||
<ProjectToBuild Include="$(SharedNativeRoot)libs\build-native.proj" Category="libs" /> | ||
</ItemGroup> | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of moving libs.native up, can you move the host subsets to build right before libs.pretest? If we are going to need to run tooling on the outputs of libs.sfx to build the host, the host subsets should come after the libs.sfx subset.
We don't have to rush this PR, if we think we are going to return do B1 anyway. Before I close this, does any of you feel that C) with pre-generated |
I would avoid pre-generated wasm-pinvoke.c in git.
This should not be that bad if the generation is done in incremental build friendly fashion. If you are touching on that the incremental builds are slow, I think it is a general problem that we have been introducing by representing more dependencies correctly in the build system. It makes the incremental builds slower and slower. It has been very apparent in some of the inner loop experiences under libraries. I think the solution for that is to introduce gestures that one can use to skip the slow dependency checks, such as |
Well, my current incremental experience on windows.
|
It seems this is yet another time when the version.c is compiled unconditionally, triggering relinking. We should fix that. |
Side note: this is because of #119639. Adding |
host.native
subset back for coreCLR browserbrowserhost
forclr.runtime
subsetbrowserhost
dependent on relevant.a
files oflibs.native+clr.runtime
subsets, instead of CMake target andadd_subdirectory
eng/liveBuilds.targets
CORERUN_LIBS_ONLY
to make libs build smaller when used from coreCLR CMakeCLR_ARTIFACTS_BIN_DIR
,CMAKE_BUILD_LIBRARIES_CONFIGURATION
,CMAKE_NET_CORE_APP_CURRENT_VERSION
,CMAKE_BUILD_RUNTIME_CONFIGURATION
SHARED_LIB_DESTINATION
andSHARED_CLR_DESTINATION
in MSBuild and pass that instead.Contributes to #120206