Skip to content

Produce a package with the host-running ILC for repos in the VMR #111443

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jan 17, 2025
Merged

Produce a package with the host-running ILC for repos in the VMR #111443

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
a2bb086
Build our in-build ILC like how we build our in-build Crossgen2 so we…
jkoritzinsky Jan 14, 2025
5602a69
Produce a package for the in-build ILC
jkoritzinsky Jan 14, 2025
0f55c82
Only mark the host packages as IsShipping=false at publish time. We w…
jkoritzinsky Jan 14, 2025
3ae3105
Always build ILCompiler_inbuild
jkoritzinsky Jan 15, 2025
ec45e4f
Merge branch 'main' of https://github.com/dotnet/runtime into host-ilc
jkoritzinsky Jan 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions eng/Publishing.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,9 @@
<Artifact Condition="'$(RuntimeFlavor)' != 'Mono' and '$(OutputRID)' != '$(NETCoreSdkRuntimeIdentifier)'"
Update="$(ArtifactsPackagesDir)**\Microsoft.NETCore.App.Crossgen2.$(NETCoreSdkRuntimeIdentifier).*.nupkg"
Visibility="Vertical" />
<Artifact Condition="'$(RuntimeFlavor)' != 'Mono' and '$(OutputRID)' != '$(NETCoreSdkRuntimeIdentifier)'"
Update="$(ArtifactsPackagesDir)**\runtime.$(NETCoreSDKRuntimeIdentifier).Microsoft.DotNet.ILCompiler.*.nupkg"
Visibility="Vertical" />
</ItemGroup>

<!--
Expand Down
3 changes: 2 additions & 1 deletion eng/Subsets.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -581,6 +581,7 @@
<_BuildHostPack Condition="'$(DotNetBuildAllRuntimePacks)' == 'true' and '$(TargetsMobile)' != 'true' and !('$(TargetsLinuxBionic)' == 'true' and '$(TargetArchitecture)' == 'arm')">true</_BuildHostPack>
<_BuildBundle Condition="'$(BuildNativeAOTRuntimePack)' != 'true' and '$(RuntimeFlavor)' == '$(PrimaryRuntimeFlavor)' and '$(TargetsMobile)' != 'true'">true</_BuildBundle>
<_BuildBundle Condition="'$(DotNetBuildAllRuntimePacks)' == 'true' and '$(TargetsMobile)' != 'true' and !('$(TargetsLinuxBionic)' == 'true' and '$(TargetArchitecture)' == 'arm')">true</_BuildBundle>
<BuildHostTools Condition="'$(DotNetBuildOrchestrator)' == 'true' and '$(OutputRID)' != '$(NETCoreSdkRuntimeIdentifier)'">true</BuildHostTools>
</PropertyGroup>

<ItemGroup>
Expand All @@ -601,7 +602,7 @@
When we're building in the VMR, we need to provide a crossgen2 that runs on the host machine for downstream repos to use to R2R their code.
In non-VMR builds, downstream repos can use the crossgen2 built for the target host SDK from another build leg, but in the VMR we need to provide one to use.
-->
<ProjectToBuild Condition="'$(RuntimeFlavor)' != 'Mono' and '$(TargetsMobile)' != 'true' and '$(TargetsLinuxBionic)' != 'true' and '$(DotNetBuildOrchestrator)' == 'true' and '$(OutputRID)' != '$(NETCoreSdkRuntimeIdentifier)'" Include="$(InstallerProjectRoot)pkg\sfx\Microsoft.NETCore.App\Microsoft.NETCore.App.Crossgen2.Host.sfxproj" Category="packs" />
<ProjectToBuild Condition="'$(RuntimeFlavor)' != 'Mono' and '$(TargetsMobile)' != 'true' and '$(TargetsLinuxBionic)' != 'true' and '$(BuildHostTools)' == 'true'" Include="$(InstallerProjectRoot)pkg\sfx\Microsoft.NETCore.App\Microsoft.NETCore.App.Crossgen2.Host.sfxproj" Category="packs" />
</ItemGroup>
<ItemGroup>
<SharedFrameworkProjectToBuild Condition="'$(_BuildHostPack)' == 'true'" Include="$(InstallerProjectRoot)pkg\archives\dotnet-nethost.proj" />
Expand Down
2 changes: 1 addition & 1 deletion eng/liveBuilds.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
<PropertyGroup>
<CoreCLRSharedFrameworkDir>$([MSBuild]::NormalizeDirectory('$(CoreCLRArtifactsPath)', 'sharedFramework'))</CoreCLRSharedFrameworkDir>
<CoreCLRILCompilerDir>$([MSBuild]::NormalizeDirectory('$(CoreCLRArtifactsPath)', 'ilc-published'))</CoreCLRILCompilerDir>
<CoreCLRCrossILCompilerDir Condition="'$(CrossBuild)' == 'true' or '$(BuildArchitecture)' != '$(TargetArchitecture)' or '$(HostOS)' != '$(TargetOS)' or '$(EnableNativeSanitizers)' != ''">$([MSBuild]::NormalizeDirectory('$(CoreCLRArtifactsPath)', '$(BuildArchitecture)', 'ilc'))</CoreCLRCrossILCompilerDir>
<CoreCLRCrossILCompilerDir>$([MSBuild]::NormalizeDirectory('$(CoreCLRArtifactsPath)', '$(BuildArchitecture)', 'ilc'))</CoreCLRCrossILCompilerDir>
<CoreCLRAotSdkDir>$([MSBuild]::NormalizeDirectory('$(CoreCLRArtifactsPath)', 'aotsdk'))</CoreCLRAotSdkDir>
<CoreCLRBuildIntegrationDir>$([MSBuild]::NormalizeDirectory('$(CoreCLRArtifactsPath)', 'build'))</CoreCLRBuildIntegrationDir>

Expand Down
3 changes: 1 addition & 2 deletions eng/testing/linker/trimmingTests.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
<Project>
<PropertyGroup>
<IlcToolsPath>$(CoreCLRILCompilerDir)</IlcToolsPath>
<IlcToolsPath Condition="'$(TargetArchitecture)' != '$(BuildArchitecture)' or '$(EnableNativeSanitizers)' != ''">$(CoreCLRCrossILCompilerDir)</IlcToolsPath>
<IlcToolsPath>$(CoreCLRCrossILCompilerDir)</IlcToolsPath>
</PropertyGroup>

<ItemGroup>
Expand Down
3 changes: 1 addition & 2 deletions eng/testing/tests.singlefile.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,7 @@
</PropertyGroup>

<PropertyGroup Condition="'$(TestNativeAot)' == 'true'">
<IlcToolsPath>$(CoreCLRILCompilerDir)</IlcToolsPath>
<IlcToolsPath Condition="'$(CrossBuild)' == 'true' or '$(TargetArchitecture)' != '$(BuildArchitecture)' or '$(EnableNativeSanitizers)' != ''">$(CoreCLRCrossILCompilerDir)</IlcToolsPath>
<IlcToolsPath>$(CoreCLRCrossILCompilerDir)</IlcToolsPath>
<SysRoot Condition="('$(CrossBuild)' == 'true' or '$(BuildArchitecture)' != '$(TargetArchitecture)') and '$(HostOS)' != 'windows'">$(ROOTFS_DIR)</SysRoot>
<IlcBuildTasksPath>$(CoreCLRILCompilerDir)netstandard/ILCompiler.Build.Tasks.dll</IlcBuildTasksPath>
<IlcSdkPath>$(CoreCLRAotSdkDir)</IlcSdkPath>
Expand Down
20 changes: 16 additions & 4 deletions src/coreclr/tools/aot/ILCompiler/ILCompiler_inbuild.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,20 @@
<Project Sdk="Microsoft.NET.Sdk">
<Project Sdk="Microsoft.NET.Sdk" DefaultTargets="Publish;PublishItemsOutputGroup">
<PropertyGroup>
<CrossHostArch>$(BuildArchitecture)</CrossHostArch>
<OutputPath>$(RuntimeBinDir)/$(CrossHostArch)/ilc/</OutputPath>
<RuntimeIdentifier>$(ToolsRID)</RuntimeIdentifier>
<CrossHostArch Condition="'$(CrossBuild)' == 'true' or '$(TargetArchitecture)' != '$(BuildArchitecture)' or '$(HostOS)' != '$(TargetOS)' or '$(EnableNativeSanitizers)' != ''">$(BuildArchitecture)</CrossHostArch>
<_IsPublishing>true</_IsPublishing>
<RuntimeIdentifier>$(NETCoreSdkRuntimeIdentifier)</RuntimeIdentifier>
<PublishDir>$(RuntimeBinDir)$(BuildArchitecture)/ilc/</PublishDir>
<SelfContained>true</SelfContained>
<PublishTrimmed>true</PublishTrimmed>
<!--
Publish with AOT on Windows as Single file doesn't work well when an app has to include DiaSymReader.Native.
Publish single-file elsewhere so we don't need to adjust our "AOT in build" logic to handle targeting the SDK RID.
-->
<PublishAot Condition="$(NETCoreSdkPortableRuntimeIdentifier.StartsWith('win-'))">true</PublishAot>
<PublishSingleFile Condition="'$(PublishAot)' != 'true'">true</PublishSingleFile>
<PublishReadyToRun>true</PublishReadyToRun>
<PublishReadyToRunComposite>true</PublishReadyToRunComposite>
<TrimmerSingleWarn>false</TrimmerSingleWarn>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm removing these in my PR because a) we don't have crossgen2 at the time it's built on community platforms and b) singlefile is enough for these internal artifacts. All official platforms end up using PublishAot in the nuget packages.

</PropertyGroup>
<Import Project="ILCompiler.props" />
</Project>
53 changes: 45 additions & 8 deletions src/installer/pkg/projects/Microsoft.DotNet.ILCompiler/Microsoft.DotNet.ILCompiler.pkgproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,55 @@
<IncludeRuntimeJson Condition="'$(DotNetBuildSourceOnly)' == 'true'">false</IncludeRuntimeJson>
<!-- Set 'ExcludeLineupReference' to avoid package dependencies due to disabling 'runtime.json'. -->
<ExcludeLineupReference Condition="'$(DotNetBuildSourceOnly)' == 'true'">true</ExcludeLineupReference>
<IsShipping Condition="'$(PackHostILCompiler)' == 'true'">false</IsShipping>
</PropertyGroup>

<Target Name="AddBuildIntegrationFiles"
DependsOnTargets="ResolveLibrariesFromLocalBuild"
BeforeTargets="GetPackageFiles"
Condition="'$(PackageTargetRuntime)' == ''">
<ItemGroup>
<File Include="$(CoreCLRBuildIntegrationDir)*" TargetPath="build" />
<File Include="$(CoreCLRILCompilerDir)netstandard\*" TargetPath="tools/netstandard" />
</ItemGroup>
</Target>

<Target Name="GetHostIlcCompilerFiles"
DependsOnTargets="ResolveLibrariesFromLocalBuild"
BeforeTargets="GetPackageFiles"
Condition="'$(PackHostILCompiler)' == 'true' and '$(PackageTargetRuntime)' != ''">

<ItemGroup>
<File Include="$(CoreCLRCrossILCompilerDir)*" TargetPath="tools" />
</ItemGroup>

<ItemGroup Condition="'$(TargetOS)' == 'linux'">
<File Include="$(SharedNativeRoot)libs\System.Globalization.Native\*" TargetPath="native/src/libs/System.Globalization.Native"/>
<File Include="$(SharedNativeRoot)libs\System.Security.Cryptography.Native\*" TargetPath="native/src/libs/System.Security.Cryptography.Native"/>
<File Include="$(SharedNativeRoot)libs\build-local.sh" TargetPath="native/src/libs/build-local.sh"/>
<File Include="$(SharedNativeRoot)minipal\*" TargetPath="native/src/minipal"/>
<File Include="$(SharedNativeRoot)libs\Common\*" TargetPath="native/src/libs/Common"/>
</ItemGroup>

<!-- exclude native symbols from ilc package (they are included in symbols package) -->
<ItemGroup>
<!-- on windows, remove the pdbs only from tools directory (both managed and native) -->
<LibPackageExcludes Include="tools\%2A%2A\%2A.pdb"/>

<LibPackageExcludes Include="%2A%2A\%2A.dbg"/>
<LibPackageExcludes Include="%2A%2A\%2A.debug"/>
<LibPackageExcludes Include="%2A%2A\%2A.dSYM"/>
<LibPackageExcludes Include="%2A%2A\%2A.dwarf"/>
</ItemGroup>

</Target>

<Target Name="GetIlcCompilerFiles"
DependsOnTargets="ResolveLibrariesFromLocalBuild"
BeforeTargets="GetPackageFiles">
BeforeTargets="GetPackageFiles"
Condition="'$(PackHostILCompiler)' != 'true' and '$(PackageTargetRuntime)' != ''">

<ItemGroup Condition="'$(PackageTargetRuntime)' != ''">
<ItemGroup>
<!-- Include all libraries (managed, static, etc.), but exclude native shared libraries (those with LibSuffix) -->
<File Include="@(LibrariesRuntimeFiles)" TargetPath="framework" Condition="'%(LibrariesRuntimeFiles.IsNative)' != 'true' or '%(LibrariesRuntimeFiles.Extension)' != '$(LibSuffix)'" />

Expand All @@ -32,19 +74,14 @@
<File Include="$(MibcOptimizationDataDir)/$(TargetOS)/$(TargetArchitecture)/**/*.mibc" TargetPath="mibc" />
</ItemGroup>

<ItemGroup Condition="'$(PackageTargetRuntime)' != '' and '$(TargetOS)' == 'linux'">
<ItemGroup Condition="'$(TargetOS)' == 'linux'">
<File Include="$(SharedNativeRoot)libs\System.Globalization.Native\*" TargetPath="native/src/libs/System.Globalization.Native"/>
<File Include="$(SharedNativeRoot)libs\System.Security.Cryptography.Native\*" TargetPath="native/src/libs/System.Security.Cryptography.Native"/>
<File Include="$(SharedNativeRoot)libs\build-local.sh" TargetPath="native/src/libs/build-local.sh"/>
<File Include="$(SharedNativeRoot)minipal\*" TargetPath="native/src/minipal"/>
<File Include="$(SharedNativeRoot)libs\Common\*" TargetPath="native/src/libs/Common"/>
</ItemGroup>

<ItemGroup Condition="'$(PackageTargetRuntime)' == ''">
<File Include="$(CoreCLRBuildIntegrationDir)*" TargetPath="build" />
<File Include="$(CoreCLRILCompilerDir)netstandard\*" TargetPath="tools/netstandard" />
</ItemGroup>

<!-- exclude native symbols from ilc package (they are included in symbols package) -->
<ItemGroup>
<!-- on windows, remove the pdbs only from tools directory (both managed and native) -->
Expand Down
1 change: 1 addition & 0 deletions src/installer/pkg/projects/nativeaot-packages.proj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
<Project Include="Microsoft.DotNet.ILCompiler\Microsoft.DotNet.ILCompiler.pkgproj" />
<ProjectReference Include="@(Project)" />
<ProjectReference Include="@(Project)" AdditionalProperties="PackageTargetRuntime=$(OutputRID)" />
<ProjectReference Condition="'$(BuildHostTools)' == 'true'" Include="@(Project)" AdditionalProperties="PackageTargetRuntime=$(NETCoreSdkRuntimeIdentifier);PackHostILCompiler=true" />
</ItemGroup>

<Import Sdk="Microsoft.Build.Traversal" Project="Sdk.targets" />
Expand Down
3 changes: 1 addition & 2 deletions src/tests/Directory.Build.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -565,8 +565,7 @@

<IlcTreatWarningsAsErrors>false</IlcTreatWarningsAsErrors>

<IlcToolsPath>$(CoreCLRILCompilerDir)</IlcToolsPath>
<IlcToolsPath Condition="'$(TargetArchitecture)' != '$(BuildArchitecture)' or '$(TargetOS)' != '$(HostOS)' or '$(EnableNativeSanitizers)' != ''">$(CoreCLRCrossILCompilerDir)</IlcToolsPath>
<IlcToolsPath>$(CoreCLRCrossILCompilerDir)</IlcToolsPath>
<IlcBuildTasksPath>$(CoreCLRILCompilerDir)netstandard/ILCompiler.Build.Tasks.dll</IlcBuildTasksPath>
<IlcSdkPath>$(CoreCLRAotSdkDir)</IlcSdkPath>
<IlcFrameworkPath>$(MicrosoftNetCoreAppRuntimePackRidLibTfmDir)</IlcFrameworkPath>
Expand Down
Loading