Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Native AOT smoke test #82

Merged
merged 38 commits into from
Sep 11, 2020
Merged

Add Native AOT smoke test #82

Show file tree
Hide file tree
Changes from 30 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
5bcbc4b
Add Native AOT smoke test
MichalStrehovsky Sep 3, 2020
4ead94c
Merge branch 'NativeAOT' into smoketest
MichalStrehovsky Sep 3, 2020
9c1d255
Try again
MichalStrehovsky Sep 3, 2020
0276669
Sigh
MichalStrehovsky Sep 4, 2020
50cb002
What else is there
MichalStrehovsky Sep 4, 2020
8b6fd62
Update CLRTest.NativeAot.targets
MichalStrehovsky Sep 4, 2020
ead11fc
Not this way
MichalStrehovsky Sep 5, 2020
c9f9109
Merge branch 'NativeAOT' into smoketest
MichalStrehovsky Sep 7, 2020
6cec9ba
Build accelerator
MichalStrehovsky Sep 7, 2020
64775ee
Do not require building CoreCLR CoreLib
MichalStrehovsky Sep 7, 2020
a93001e
Do not run xunit with corerun
MichalStrehovsky Sep 7, 2020
6eaef11
Copy over JIT and JitInterface
MichalStrehovsky Sep 7, 2020
71338af
Fix up build script
MichalStrehovsky Sep 7, 2020
4b7a62d
Ok let's see
MichalStrehovsky Sep 7, 2020
44b8b4d
Maybe get linux support
MichalStrehovsky Sep 7, 2020
b640780
Revert "Maybe get linux support"
MichalStrehovsky Sep 8, 2020
a236229
Revert "Build accelerator"
MichalStrehovsky Sep 8, 2020
806b379
Add more tests
MichalStrehovsky Sep 8, 2020
34ec632
Always build libs as Release
MichalStrehovsky Sep 8, 2020
2d2c084
Linux support
MichalStrehovsky Sep 8, 2020
d0c3ca8
Merge branch 'NativeAOT' into smoketest
MichalStrehovsky Sep 9, 2020
f79f81c
Maybe fix linux
MichalStrehovsky Sep 9, 2020
0e28c08
Yaml bs
MichalStrehovsky Sep 9, 2020
ae0cd8d
Revert "Revert "Build accelerator""
MichalStrehovsky Sep 9, 2020
7b27d04
Update ConfigurablePInvokePolicy.cs
MichalStrehovsky Sep 9, 2020
da4fc0b
Sigh
MichalStrehovsky Sep 9, 2020
4740d5f
Revert "Revert "Revert "Build accelerator"""
MichalStrehovsky Sep 9, 2020
aa7c9a1
Enable debug symbols
MichalStrehovsky Sep 9, 2020
8fbc531
Try to fix linux
MichalStrehovsky Sep 9, 2020
582faad
Do we need this?
MichalStrehovsky Sep 9, 2020
4ce9680
Revert "Do we need this?"
MichalStrehovsky Sep 10, 2020
790737a
Don't run tests on Linux
MichalStrehovsky Sep 10, 2020
7c9238e
Merge branch 'NativeAOT' into smoketest
MichalStrehovsky Sep 10, 2020
27578b7
CR feedback
MichalStrehovsky Sep 10, 2020
4de5b88
Fix conflict with master
MichalStrehovsky Sep 10, 2020
33c4c6e
Fix building JIT without runtime on Linux
MichalStrehovsky Sep 10, 2020
77a2599
Do mscorrc differently
MichalStrehovsky Sep 11, 2020
e7646b4
Add comment
MichalStrehovsky Sep 11, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions dotnet.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,5 +23,9 @@ source $scriptroot/eng/common/tools.sh
InitializeDotNetCli true # Install
__dotnetDir=${_InitializeDotNetCli}

# Temporarily make this dotnet more permanent
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to send PR with this change to dotnet/runtime too.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is only needed because we don't publish ilc as a selfcontained app. Do you think it's still valuable in dotnet/runtime?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought that you have added this for your own convenience to make dotnet.sh work on anything. Is it the case, or is this needed to make the CI work as well?

If this is for convenience, I think it would be useful in dotnet/runtime too.
If this is a temporary workaround to make CI work, it would be nice to have better comments on it that describe what it is for and what it would take to remove it.

export DOTNET_ROLL_FORWARD=Major
export DOTNET_ROOT=${__dotnetDir}

dotnetPath=${__dotnetDir}/dotnet
${dotnetPath} "$@"
89 changes: 2 additions & 87 deletions eng/pipelines/runtimelab.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ jobs:
jobParameters:
timeoutInMinutes: 90
testGroup: innerloop
buildArgs: -s clr+libs+installer -c debug -runtimeConfiguration Checked
buildArgs: -s nativeaot+libs+installer -lc release -rc checked
extraStepsTemplate: /eng/pipelines/runtimelab/runtimelab-post-build-steps.yml

#
Expand All @@ -81,7 +81,7 @@ jobs:
jobParameters:
timeoutInMinutes: 90
testGroup: innerloop
buildArgs: -s clr+libs+libs.tests+installer -c $(_BuildConfig) /p:ArchiveTests=true
buildArgs: -s nativeaot+libs+installer -c $(_BuildConfig) /p:ArchiveTests=true
extraStepsTemplate: /eng/pipelines/runtimelab/runtimelab-post-build-steps.yml
extraStepsParameters:
uploadTests: true
Expand All @@ -100,88 +100,3 @@ jobs:
nameSuffix: All_Configurations
buildArgs: -s clr.runtime+libs -c $(_BuildConfig) -allConfigurations

#
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@joperezr The CI infrastructure in the NativeAOT branch is going to be a bit different from how CI is done in the runtime repo (we can't do AOT compilation on the Helix machines because they miss some big prerequisites, so we'll do the testing on the build machine - this should be similar how the current plan for iOS testing in the runtime repo will be).

Do you think it makes sense to try to share the runtimelab.yml file, or just make a new one to avoid constant merge conflicts?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It’s ok to have another yml file if desired, but that will mean that we have to setup a whole new pipeline on AzDO for this branch so just keep thag in mind, since AzDO pipelines are tied to only one yml file. On the other hand, if you do decide to change runtimelab.yml to suit your needs, we could just ignore conflicts from that file and keep branch’s version always.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, you shouldn’t expect to get many conflicts as the idea is to have runtimelab.yml as stable as possible, which in fact was the whole reason we created that as opposed to using runtime.yml which gets changed much more frequently.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I didn't know that extra yaml file would be work - I'll just keep this and we'll ignore conflicts in the future integrations.

# CoreCLR Test builds using live libraries release build
#
- template: /eng/pipelines/common/platform-matrix.yml
parameters:
jobTemplate: /eng/pipelines/common/templates/runtimes/build-test-job.yml
buildConfig: Checked
platforms:
- CoreClrTestBuildHost # Either OSX_x64 or Linux_x64
jobParameters:
testGroup: innerloop
liveLibrariesBuildConfig: Release
dependsOn:
- build_Linux_x64_Checked_
- build_Linux_x64_Release_

#
# CoreCLR Test executions using live libraries
#
- template: /eng/pipelines/common/platform-matrix.yml
parameters:
jobTemplate: /eng/pipelines/common/templates/runtimes/run-test-job.yml
buildConfig: Checked
platforms:
- Linux_x64
helixQueueGroup: pr
helixQueuesTemplate: /eng/pipelines/coreclr/templates/helix-queues-setup.yml
jobParameters:
testGroup: innerloop
liveLibrariesBuildConfig: Release
dependsOn:
- coreclr_common_test_build_p0_AnyOS_AnyCPU_Checked

- template: /eng/pipelines/common/platform-matrix.yml
parameters:
jobTemplate: /eng/pipelines/common/templates/runtimes/run-test-job.yml
buildConfig: Checked
platforms:
- Windows_NT_x64
helixQueueGroup: pr
helixQueuesTemplate: /eng/pipelines/coreclr/templates/helix-queues-setup.yml
jobParameters:
testGroup: innerloop
liveLibrariesBuildConfig: Release
dependsOn:
- coreclr_common_test_build_p0_AnyOS_AnyCPU_Checked
- build_Windows_NT_x64_Checked_
- build_Windows_NT_x64_Release_

#
# Libraries Release Test Execution against a release coreclr runtime
#
- template: /eng/pipelines/common/platform-matrix.yml
parameters:
jobTemplate: /eng/pipelines/libraries/run-test-job.yml
buildConfig: Release
platforms:
- Linux_x64
helixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.yml
jobParameters:
isFullMatrix: false
isOfficialBuild: false
testScope: innerloop
liveRuntimeBuildConfig: Release
dependsOnTestBuildConfiguration: Release
dependsOnTestArchitecture: x64
dependsOn:
- build_Linux_x64_Release_

- template: /eng/pipelines/common/platform-matrix.yml
parameters:
jobTemplate: /eng/pipelines/libraries/run-test-job.yml
buildConfig: Release
platforms:
- Windows_NT_x64
helixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.yml
jobParameters:
isFullMatrix: false
isOfficialBuild: false
testScope: innerloop
liveRuntimeBuildConfig: Release
dependsOnTestBuildConfiguration: Release
dependsOnTestArchitecture: x64
dependsOn:
- build_Windows_NT_x64_Release_
57 changes: 8 additions & 49 deletions eng/pipelines/runtimelab/runtimelab-post-build-steps.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,53 +7,12 @@ parameters:

steps:
# Build coreclr native test output
- script: $(Build.SourcesDirectory)/src/tests/build$(scriptExt) skipstressdependencies skipmanaged skipgeneratelayout $(buildConfigUpper) ${{ parameters.archType }}
displayName: Build native test components
- script: $(Build.SourcesDirectory)/src/tests/build$(scriptExt) skipstressdependencies $(buildConfigUpper) ${{ parameters.archType }}
displayName: Build tests

# Copy all build output into artifacts staging directory
- template: /eng/pipelines/libraries/prepare-for-bin-publish.yml

# Zip CoreCLR Build Output
- template: /eng/pipelines/common/upload-artifact-step.yml
parameters:
rootFolder: $(Build.SourcesDirectory)/artifacts/bin/coreclr/${{ parameters.osGroup }}.${{ parameters.archType }}.$(buildConfigUpper)
archiveType: $(archiveType)
tarCompression: $(tarCompression)
includeRootFolder: false
archiveExtension: $(archiveExtension)
artifactName: CoreCLRProduct__${{ parameters.osGroup }}${{ parameters.osSubgroup }}_${{ parameters.archType }}_${{ parameters.buildConfig }}
displayName: 'CoreCLR product build'

# Zip Test Build
- ${{ if eq(parameters.uploadTests, true) }}:
- template: /eng/pipelines/common/upload-artifact-step.yml
parameters:
rootFolder: $(Build.SourcesDirectory)/artifacts/helix
includeRootFolder: true
archiveType: $(archiveType)
archiveExtension: $(archiveExtension)
tarCompression: $(tarCompression)
artifactName: libraries_test_assets_${{ parameters.osGroup }}_${{ parameters.archType }}_${{ parameters.buildConfig }}
displayName: Test Assets

# Zip product native assets for use by Tests
- template: /eng/pipelines/common/upload-artifact-step.yml
parameters:
rootFolder: $(Build.SourcesDirectory)/artifacts/tests/coreclr/obj/${{ parameters.osGroup }}.${{ parameters.archType }}.$(buildConfigUpper)
includeRootFolder: false
archiveType: $(archiveType)
tarCompression: $(tarCompression)
archiveExtension: $(archiveExtension)
artifactName: CoreCLRNativeTestArtifacts_${{ parameters.osGroup }}${{ parameters.osSubgroup }}_${{ parameters.archType }}_${{ parameters.buildConfig }}
displayName: 'native test components'

# Zip Libraries Build Output
- template: /eng/pipelines/common/upload-artifact-step.yml
parameters:
rootFolder: $(Build.ArtifactStagingDirectory)/artifacts
archiveType: $(archiveType)
tarCompression: $(tarCompression)
includeRootFolder: false
archiveExtension: $(archiveExtension)
artifactName: libraries_bin_${{ parameters.osGroup }}${{ parameters.osSubgroup }}_${{ parameters.archType }}_${{ parameters.buildConfig }}
displayName: Build Assets
- ${{ if eq(parameters.osGroup, 'Windows_NT') }}:
- script: $(Build.SourcesDirectory)/src/tests/run$(scriptExt) runnativeaottests $(buildConfigUpper) ${{ parameters.archType }}
displayName: Run tests
- ${{ if ne(parameters.osGroup, 'Windows_NT') }}:
- script: $(Build.SourcesDirectory)/src/tests/run$(scriptExt) --runnativeaottests $(buildConfigUpper) ${{ parameters.archType }}
displayName: Run tests
1 change: 0 additions & 1 deletion src/coreclr/src/nativeaot/BuildIntegration/Microsoft.NETCore.Native.Unix.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,6 @@ The .NET Foundation licenses this file to you under the MIT license.

<ItemGroup>
<NativeLibrary Include="$(IlcPath)/framework/libSystem.Native$(NativeLibraryExtension)" />
<NativeLibrary Include="$(IlcPath)/framework/libSystem.Globalization.Native$(NativeLibraryExtension)" />
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need this. We need to fix the build to start building it again.

cc @VSadov

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wondering - Is this going to be used or just to make the build happy?
Libraries in coreclr hardcode "QCall" at the moment. AOT uses different libraries?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Native AOT needs a regular .a in the package. It does not need any tables with mapping, etc. The native linker that produces the final native AOT binary will take care of stitching everything together.

Libraries in coreclr hardcode "QCall"

"QCall" is hardcoded just for the System.Globalization.Native in coreclr. Everything else uses actual name.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider out of scope for this pull request and disable testing on Linux?

Or is this as easy as adding Globalization.Native to CMakeLists.txt and treating DllImport("QCall") same as the other libSystem imports (i.e. have linker deal with it)?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider out of scope for this pull request and disable testing on Linux?

I think so.

<NativeLibrary Include="$(IlcPath)/framework/libSystem.IO.Compression.Native$(NativeLibraryExtension)" />
<NativeLibrary Include="$(IlcPath)/framework/libSystem.Net.Security.Native$(NativeLibraryExtension)" />
<NativeLibrary Include="$(IlcPath)/framework/libSystem.Security.Cryptography.Native.Apple$(NativeLibraryExtension)" Condition="'$(TargetOS)' == 'OSX'"/>
Expand Down
3 changes: 2 additions & 1 deletion src/coreclr/src/tools/aot/ILCompiler/ConfigurablePInvokePolicy.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,8 @@ public override bool GenerateDirectCall(string importModule, string methodName)
methodName == "LoadLibraryExW" ||
methodName == "GetProcAddress" ||
methodName == "SetLastError" ||
methodName == "GetLastError")
methodName == "GetLastError" ||
methodName == "LocalAlloc")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add LocalFree here as well to make it symmetric? Also, GetProcessHeap and HeapAlloc do not need to be in the list now.

{
return true;
}
Expand Down
27 changes: 22 additions & 5 deletions src/coreclr/src/tools/aot/ILCompiler/ILCompiler.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,14 +16,18 @@
<Configurations>Debug;Release;Checked</Configurations>
</PropertyGroup>

<PropertyGroup>
<LibraryNamePrefix>lib</LibraryNamePrefix>
<LibraryNamePrefix Condition="$([MSBuild]::IsOsPlatform('WINDOWS'))"></LibraryNamePrefix>
<LibraryNameExtension Condition="$([MSBuild]::IsOsPlatform('WINDOWS'))">.dll</LibraryNameExtension>
<LibraryNameExtension Condition="$([MSBuild]::IsOsPlatform('LINUX'))">.so</LibraryNameExtension>
<LibraryNameExtension Condition="$([MSBuild]::IsOsPlatform('OSX'))">.dylib</LibraryNameExtension>
</PropertyGroup>

<PropertyGroup>
<ObjectWriterVersion>1.0.0-alpha-28820-01</ObjectWriterVersion>
<CoreDisToolsVersion>1.0.1-prerelease-00005</CoreDisToolsVersion>

<ObjWriterLibrary Condition="'$(TargetsWindows)' == 'true'">objwriter.dll</ObjWriterLibrary>
<ObjWriterLibrary Condition="'$(TargetsOSX)' == 'true'">libobjwriter.dylib</ObjWriterLibrary>
<ObjWriterLibrary Condition="'$(ObjWriterLibrary)' == ''">libobjwriter.so</ObjWriterLibrary>

<!-- CoreDisTools are used in debugging visualizers. The package only exists on Windows -->
<IncludeCoreDisTools Condition="'$(TargetsWindows)' == 'true' and '$(Configuration)' != 'Release'">true</IncludeCoreDisTools>
</PropertyGroup>
Expand All @@ -38,7 +42,7 @@
</ItemGroup>

<ItemGroup>
<Content Include="$(NuGetPackageRoot)\runtime.$(DefaultAppHostRuntimeIdentifier).microsoft.dotnet.objectwriter\$(ObjectWriterVersion)\runtimes\$(DefaultAppHostRuntimeIdentifier)\native\$(ObjWriterLibrary)">
<Content Include="$(NuGetPackageRoot)\runtime.$(DefaultAppHostRuntimeIdentifier).microsoft.dotnet.objectwriter\$(ObjectWriterVersion)\runtimes\$(DefaultAppHostRuntimeIdentifier)\native\$(LibraryNamePrefix)objwriter$(LibraryNameExtension)">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Visible>false</Visible>
<Pack>false</Pack>
Expand All @@ -49,6 +53,19 @@
<Visible>false</Visible>
<Pack>false</Pack>
</Content>

<Content Include="$(BinDir)\$(LibraryNamePrefix)jitinterface$(LibraryNameExtension)">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Visible>false</Visible>
<Pack>false</Pack>
</Content>

<Content Include="$(BinDir)\$(LibraryNamePrefix)clrjit$(LibraryNameExtension)">
<Link>$(LibraryNamePrefix)clrjitilc$(LibraryNameExtension)</Link>
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Visible>false</Visible>
<Pack>false</Pack>
</Content>
</ItemGroup>

<ItemGroup>
Expand Down
11 changes: 11 additions & 0 deletions src/coreclr/tests/runtest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,7 @@
parser.add_argument("--ilasmroundtrip", dest="ilasmroundtrip", action="store_true", default=False)
parser.add_argument("--run_crossgen_tests", dest="run_crossgen_tests", action="store_true", default=False)
parser.add_argument("--run_crossgen2_tests", dest="run_crossgen2_tests", action="store_true", default=False)
parser.add_argument("--run_nativeaot_tests", dest="run_nativeaot_tests", action="store_true", default=False)
parser.add_argument("--large_version_bubble", dest="large_version_bubble", action="store_true", default=False)
parser.add_argument("--precompile_core_root", dest="precompile_core_root", action="store_true", default=False)
parser.add_argument("--skip_test_run", dest="skip_test_run", action="store_true", default=False, help="Does not run tests. Useful in conjunction with --precompile_core_root")
Expand Down Expand Up @@ -909,6 +910,11 @@ def run_tests(args,
print("Setting RunCrossGen2=true")
os.environ["RunCrossGen2"] = "true"

if args.run_nativeaot_tests:
print("Running tests Native AOT")
print("Setting RunNativeAot=true")
os.environ["RunNativeAot"] = "true"

if args.large_version_bubble:
print("Large Version Bubble enabled")
os.environ["LargeVersionBubble"] = "true"
Expand Down Expand Up @@ -1083,6 +1089,11 @@ def setup_args(args):
lambda unused: True,
"Error setting run_crossgen2_tests")

coreclr_setup_args.verify(args,
"run_nativeaot_tests",
lambda unused: True,
"Error setting run_nativeaot_tests")

coreclr_setup_args.verify(args,
"precompile_core_root",
lambda arg: True,
Expand Down
4 changes: 4 additions & 0 deletions src/coreclr/tests/src/CLRTest.Execute.Bash.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -282,6 +282,10 @@ then
else
LAUNCHER="$_DebuggerFullPath $(_CLRTestRunFile)"
fi
if [ ! -z "$RunNativeAot" ]
then
LAUNCHER="$_DebuggerFullPath"
fi

$(BashIlrtTestLaunchCmds)

Expand Down
3 changes: 3 additions & 0 deletions src/coreclr/tests/src/CLRTest.Execute.Batch.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -297,6 +297,9 @@ IF NOT "%CLRCustomTestLauncher%"=="" (
) ELSE (
set LAUNCHER=%_DebuggerFullPath% $(_CLRTestRunFile)
)
IF NOT "%RunNativeAot%"=="" (
set LAUNCHER=%_DebuggerFullPath%
)

$(BatchIlrtTestLaunchCmds)

Expand Down
1 change: 1 addition & 0 deletions src/coreclr/tests/src/CLRTest.Execute.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ This file contains the logic for providing Execution Script generation.

<Import Project="CLRTest.Jit.targets" />
<Import Project="CLRTest.CrossGen.targets" />
<Import Project="CLRTest.NativeAot.targets" />
<Import Project="CLRTest.GC.targets" />
<Import Project="CLRTest.Execute.*.targets" />
<Import Project="CLRTest.MockHosting.targets" Condition="'$(RequiresMockHostPolicy)' == 'true'" />
Expand Down
Loading