Shift Most of Wasm AOT test build to helix #48226

steveisok · 2021-02-12T16:41:25Z

Since AOT'ing each test suite takes between 3-9 min, we need to shift the burden over to helix.

This is done by:

building the test assemblies on the build machine
- the wasm part of the build is not executed on the build machine,
  because it has the AOT build part
Zip up the test assembly+friends, and any bits required to run the wasm
app build for that on helix (eg. emsdk, wasm app targets, cross compiler etc)
Send all this to helix, and use a custom aot-build.proj
- which recreates all the build inputs for the WasmBuildApp target
  using the paths for the assets on helix
- then we can run WasmBuildApp for the build, resulting in a wasm app
  bundle.
Run the tests!

We already have the bits required for building wasm apps on helix, supported
for Wasm.Build.Tests, which we can use here too.

Trimming:

Since, AOT can be so expensive, we use EnableAggressiveTrimming=true(EAT), but
that means that we could have issues due to trimming.
And it can sometimes be unclear whether the build/test failures are due to trimming
or AOT.
Because these builds+test runs are different from other builds, owing to the
"build partially on helix" step, a normal EAT build would not be the same as
to help with testing this, we add two lanes to runtime-staging:
- *_Mono_AOT: builds AOT+EAT on helix
- *_Mono_EAT: builds EAT, on helix
  - this is required because we want to run almost the same kinda
    build: 1. build test assembly; 2. send to helix; 3. build wasm app; 4. run tests
This should effectively mean that we can see which errors might be due to EAT, and
which are clearly because of EAT+AOT.

Since AOT'ing each test suite takes between 3-9 min, we need to shift the burden over to helix.

ghost · 2021-02-12T16:41:33Z

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

Since AOT'ing each test suite takes between 3-9 min, we need to shift the burden over to helix.

Author:	steveisok
Assignees:	-
Labels:	`area-Infrastructure-mono`
Milestone:	-

…nto build-wasm-aot-helix

…tests

…m run with EnableAggressiveTrimming enabled

radical · 2021-04-15T20:19:38Z

eng/testing/tests.wasm.targets

  <PropertyGroup Condition="'$(UseDefaultBlazorWASMFeatureSwitches)' == 'true'">
    <EventSourceSupport>false</EventSourceSupport>
-    <UseSystemResourceKeys>true</UseSystemResourceKeys>
+    <UseSystemResourceKeys>false</UseSystemResourceKeys>
    <EnableUnsafeUTF7Encoding>false</EnableUnsafeUTF7Encoding>
    <HttpActivityPropagationSupport>false</HttpActivityPropagationSupport>

    <!-- we want to default to what Blazor has, except if we are building in Debug config -->
-    <DebuggerSupport Condition="'$(Configuration)' != 'Debug'">false</DebuggerSupport>
+    <DebuggerSupport Condition="'$(DebuggerSupport)' == '' and '$(Configuration)' != 'Debug'">false</DebuggerSupport>
  </PropertyGroup>


Some these need to have values different from what UseDefaultBlazorWasmFeatureSwitches wants:

DebuggerSupport - this is needed by some library tests because they depend on pdbs, or debugger attributes

Currently, I am setting this in the specific projects that need it

UseSystemResourceKeys - IIUC, this is needed so that the resources don't get trimmed, which is needed by some library tests

Setting these only for some library projects makes it kinda inconsistent. And I'm wondering if that affects what the intention of UseDefaultBlazorWasmFeatureSwitches is.
So, do we need switch at all? If so, then what should be done here?

/cc @lewing @akoeplinger

I agree with @radical, the intention of UseDefaultBlazorWASMFeatureSwitches is to make us consistent with Blazor. I think overriding in the individual library test projects where necessary is fine for now.

radical · 2021-04-15T20:39:25Z

eng/pipelines/common/platform-matrix.yml

@@ -219,7 +219,7 @@ jobs:
      targetRid: browser-wasm
      platform: Browser_wasm
      container:
-        image: ubuntu-18.04-webassembly-20210223133559-4800846
+        image: ubuntu-18.04-webassembly-20210309143118-005aab4


@steveisok @mdh1418 Do you know why this change was made? main is still using the older version.

judging from the date it's to bring in dotnet/dotnet-buildtools-prereqs-docker@f25f6cd

…LoadContext The test project explicitly copies, and needs `mscorlib.dll`, which gets trimmed out with `EnableAggressiveTrimming=true`. Preserve that.

Fails because it can't find `xunit.runner.utility.netcoreapp10`

`InlineDataDiscoverer`, and `MemberDataDiscoverer` were getting trimmed, which would cause some tests to get skipped. For example, ~9k tests in `System.Data.Common.Tests`

…dTest Tracked in dotnet#51369

…MetadataLoadContext" This reverts commit 79fa75b.

This reverts commit 4742f4b.

…ionTest.RegressionTest1" This reverts commit 867a381.

…t.XmlLoadTest" This reverts commit f33e181.

…es.Tests

…steve

radical · 2021-04-21T06:12:29Z

Failing test in runtime (Libraries Test Run release mono Linux x64 Debug): #51588

Failing test in Libraries Test Run release coreclr Linux x64 Debug is possibly: #48658

Failing test in Libraries Test Run release mono OSX x64 Debug: #51611

akoeplinger · 2021-04-21T16:42:18Z

eng/pipelines/common/platform-matrix.yml

@@ -219,7 +219,7 @@ jobs:
      targetRid: browser-wasm
      platform: Browser_wasm
      container:
-        image: ubuntu-18.04-webassembly-20210223133559-4800846
+        image: ubuntu-18.04-webassembly-20210309143118-005aab4


judging from the date it's to bring in dotnet/dotnet-buildtools-prereqs-docker@f25f6cd

akoeplinger · 2021-04-21T16:46:19Z

eng/testing/tests.wasm.targets

  <PropertyGroup Condition="'$(UseDefaultBlazorWASMFeatureSwitches)' == 'true'">
    <EventSourceSupport>false</EventSourceSupport>
-    <UseSystemResourceKeys>true</UseSystemResourceKeys>
+    <UseSystemResourceKeys>false</UseSystemResourceKeys>
    <EnableUnsafeUTF7Encoding>false</EnableUnsafeUTF7Encoding>
    <HttpActivityPropagationSupport>false</HttpActivityPropagationSupport>

    <!-- we want to default to what Blazor has, except if we are building in Debug config -->
-    <DebuggerSupport Condition="'$(Configuration)' != 'Debug'">false</DebuggerSupport>
+    <DebuggerSupport Condition="'$(DebuggerSupport)' == '' and '$(Configuration)' != 'Debug'">false</DebuggerSupport>
  </PropertyGroup>


I agree with @radical, the intention of UseDefaultBlazorWASMFeatureSwitches is to make us consistent with Blazor. I think overriding in the individual library test projects where necessary is fine for now.

akoeplinger · 2021-04-21T16:47:36Z

eng/testing/tests.wasm.targets

+    </ItemGroup>
+
+    <!-- To recreate the original project on helix, we need to set the wasm properties also, same as the
+         library test project. Eg. $(InvariantGlobalization) -->


do we have a way of knowing the list of properties is complete or when we need to update it?

Um .. not really. So, when something breaks? I didn't add all the properties either, because I want to do that as it comes up.

akoeplinger · 2021-04-21T16:50:53Z

src/libraries/System.Collections.NonGeneric/tests/System.Collections.NonGeneric.Tests.csproj

@@ -2,6 +2,7 @@
  <PropertyGroup>
    <IncludeRemoteExecutor>true</IncludeRemoteExecutor>
    <TargetFrameworks>$(NetCoreAppCurrent)</TargetFrameworks>
+    <DebuggerSupport Condition="'$(DebuggerSupport)' == '' and '$(TargetOS)' == 'Browser'">true</DebuggerSupport>


curious why does this need .pdb?

Eg:

fail: [FAIL] System.Collections.Tests.HashtableTests.DebuggerAttribute info: System.InvalidOperationException : Expected one DebuggerDisplayAttribute on System.Collections.Hashtable. info: at System.Diagnostics.DebuggerAttributes.ValidateDebuggerDisplayReferences(Object obj) info: at System.Collections.Tests.HashtableTests.DebuggerAttribute() info: at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) fail: [FAIL] System.Collections.Tests.QueueTests.DebuggerAttribute_NullQueue_ThrowsArgumentNullException info: System.InvalidOperationException : Expected one DebuggerTypeProxyAttribute on System.Collections.Queue. info: at System.Diagnostics.DebuggerAttributes.GetProxyType(Type type, Type[] genericTypeArguments) info: at System.Diagnostics.DebuggerAttributes.ValidateDebuggerTypeProxyProperties(Type type, Type[] genericTypeArguments, Object obj) info: at System.Diagnostics.DebuggerAttributes.ValidateDebuggerTypeProxyProperties(Type type, Object obj) info: at System.Collections.Tests.QueueTests.DebuggerAttribute_NullQueue_ThrowsArgumentNullException() info: at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

akoeplinger · 2021-04-21T16:52:44Z

src/tasks/WasmBuildTasks/GenerateAOTProps.cs

@@ -0,0 +1,55 @@
+// Licensed to the .NET Foundation under one or more agreements.


this should ideally be added to

runtime/src/tasks/tasks.proj

Lines 8 to 11 in d08f7f2

<ProjectReference Remove="$(MSBuildThisFileDirectory)WasmAppBuilder\WasmAppBuilder.csproj"

Condition="'$(TargetOS)' != 'Browser'" />

<ProjectReference Remove="$(MSBuildThisFileDirectory)WasmBuildTasks\WasmBuildTasks.csproj"

Condition="'$(TargetOS)' != 'Browser'" />

I'm not sure I understand. This file will get built with WasmBuildTasks. What should be added to tasks.proj?

…steve

lewing

looks green to me, there is some minor whitespace damage but this was a long road so lets land it.

steveisok and others added 9 commits February 9, 2021 14:58

Unraveling helix

438d530

More helix setup

4ccf81b

Trying to keep directory names

83760ea

Should be able to generate a wasm app on an island

0b15afc

More tweaks

adcd026

Better path attempts. Helix is at least trying to aot now

9003dc3

Paths are correct - AOT System.Buffers works on helix

f2dc5df

Shift Most of Wasm AOT test build to helix

3ba7a82

Since AOT'ing each test suite takes between 3-9 min, we need to shift the burden over to helix.

Merge remote-tracking branch 'upstream/master' into build-wasm-aot-helix

281cb1a

dotnet-issue-labeler bot added the area-Infrastructure-mono label Feb 12, 2021

Steve Pfister and others added 11 commits February 12, 2021 11:49

Reenable all tests

ed324a1

Skip functional tests for now

62b9c3e

Copy emscripten in repo so the helix payload can be written to disk

ed62416

update to th right darc feed for the time being

4b6f7cb

Merge remote-tracking branch 'upstream/master' into build-wasm-aot-helix

03d33c4

Getting deps right

e1cd904

Merge remote-tracking branch 'upstream/master' into build-wasm-aot-helix

7e0661c

Path adjustments... Should work nicely

4e7c715

Merge branch 'build-wasm-aot-helix' of github.com:steveisok/runtime i…

573a5e7

…nto build-wasm-aot-helix

Merge remote-tracking branch 'upstream/master' into build-wasm-aot-helix

cac00aa

Tweak linker settings to meet tests expectations

06a4738

Base automatically changed from master to main March 1, 2021 09:07

Mitchell Hwang and others added 6 commits March 1, 2021 09:59

[testing] Add more assemblies to depend on during linking of library …

4f17c4e

…tests

Temporarily EnableAggressiveTrimming to discern AOT failures

c0bb31b

Merge remote-tracking branch 'upstream/main' into build-wasm-aot-helix

6e7905f

Hardcoding pays a price ;-)

b4482b6

[wasm] ReAdd WasmSrcPath

f60f641

[wasm] Exclude System.Reflection.MetadataLoadContext from non AOT was…

21dec7f

…m run with EnableAggressiveTrimming enabled

System.ComponentModel.TypeConverter: fix path to il descriptors file

b09c9cb

radical reviewed Apr 15, 2021

View reviewed changes

radical added 4 commits April 15, 2021 19:58

[wasm] Fix build errors due to trimming in System.Reflection.Metadata…

79fa75b

…LoadContext The test project explicitly copies, and needs `mscorlib.dll`, which gets trimmed out with `EnableAggressiveTrimming=true`. Preserve that.

Skip xunit trimming descriptors for wasm build tests

0ec0210

Fails because it can't find `xunit.runner.utility.netcoreapp10`

[wasm] Fix tests skipped, or not discovered due to trimming

f81fde9

`InlineDataDiscoverer`, and `MemberDataDiscoverer` were getting trimmed, which would cause some tests to get skipped. For example, ~9k tests in `System.Data.Common.Tests`

[wasm] Disable broken test System.Data.Tests.XmlDataReaderTest.XmlLoa…

f33e181

…dTest Tracked in dotnet#51369

radical force-pushed the build-wasm-aot-helix branch from 88b8234 to f33e181 Compare April 16, 2021 10:31

radical added 11 commits April 20, 2021 12:48

Merge remote-tracking branch 'origin/main' into build-wasm-aot-helix

fec5a71

Revert "[wasm] Fix build errors due to trimming in System.Reflection.…

77184f6

…MetadataLoadContext" This reverts commit 79fa75b.

Revert "[wasm] Fix trimming errors for System.Runtime.Loader.Tests"

5021fec

This reverts commit 4742f4b.

[wasm] Disable System.Reflection.MetadataLoadContext tests

da1aca9

Revert "[wasm] Disable broken test System.Xml.Tests.CTransformRegress…

ec9255c

…ionTest.RegressionTest1" This reverts commit 867a381.

Revert "[wasm] Disable broken test System.Data.Tests.XmlDataReaderTes…

29aed77

…t.XmlLoadTest" This reverts commit f33e181.

[wasm] Disable failing tests build for System.ComponentModel.Primitiv…

50dddbb

…es.Tests

Disable System.Data.Common.Tests

160f053

Disable System.Text.Json.Tests

17693cd

disable System.Xml.Xsl.XslCompiledTransformApi.Tests

fcfb527

Merge remote-tracking branch 'origin/main' into build-wasm-aot-helix-…

04e7f99

…steve

runfoapp bot mentioned this pull request Apr 21, 2021

Test System.ComponentModel.Tests.ReflectionCachesUpdateHandlerTests.ReflectionCachesUpdateHandler_CachesCleared is failing on mono OSX x64 #51588

Closed

akoeplinger reviewed Apr 21, 2021

View reviewed changes

radical added 2 commits April 21, 2021 16:17

Merge remote-tracking branch 'origin/main' into build-wasm-aot-helix-…

4f95a05

…steve

[wasm] Disable System.ComponentModel.TypeConverter.Tests

6bee3b7

lewing approved these changes Apr 22, 2021

View reviewed changes

radical merged commit 6ca3d91 into dotnet:main Apr 22, 2021

karelz added this to the 6.0.0 milestone May 20, 2021

mdh1418 mentioned this pull request May 27, 2021

[libraries] Move library tests Feature Switches defaults to Functional tests #53253

Merged

ghost locked as resolved and limited conversation to collaborators Jun 19, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Shift Most of Wasm AOT test build to helix #48226

Shift Most of Wasm AOT test build to helix #48226

steveisok commented Feb 12, 2021 •

edited by radical

Loading

ghost commented Feb 12, 2021

radical Apr 15, 2021

akoeplinger Apr 21, 2021

radical Apr 15, 2021

akoeplinger Apr 21, 2021

radical commented Apr 21, 2021

akoeplinger Apr 21, 2021

akoeplinger Apr 21, 2021

akoeplinger Apr 21, 2021 •

edited

Loading

radical Apr 21, 2021

akoeplinger Apr 21, 2021

radical Apr 21, 2021

akoeplinger Apr 21, 2021

radical Apr 21, 2021

lewing left a comment

		@@ -0,0 +1,55 @@
		// Licensed to the .NET Foundation under one or more agreements.

	<ProjectReference Remove="$(MSBuildThisFileDirectory)WasmAppBuilder\WasmAppBuilder.csproj"
	Condition="'$(TargetOS)' != 'Browser'" />
	<ProjectReference Remove="$(MSBuildThisFileDirectory)WasmBuildTasks\WasmBuildTasks.csproj"
	Condition="'$(TargetOS)' != 'Browser'" />

Shift Most of Wasm AOT test build to helix #48226

Shift Most of Wasm AOT test build to helix #48226

Conversation

steveisok commented Feb 12, 2021 • edited by radical Loading

This is done by:

Trimming:

ghost commented Feb 12, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

radical commented Apr 21, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akoeplinger Apr 21, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lewing left a comment

Choose a reason for hiding this comment

steveisok commented Feb 12, 2021 •

edited by radical

Loading

akoeplinger Apr 21, 2021 •

edited

Loading