[wasm] wasi: Enable library tests on CI #81052

radical · 2023-01-23T21:51:13Z

Adds tasks/targets to run a wasi console sample, and tests on CI - for linux, and windows
Only few tests are enabled by default
Other tests fail due to GC issues, missing fs/network access etc
Also, adds runtime-wasm-optional pipeline which will have the wasi build with all the tests enabled, and will be RED. This is disabled right now.

Some differences from browser/wasm case:

    [wasi] Use the real entrypoint even with async Main

    When the main assembly has a async main, roslyn generates a wrapper
    method which invokes the async main. And in case of wasm that
    async-main method is invoked so it can yield to the host.

    But in case of wasi we invoke the actual entrypoint.

Fixes #72641 .

ghost · 2023-01-23T21:51:29Z

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

null

Author:	radical
Assignees:	-
Labels:	`NO-MERGE`, `NO-REVIEW`, `arch-wasm`
Milestone:	-

pavelsavara · 2023-01-30T11:32:54Z

I would like to see this merged so that I could start looking at the tests. How can I help ?

radical · 2023-01-30T22:27:30Z

update: I'm cleaning this up now. I have to check the various runtime/native changes/cmake changes to make sure that they are still needed, after the earlier wasi PR being merged. I will open this up for review in a few days.

pavelsavara

good work!

This is large PR and I made many comments, but I don't insist on any of them.
I would like to merge it soon and fix things incrementally on top of this in further PRs.

eng/pipelines/common/platform-matrix.yml

eng/testing/WasiRunnerTemplate.sh

pavelsavara · 2023-01-31T09:50:37Z

eng/testing/tests.browser.targets

@@ -0,0 +1,344 @@
+<Project TreatAsLocalProperty="ArchiveTests">


Is this just rename ? Or is there something to review in detail ?

There are bits that I'm extracting from this to the base targets file. But this is very much WIP, so tests passing is the check for the changes right now!

pavelsavara · 2023-01-31T09:51:54Z

eng/testing/tests.wasi.targets

+    <BundleTestAppTargets>$(BundleTestAppTargets);BundleTestWasmApp</BundleTestAppTargets>
+    <DebuggerSupport Condition="'$(DebuggerSupport)' == '' and '$(Configuration)' == 'Debug'">true</DebuggerSupport>
+
+    <!-- set this when provisioning emsdk on CI -->


wasi SDK & WASI_SDK_PATH

pavelsavara · 2023-01-31T09:53:13Z

eng/testing/tests.wasi.targets

+    -->
+    <WasmNativeStrip Condition="'$(WasmNativeStrip)' == ''">false</WasmNativeStrip>
+    <WasmEmitSymbolMap Condition="'$(WasmEmitSymbolMap)' == '' and '$(RunAOTCompilation)' != 'true'">true</WasmEmitSymbolMap>
+    <WasmMainAssemblyFileName Condition="'$(WasmMainAssemblyFileName)' == ''">WasmTestRunner.dll</WasmMainAssemblyFileName>


Q: is WasmTestRunner.dll identical to browser ? If not, should they have different project or name ?

pavelsavara · 2023-01-31T10:51:12Z

src/mono/wasi/build/WasiApp.targets

+      <WasmNativeAsset Include="$(MicrosoftNetCoreAppRuntimePackRidNativeDir)$(WasmIcuDataFileName)" Condition="'$(InvariantGlobalization)' != 'true'" />
+      <WasmNativeAsset Include="$(MicrosoftNetCoreAppRuntimePackRidNativeDir)dotnet.timezones.blat" />
+
+      <WasmFilesToIncludeInFileSystem Include="@(WasmNativeAsset)" Condition="'%(WasmNativeAsset.FileName)%(WasmNativeAsset.Extension)' == 'dotnet.js.symbols'" />


dotnet.js.symbols could be removed. dotnet.timezones.blat needs to be discussed in context of VFS #81418

pavelsavara · 2023-01-31T10:51:33Z

src/mono/wasi/build/WasiApp.targets

+    <RemoveDir Directories="$(WasmAppDir)" />
+    <WasmAppBuilder
+      AppDir="$(WasmAppDir)"
+      MainJS="$(WasmMainJSPath)"


pavelsavara · 2023-01-31T10:52:10Z

src/mono/wasi/build/WasiApp.targets

+    <WriteLinesToFile File="$(WasmAppDir)\.stamp" Lines="" Overwrite="true" />
+  </Target>
+
+  <Target Name="_GenerateRunV8Script">


do we need wasmtime script ?

src/mono/wasi/mono-include/driver.h

src/tasks/WasmAppBuilder/wasi/EmitWasmBundleObjectFile.cs

radical · 2023-01-31T16:55:46Z

I've read the feedback on all the tests targets, and wasi targets files, but ignoring them for now, because they are very much in WIP. Most of them are copies of the existing wasm files, and will get pruned in future PRs.

radical · 2023-02-01T15:41:18Z

/azp run runtime-wasm

azure-pipelines · 2023-02-01T15:41:35Z

Azure Pipelines successfully started running 1 pipeline(s).

src/mono/wasi/wasi.proj

pavelsavara · 2023-02-03T13:42:39Z

/azp run runtime-wasm

azure-pipelines · 2023-02-03T13:42:50Z

Azure Pipelines successfully started running 1 pipeline(s).

* also for relink

pavelsavara · 2023-02-03T17:21:51Z

src/mono/mono/metadata/sgen-mono.c

@@ -2293,7 +2293,7 @@ sgen_client_scan_thread_data (void *start_nursery, void *end_nursery, gboolean p
 {
 	scan_area_arg_start = start_nursery;
 	scan_area_arg_end = end_nursery;
-#ifdef HOST_WASM
+#if defined(HOST_WASM) || defined(HOST_WASI)


should be rather HOST_BROWSER ?

- asserts about order of memory segments - further unified re-link and get rig of --stack-first

radical · 2023-02-03T19:09:34Z

Wasi tests are failing on g_assert (heap_base - stack_size_pad == data_end);
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-81052-merge-b56c232ac68c49339e/Microsoft.XmlSerializer.Generator.Tests/1/console.e72d0849.log?helixlogtype=result

the current memory layout: ``` /* * | -- increasing address ---> | * | data (data_end)| stack |(heap_base) heap | */ ``` .. based on which we can calculate a reasonable stack size. Tests were failing with the earlier code on the following assert: ```c size_t stack_size_pad = wasm_get_stack_size() + 12; // there is padding g_assert (heap_base - stack_size_pad == data_end); ```

radical · 2023-02-04T06:11:04Z

/azp run runtime-wasm

azure-pipelines · 2023-02-04T06:11:15Z

Azure Pipelines successfully started running 1 pipeline(s).

radical added NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) NO-REVIEW Experimental/testing PR, do NOT review it arch-wasm WebAssembly architecture labels Jan 23, 2023

dotnet-issue-labeler bot added the area-Build-mono label Jan 23, 2023

ghost assigned radical Jan 23, 2023

radical force-pushed the aj-wasi_build branch 2 times, most recently from acf4e33 to 270002c Compare January 25, 2023 22:46

build-analysis bot mentioned this pull request Jan 26, 2023

Tracking issue for CI build timeouts #76454

Closed

radical added 8 commits January 31, 2023 07:19

Add tasks from https://github.com/SteveSandersonMS/dotnet-wasi-sdk

a564ab2

build changes

5db6d5c

add test build changes

023226a

test fixes

ce9dff4

add more wasi build files

8e59b0d

fix wasi sample

03b6c28

add missing files

5d977c2

disable non-wasi/wasm builds

d918291

radical force-pushed the aj-wasi_build branch from 84553f5 to d918291 Compare January 31, 2023 07:38

radical removed NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) NO-REVIEW Experimental/testing PR, do NOT review it labels Jan 31, 2023

radical changed the title ~~WIP: [wasm] Wasi build improvements~~ [wasm] wasi: Enable library tests on CI Jan 31, 2023

pavelsavara reviewed Jan 31, 2023

View reviewed changes

radical added 5 commits January 31, 2023 16:59

address feedback from @pavelsavara

7d659e3

Merge remote-tracking branch 'origin/main' into wasi_tests

d7f70d2

cleanup

396e4df

restore ci builds

fd211a8

fix the optional build

7ad2c72

radical added 2 commits February 1, 2023 23:31

[wasm] fix wasi tests on windows/CI

0ea5bab

[wasm] Add wasi tests for CI/win

0fd51b8

radical mentioned this pull request Feb 2, 2023

WASI support tracking #65895

Open

27 tasks

This was referenced Feb 2, 2023

Build fails with "eng/common/tools.sh: line 474: 537 Segmentation fault" #76759

Closed

arm64 linux build machines seem to be missing key components? dotnet/arcade#12395

Closed

pavelsavara reviewed Feb 2, 2023

View reviewed changes

src/mono/wasi/wasi.proj Show resolved Hide resolved

Merge remote-tracking branch 'origin/main' into wasi_tests

efea8b9

radical mentioned this pull request Feb 2, 2023

[wasi] Enabling icu in wasi.proj breaks tests #81560

Closed

radical added the os-wasi Related to WASI variant of arch-wasm label Feb 2, 2023

build-analysis bot mentioned this pull request Feb 2, 2023

Test failure Loader\\classloader\\DictionaryExpansion\\DictionaryExpansion\\DictionaryExpansion.cmd #75244

Closed

fix stack range again

d97cb02

build-analysis bot mentioned this pull request Feb 3, 2023

Roslyn source generator crash on mono/linux/arm64 #81123

Closed

* stack first

a2239d4

* also for relink

pavelsavara reviewed Feb 3, 2023

View reviewed changes

- back to assembly in .S file. extern doesn't work is zero

bf21da7

- asserts about order of memory segments - further unified re-link and get rig of --stack-first

pavelsavara approved these changes Feb 3, 2023

View reviewed changes

radical added 3 commits February 4, 2023 06:08

fix typos in makefile

6fbd475

Run System.Collections.Tests also, which would test the oom case

3746b09

radical mentioned this pull request Feb 4, 2023

[wasm][debugger] Breakpoint test failing with multithreaded build - CreateGoodBreakpointAndHitGoToWasmPageWithoutAssetsComeBackAndHitAgain #81646

Closed

radical merged commit f8daacf into dotnet:main Feb 4, 2023

radical deleted the aj-wasi_build branch February 4, 2023 22:35

ghost locked as resolved and limited conversation to collaborators Mar 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[wasm] wasi: Enable library tests on CI #81052

[wasm] wasi: Enable library tests on CI #81052

radical commented Jan 23, 2023 •

edited

Loading

ghost commented Jan 23, 2023

pavelsavara commented Jan 30, 2023

radical commented Jan 30, 2023

pavelsavara left a comment •

edited

Loading

pavelsavara Jan 31, 2023

radical Jan 31, 2023

pavelsavara Jan 31, 2023

pavelsavara Jan 31, 2023

pavelsavara Jan 31, 2023

pavelsavara Jan 31, 2023

pavelsavara Jan 31, 2023

radical commented Jan 31, 2023

radical commented Feb 1, 2023

azure-pipelines bot commented Feb 1, 2023

pavelsavara commented Feb 3, 2023

azure-pipelines bot commented Feb 3, 2023

pavelsavara Feb 3, 2023

radical commented Feb 3, 2023

radical commented Feb 4, 2023

azure-pipelines bot commented Feb 4, 2023

		@@ -0,0 +1,344 @@
		<Project TreatAsLocalProperty="ArchiveTests">

[wasm] wasi: Enable library tests on CI #81052

[wasm] wasi: Enable library tests on CI #81052

Conversation

radical commented Jan 23, 2023 • edited Loading

ghost commented Jan 23, 2023

pavelsavara commented Jan 30, 2023

radical commented Jan 30, 2023

pavelsavara left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

radical commented Jan 31, 2023

radical commented Feb 1, 2023

azure-pipelines bot commented Feb 1, 2023

pavelsavara commented Feb 3, 2023

azure-pipelines bot commented Feb 3, 2023

Choose a reason for hiding this comment

radical commented Feb 3, 2023

radical commented Feb 4, 2023

azure-pipelines bot commented Feb 4, 2023

radical commented Jan 23, 2023 •

edited

Loading

pavelsavara left a comment •

edited

Loading