[wasm] Bump emscripten to 2.0.23 #53603

radekdoulik · 2021-06-02T14:23:57Z

No description provided.

This reverts commit ec1ae53.

This reverts commit 356b3ff.

ghost · 2021-06-02T14:24:03Z

Tagging subscribers to this area: @carlossanlop
See info in area-owners.md if you want to be subscribed.

Issue Details

Author:	radekdoulik
Assignees:	-
Labels:	`area-System.IO`
Milestone:	-

ghost · 2021-06-02T14:24:31Z

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

Author:	radekdoulik
Assignees:	-
Labels:	`arch-wasm`, `area-Build-mono`
Milestone:	-

This reverts commit eb2f954. The fix is not present in 2.0.23

This reverts commit 8be39f5. The fix is not present in 2.0.23

radekdoulik · 2021-06-09T14:45:03Z

The failing System.Net.Primitives.Unit.Tests.CookieContainerTest.SetCookies_Success is #53931

radekdoulik · 2021-06-09T14:50:46Z

The fix for FS bug, where it ignores permissions didn't make it into 2.0.23, even if it looks like it from the commit history.

I still see this in upstream\emscripten\src\library_fs.js:

  $FS__postset: function() {
    // TODO: do we need noFSInit?
    addAtInit('if (!Module["noFSInit"] && !FS.init.initialized) FS.init();');
    addAtMain('FS.ignorePermissions = false;');
...

addAtMain shouldn't be used here.

Context: emscripten-core/emscripten@a56817a

radical · 2021-06-09T14:52:28Z

Do we know why the AOT tests are failing?
for example - System.Xml.XPath.XmlDocument.Tests failed (received SIGKILL (-9))
and System.Data.Common.Tests

radical · 2021-06-09T15:00:45Z

Some of them are timing out, due to the recent increase in build time. We should bump the timeout, to we can at least get the tests running, and not miss any regressions. I would add

diff --git a/src/libraries/sendtohelixhelp.proj b/src/libraries/sendtohelixhelp.proj
index 84c140a2182..1dd6716f2e8 100644
--- a/src/libraries/sendtohelixhelp.proj
+++ b/src/libraries/sendtohelixhelp.proj
@@ -29,6 +29,7 @@
     <_workItemTimeout Condition="'$(Scenario)' == '' and '$(_workItemTimeout)' == '' and ('$(TargetArchitecture)' == 'arm64' or '$(TargetArchitecture)' == 'arm')">00:45:00</_workItemTimeout>
     <_workItemTimeout Condition="'$(Scenario)' != '' and '$(_workItemTimeout)' == '' and ('$(TargetArchitecture)' == 'arm64' or '$(TargetArchitecture)' == 'arm')">01:00:00</_workItemTimeout>
     <_workItemTimeout Condition="'$(Scenario)' == 'BuildWasmApps' and '$(_workItemTimeout)' == ''">01:00:00</_workItemTimeout>
+    <_workItemTimeout Condition="'$(TargetOS)' == 'Browser' and '$(NeedsToBuildWasmAppsOnHelix)' == 'true'">01:00:00</_workItemTimeout>
     <_workItemTimeout Condition="'$(Scenario)' == '' and '$(_workItemTimeout)' == ''">00:15:00</_workItemTimeout>
     <_workItemTimeout Condition="'$(Scenario)' != '' and '$(_workItemTimeout)' == ''">00:30:00</_workItemTimeout>

…ipten-2-0-23-bump

To be able to use this image before dotnet/runtime#53603 is finished

radical · 2021-06-10T16:38:38Z

8 libraries are failing with AOT. Some are timing out, some with emcc getting killed with:

emcc : error : '/datadisks/disk1/work/B1410964/p/build/emsdk/upstream/bin/wasm-ld @/datadisks/disk1/work/B1410964/t/emscripten_icjo4j9w.rsp' failed (received SIGKILL (-9))

radekdoulik · 2021-06-17T13:00:13Z

In the last run there are 7 test jobs failing, only 1 of them receiving signal SIGKILL, System.Data.Common.Tests.

On linux helix machine, the limits are set like this:

+ ulimit -a
time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        8192
coredump(blocks)     unlimited
memory(kbytes)       unlimited
locked memory(kbytes) 16384
process              31703
nofiles              16384
vmemory(kbytes)      unlimited
locks                unlimited
rtprio               0

and the system has cca 8GB of memory and 128GB of storage space for /datadisks/disk1 mount.

radekdoulik · 2021-06-17T19:37:18Z

Looks like the wasm-ld is indeed causing issues here. It is always this tool, which gets killed or is timeouting.

I tried to build the first library test, which timeouts - Microsoft.Extensions.Configuration.Json. It builds on my machine, but the memory usage during wasm-ld run is peaking around 12.3GB. The helix system has only 8GB, so that might be the issue here. It took cca 16 minutes to build. I will try to build it with 2.0.21 as well to see how the memory usage compares.

Note: the local build might be behaving differently compared to helix, I see 81 assemblies AOT'ed after linking vs 61 on helix.

radical · 2021-06-17T19:45:53Z

Do you think #54053 might help? It builds dll -> .bc -> .o with Oz, and then links with O0.

I tried to change EmccLinkOptimizationFlag property to -O0 and the memory peaked around 12.7GB as well. Not saure whether it is good enough test though.

radekdoulik · 2021-06-17T21:15:14Z

The 2.0.21 build of the same lib test peaked around 8.8GB of memory. So it looks like regression of wasm-ld. I will try to get more details and open an issue in emscripten repo.

lewing · 2021-06-17T21:33:07Z

cc @marek-safar

…ipten-2-0-23-bump

This reverts commit 67dd775.

radekdoulik · 2021-06-18T10:48:19Z

It might be the new string tail merging in wasm-ld:

- wasm-ld will now perform string tail merging in debug string sections as well
  as regular data sections.   This behaviour can be be disabled with `-Wl,-O0`.
  This should significantly reduce the size of dwarf debug information in the
  wasm binary.

Let see if the -Wl,-O0 changes the memory load.

radekdoulik · 2021-06-18T19:04:39Z

-Wl,O0 doesn't help with the memory load, neither does -Oz -Wl,-O0.

Merging current main improves it a bit, we are still at 10.9GB peek.

I have opened emscripten-core/emscripten#14485 and will try to dig deeper to see if I can spot what might be causing the higher memory usage.

To be sure it behaves the same as in local build

radekdoulik · 2021-06-21T18:46:56Z

Lukas Lansky opened a ticket for us to get more memory (and cores as side effect) https://github.com/dotnet/core-eng/issues/13425

It looks like it reduces the memory load a lot

lewing · 2021-06-23T05:34:14Z

Nice to see this green, please get in mergeable condition so we can land it.

And reset the default value

This reverts commit fec0e55.

…ipten-2-0-23-bump

lewing · 2021-06-24T04:59:24Z

browser failure is #53957

radekdoulik · 2021-06-24T15:17:44Z

CoreCLR Pri0 Runtime Tests Run windows arm64 checked - failure is the same seen in #54053 and doesn't contain logs

…bugger2 * origin/main: (107 commits) Disable MacCatalyst arm64 PR test runs on staging pipeline (dotnet#54678) [WASM] Fix async/await in config loading (dotnet#54652) Fix for heap_use_after_free flagged by sanitizer (dotnet#54679) [wasm] Bump emscripten to 2.0.23 (dotnet#53603) Fix compiler references when building inside VS (dotnet#54614) process more TLS frames at one when available (dotnet#50815) Add PeriodicTimer (dotnet#53899) UdpClient with span support (dotnet#53429) exclude fragile tests (dotnet#54671) get last error before calling a method that might fail as well (dotnet#54667) [FileStream] add tests for device and UNC paths (dotnet#54545) Fix sporadic double fd close (dotnet#54660) Remove Version.Clone from AssemblyName.Clone (dotnet#54621) [wasm] Enable fixed libraries tests (dotnet#54641) [wasm] Fix blazor/aot builds (dotnet#54651) [mono][wasm] Fix compilation error on wasm (dotnet#54659) Fix telemetry for Socket connects to Dns endpoints (dotnet#54071) [wasm] Build static components; include hot_reload in runtime (dotnet#54568) [wasm][debugger] Reuse debugger-agent on wasm debugger (dotnet#52300) Put Crossgen2 in sync with dotnet#54235 (dotnet#54438) ...

radekdoulik added 5 commits June 2, 2021 16:18

Revert "Add ActiveIssue to the MemoryMappedFiles tests"

8be39f5

This reverts commit ec1ae53.

Revert "Add ActiveIssue attr to the FileSystem tests"

eb2f954

This reverts commit 356b3ff.

Bump emscripten version to 2.0.23

3eb7dbe

Use newer docker images with 2.0.23

03abd7f

Update docs

9fdb05d

dotnet-issue-labeler bot added the area-System.IO label Jun 2, 2021

radekdoulik added area-Build-mono arch-wasm WebAssembly architecture and removed area-System.IO labels Jun 2, 2021

radekdoulik added 3 commits June 9, 2021 11:23

Use 2.0.23 emscripten nuget packages

bf29544

Revert "Revert "Add ActiveIssue attr to the FileSystem tests""

739af16

This reverts commit eb2f954. The fix is not present in 2.0.23

Revert "Revert "Add ActiveIssue to the MemoryMappedFiles tests""

c0e0175

This reverts commit 8be39f5. The fix is not present in 2.0.23

radekdoulik marked this pull request as ready for review June 9, 2021 14:43

radekdoulik requested a review from marek-safar as a code owner June 9, 2021 14:43

radekdoulik requested review from lewing, radical, pavelsavara and akoeplinger June 9, 2021 14:46

radekdoulik requested a review from kg June 9, 2021 14:52

radekdoulik added 3 commits June 9, 2021 17:10

Increase timeout for AOT tests

ff8848a

Add description of emscripten bump to README

02b5ee2

Merge remote-tracking branch 'remotes/origin/main' into pr-wasm-emscr…

0fef9d5

…ipten-2-0-23-bump

radekdoulik added a commit to radekdoulik/dotnet-buildtools-prereqs-docker that referenced this pull request Jun 10, 2021

Downgrade emscripten

3b93e1d

To be able to use this image before dotnet/runtime#53603 is finished

Reduce platform matrix

fec0e55

Lets try one more build with doubled timeout

67dd775

radekdoulik added 2 commits June 18, 2021 11:00

Merge remote-tracking branch 'remotes/origin/main' into pr-wasm-emscr…

38041f4

…ipten-2-0-23-bump

Revert "Lets try one more build with doubled timeout"

e9dfd01

This reverts commit 67dd775.

Try -Wl,-O0 on CI

de4037e

To be sure it behaves the same as in local build

Use -Wl,-lto-O0 do lower link time optimization

7c1dff2

It looks like it reduces the memory load a lot

radekdoulik added 6 commits June 23, 2021 10:42

Set EmccLinkOptimizationFlag for AOT tests

b910ec6

And reset the default value

Escape commas

db0cd09

Revert "Reduce platform matrix"

91e4670

This reverts commit fec0e55.

Remove resource info retrieval

3c57b20

Merge remote-tracking branch 'remotes/origin/main' into pr-wasm-emscr…

1f9036a

…ipten-2-0-23-bump

Merge remote-tracking branch 'remotes/origin/main' into pr-wasm-emscr…

60b00e0

…ipten-2-0-23-bump

lewing approved these changes Jun 24, 2021

View reviewed changes

Bump emsdk versions

49822e6

radekdoulik merged commit 055a38a into dotnet:main Jun 24, 2021

This was referenced Jun 28, 2021

[wasm][aot] Link step failing for Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests - wasm-opt getting killed #51961

Closed

[wasm][aot] Enable System.Text.Json tests for Wasm AOT #54833

Merged

ghost locked as resolved and limited conversation to collaborators Jul 24, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[wasm] Bump emscripten to 2.0.23 #53603

[wasm] Bump emscripten to 2.0.23 #53603

radekdoulik commented Jun 2, 2021

ghost commented Jun 2, 2021

ghost commented Jun 2, 2021

radekdoulik commented Jun 9, 2021

radekdoulik commented Jun 9, 2021

radical commented Jun 9, 2021 •

edited

Loading

radical commented Jun 9, 2021

radical commented Jun 10, 2021

radekdoulik commented Jun 17, 2021 •

edited

Loading

radekdoulik commented Jun 17, 2021

radical commented Jun 17, 2021 •

edited by radekdoulik

Loading

radekdoulik commented Jun 17, 2021

lewing commented Jun 17, 2021

radekdoulik commented Jun 18, 2021

radekdoulik commented Jun 18, 2021 •

edited

Loading

radekdoulik commented Jun 21, 2021

lewing commented Jun 23, 2021

lewing commented Jun 24, 2021

radekdoulik commented Jun 24, 2021

[wasm] Bump emscripten to 2.0.23 #53603

[wasm] Bump emscripten to 2.0.23 #53603

Conversation

radekdoulik commented Jun 2, 2021

ghost commented Jun 2, 2021

ghost commented Jun 2, 2021

radekdoulik commented Jun 9, 2021

radekdoulik commented Jun 9, 2021

radical commented Jun 9, 2021 • edited Loading

radical commented Jun 9, 2021

radical commented Jun 10, 2021

radekdoulik commented Jun 17, 2021 • edited Loading

radekdoulik commented Jun 17, 2021

radical commented Jun 17, 2021 • edited by radekdoulik Loading

radekdoulik commented Jun 17, 2021

lewing commented Jun 17, 2021

radekdoulik commented Jun 18, 2021

radekdoulik commented Jun 18, 2021 • edited Loading

radekdoulik commented Jun 21, 2021

lewing commented Jun 23, 2021

lewing commented Jun 24, 2021

radekdoulik commented Jun 24, 2021

radical commented Jun 9, 2021 •

edited

Loading

radekdoulik commented Jun 17, 2021 •

edited

Loading

radical commented Jun 17, 2021 •

edited by radekdoulik

Loading

radekdoulik commented Jun 18, 2021 •

edited

Loading