[Wasm][AOT] The AOT compiler fails silently when running on Windows #57141

jeromelaban · 2021-08-10T16:00:18Z

Description

The AOT compiler stops working at an unknown location, without any error message.

Here's how to reproduce it:

cd extracted_archive
runtimes\browser-wasm/native/cross/browser-wasm/mono-aot-cross.exe --debug -v  -O=gsharedvt --aot=dedup-skip,llvmonly,asmonly,no-opt,static,direct-icalls,deterministic,nodebug,llvm-path="[...]\emsdk-2.0.23\emsdk/upstream/bin",interp,,depfile=./linker-out/Uno.UI.FluentTheme.dll.depfile,llvm-outfile=./Uno.UI.FluentTheme.dll.bc.tmp .\linker-out\Uno.UI.FluentTheme.dll

With -v, the last two output lines are:

rySingleton__2__Get_2_5287b__959_0_Windows_UI_Xaml_Media_SolidColorBrush
converting llvm method void Uno.UI.FluentTheme.GlobalStaticResources/ResourceDictionarySingleton__2:<Get_2_5288>b__960_0 (Windows.UI.Xaml.Media.SolidColorBrush)
void Uno.UI.FluentTheme.GlobalStaticResources/ResourceDictionarySingleton__2:<Get_2_5288>b__960_0 (Windows.UI.Xaml.Media.SolidColorBrush) emitted as Uno_UI_FluentTheme_Uno_UI_FluentTheme_GlobalStaticResources_ResourceDictionarySingleton__2__Get_2_5288b__960_0_Windows_UI_Xaml_Media_SolidColorBrush
converting llvm method object Uno.UI.FluentTheme.GlobalStaticResources/ResourceDictionarySingleton__2:<get_themeresources_6ae50ee5d890e499f2c9aa02298c937a_ResourceDictionary>b__963_0 (object)

The same command finishes properly when running under linux.

Configuration

6696065

Regression?

Unsure

Other information

The text was updated successfully, but these errors were encountered:

dotnet-issue-labeler · 2021-08-10T16:26:19Z

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

ghost · 2021-08-10T19:30:46Z

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

Description

The AOT compiler stops working at an unknown location, without any error message.

aot-issue-assets.zip

Here's how to reproduce it:

cd extracted_archive
runtimes\browser-wasm/native/cross/browser-wasm/mono-aot-cross.exe --debug -v  -O=gsharedvt --aot=dedup-skip,llvmonly,asmonly,no-opt,static,direct-icalls,deterministic,nodebug,llvm-path="[...]\emsdk-2.0.23\emsdk/upstream/bin",interp,,depfile=./linker-out/Uno.UI.FluentTheme.dll.depfile,llvm-outfile=./Uno.UI.FluentTheme.dll.bc.tmp .\linker-out\Uno.UI.FluentTheme.dll

With -v, the last two output lines are:

rySingleton__2__Get_2_5287b__959_0_Windows_UI_Xaml_Media_SolidColorBrush
converting llvm method void Uno.UI.FluentTheme.GlobalStaticResources/ResourceDictionarySingleton__2:<Get_2_5288>b__960_0 (Windows.UI.Xaml.Media.SolidColorBrush)
void Uno.UI.FluentTheme.GlobalStaticResources/ResourceDictionarySingleton__2:<Get_2_5288>b__960_0 (Windows.UI.Xaml.Media.SolidColorBrush) emitted as Uno_UI_FluentTheme_Uno_UI_FluentTheme_GlobalStaticResources_ResourceDictionarySingleton__2__Get_2_5288b__960_0_Windows_UI_Xaml_Media_SolidColorBrush
converting llvm method object Uno.UI.FluentTheme.GlobalStaticResources/ResourceDictionarySingleton__2:<get_themeresources_6ae50ee5d890e499f2c9aa02298c937a_ResourceDictionary>b__963_0 (object)

The same command finishes properly when running under linux.

Configuration

6696065

Regression?

Unsure

Other information

Author:	jeromelaban
Assignees:	-
Labels:	`arch-wasm`, `untriaged`, `area-Codegen-AOT-mono`
Milestone:	-

radical · 2021-08-10T19:31:09Z

cc @vargaz @radekdoulik

radekdoulik · 2021-08-11T14:15:54Z

I have tried to run it with valgrind on linux. No errors, just leaks.

MONO_PATH=linker-out valgrind .../artifacts/bin/mono/Browser.wasm.Release/cross/browser-wasm/mono-aot-cross -O=gsharedvt --aot=dedup-skip,llvmonly,asm
only,no-opt,static,direct-icalls,deterministic,nodebug,llvm-path=".../src/mono/wasm/emsdk/upstream/bin",interp,,depfile=./linker-out/Uno.UI.FluentTheme.dll.depfile,llvm-outfile=./Uno.U
I.FluentTheme.dll.bc.tmp ./linker-out/Uno.UI.FluentTheme.dll
==47864== Memcheck, a memory error detector
==47864== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==47864== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==47864== Command: /home/rodo/wa-main/artifacts/bin/mono/Browser.wasm.Release/cross/browser-wasm/mono-aot-cross -O=gsharedvt --aot=dedup-skip,llvmonly,asmonly,no-opt,static,direct-icalls,deterministic,nodebug,llvm-path=/home/rodo/wa-main/src/mono/wasm/emsdk/upstream/bin,interp,,depfile=./linker-out/Uno.UI.FluentTheme.dll.depfile,llvm-outfile=./Uno.UI.FluentTheme.dll.bc.tmp ./linker-out/Uno.UI.FluentTheme.dll
==47864==
Mono Ahead of Time compiler - compiling assembly /mnt/c/Users/rodo/git/repro/linker-out/Uno.UI.FluentTheme.dll
Compiled: 12703/12703
Output file: '/tmp/mono_aot_X7RNFs'.
Linking symbol: 'mono_aot_module_Uno_UI_FluentTheme_info'.
LLVM output file: './Uno.UI.FluentTheme.dll.bc.tmp'.
JIT time: 145513 ms, Generation time: 34348 ms, Assembly+Link time: 8 ms.
==47864==
==47864== HEAP SUMMARY:
==47864==     in use at exit: 878,549,022 bytes in 8,425,521 blocks
==47864==   total heap usage: 15,594,719 allocs, 7,169,198 frees, 7,687,254,708 bytes allocated
==47864==
==47864== LEAK SUMMARY:
==47864==    definitely lost: 16,538,022 bytes in 267,933 blocks
==47864==    indirectly lost: 38,751 bytes in 922 blocks
==47864==      possibly lost: 352,421,007 bytes in 3,015,908 blocks
==47864==    still reachable: 509,551,242 bytes in 5,140,758 blocks
==47864==                       of which reachable via heuristic:
==47864==                         stdstring          : 270 bytes in 6 blocks
==47864==                         newarray           : 46 bytes in 2 blocks
==47864==         suppressed: 0 bytes in 0 blocks
==47864== Rerun with --leak-check=full to see details of leaked memory
==47864==
==47864== For lists of detected and suppressed errors, rerun with: -s
==47864== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

jeromelaban · 2021-08-11T14:25:52Z

I forgot to add to this issue, the debugger caught this, when attached:

Unhandled exception at 0x00007FF7D8887021 in mono-aot-cross.exe: 0xC00000FD: Stack overflow (parameters: 0x0000000000000001, 0x00000049D8C03FF8).

I don't have symbols to provide though, as the build does not seem to provide them.

radekdoulik · 2021-08-13T18:43:25Z

I have opened #57320 for the debug symbols on windows

lewing · 2021-10-07T01:21:26Z

@imhameed any thoughts on how to proceed here?

Context: dotnet#57320 Setting `CMAKE_MSVC_RUNTIME_LIBRARY` to `MultiThreaded` in `Debug` configuration allows us to link with llvm built in `Release` config without symbols. That results in `-MT` option used for compilation and getting rid of errors like: error LNK2038: mismatch detected for 'RuntimeLibrary': value 'MT_StaticRelease' doesn't match value 'MTd_StaticDebug' in monosgen-2.0.lib(...) Also sets `_ITERATOR_DEBUG_LEVEL` to get rid of errors like: error LNK2038: mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '0' doesn't match value '2' in monosgen-2.0.lib(...) And finally removing _DEBUG define. Without removing this define the `-MT` option doesn't have an effect on the runtime selection and we still end up with `MTd_StaticDebug`. I am not sure about all implications of removing `_DEBUG` define. It works fine so far and produces executables with debug information, like `mono-aot-cross.exe` with `mono-aot-cross.pdb`. That allows us to debug bugs specific to windows, like dotnet#57141

radekdoulik · 2021-10-11T19:39:11Z

I was able to debug it on Windows. It crashes like this:

Unhandled exception at 0x00007FFE7192AA7F (KernelBase.dll) in mono-aot-cross.exe: 0xC00000FD: Stack overflow (parameters: 0x0000000000000001, 0x000000732B0C3FD8).

 	KernelBase.dll!00007ffe7192aa7f()	Unknown
 	mono-aot-cross.exe!write_text_ansi_nolock(const int fh, const char * const buffer, const unsigned int buffer_size) Line 415	C++
 	mono-aot-cross.exe!_write_nolock(int fh, const void * buffer, unsigned int buffer_size) Line 679	C++
 	mono-aot-cross.exe!_write(int fh, const void * buffer, unsigned int size) Line 64	C++
 	mono-aot-cross.exe!write_buffer_nolock<char>(const char c, const __crt_stdio_stream stream) Line 74	C++
 	[Inline Frame] mono-aot-cross.exe!common_flush_and_write_nolock(const int) Line 174	C++
 	mono-aot-cross.exe!__acrt_stdio_flush_and_write_narrow_nolock(int c, _iobuf * stream) Line 190	C++
 	[Inline Frame] mono-aot-cross.exe!__crt_char_traits<char>::puttc_nolock(const char &) Line 109	C++
 	[Inline Frame] mono-aot-cross.exe!__crt_stdio_output::stream_output_adapter<char>::write_character_without_count_update(const char c) Line 166	C++
 	mono-aot-cross.exe!__crt_stdio_output::output_adapter_common<char,__crt_stdio_output::stream_output_adapter<char>>::write_string_impl(const char * const string, const int length, int * const count_written, __crt_deferred_errno_cache & status) Line 86	C++
 	[Inline Frame] mono-aot-cross.exe!__crt_stdio_output::stream_output_adapter<char>::write_string(const char * const string, const int length, int * const count_written, __crt_deferred_errno_cache &) Line 182	C++
 	mono-aot-cross.exe!__crt_stdio_output::output_processor<char,__crt_stdio_output::stream_output_adapter<char>,__crt_stdio_output::standard_base<char,__crt_stdio_output::stream_output_adapter<char>>>::write_stored_string_tchar(char __formal) Line 2707	C++
 	mono-aot-cross.exe!__crt_stdio_output::output_processor<char,__crt_stdio_output::stream_output_adapter<char>,__crt_stdio_output::standard_base<char,__crt_stdio_output::stream_output_adapter<char>>>::state_case_type() Line 2150	C++
 	mono-aot-cross.exe!__crt_stdio_output::output_processor<char,__crt_stdio_output::stream_output_adapter<char>,__crt_stdio_output::standard_base<char,__crt_stdio_output::stream_output_adapter<char>>>::process() Line 1699	C++
 	mono-aot-cross.exe!common_vfprintf::__l2::<lambda>() Line 49	C++
 	mono-aot-cross.exe!__crt_seh_guarded_call<int>::operator()<void <lambda>(void),int <lambda>(void) &,void <lambda>(void)>(__acrt_lock_stream_and_call::__l2::void <lambda>(void) && setup, common_vfprintf::__l2::int <lambda>(void) & action, __acrt_lock_stream_and_call::__l2::void <lambda>(void) && cleanup) Line 204	C++
 	[Inline Frame] mono-aot-cross.exe!__acrt_lock_stream_and_call(_iobuf * const) Line 252	C++
 	[Inline Frame] mono-aot-cross.exe!common_vfprintf(const unsigned __int64 options, _iobuf * const stream, const char * const format, __crt_locale_pointers * const locale, char * const arglist) Line 37	C++
 	mono-aot-cross.exe!__stdio_common_vfprintf(unsigned __int64 options, _iobuf * stream, const char * format, __crt_locale_pointers * locale, char * arglist) Line 62	C++
 	[External Code]	
>	mono-aot-cross.exe!mono_print_ins_index(int i, MonoInst * ins) Line 429	C
 	mono-aot-cross.exe!mono_print_ins(MonoInst * ins) Line 771	C
 	mono-aot-cross.exe!mono_ssa_rename_vars(MonoCompile * cfg, int max_vars, MonoBasicBlock * bb, int * originals_used, MonoInst * * stack, unsigned int * lvreg_stack, int * lvreg_defined, RenameInfo * stack_history, int stack_history_size) Line 188	C
...
 	mono-aot-cross.exe!mono_ssa_rename_vars(MonoCompile * cfg, int max_vars, MonoBasicBlock * bb, int * originals_used, MonoInst * * stack, unsigned int * lvreg_stack, int * lvreg_defined, RenameInfo * stack_history, int stack_history_size) Line 322	C
 	mono-aot-cross.exe!mono_ssa_compute(MonoCompile * cfg) Line 477	C
 	mono-aot-cross.exe!mini_method_compile(_MonoMethod * method, unsigned int opts, JitFlags flags, int parts, int aot_method_index) Line 3671	C
 	mono-aot-cross.exe!compile_method(MonoAotCompile * acfg, _MonoMethod * method) Line 8945	C
 	mono-aot-cross.exe!compile_methods(MonoAotCompile * acfg) Line 12421	C
 	mono-aot-cross.exe!mono_compile_assembly(_MonoAssembly * ass, unsigned int opts, const char * aot_options, void * * * global_aot_state) Line 14193	C
 	mono-aot-cross.exe!main_thread_handler(void * user_data) Line 1429	C
 	mono-aot-cross.exe!mono_main(int argc, char * * argv) Line 2678	C
 	mono-aot-cross.exe!mono_main_with_options(int argc, char * * argv) Line 55	C
 	mono-aot-cross.exe!main(int _argc, char * * _argv) Line 380	C
 	[External Code]

The mono_ssa_rename_vars is called recursively and can go quite deep. Like 2700+ recursive calls before it crashed in this case.

I have checked the output with #define DEBUG_SSA 1 and it looks like it works the same way as on linux.

Then I checked the stack sizes and stack size set for our windows executable is 1M. While on linux the default is 8M.

I am going to set 8M stack size on windows too. OTOH it might be good idea to remove the recursion of mono_ssa_rename_vars on the stack, so I will open an issue for that too as long term improvement.

Fix dotnet#57141 in windows build. Set the stack size (reserve) to 8MB, which is usually a default on linux. This way we should get similar behavior on windows.

radekdoulik · 2021-10-11T20:13:33Z

Above PR fixes it for windows build. I think we might need to set it also for linux build, where we also produce windows executables, IIRC. I will check that.

radekdoulik · 2021-10-12T17:04:42Z

The windows executables for the nuget are built on windows, so #60265 should fix this issue.

Fix #57141 in windows build. Set the stack size (reserve) to 8MB, which is usually a default on linux. This way we should get similar behavior on windows. > dumpbin.exe /headers artifacts\bin\mono\Browser.wasm.Debug\cross\browser-wasm\mono-aot-cross.exe|Select-String -Pattern stack 800000 size of stack reserve 1000 size of stack commit

Fix #57141 in windows build. Set the stack size (reserve) to 8MB, which is usually a default on linux. This way we should get similar behavior on windows.

Fix #57141 in windows build. Set the stack size (reserve) to 8MB, which is usually a default on linux. This way we should get similar behavior on windows. Co-authored-by: Radek Doulik <[email protected]>

dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Aug 10, 2021

jeromelaban mentioned this issue Aug 10, 2021

[WebAssembly] .NET 5 support unoplatform/uno#4613

Closed

radical added arch-wasm WebAssembly architecture area-Codegen-AOT-mono labels Aug 10, 2021

lewing modified the milestones: 5.0.x, 6.0.0 Aug 10, 2021

lewing removed the untriaged New issue has not been triaged by the area owner label Aug 10, 2021

lewing assigned radekdoulik Aug 10, 2021

radekdoulik modified the milestones: 6.0.0, 7.0.0 Aug 13, 2021

jeromelaban mentioned this issue Sep 17, 2021

[WASM][Windows] Uno.UI.FluentTheme.v2 assembly fails to AOT under Windows unoplatform/uno#6152

Closed

24 tasks

radekdoulik mentioned this issue Oct 11, 2021

[mono/win] Build native runtime as Debug again #60263

Closed

radekdoulik mentioned this issue Oct 11, 2021

[mono/win] Increase stack size (reserve) to 8MB #60265

Merged

ghost added the in-pr There is an active PR which will close this issue when it is merged label Oct 11, 2021

radekdoulik mentioned this issue Oct 11, 2021

[mono] Improvement: avoid deep recursion in mono_ssa_rename_vars #60266

Closed

radekdoulik closed this as completed in #60265 Oct 15, 2021

ghost removed the in-pr There is an active PR which will close this issue when it is merged label Oct 15, 2021

github-actions bot pushed a commit that referenced this issue Oct 18, 2021

[mono/win] Increase stack size (reserve) to 8MB

05f2892

Fix #57141 in windows build. Set the stack size (reserve) to 8MB, which is usually a default on linux. This way we should get similar behavior on windows.

radical mentioned this issue Nov 1, 2021

[Wasm][AOT] Precompilation fails using Syncfusion.Blazor.dll #61053

Closed

ghost locked as resolved and limited conversation to collaborators Nov 14, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Wasm][AOT] The AOT compiler fails silently when running on Windows #57141

[Wasm][AOT] The AOT compiler fails silently when running on Windows #57141

jeromelaban commented Aug 10, 2021

dotnet-issue-labeler bot commented Aug 10, 2021

ghost commented Aug 10, 2021

Description

Configuration

Regression?

Other information

radical commented Aug 10, 2021

radekdoulik commented Aug 11, 2021

jeromelaban commented Aug 11, 2021

radekdoulik commented Aug 13, 2021

lewing commented Oct 7, 2021

radekdoulik commented Oct 11, 2021

radekdoulik commented Oct 11, 2021

radekdoulik commented Oct 12, 2021

[Wasm][AOT] The AOT compiler fails silently when running on Windows #57141

[Wasm][AOT] The AOT compiler fails silently when running on Windows #57141

Comments

jeromelaban commented Aug 10, 2021

Description

Configuration

Regression?

Other information

dotnet-issue-labeler bot commented Aug 10, 2021

ghost commented Aug 10, 2021

Description

Configuration

Regression?

Other information

radical commented Aug 10, 2021

radekdoulik commented Aug 11, 2021

jeromelaban commented Aug 11, 2021

radekdoulik commented Aug 13, 2021

lewing commented Oct 7, 2021

radekdoulik commented Oct 11, 2021

radekdoulik commented Oct 11, 2021

radekdoulik commented Oct 12, 2021