Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Methodical_others test JIT/Methodical/Coverage/copy_prop_byref_to_native_int crashing #69832

Open
ericstj opened this issue May 25, 2022 · 8 comments
Assignees
Labels
area-Codegen-AOT-mono disabled-test The test is disabled in source code against the issue
Milestone

Comments

@ericstj
Copy link
Member

ericstj commented May 25, 2022

Failures 7/5-8/5 (incl. PRs) based on Runfo in last 30 days:

Day Run Failure
8/4 PR #73174 - 1923945 Crash dump
8/4 Rolling run 1923908 Crash dump
8/4 PR #73073 - 1922939 fail: Application has finished with exit code 1 but 0 was expected - likely infra hiccup
8/3 PR #73265 - 1921135 Crash dump
8/2 PR #72443 - 1918306 Crash dump
8/2 PR #72851 - 1917737 Crash dump
8/2 PR #72143 - 1917321 Crash dump
7/30 PR #72934 - 1913487 No clear error - likely infra hiccup
7/29 PR #49576 - 1912700 2x Segmentation fault - most likely caused by the PR
7/29 PR #73009 - 1912477 Crash dump
7/29 PR #49576 - 1912185 2x Segmentation fault - most likely caused by the PR
7/29 PR #49576 - 1910538 2x Segmentation fault - most likely caused by the PR
7/28 PR #67272 - 1909374 Unhandled Exception
7/28 PR #72762 - 1908782 Unhandled Exception
7/28 PR #69542 - 1908613 Crash dump
7/26 PR #71661 - 1905330 Unhandled Exception
7/26 Rolling run 1904930 Crash dump
7/24 PR #72448 - 1900699 Crash dump
7/24 PR #72732 - 1900156 Crash dump
7/22 PR #72672 - 1897479 Unhandled Exception
7/20 PR #72482 - 1893721 Crash dump
7/20 PR #72529 - 1892945 Crash dump
7/19 PR #72394 - 1891289 Crash dump
7/19 PR #72452 - 1891146 Unhandled Exception
7/16 PR #71827 - 1885040 Crash dump
7/16 PR #72275 - 1885005 Crash dump
7/15 PR #72021 - 1882959 Crash dump
4/5-7/14 Logs not available

Unhandled Exception

Example: Console log

16:13:46.844 Running test: JIT/Methodical/Coverage/copy_prop_byref_to_native_int/copy_prop_byref_to_native_int.cmd

Unhandled Exception:
System.NullReferenceException: Object reference not set to an instance of an object
   at System.Net.Sockets.SocketAsyncContext.OperationQueue`1.ProcessAsyncOperation(TOperation_REF op)
   at System.Net.Sockets.SocketAsyncContext.ProcessAsyncReadOperation(ReadOperation op)
   at System.Net.Sockets.SocketAsyncContext.ReadOperation.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Net.Sockets.SocketAsyncContext.AsyncOperation.Process()
   at System.Net.Sockets.SocketAsyncContext.HandleEvents(SocketEvents events)
   at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()
[ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: Object reference not set to an instance of an object
   at System.Net.Sockets.SocketAsyncContext.OperationQueue`1.ProcessAsyncOperation(TOperation_REF op)
   at System.Net.Sockets.SocketAsyncContext.ProcessAsyncReadOperation(ReadOperation op)
   at System.Net.Sockets.SocketAsyncContext.ReadOperation.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Net.Sockets.SocketAsyncContext.AsyncOperation.Process()
   at System.Net.Sockets.SocketAsyncContext.HandleEvents(SocketEvents events)
   at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()

Crash dump

Observed in https://dev.azure.com/dnceng/public/_build/results?buildId=1789473

Console output: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-69813-merge-212c7ffc453c4eb2a8/Methodical_others/1/console.b7c2b95a.log?helixlogtype=result

Relevant log snippet:

22:49:29.737 Running test: JIT/Methodical/Coverage/copy_prop_byref_to_native_int/copy_prop_byref_to_native_int.cmd

=================================================================
	Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
	Native stacktrace:
=================================================================
	0x7fa4c2a98722 - Unknown
	0x7fa4c2a39a2e - Unknown
	0x7fa4c2996d71 - Unknown
	0x7fa4c4e52980 - Unknown
	0x7fa4c29e5494 - Unknown
	0x7fa4bed63df3 - Unknown

=================================================================
	External Debugger Dump:
=================================================================
[New LWP 26773]
[New LWP 26774]
[New LWP 26775]
[New LWP 26776]
[New LWP 26781]
[New LWP 26785]
[New LWP 26786]
[New LWP 26787]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
warning: the debug information found in "/datadisks/disk1/work/B4770A02/p/libcoreclr.so.dbg" does not match "/datadisks/disk1/work/B4770A02/p/libcoreclr.so" (CRC mismatch).

0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x562701fd5188) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88	../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.
  Id   Target Id         Frame 
* 1    Thread 0x7fa4c5479740 (LWP 26772) "corerun" 0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x562701fd5188) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  2    Thread 0x7fa4c13ff700 (LWP 26773) "SGen worker" 0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa4c40d95c8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  3    Thread 0x7fa4be6cc700 (LWP 26774) "corerun" 0x00007fa4c4224bb9 in __GI___poll (fds=0x7fa4b8002d50, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
  4    Thread 0x7fa4be4cb700 (LWP 26775) "Finalizer" 0x00007fa4c4e507c6 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7fa4c40ca530) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  5    Thread 0x7fa4b63fe700 (LWP 26776) "corerun" 0x00007fa4c4e51474 in __libc_read (fd=7, buf=0x7fa4b63fdee7, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:27
  6    Thread 0x7fa4a8194700 (LWP 26781) ".NET Sockets" 0x00007fa4c4231947 in epoll_wait (epfd=13, events=0x5627021f5630, maxevents=32, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
  7    Thread 0x7fa4a7d8d700 (LWP 26785) ".NET ThreadPool" 0x00007fa4c4e4dfb9 in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x7fa4a7d8c6f0, expected=0, futex_word=0x7fa4a7d8c828) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
  8    Thread 0x7fa4a7b8c700 (LWP 26786) ".NET ThreadPool" 0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa49c002c90) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  9    Thread 0x7fa4a7b4b700 (LWP 26787) ".NET ThreadPool" 0x00007fa4c4e5232a in __waitpid (pid=26788, stat_loc=0x7fa4a7b491c0, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30

Thread 9 (Thread 0x7fa4a7b4b700 (LWP 26787)):
#0  0x00007fa4c4e5232a in __waitpid (pid=26788, stat_loc=0x7fa4a7b491c0, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1  0x00007fa4c2a98867 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#2  0x00007fa4c2a39a2e in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#3  0x00007fa4c2996d71 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4  <signal handler called>
#5  0x00007fa4c29e5494 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#6  0x00007fa4bed63df3 in System_Runtime_CompilerServices_AsyncTaskMethodBuilder_1_TResult_INT_SetExistingTaskResult_System_Threading_Tasks_Task_1_TResult_INT_TResult_INT () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#7  0x00005627026707f8 in ?? ()
#8  0x00007fa4c143f198 in ?? ()
#9  0x00007fa4c143f620 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7fa4a7b8c700 (LWP 26786)):
#0  0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa49c002c90) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fa49c002c40, cond=0x7fa49c002c68) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fa49c002c68, mutex=0x7fa49c002c40) at pthread_cond_wait.c:655
#3  0x00007fa4bcf3d1fc in SystemNative_LowLevelMonitor_Wait (monitor=0x7fa49c002c40) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155
#4  0x00007fa4bf0ee1e6 in wrapper_managed_to_native_Interop_Sys_LowLevelMonitor_Wait_intptr () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#5  0x00007fa4bea252df in System_Threading_WaitSubsystem_ThreadWaitInfo_Wait_int_bool_bool () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#6  0x00007fa4bea2688e in System_Threading_WaitSubsystem_WaitableObject_Wait_Locked_System_Threading_WaitSubsystem_ThreadWaitInfo_int_bool_bool () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#7  0x00007fa4bea265fb in System_Threading_WaitSubsystem_WaitableObject_Wait_System_Threading_WaitSubsystem_ThreadWaitInfo_int_bool_bool () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#8  0x00007fa4a7b8bbb0 in ?? ()
#9  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7fa4a7d8d700 (LWP 26785)):
#0  0x00007fa4c4e4dfb9 in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x7fa4a7d8c6f0, expected=0, futex_word=0x7fa4a7d8c828) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#1  __pthread_cond_wait_common (abstime=0x7fa4a7d8c7a8, mutex=0x7fa4a000b800, cond=0x7fa4a7d8c800) at pthread_cond_wait.c:533
#2  __pthread_cond_timedwait (cond=0x7fa4a7d8c800, mutex=0x7fa4a000b800, abstime=0x7fa4a7d8c7a8) at pthread_cond_wait.c:667
#3  0x00007fa4c292a474 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4  0x00007fa4c292fb74 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#5  0x00007fa4bef1e863 in wrapper_managed_to_native_System_Threading_LowLevelLifoSemaphore_TimedWaitInternal_intptr_int () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#6  0x00007fa4be9fb9cd in System_Threading_LowLevelLifoSemaphore_WaitForSignal_int () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#7  0x00007fa4c1441d08 in ?? ()
#8  0x0000000000000002 in ?? ()
#9  0xfffeffffffffffff in ?? ()
#10 0x0000000000004e20 in ?? ()
#11 0x0000000000000046 in ?? ()
#12 0x00007fa4c410ee30 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#13 0x00007fa4be9fb6db in System_Threading_LowLevelLifoSemaphore_Wait_int_bool () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#14 0x00007fa4c1441998 in ?? ()
#15 0x00007fa4a7d8c9d0 in ?? ()
#16 0x00007fa4a7d8ca08 in ?? ()
#17 0x00007fa4a7d8cdc8 in ?? ()
#18 0x00007fa4c410ee30 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#19 0x00007fa4a7d8ca80 in ?? ()
#20 0x00007fa4bea20b1d in System_Threading_PortableThreadPool_WorkerThread_WorkerThreadStart () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 6 (Thread 0x7fa4a8194700 (LWP 26781)):
#0  0x00007fa4c4231947 in epoll_wait (epfd=13, events=0x5627021f5630, maxevents=32, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1  0x00007fa4bcf37f9e in WaitForSocketEventsInner (port=13, buffer=0x5627021f5630, count=0x7fa4a81939d4) at /__w/1/s/src/native/libs/System.Native/pal_networking.c:2717
#2  0x00007fa4bcf37ebf in SystemNative_WaitForSocketEvents (port=13, buffer=0x5627021f5630, count=0x7fa4a81939d4) at /__w/1/s/src/native/libs/System.Native/pal_networking.c:3017
#3  0x00007fa4a8eef485 in wrapper_managed_to_native_Interop_Sys_WaitForSocketEvents_intptr_Interop_Sys_SocketEvent__int_ (param0=13, param1=94725539321392, param2=140345171589588) at /_/src/libraries/System.Collections.Concurrent/src/System/Collections/Concurrent/ConcurrentDictionary.cs:2264
#4  0x00007fa4a8e30f1c in System_Net_Sockets_SocketAsyncEngine_EventLoop () at /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs:184
#5  0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7fa4b63fe700 (LWP 26776)):
#0  0x00007fa4c4e51474 in __libc_read (fd=7, buf=0x7fa4b63fdee7, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:27
#1  0x00007fa4bcf3c60e in SignalHandlerLoop (arg=0x562702331280) at /__w/1/s/src/native/libs/System.Native/pal_signal.c:323
#2  0x00007fa4c4e476db in start_thread (arg=0x7fa4b63fe700) at pthread_create.c:463
#3  0x00007fa4c423161f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7fa4be4cb700 (LWP 26775)):
#0  0x00007fa4c4e507c6 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7fa4c40ca530) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0x7fa4c40ca530, abstime=0x0) at sem_waitcommon.c:111
#2  0x00007fa4c4e508b8 in __new_sem_wait_slow (sem=0x7fa4c40ca530, abstime=0x0) at sem_waitcommon.c:181
#3  0x00007fa4c29011c6 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4  0x00007fa4c28da2ba in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#5  0x00007fa4c28da149 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#6  0x00007fa4c4e476db in start_thread (arg=0x7fa4be4cb700) at pthread_create.c:463
#7  0x00007fa4c423161f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7fa4be6cc700 (LWP 26774)):
#0  0x00007fa4c4224bb9 in __GI___poll (fds=0x7fa4b8002d50, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007fa4c2b5f84a in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#2  0x00007fa4c2b5cd45 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#3  0x00007fa4c2b5b5a9 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4  0x00007fa4c2b5ec61 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#5  0x00007fa4c4e476db in start_thread (arg=0x7fa4be6cc700) at pthread_create.c:463
#6  0x00007fa4c423161f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7fa4c13ff700 (LWP 26773)):
#0  0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa4c40d95c8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fa4c40d9578, cond=0x7fa4c40d95a0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fa4c40d95a0, mutex=0x7fa4c40d9578) at pthread_cond_wait.c:655
#3  0x00007fa4c297ee73 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4  0x00007fa4c4e476db in start_thread (arg=0x7fa4c13ff700) at pthread_create.c:463
#5  0x00007fa4c423161f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7fa4c5479740 (LWP 26772)):
#0  0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x562701fd5188) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x562701fd5138, cond=0x562701fd5160) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x562701fd5160, mutex=0x562701fd5138) at pthread_cond_wait.c:655
#3  0x00007fa4c292a490 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4  0x00007fa4c28df52a in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#5  0x00007fa4c28df3a8 in mono_w32handle_wait_one () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#6  0x00007fa4c2902bb4 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#7  0x00007fa4c2892b93 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#8  0x00007fa4bef1d79d in wrapper_managed_to_native_System_Threading_Monitor_Monitor_wait_object_int_bool () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#9  0x00007fa4be9f5986 in System_Threading_Monitor_ObjWait_int_object () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#10 0x0000000000000001 in ?? ()
#11 0x00007fa4c1440a30 in ?? ()
#12 0x0000000000000000 in ?? ()

=================================================================
	Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x7fa4c29e5494):0x7fa4c29e5484  57 01 48 8b 40 38 48 8b 80 98 00 00 00 48 8b 38  [email protected]
0x7fa4c29e5494  41 8b 46 10 8d 34 c5 08 00 00 00 e8 6c 91 f1 ff  A.F..4......l...
0x7fa4c29e54a4  48 89 04 24 41 83 7e 10 00 0f 8e 80 00 00 00 48  H..$A.~........H
0x7fa4c29e54b4  63 c5 48 89 44 24 10 f7 d8 48 89 44 24 08 48 8d  c.H.D$...H.D$.H.

=================================================================
	Managed Stacktrace:
=================================================================
=================================================================
JIT/Methodical/Methodical_others/Methodical_others.sh: line 382: 26772 Aborted                 (core dumped) $LAUNCHER $ExePath "${CLRTestExecutionArguments[@]}"

Crashdump: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-69813-merge-212c7ffc453c4eb2a8/Methodical_others/1/core.1000.26772?helixlogtype=result

Runfo Tracking Issue: methodical_others work item

Build Definition Kind Run Name

Build Result Summary

Day Hit Count Week Hit Count Month Hit Count
0 0 0
@ghost ghost added the untriaged New issue has not been triaged by the area owner label May 25, 2022
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@ericstj ericstj added area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' labels May 25, 2022
@ghost
Copy link

ghost commented May 25, 2022

Tagging subscribers to this area: @JulieLeeMSFT
See info in area-owners.md if you want to be subscribed.

Issue Details

Observed in https://dev.azure.com/dnceng/public/_build/results?buildId=1789473

Console output: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-69813-merge-212c7ffc453c4eb2a8/Methodical_others/1/console.b7c2b95a.log?helixlogtype=result

Relevant log snippet:

22:49:29.737 Running test: JIT/Methodical/Coverage/copy_prop_byref_to_native_int/copy_prop_byref_to_native_int.cmd

=================================================================
	Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
	Native stacktrace:
=================================================================
	0x7fa4c2a98722 - Unknown
	0x7fa4c2a39a2e - Unknown
	0x7fa4c2996d71 - Unknown
	0x7fa4c4e52980 - Unknown
	0x7fa4c29e5494 - Unknown
	0x7fa4bed63df3 - Unknown

=================================================================
	External Debugger Dump:
=================================================================
[New LWP 26773]
[New LWP 26774]
[New LWP 26775]
[New LWP 26776]
[New LWP 26781]
[New LWP 26785]
[New LWP 26786]
[New LWP 26787]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
warning: the debug information found in "/datadisks/disk1/work/B4770A02/p/libcoreclr.so.dbg" does not match "/datadisks/disk1/work/B4770A02/p/libcoreclr.so" (CRC mismatch).

0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x562701fd5188) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88	../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.
  Id   Target Id         Frame 
* 1    Thread 0x7fa4c5479740 (LWP 26772) "corerun" 0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x562701fd5188) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  2    Thread 0x7fa4c13ff700 (LWP 26773) "SGen worker" 0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa4c40d95c8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  3    Thread 0x7fa4be6cc700 (LWP 26774) "corerun" 0x00007fa4c4224bb9 in __GI___poll (fds=0x7fa4b8002d50, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
  4    Thread 0x7fa4be4cb700 (LWP 26775) "Finalizer" 0x00007fa4c4e507c6 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7fa4c40ca530) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  5    Thread 0x7fa4b63fe700 (LWP 26776) "corerun" 0x00007fa4c4e51474 in __libc_read (fd=7, buf=0x7fa4b63fdee7, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:27
  6    Thread 0x7fa4a8194700 (LWP 26781) ".NET Sockets" 0x00007fa4c4231947 in epoll_wait (epfd=13, events=0x5627021f5630, maxevents=32, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
  7    Thread 0x7fa4a7d8d700 (LWP 26785) ".NET ThreadPool" 0x00007fa4c4e4dfb9 in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x7fa4a7d8c6f0, expected=0, futex_word=0x7fa4a7d8c828) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
  8    Thread 0x7fa4a7b8c700 (LWP 26786) ".NET ThreadPool" 0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa49c002c90) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  9    Thread 0x7fa4a7b4b700 (LWP 26787) ".NET ThreadPool" 0x00007fa4c4e5232a in __waitpid (pid=26788, stat_loc=0x7fa4a7b491c0, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30

Thread 9 (Thread 0x7fa4a7b4b700 (LWP 26787)):
#0  0x00007fa4c4e5232a in __waitpid (pid=26788, stat_loc=0x7fa4a7b491c0, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1  0x00007fa4c2a98867 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#2  0x00007fa4c2a39a2e in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#3  0x00007fa4c2996d71 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4  <signal handler called>
#5  0x00007fa4c29e5494 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#6  0x00007fa4bed63df3 in System_Runtime_CompilerServices_AsyncTaskMethodBuilder_1_TResult_INT_SetExistingTaskResult_System_Threading_Tasks_Task_1_TResult_INT_TResult_INT () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#7  0x00005627026707f8 in ?? ()
#8  0x00007fa4c143f198 in ?? ()
#9  0x00007fa4c143f620 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7fa4a7b8c700 (LWP 26786)):
#0  0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa49c002c90) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fa49c002c40, cond=0x7fa49c002c68) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fa49c002c68, mutex=0x7fa49c002c40) at pthread_cond_wait.c:655
#3  0x00007fa4bcf3d1fc in SystemNative_LowLevelMonitor_Wait (monitor=0x7fa49c002c40) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155
#4  0x00007fa4bf0ee1e6 in wrapper_managed_to_native_Interop_Sys_LowLevelMonitor_Wait_intptr () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#5  0x00007fa4bea252df in System_Threading_WaitSubsystem_ThreadWaitInfo_Wait_int_bool_bool () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#6  0x00007fa4bea2688e in System_Threading_WaitSubsystem_WaitableObject_Wait_Locked_System_Threading_WaitSubsystem_ThreadWaitInfo_int_bool_bool () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#7  0x00007fa4bea265fb in System_Threading_WaitSubsystem_WaitableObject_Wait_System_Threading_WaitSubsystem_ThreadWaitInfo_int_bool_bool () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#8  0x00007fa4a7b8bbb0 in ?? ()
#9  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7fa4a7d8d700 (LWP 26785)):
#0  0x00007fa4c4e4dfb9 in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x7fa4a7d8c6f0, expected=0, futex_word=0x7fa4a7d8c828) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#1  __pthread_cond_wait_common (abstime=0x7fa4a7d8c7a8, mutex=0x7fa4a000b800, cond=0x7fa4a7d8c800) at pthread_cond_wait.c:533
#2  __pthread_cond_timedwait (cond=0x7fa4a7d8c800, mutex=0x7fa4a000b800, abstime=0x7fa4a7d8c7a8) at pthread_cond_wait.c:667
#3  0x00007fa4c292a474 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4  0x00007fa4c292fb74 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#5  0x00007fa4bef1e863 in wrapper_managed_to_native_System_Threading_LowLevelLifoSemaphore_TimedWaitInternal_intptr_int () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#6  0x00007fa4be9fb9cd in System_Threading_LowLevelLifoSemaphore_WaitForSignal_int () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#7  0x00007fa4c1441d08 in ?? ()
#8  0x0000000000000002 in ?? ()
#9  0xfffeffffffffffff in ?? ()
#10 0x0000000000004e20 in ?? ()
#11 0x0000000000000046 in ?? ()
#12 0x00007fa4c410ee30 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#13 0x00007fa4be9fb6db in System_Threading_LowLevelLifoSemaphore_Wait_int_bool () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#14 0x00007fa4c1441998 in ?? ()
#15 0x00007fa4a7d8c9d0 in ?? ()
#16 0x00007fa4a7d8ca08 in ?? ()
#17 0x00007fa4a7d8cdc8 in ?? ()
#18 0x00007fa4c410ee30 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#19 0x00007fa4a7d8ca80 in ?? ()
#20 0x00007fa4bea20b1d in System_Threading_PortableThreadPool_WorkerThread_WorkerThreadStart () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 6 (Thread 0x7fa4a8194700 (LWP 26781)):
#0  0x00007fa4c4231947 in epoll_wait (epfd=13, events=0x5627021f5630, maxevents=32, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1  0x00007fa4bcf37f9e in WaitForSocketEventsInner (port=13, buffer=0x5627021f5630, count=0x7fa4a81939d4) at /__w/1/s/src/native/libs/System.Native/pal_networking.c:2717
#2  0x00007fa4bcf37ebf in SystemNative_WaitForSocketEvents (port=13, buffer=0x5627021f5630, count=0x7fa4a81939d4) at /__w/1/s/src/native/libs/System.Native/pal_networking.c:3017
#3  0x00007fa4a8eef485 in wrapper_managed_to_native_Interop_Sys_WaitForSocketEvents_intptr_Interop_Sys_SocketEvent__int_ (param0=13, param1=94725539321392, param2=140345171589588) at /_/src/libraries/System.Collections.Concurrent/src/System/Collections/Concurrent/ConcurrentDictionary.cs:2264
#4  0x00007fa4a8e30f1c in System_Net_Sockets_SocketAsyncEngine_EventLoop () at /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs:184
#5  0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7fa4b63fe700 (LWP 26776)):
#0  0x00007fa4c4e51474 in __libc_read (fd=7, buf=0x7fa4b63fdee7, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:27
#1  0x00007fa4bcf3c60e in SignalHandlerLoop (arg=0x562702331280) at /__w/1/s/src/native/libs/System.Native/pal_signal.c:323
#2  0x00007fa4c4e476db in start_thread (arg=0x7fa4b63fe700) at pthread_create.c:463
#3  0x00007fa4c423161f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7fa4be4cb700 (LWP 26775)):
#0  0x00007fa4c4e507c6 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7fa4c40ca530) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0x7fa4c40ca530, abstime=0x0) at sem_waitcommon.c:111
#2  0x00007fa4c4e508b8 in __new_sem_wait_slow (sem=0x7fa4c40ca530, abstime=0x0) at sem_waitcommon.c:181
#3  0x00007fa4c29011c6 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4  0x00007fa4c28da2ba in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#5  0x00007fa4c28da149 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#6  0x00007fa4c4e476db in start_thread (arg=0x7fa4be4cb700) at pthread_create.c:463
#7  0x00007fa4c423161f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7fa4be6cc700 (LWP 26774)):
#0  0x00007fa4c4224bb9 in __GI___poll (fds=0x7fa4b8002d50, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007fa4c2b5f84a in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#2  0x00007fa4c2b5cd45 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#3  0x00007fa4c2b5b5a9 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4  0x00007fa4c2b5ec61 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#5  0x00007fa4c4e476db in start_thread (arg=0x7fa4be6cc700) at pthread_create.c:463
#6  0x00007fa4c423161f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7fa4c13ff700 (LWP 26773)):
#0  0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa4c40d95c8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fa4c40d9578, cond=0x7fa4c40d95a0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fa4c40d95a0, mutex=0x7fa4c40d9578) at pthread_cond_wait.c:655
#3  0x00007fa4c297ee73 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4  0x00007fa4c4e476db in start_thread (arg=0x7fa4c13ff700) at pthread_create.c:463
#5  0x00007fa4c423161f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7fa4c5479740 (LWP 26772)):
#0  0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x562701fd5188) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x562701fd5138, cond=0x562701fd5160) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x562701fd5160, mutex=0x562701fd5138) at pthread_cond_wait.c:655
#3  0x00007fa4c292a490 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4  0x00007fa4c28df52a in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#5  0x00007fa4c28df3a8 in mono_w32handle_wait_one () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#6  0x00007fa4c2902bb4 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#7  0x00007fa4c2892b93 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#8  0x00007fa4bef1d79d in wrapper_managed_to_native_System_Threading_Monitor_Monitor_wait_object_int_bool () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#9  0x00007fa4be9f5986 in System_Threading_Monitor_ObjWait_int_object () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#10 0x0000000000000001 in ?? ()
#11 0x00007fa4c1440a30 in ?? ()
#12 0x0000000000000000 in ?? ()

=================================================================
	Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x7fa4c29e5494):0x7fa4c29e5484  57 01 48 8b 40 38 48 8b 80 98 00 00 00 48 8b 38  [email protected]
0x7fa4c29e5494  41 8b 46 10 8d 34 c5 08 00 00 00 e8 6c 91 f1 ff  A.F..4......l...
0x7fa4c29e54a4  48 89 04 24 41 83 7e 10 00 0f 8e 80 00 00 00 48  H..$A.~........H
0x7fa4c29e54b4  63 c5 48 89 44 24 10 f7 d8 48 89 44 24 08 48 8d  c.H.D$...H.D$.H.

=================================================================
	Managed Stacktrace:
=================================================================
=================================================================
JIT/Methodical/Methodical_others/Methodical_others.sh: line 382: 26772 Aborted                 (core dumped) $LAUNCHER $ExePath "${CLRTestExecutionArguments[@]}"

Crashdump: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-69813-merge-212c7ffc453c4eb2a8/Methodical_others/1/core.1000.26772?helixlogtype=result

Author: ericstj
Assignees: -
Labels:

area-CodeGen-coreclr, blocking-clean-ci, untriaged

Milestone: -

@ericstj ericstj added area-Codegen-AOT-mono and removed area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI labels May 25, 2022
@ericstj
Copy link
Member Author

ericstj commented May 25, 2022

Changing the tag as this happened in llvmfullaot

@marek-safar marek-safar removed the untriaged New issue has not been triaged by the area owner label Jul 20, 2022
@marek-safar marek-safar added this to the 7.0.0 milestone Jul 20, 2022
@karelz
Copy link
Member

karelz commented Aug 5, 2022

This happens more than 1x per day. We should disable the test ASAP. @vargaz can you please do it? Thanks!

@karelz
Copy link
Member

karelz commented Aug 9, 2022

Test disabled in PR #73511 - not blocking CI anymore.
Note: Disabled only on Mono interpreter by accident.

@karelz karelz added disabled-test The test is disabled in source code against the issue and removed blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' labels Aug 9, 2022
@SamMonoRT SamMonoRT modified the milestones: 7.0.0, 8.0.0 Aug 10, 2022
@SamMonoRT
Copy link
Member

Assigning to @jandupej and moving to 8.0.0 We will triage the fix and consider backporting to re-enable the tests.

@karelz
Copy link
Member

karelz commented Aug 26, 2022

Test disabled on all Mono configs in main (8.0) in PR #74265 and in 7.0 (RC2) in PR #74636

Note: It happened in release/7.0 rolling run 1967536 on 8/25 - therefore backporting the disabling test into 7.0 branch.

@jandupej jandupej modified the milestones: 8.0.0, 9.0.0 Jul 12, 2023
@SamMonoRT SamMonoRT assigned matouskozak and unassigned jandupej Oct 25, 2023
@matouskozak matouskozak modified the milestones: 9.0.0, 10.0.0 Jul 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-Codegen-AOT-mono disabled-test The test is disabled in source code against the issue
Projects
None yet
Development

No branches or pull requests

8 participants