[ASan] Disable InstallAtForkHandler on Linux/sparc64

[rorth]

When SPARC Asan testing is enabled by PR #107405, many Linux/sparc64 tests just hang like

#0  0xf7ae8e90 in syscall () from /usr/lib32/libc.so.6
#1  0x701065e8 in __sanitizer::FutexWait(__sanitizer::atomic_uint32_t*, unsigned int) ()
    at compiler-rt/lib/sanitizer_common/sanitizer_linux.cpp:766
#2  0x70107c90 in Wait ()
    at compiler-rt/lib/sanitizer_common/sanitizer_mutex.cpp:35
#3  0x700f7cac in Lock ()
    at compiler-rt/lib/asan/../sanitizer_common/sanitizer_mutex.h:196
#4  Lock ()
    at compiler-rt/lib/asan/../sanitizer_common/sanitizer_thread_registry.h:98
#5  LockThreads ()
    at compiler-rt/lib/asan/asan_thread.cpp:489
#6  0x700e9c8c in __asan::BeforeFork() ()
    at compiler-rt/lib/asan/asan_posix.cpp:157
#7  0xf7ac83f4 in ?? () from /usr/lib32/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

It turns out that this happens in tests using internal_fork (e.g. invoking llvm-symbolizer): unlike most other Linux targets, which use clone, Linux/sparc64 has to use __fork instead. While clone doesn't trigger pthread_atfork handlers, __fork obviously does, causing the hang.

To avoid this, this patch disables InstallAtForkHandler and lets the ASan tests run to completion.

Tested on sparc64-unknown-linux-gnu.

[llvmbot]

@llvm/pr-subscribers-compiler-rt-sanitizer

Author: Rainer Orth (rorth)

Changes

When SPARC Asan testing is enabled by PR #107405, many Linux/sparc64 tests just hang like

#<!-- -->0  0xf7ae8e90 in syscall () from /usr/lib32/libc.so.6
#<!-- -->1  0x701065e8 in __sanitizer::FutexWait(__sanitizer::atomic_uint32_t*, unsigned int) ()
    at compiler-rt/lib/sanitizer_common/sanitizer_linux.cpp:766
#<!-- -->2  0x70107c90 in Wait ()
    at compiler-rt/lib/sanitizer_common/sanitizer_mutex.cpp:35
#<!-- -->3  0x700f7cac in Lock ()
    at compiler-rt/lib/asan/../sanitizer_common/sanitizer_mutex.h:196
#<!-- -->4  Lock ()
    at compiler-rt/lib/asan/../sanitizer_common/sanitizer_thread_registry.h:98
#<!-- -->5  LockThreads ()
    at compiler-rt/lib/asan/asan_thread.cpp:489
#<!-- -->6  0x700e9c8c in __asan::BeforeFork() ()
    at compiler-rt/lib/asan/asan_posix.cpp:157
#<!-- -->7  0xf7ac83f4 in ?? () from /usr/lib32/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

It turns out that this happens in tests using internal_fork (e.g. invoking llvm-symbolizer): unlike most other Linux targets, which use clone, Linux/sparc64 has to use __fork instead. While clone doesn't trigger pthread_atfork handlers, __fork obviously does, causing the hang.

To avoid this, this patch disables InstallAtForkHandler and lets the ASan tests run to completion.

Tested on sparc64-unknown-linux-gnu.

---

Full diff: https://github.com/llvm/llvm-project/pull/108542.diff

1 Files Affected:

• (modified) compiler-rt/lib/asan/asan_posix.cpp (+5-1)

diff --git a/compiler-rt/lib/asan/asan_posix.cpp b/compiler-rt/lib/asan/asan_posix.cpp
index 76564538bd5d77..c42c0472592b78 100644
--- a/compiler-rt/lib/asan/asan_posix.cpp
+++ b/compiler-rt/lib/asan/asan_posix.cpp
@@ -171,7 +171,11 @@ static void AfterFork(bool fork_child) {
 }
 
 void InstallAtForkHandler() {
-#  if SANITIZER_SOLARIS || SANITIZER_NETBSD || SANITIZER_APPLE
+#  if SANITIZER_SOLARIS || SANITIZER_NETBSD || SANITIZER_APPLE || \
+      (SANITIZER_LINUX && SANITIZER_SPARC)
+  // While other Linux targets use clone in internal_fork which doesn't
+  // trigger pthread_atfork handlers, Linux/sparc64 uses __fork, causing a
+  // hang.
   return;  // FIXME: Implement FutexWait.
 #  endif
   pthread_atfork(