SIGSEGV in alloc.nim addToSharedFreeList() in heavily threaded code

[zevv]

Description

In one of my projects I'm making quite heavy use of threads; I am only able to get this code to run stable when using -d:useMalloc; when using the default Nim memory allocator I can consistently reproduce crashes.

Unfortunately I am not able to reduce to a minimal reproduction example yet.

I have no proof the Nim allocator is the root cause of the problem and not my code, but the code runs 100% clean under valgrind, helgrind, asan and tsan when using -d:useMalloc, but is problematic under all of these tools when using the Nim allocator.

Symptoms are at least a SIGSEGV at runtime. gdb output, valgrind and helgrind logs are attached below. gdb backtrace indicates that addToSharedFreeList() gets called with nil for f.

Helgrind seems to indicate a race between compensateCounters() and addToSharedFreeList()

Not likely to be related, but ringabout mentioned a "Weird bug 2:" related to the compensateCounters() function in #20884 recently here: #20884 (comment)

valgrind.log
helgrind.log

Nim Version

Nim Compiler Version 1.7.3 [Linux: amd64]
Compiled at 2022-12-06
Copyright (c) 2006-2022 by Andreas Rumpf

git hash: 047c9fb08a4daae1fac251e43cef9c7ec5350110
active boot switches: -d:release -d:danger

Current Output

Gdb trace of the problem:


#0  0x000055555555a1c1 in addToSharedFreeList__system_5075 (c=c@entry=0x7ffff5401000, f=0x0)
    at /home/ico/external/Nim/lib/system/alloc.nim:764
#1  0x000055555555b833 in compensateCounters__system_5113 (a=a@entry=0x7ffff567cb50, 
    c=c@entry=0x7ffff5401000, size=size@entry=32) at /home/ico/external/Nim/lib/system/alloc.nim:791
#2  0x000055555555bdef in rawAlloc__system_5186 (a=a@entry=0x7ffff567cb50, 
    requestedSize=requestedSize@entry=32) at /home/ico/external/Nim/lib/system/alloc.nim:854
#3  0x000055555555bebf in alloc__system_5401 (allocator=0x7ffff567cb50, size=size@entry=32)
    at /home/ico/external/Nim/lib/system/alloc.nim:1048
#4  0x000055555555bef9 in allocImpl__system_1767 (size=size@entry=32)
    at /home/ico/external/Nim/lib/system/alloc.nim:1124
#5  0x000055555555bf07 in allocSharedImpl (size=size@entry=32)
    at /home/ico/external/Nim/lib/system/alloc.nim:1180
#6  0x000055555555bf1a in allocShared0Impl__system_1782 (size=32)
    at /home/ico/external/Nim/lib/system/alloc.nim:1183
#7  0x000055555555cbb4 in alignedAlloc0__system_1933 (size=size@entry=32, align=align@entry=8)
    at /home/ico/external/Nim/lib/system/memalloc.nim:351
#8  0x000055555555cf66 in nimNewObj (size=size@entry=24, alignment=alignment@entry=8)
    at /home/ico/external/Nim/lib/system/arc.nim:66
#9  0x00005555555771fa in addFd2281709574__OOZnimactorsZlibZevq_8222 (continuation=0x7fffd758f268)
    at /home/ico/sandbox/prjs/nimactors/nimactors/api.nim:82
#10 0x00005555555710ee in workerThread__OOZnimactorsZpool_1683 (worker=<optimized out>)
    at /home/ico/sandbox/prjs/nimactors/nimactors/pool.nim:408
#11 0x000055555555fdea in threadProcWrapDispatch__OOZnimactorsZpool_2178 (thrd=thrd@entry=0x7ffff7e32420)
    at /home/ico/external/Nim/lib/system/threadimpl.nim:74
#12 0x000055555555fe8a in threadProcWrapStackFrame__OOZnimactorsZpool_2168 (thrd=thrd@entry=0x7ffff7e32420)
    at /home/ico/external/Nim/lib/system/threadimpl.nim:100
#13 0x0000555555557fd5 in threadProcWrapper__OOZnimactorsZpool_2154 (closure=0x7ffff7e32420)
    at /home/ico/external/Nim/lib/system/threadimpl.nim:106
#14 0x00007ffff7c8784a in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#15 0x00007ffff7d0b2cc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81```

Expected Output

Clean run, no crash in nim allocator

Possible Solution

No response

Additional Information

No response

[Araq]

We found the same bug but don't have an easy to reproduce example either, unfortunately.

[zevv]

Ok, cool, thanks for confirming. I assume this ticket is a duplicate, what is the original?

I do have some code to run that reproduces 80% of the time, but it's not nice and standalone; let me know if you are interested, I can give instructions how to run.

[Araq]

I assume this ticket is a duplicate, what is the original?

There is not one yet. @ringabout is also investigating.

[zevv]

Verified to fix my case, thanks