tls: the pthread key of the thread local slots is created once

[Jarred-Sumner]

Where a thread local is a pthread key (macOS, MI_TLS_MODEL_PTHREADS), the key is created when it is first set: mi_pthread_key_set looks at the key, and when it is not there yet _mi_pthread_key_create creates one into it and stores the value under it. No synchronization between the look and the create.

The key of the thread local slots (threadlocal.c: the theaps of the heaps besides the main one hang off it) is first set when such a heap is first used, and a program can do that on many threads at once. Two threads both find no key, both create one, the second replaces the first in the variable, and the first thread's slots are now under a key that nobody reads any more:

• a debug build stops in heap.c:mi_heap_init_theap, assertion: "theap == (mi_theap_t*)_mi_thread_local_get(heap->theap)". That is the failure of test-park-handoff-eager on the macOS Intel runners, seen twice (on bun-dev3-v2 itself, run 34803968831, and on profile: a theap starts over with each mi_profiler_start #47's first run): it is not the park code, the test that fails is test_exit_while_swept_with_dyn_tls, whose eight threads each do mi_heap_new + mi_heap_malloc and are the first in that process to use another heap than the main one;
• a release build has no assertion: the thread makes a second theap for its heap, the first one and its pages stay until the thread ends, and a pthread key leaks.

The change

• _mi_pthread_key_create_once creates the key aside and stores it with a compare and swap; a thread that was not first deletes its key and uses the one that was (also when its own pthread_key_create fails). mi_pthread_key_set calls it instead of _mi_pthread_key_create, which stays as it is for the two keys that _mi_tls_slots_init creates under its own once.
• The key of mi_define_thread_local is an _Atomic(pthread_key_t), as _mi_theap_default_key in prim-tls.c is: read with a relaxed load where a value is read, with acquire where one is set (about once a thread).

Platforms with real thread locals are not affected (the macro defines a plain mi_decl_thread variable there).

Test

test-tls-key-race: one process, eight threads that make their heaps, wait for each other, and allocate from them at the same moment. A thread that lost its slots does not notice (it makes new ones and a second theap for its heap), so the test counts the theaps that were made: two a thread. A debug build has a hook for it like the other mi_debug_stall_in_* ones, mi_debug_stall_in_pthread_key_create: the first thread that comes to create a key waits for a second one, so both have found no key. With MI_TLS_MODEL_PTHREADS and MI_DEBUG=FULL:

this PR	"8 threads made 16 theaps, 0 of them saw its heap change; 2 created the key at the same time", 100 of 100 runs (3 now and then)
with the last key winning, as before	MI_DEBUG=FULL: stops on heap.c:84 or page.c:259, 12 of 12; MI_DEBUG=ON: "made 17 theaps", 12 of 12

A release build has no hook and only the natural window, a few instructions wide; to see the unpatched code fail there I put a usleep(2000) in front of pthread_key_create (not part of this PR): a forking version of the test, 8 threads in each of 50 children, failed in 50 of 50 on the assertion above, and in 0 of 100 with the compare and swap and the same sleep; without the sleep 0 of 400 either way on a 64 core Linux machine (the macOS Intel runners hit it twice in some dozens of runs).

Locally, all of ctest: release 34 of 34, debug (MI_DEBUG=FULL) 35 of 35, debug as C++ with MI_TLS_MODEL_PTHREADS 35 of 35, MI_TLS_MODEL_PTHREADS debug 35 of 35 and release 34 of 34; the test alone under the thread sanitizer with and without MI_TLS_MODEL_PTHREADS: no report.

[Jarred-Sumner]

Cost

Measured after the merge, for the record: f3171b42 (this PR's head) against its base dbefe024. Release builds with clang 21, perf stat -e instructions:u, pinned to the same 8 cores, the two builds interleaved, medians of 7 runs (5 for the default TLS rows), range in brackets where it is not zero.

What changed in the generated code

• Default Linux build (real TLS, the changed helpers are not compiled): the disassembly of every object in libmimalloc.a is the same byte for byte, but for libc.c.o, which gains the one new function _mi_pthread_key_create_once (+176 bytes of text, 241418 -> 241594 for the whole library; nothing calls it in this configuration).
• -DMI_TLS_MODEL=PTHREADS on Linux x86-64: only threadlocal.c.o (2315 -> 2283 bytes of text) and libc.c.o (+176) differ; alloc.c.o, heap.c.o, theap.c.o, init.c.o, page.c.o and the rest are the same.
  • _mi_thread_local_get and mi_thread_local_get_regular (what mi_heap_malloc on a heap that is not the main one goes through for every call) are the same instruction for instruction: cmp; jne; mov key(%rip),%edi; jmp pthread_getspecific. The relaxed atomic load is the same mov.
  • _mi_thread_local_set is three instructions shorter (it no longer passes a destructor argument along); the acquire load of the key is a plain mov on x86.
  • _mi_thread_locals_done (process teardown) uses xchg where it had a load and a store.
  • No fence anywhere.
• arm64 macOS (where the pthread path is the default; threadlocal.c, libc.c and heap.c compiled with --target=arm64-apple-macos13 -O3, as C and as C++):
  • heap.c is the same file of assembly before and after.
  • _mi_thread_local_get is the same instruction for instruction: cmp x0,#1; b.ne; adrp; ldr x0,[x8,key@PAGEOFF]; cbz; b _pthread_getspecific. The relaxed load is the same ldr.
  • Where a value is set (_mi_thread_local_set, mi_thread_local_set_expand, _mi_thread_locals_thread_done: once per thread and heap, and at thread exit), the key is now read with add x8,x8,key@PAGEOFF; ldapr x0,[x8] where it was one ldr x0,[x8,key@PAGEOFF]: one more instruction, and no dmb. In the two functions that read the key and then set it, the adrp is repeated after the call in between where the base kept the page address in a callee-saved register; neither is in a loop.
  • _mi_thread_locals_done uses swp where it had ldr + str.

Instructions (G = 10^9 user instructions)

-DMI_TLS_MODEL=PTHREADS	base dbefe024	head f3171b42
allocator micro (50 M random malloc/free of 16..1039 bytes)	2.5686	2.5686	0.000%
20 M x mi_malloc(128 KiB) + mi_free	5.4713	5.4713	0.000%
8 threads x 100 k x (mi_heap_new, 4 x mi_heap_malloc, mi_free, mi_heap_destroy)	24.4511 [24.4445-24.4638]	24.4513 [24.4462-24.4547]	+0.001%
the same on one thread	2.9425	2.9422	-0.010% (3 instructions per iteration)
8 threads x 10 M x (mi_heap_malloc + mi_free) on a heap of the thread's own	6.4870	6.4870	0.000%
the same on one thread	0.8112	0.8112	0.000%

default TLS (Linux)	base	head
allocator micro	1.9932	1.9932	0.000%
128 KiB pair	5.0313	5.0313	0.000%
heaps, 8 threads	23.8768 [23.8696-23.8841]	23.8750 [23.8699-23.9151]	-0.008%
heaps, 1 thread	2.8699	2.8699	0.000%
mi_heap_malloc + mi_free, 1 thread	0.5509	0.5509	0.000%

Bun (release build of the same tree, the mimalloc commit the only difference; Linux x64, so the changed path is not compiled)

	pinned to dbefe024	pinned to f3171b42
bun -e 1 (9 runs)	10.01 M [10.00-10.03]	10.02 M [10.01-10.03]
JSON.stringify + JSON.parse loop (5 runs)	4789.8 M [4787.1-4794.4]	4789.4 M [4787.0-4794.4]
allocation loop (5 runs)	5771.6 M [5742.9-5804.2]	5792.8 M [5783.7-5817.3] (ranges overlap; this one moves by that much between runs of one binary)

Not measured: a run on Apple hardware. What is above for arm64 is the generated code, not a timing.