profile: MI_PROFILE=1 without a profiler costs what MI_PROFILE=0 costs (but for 4 instructions in the generic path); mi_profiler_start reaches the running threads

[Jarred-Sumner]

Where MI_PROFILE=1 costs something without a profiler

Bun's flags (C++20, -O3 -march=nehalem, MI_FREE_USE_PAGEMAP=1), clang 21, a malloc/free micro-benchmark (50 M iterations over 256 slots, 16 to 1039 bytes, 25 M mi_malloc of which 1.23 M reach _mi_malloc_generic), user instructions:

	MI_PROFILE=0	MI_PROFILE=1, no profiler
bun-dev3-v2	2.25417 G	2.28706 G (+1.46%)
this branch	2.25419 G	2.25916 G (+0.22%)

mi_malloc and mi_free compile to the same instructions with and without MI_PROFILE (only the address of __mi_page_map differs), and callgrind counts the same for them and for main: 421,582,951 and 576,011,818 in both builds. All of the 32.9 M instructions are in the generic path:

• mi_page_update_sample_countdown: 25.3 M. It runs when a free list is refilled (898,081 times, 28 instructions): find the theap of the page, leave if sample_rate == 0, move last_alloc up.
• _mi_malloc_generic: 7.6 M, 6 instructions per call: the check of the countdown before, and of sample_rate and the countdown after mi_page_queue_find_free.

What a theap that does not sample runs now

• mi_page_update_sample_countdown is what it is with MI_SAMPLE==0: the reset when alloc_count gets to 0x8000. The blocks that the fast path took from a page are charged to the sample countdown by mi_theap_count_page_allocs, which only a theap that samples calls: for the first page of the size class before mi_page_queue_find_free refills it (mi_theap_refill_is_sample, out of line), and as before in mi_theap_page_update_stats. (At the end of mi_malloc_generic_fallback every theap calls it: that is where the block of a sample comes from, with sample_rate set aside by _mi_malloc_generic_no_sample; last_alloc has to move up there or the block is counted a second time.) If that uses up the countdown, or it was used up already, the allocation in progress goes to mi_malloc_generic_fallback, which takes the sample as it did (or returns NULL for MI_MALLOC_GENERIC_BLOCK_START).
• In _mi_malloc_generic that is one test, theap->sample_rate != 0, before mi_page_queue_find_free.
• A theap that does not sample does not move last_alloc up either. When its rate goes from 0 to something (_mi_theap_update_sample_rate), _mi_theap_sync_sample_counts sets last_alloc = alloc_count on its pages: what they handed out before is not for the first period.
• Its countdown is MI_SAMPLE_COUNTDOWN_MAX (_mi_theap_init, and _mi_theap_update_sample_rate when the rate goes to 0), not -1 or 0: nothing in the generic path puts it right any more, and alloc-aligned.c asks mi_theap_should_sample before its naturally aligned path.

The 4 instructions per generic call that are left: that test (2), the limit below (1, clang does not fold the relaxed atomic load into the compare), and one that the compiler adds around them. Without any: a theap that samples has a limit of 0 and takes another path than the fast one of _mi_malloc_generic, which then has no code for sampling at all (+0.056%, the load of the limit). That path runs the head of _mi_malloc_generic for nothing, and an attached profiler cost 2% more in an allocation benchmark of Bun: not taken.

mi_profiler_start reaches the threads that are running

A theap looked at its profiler every 1000 generic allocations (mi_malloc_generic_admin): with small blocks that is many megabytes after mi_profiler_start. Now:

• _mi_malloc_generic takes its fast path while ++generic_count < theap->generic_fast_limit (a field, MI_GENERIC_FAST_LIMIT = 1000, instead of the constant).
• mi_profiler_start stores -1 there for the theaps (not the detached ones, which hold meta data) of the heaps that have the profiler, in the current and the main sub-process, under heaps_lock and theaps_lock. Their next generic allocation goes to mi_malloc_generic_fallback, which calls _mi_theap_update_profiling (the code that was in mi_malloc_generic_admin, which calls it too). That sets the limit to 0 while it looks and back to 1000 with a compare-and-swap: a request that comes in while it looks stays.
• A theap that is made while mi_profiler_start runs leaves the request with itself if the profiler is enabled once it is on the list of its heap and its own look did not see that.
• It starts with a full period for the profiler as well (profile_sample_countdown), as _mi_theap_init does: with guarded sampling on, the first guarded sample point was a profiler sample for a period that nobody requested.
• _mi_theap_update_profiling does not start a detached theap: mi_process_theap_meta is on the list of the main heap, and mi_malloc_generic_admin would have started to sample it after 1000 generic allocations ("no sampling for meta data", init.c).
• Inside _mi_malloc_generic_no_sample (which sets sample_rate aside and puts it back) a start or stop is put off to the next generic allocation (it leaves the request for it: when every allocation is a sample, mi_malloc_generic_admin can get there inside of it every time, and a mi_profiler_stop was never seen). _mi_theap_malloc_profiled looks as well when it finds the profiler stopped. _mi_malloc_generic_no_sample on a theap that does not sample sets nothing aside and just allocates.
• The meta-data theaps get a countdown of MI_SAMPLE_COUNTDOWN_MAX instead of 0.

With a profiler attached

A running profiler costs a bit more than before. Same micro-benchmark with a profiler at 512 KiB (26,900 samples): 2.87383 G before, 2.88457 G (+0.37%). An allocation benchmark of Bun (3 M ArrayBuffers, JIT on, user instructions, average of 5): without a profile 6.0253 G to 6.0138 G; with one at 512 KiB 6.1229 G to 6.1653 G, which is 2.5% over a run without a profile where it was 1.6%. That is paid only while a profile runs; what is saved is paid by everyone all the time. Sampled over requested bytes for allocations of 12,000 to 200,000 bytes at 128 KiB: 1.021, 1.022, 1.098, 1.012, 1.000, as before. test-profile prints the same figures as before within a few percent (release: small share 0.585 to 0.620 for 0.699, 511 to 493 samples of aligned allocations for 553).

Tests

test_profiler_start_with_running_threads: four threads allocate (250,000 mi_malloc each, 16 to 1039 bytes, 256 slots), a fifth starts a profiler at 256 KiB; each thread gets its first sample after 264 K to 333 K bytes of what it requests from then on (bounds: half a period to two). Without the request: none within 16 periods on two threads, 3 and 4 MiB on the others. Without _mi_theap_sync_sample_counts: after 546 to 14,439 bytes.

test_profiler_aligned_late_start now gets all 3000 samples (2002 before: the first 998 allocations were before the theap looked).

ctest with the flags and environment of .github/workflows/test.yaml: Release, Debug MI_DEBUG=FULL, Secure, Debug and Release as C++, ASAN, UBSAN, TSAN, Debug and Release MI_GUARDED=FULL, Debug and Release MI_FREE_USE_PAGEMAP=ON, Debug and Release MI_SECURE=FULL, TLS pthreads, SIMD, Release MI_PROFILE=OFF, Debug MI_DEBUG=FULL with MI_GUARDED=OFF (coarse sampling with asserts).

Seen on the way, not changed

test_profiler_start_with_running_threads fails when it is run by hand with MIMALLOC_GUARDED_SAMPLE_RATE below its rate in a MI_GUARDED build (ctest sets 0 for test-profile): the profiler gets its samples early there, before and after this change (_mi_theap_malloc_sampled takes all bytes since the last profiler sample off profile_sample_countdown at every guarded sample point, see the notes of #43).

The block of a sample below MI_SMALL_MAX_OBJ_SIZE is counted a second time when its page is next counted (it comes from the fast path of _mi_malloc_generic inside _mi_malloc_generic_no_sample, where last_alloc does not move): 3% for allocations of 2000 bytes at 128 KiB, as before.