profile: a theap starts over with each mi_profiler_start

[Jarred-Sumner]

A theap that was sampling before a mi_profiler_stop, and that got to neither the stop nor the next mi_profiler_start in its generic path, went on with what it had: the rate that on_alloc had given it, how far it was into that period, and the bytes requested since its last sample. _mi_theap_update_profiling only acted when "the profiler is enabled" and "the theap has a rate" disagreed, which they do not for such a theap. Two effects:

• A session at a small rate after one at a large rate sees nothing of that thread until the old period has run out. start (on_alloc returns 256 MiB); a thread allocates 32 MiB; stop; start (on_alloc returns 64 KiB); the thread allocates 32 MiB: 0 of those 32 MiB are sampled.
• At equal rates its first sample of the new session reports bytes from before the start: 1114112 bytes with 655360 requested since the start.

Found with a heap profiler on these hooks that draws its intervals from an exponential distribution (a long draw is the first case) and that stops and starts in one process.

The change

• mi_profiler_t.reserved is (epoch << 1) | enabled (one word, so both are read together). A start of a profiler that is not running takes a new epoch from one counter of the process, so two profilers never have the same one.
• A theap keeps the epoch that its profile rate and countdowns belong to (profile_sample_epoch). _mi_theap_update_profiling starts it over (the initial rate, a full period, nothing requested, and with MI_SAMPLE==1 the counts of its pages synced) also when the profiler is enabled and the epoch is another one. Starting a period is one function for _mi_theap_init and _mi_theap_update_profiling; "has to look" is one predicate, also for the second look of _mi_theap_init (for a start or a stop that came in between the first one and the theap being on the heap's list).
• _mi_theap_malloc_profiled looks at the epoch too: with MI_SAMPLE==2 a sample that comes due in alloc.c does not pass mi_malloc_generic_fallback, and the look inside _mi_malloc_generic_no_sample is put off. A sample that belongs to the period before the start is not taken.
• The theap of the thread that calls mi_profiler_start starts over through _mi_theap_update_profiling instead of getting a rate of 1 with the countdowns it had.
• mi_profiler_stop leaves the request to look, as mi_profiler_start does, so a theap stops counting what it hands out at its next generic allocation and not at its next sample or 1000 generic allocations later. It waits for no lock for that (what is behind a lock that is taken is left out; a theap finds out by itself as before), so it can still be called from on_alloc, from a visitor of the heaps or late in the exit of the process. mi_heap_profile and mi_subproc_profile disable the profiler they are given without it, as before.

Cost

Nothing on the fast paths and nothing for a theap that does not sample: the comparison is in _mi_theap_update_profiling, which runs on a request and every 1000 generic allocations, and in _mi_theap_malloc_profiled, per sample. clang 21, -O3 -march=nehalem, C++, MI_FREE_USE_PAGEMAP, user-mode instructions (runs of one build agree to 200):

	before (69050cf3)	this
25 M mi_malloc + 25 M mi_free of 16 to 1039 bytes, no profiler	2.030466 G	2.030470 G
20 M mi_malloc(128 KiB) + mi_free, no profiler, per pair	261.68	261.68
the same with a profiler running at 512 KiB (4 M samples), per pair	334.85	335.46 (3 instructions per sample)
the same after a start and a stop, per pair	261.68	261.69

In bun (release build with this commit as the pin against the same build with 69050cf3, 8 cores, interleaved runs, medians and ranges): bun -e 1 9.73 M (9.72..9.74) against 9.72 M (9.72..9.74); JSON.stringify + JSON.parse of a 200-user object 3000 times 4789.8 M (4785.4..4792.2) against 4787.3 M (4785.4..4793.3); 3M ArrayBuffers 5802 M (5774..5807) against 5792 M (5781..5803), and with a heap profile running at 512 KiB 5965 M (5932..5968) against 5959 M (5957..5969). (Measured with the first commit; the second one adds the 3 instructions per sample.)

Test

test-profile: "a thread starts over when the profiler is started again", one thread that does not allocate between a stop and the next start:

1. a period at a rate of 256 MiB, then 32 MiB at a rate of 64 KiB of another profiler that is started for the first time. On 69050cf3, and with an epoch per profiler: 0 bytes sampled. Here 33488896.
2. a stop half way into a period of 1 MiB, with blocks of a page in between. On 69050cf3 the first sample of the next start reports 1114112 bytes with 655360 requested since the start; here it reports no more than the thread got since the start.
3. a rate at which every allocation is a sample, then one at which none is: 0 samples. With MI_SAMPLE==2 and without the look in _mi_theap_malloc_profiled: 1.

The bounds are in bytes of blocks, which is what a build that counts the blocks of a page when it is refilled counts, and what MI_SECURE makes larger than the request. (The first version of the test compared with requested bytes and failed with MI_SECURE=ON; an earlier version of this text said that only test-emulated-tls failed there, which was wrong.)

Not covered by the test: that a theap which starts over syncs the counts of its pages (MI_SAMPLE==1). Without it, the blocks that a page handed out since it was last refilled (up to 8 KiB a page) are counted against the first period after the start; I did not get a test to tell the two apart, the first sample moves by less than the bounds allow.

Locally, test-profile in release, debug (MI_DEBUG=FULL), secure, release and debug with MI_GUARDED=FULL, release and debug as C++, TLS pthreads, ASAN, UBSAN and TSAN (no report): 16 of 16. All of ctest: release 33 of 33, debug (MI_DEBUG=FULL) 34 of 34, guarded release 33 of 33 and debug 34 of 34; MI_SECURE=ON all but test-emulated-tls, which fails the same way on 69050cf3 on this machine.