arena: what is freed during a purge pass is purged by the next pass

[robobun]

Stacked on #38: review the second and third commit.

Problem

• Memory freed while the scavenger is in a purge pass is never purged by it. It stays until a thread parks or something forces a collect. A repro prints 266 of 320 MiB purged after 65 s.
• _mi_arenas_try_purge (src/arena.c) keeps its old subproc->purge_expire during a pass, so a free in that time arms the arena only. It then skips its update, and mi_scavenger_run sets the value to 0.
• No later free re-arms the scavenger, because the arena is armed.

Fix

• A pass resets subproc->purge_expire first, as mi_arena_try_purge does for an arena. A free behind the pass then arms it and wakes the scavenger. At its end the pass puts back the earliest pending arena expire.
• The scavenger no longer clears the value. It retries in 1 ms if another thread is in a pass. After 30 s idle it looks at the arenas.
• The fork child handler makes a pass due if fork() cut one off. Process exit on Windows releases the purge guard (third commit).
• Verified: new test/test-purge-behind-pass.c (5 of 5 fail before, 30 of 30 pass after). Full ctest passes.

Background

• A purge returns freed arena slices to the OS (madvise), purge_delay (100 ms) after the free.
• arena->purge_expire is when an arena is due. A free sets it only from 0: it arms the arena.
• subproc->purge_expire is the earliest of these. The scavenger thread sleeps until then. A free sets it only from 0, after it armed the arena.

Notes

The 30 s timeout did not help. With purge_expire at 0 the scavenger picks a 30 s wait and waits again. It never looked at the arenas on that path, so the comment there ("any per-arena expiry that did not propagate to subproc is still eventually purged") was not true. The repro still has 64 MiB unpurged after two of those timeouts.

Where it came from. Commit 3fdbfb3 ("scavenger: 30s safety-net timeout, exact expire-now wait") made _mi_arenas_try_purge install the earliest pending arena expire with a CAS at the end of a pass, and made the scavenger CAS the value to 0 after that. Upstream left the stale value in place and polled.

Two defects, one protocol. (1) With visit_all, max_purge_count = max_arena, so the break for it fires on the last arena. With one arena each pass that purges something ends with all_visited = false. The final update and the read of that arena's expire are both skipped. (2) With that fixed there is still a window. A free into arena 0 after the pass read its expire, and before the final CAS, arms arena 0 alone. The window spans the passes over all later arenas. The reset at the start closes both.

Ordering. The pass resets subproc->purge_expire (CAS loop), then reads each arena expire with a CAS (0 -> 0) after its walk. A free does CAS(arena: 0 -> E), then CAS(subproc: 0 -> E). The two CAS operations on the arena expire are ordered. If the free is first, the pass reads E and puts it back. If the pass is first, the free reads its write, so it also sees the reset and its second CAS succeeds. A plain load in the pass allows both to miss (store buffering), as the comment on scavenger_wake in mi_scavenger_run describes for the park path.

The scavenger. The old CAS to 0 was an ABA hazard by itself: _mi_arenas_purge_now stores the current time, which can be the value the scavenger read. A first version here backed off exponentially while purge_expire stayed due and unchanged. A stress run showed the wait pinned at its 16 s cap, because parks wake the loop at kHz rates and each iteration doubled it. One round in 60 then stalled with a due purge_expire. The final version uses the return value from #38: dropped means retry in 1 ms, not dropped and not reset means purging got switched off at run time.

mi_subproc_schedule_purge wakes the scavenger on each change. A thread other than the scavenger can run the pass (mi_collect, a thread exit, a poll without visit_all). The scavenger can read 0 during that pass and sleep for 30 s. The wake covers that. On the scavenger's own thread it is one futex call with no waiter.

Fork. With the reset at the start, a fork() in the middle of a pass leaves the child with purge_expire at 0 and nobody to put back what is pending. Before this change the stale due value stayed, so the child's next poll ran a pass. _mi_arenas_forked_child now reports whether it released a held guard, and _mi_process_fork_child makes a pass due for each sub-process in that case. The second part of the test covers it: without these lines it fails 2 of 3 runs.

Second commit: process exit on Windows. mi_process_done runs in the process detach callback on Windows, after the system terminated every other thread. A thread that was in a pass then holds the purge guard for good, and the forced collect in mi_process_done_once waits for it without end (the wait is from #38). The new test returns from main while the scavenger is still in its last pass, so the Windows Debug jobs of the first CI run here hung at exit (5 of 6 local runs on Windows x64, 0 of 8 with the second commit, full Debug ctest 31 of 31). mi_process_done_once now releases the guard on Windows. _mi_arenas_forked_child becomes _mi_arenas_purge_guard_reset, as it serves both cases.

In bun. JSC registers its structure heap as a second, exclusive arena (mi_manage_os_memory_ex in StructureAlignedMemoryAllocator.cpp). That arena rarely has anything to purge, so a pass over both ends with all_visited = true, reads the re-armed expire of the main arena after its walk, and installs it. That hides the first defect in bun's default configuration: the same script that loses 128 MB with one arena gets its second pass on time with two. It shows when each arena purges in the same pass, and with one arena (Malloc=1 makes JSC allocate through malloc, which is mimalloc on Linux): then bun keeps purged: 274 MB of 384 MB until the loop goes idle.

Stress (local harness, not in the PR). 8 to 16 threads that stay alive allocate and free 1 to 8 MiB blocks, with and without mi_collect(true), mi_collect(false) and short parks. They pause without a park or an exit. The main thread reads the arena and sub-process expires and the purge bitmaps. On the base (8565de2): 15 of 15 rounds end with armed arenas=1 subproc expire=0 and about 530 MiB of free, committed ranges with their purge bit set. With this change: 750 rounds in three configurations, 0 stuck, 0 ranges left, quiet after 103 ms at most. Also with MIMALLOC_ARENA_RESERVE=65536 (9 arenas).

More purge activity. Ranges that used to stay resident by accident are now purged purge_delay after they were freed. That is what the option promises.

Not changed, seen on the way (upstream windows, all narrow). mi_arena_schedule_purge sets the purge bits after it arms the arena, and _mi_arenas_free marks the slices free after that. A forced pass can reset an arena expire that a free set a moment before, with the bits of that free landing behind the walk. A walk that meets purge bits before the slices are marked free clears them without a purge. Each heals on the next free into the arena.

Local runs (Linux x64, clang 17 and gcc 14). Full ctest: Release 31 of 31, Debug 32 of 32. The purge, park and fork tests also pass with ASAN, UBSAN, MI_GUARDED, MI_SECURE and MI_USE_CXX. The new test passes 30 of 30 runs, 10 of 10 pinned to one CPU, and with MIMALLOC_PURGE_DELAY=10 and 1000. TSAN does not run in my container (the runtime cannot disable ASLR there), so that is for CI here.

The bun side is oven-sh/bun#42543 for #38. A bun PR for this one follows.