page: hole purging in large (4 MiB) pages, held back while the page is in use (by the sweep's epoch: no clock on the allocating side)

[Jarred-Sumner]

Problem

Free blocks in large (4 MiB) pages, which hold the blocks of about 84 KiB up to 512 KiB, are never given back while one block of the page is live. Hole purging skips these pages on a 4 KiB OS page: page->purged has 256 bits and a 4 MiB page has 1024 OS pages (mi_page_can_purge_holes is false, the sweep counts the page as ineligible). Measured with mincore over every free block in such pages, 97% of the free space is resident, and it does not move while the process is idle:

workload (Bun, release, Linux x64)	large pages	live	free	of it resident at idle
100..500 KiB typed arrays, strings and Buffers, 1 in 12 kept, idle 10 s	33	49.2 MB	73.1 MB	71.6 MB
Bun.serve, 256 KiB request bodies, 70k requests, 40 bodies kept, idle 60 s	10	11.1 MB	21.3 MB	20.8 MB
same server, nothing kept, idle 60 s	5	1.1 MB	8.8 MB	8.4 MB

The pages of the last row exist in every Bun process: they are pinned by about 1 MB of allocations made at start-up. A session of a large CLI application leaves 6.5 to 7.6 MB free in 9 to 10 such pages; what it allocates in that range (187 MB over 918 allocations in 47 s) is Vec and Vector growth, string builders, Uint8Array chunks and JIT compiler temporaries, all written in full before they are freed.

Change

1. The unit of the purge bitmap is per page. mi_page_purge_unit(page) is the OS page where the block area of the page then needs at most MI_PAGE_PURGE_BITS bits, and else the smallest power-of-two multiple of the OS page for which it does: 16 KiB for a 4 MiB page on a 4 KiB OS page. Bit k of page->purged names unit k, counted from align_down(page_start, unit). Nothing changes for small and medium pages on any OS page size, for large pages where the OS page is 16 KiB or more, and on 32-bit (a large page is 1 MiB there): the unit is the OS page in each of those.

Everything else is what is there already, with "OS page" read as "unit": a unit is discarded only when it lies wholly inside the block area and every block that overlaps it is free (mi_page_purge_holes_walk), a block is purged exactly when it overlaps a discarded unit, a purged block is off every free list until _mi_page_unpurge_run brings its run back, _mi_os_discard never touches commit state, _mi_page_unpurge_all runs when the page goes back to the arena, the counters and the sweep's rate limit. Nothing in mi_malloc or mi_free.

• Block sizes. The rule does not depend on how the block size relates to the unit; a block that shares a unit with a live neighbour keeps that unit (test_unit_arithmetic checks the arithmetic for 8, 16 and 64 KiB units under block sizes that are and are not multiples of them, with and without an offset of the first block). In practice each free block goes as a whole: the size classes of a large page on 64-bit are 96, 112, 128, 160, 192, 224, 256, 320, 384, 448 and 512 KiB, all multiples of 16 KiB, and the blocks start at the slice (the page meta data is separate from the page for these block sizes both with the page map, MI_FREE_USE_PAGEMAP, which is how Bun builds, and with upstream's aligned meta data).
• Page header, unformed tail, guard page. A unit that is not wholly inside [page_start, page_start + capacity*block_size) is never discarded, as before; that covers a header in front of the blocks, the blocks that are not formed yet (discarded by their own OS-page-granular mechanism, which is unchanged and already worked for large pages) and the guard page of MI_SECURE=FULL.
• First word and zeroing. A purged block carries no free-list link (that is why it is off the lists); mi_page_unpurge_range writes the links when the run comes back, which touches one OS page per block. page->free_is_zero is cleared on every purge and un-purge, as before, so mi_zalloc of a block that was purged zeroes it like any reused block and never sees stale bytes (on macOS and Windows a discarded page may keep its contents). It does not skip the memset on Linux where the kernel already zeroed the pages: that would need per-run zero knowledge, which the mechanism does not have for any page size today.
• Other threads, abandoned pages, heap release. A free from another thread only pushes on xthread_free; the bitmap is touched by the owner or by the sweep that claimed the page, as before. Abandoned large pages in the abandoned map are swept like the others; a page that was abandoned while full is still not visited (unchanged, a separate change). mi_heap_destroy, mi_heap_delete and a page that becomes free all go through _mi_page_unpurge_all, so the discarded-bytes accounting returns to where it was (tested). The committed statistic is not involved: a discard is madvise(MADV_DONTNEED) / MADV_FREE_REUSABLE / MEM_RESET + VirtualUnlock and never a decommit.
• Huge pages are singletons and stay out.

2. The free blocks of a large page stay while the page is in use, and only the sweep reads a clock for that. With (1) alone, a Bun.serve with 256 KiB bodies under sustained load discarded and refaulted 3 GB in 20 s: its event loop parks between requests, a park that is 100 ms after the last sweep is swept, and the buffers it had just freed were taken again at once. Kernel instructions per request went from 115 k to 152 k, page faults from 0.9 to 8.4 per request, throughput down by about a tenth. A thread parks, and is swept, far more often than it is idle. used does not tell a page that is in use from one that is not: it is the same at every park of a busy server.

The first version of this PR had the allocating thread leave a reading of CLOCK_MONOTONIC_COARSE in the page. That is gone: nothing on the allocating side reads a clock or makes a call, and _mi_clock_now_coarse with its plumbing is deleted (nothing else used it; stats.c is identical to bun-dev3-v2 again, so there is no macOS or Windows variant of it to get right either).

The sweep owns the time instead. It counts epochs: a process-wide counter (mi_holes_epoch in page.c) that an idle sweep moves on, in _mi_page_purge_holes_sweep_begin, when the current epoch has lasted for purge_holes_min_interval (100 ms) by the clock that the sweep reads anyway.

• Allocation (mi_malloc_generic_fallback, the out-of-line path that every allocation above 10 KiB takes; same place and same gate as before, block_size > MI_MEDIUM_MAX_OBJ_SIZE && page->reserved > 1): a relaxed load of the epoch, and a store of it into page->swept_state, under the tag that no (capacity,used) state and not MI_PAGE_SWEPT_NONE can have, only when the page does not carry that value already. A page that is allocated from all the time is written to once per epoch; when nothing sweeps (purge_holes off, or a process that never parks) the epoch stands still and the page is written to once. There is no separate check of the option: it would cost what the load and compare cost.
• Sweep (_mi_page_purge_holes): a large page whose epoch is the current one or the one before keeps its free blocks and the blocks it has not formed yet; one that is two or more epochs old is swept. An epoch lasts for the interval at least and only a sweep ends it, so with the third epoch there to be seen, the whole of the second one lies between the allocation and now: those blocks were left alone for the interval at least.
• Net effect: a busy thread puts each new epoch in its pages within the interval and keeps its buffers for good. After the last allocation the free space goes back between one and two intervals later, where something sweeps that often.

No epoch is shorter than the interval, whoever looks and wherever a sweeper is preempted; that is what the rule rests on. Any number of threads sweep at the same time (the scavenger for the parked ones, each thread that calls mi_on_thread_idle itself). mi_holes_epoch_advance: read the time the epoch began; if that is the interval ago, claim the move with a CAS of that time to a MOVING mark; increment the epoch; read the clock AGAIN and store that as the time the new epoch began. The epoch begins for everybody else at the increment, so its recorded beginning is never before its real one, and until it is recorded nobody else moves the epoch: a sweeper that is held up in between delays the next epoch and cannot shorten this one. (A sweeper that finds the mark waits for it, bounded, so that it sees the new epoch.) To the millisecond, as the clock is. A fork that lands while another thread holds the mark: the child records a new beginning (_mi_page_purge_holes_forked_child).

32 bits of the epoch are kept in the page and compared modulo 2^32: a page that was left for a multiple of 2^32 epochs (13 years of sweeping at 100 ms) reads as young once more and goes two epochs later. The epoch is process-wide like the clock and the option; a page of any subprocess compares with it. A page that reaches a sweeper does so through a release that the sweeper acquired (the park, the claim of an abandoned page), so the sweeper never reads an epoch older than the one in the page; if it did, the page would read as very old and go at once, which is a page fault and not an error. Stamps are written by the owner of a page only, never while a sweep has the page.

A thread that stays parked has no next sweep, and one sweep ends one epoch at the most, so the first sweep after an allocation leaves the page however long ago the allocation was. In the handoff (_mi_theap_sweep_parked) a park in which such blocks were left is therefore not marked as done. How that terminates:

• it is swept again purge_holes_min_interval after the END of the sweep before (holes_sweep_last is written after the work now), which is when that sweep can end the epoch that the one before began; the scavenger sleeps until then (the due time it gets back), nothing polls;
• until a sweep BEGAN two epochs on from the first sweep of the park (tld->holes_park_epoch, holes_sweep_epoch): everything the thread left when it parked is from the epoch of that first sweep at the latest, so such a sweep saw all of it as old and took it. What it still leaves was allocated from since, by other threads (the abandoned pages that the sweep passes as well), and is for whoever sweeps after that;
• that is the second sweep of the park when the first one ended an epoch and the third when it did not; more only where a sweeper was held up for an interval while it moved the epoch, and MI_PARK_SWEEPS_MAX = 6 whatever happens. park_swept counts them;
• with several parked threads: a thread that was just swept is not due for an interval, so the loop claims the next one in the list that is; each costs a bounded number of sweeps per park and at most one per interval, which is what a thread that parks all the time costs already.

The inline mi_on_thread_idle() follows the same rule, which is what its callers need from a busy loop (Bun's Windows event loop calls it at most every 100 ms: the buffers must stay). Nothing comes back for what an inline sweep leaves, so there is mi_on_thread_idle_pending(): true when the last sweep of this thread's heaps left such blocks, for a caller that is about to block for good to call mi_on_thread_idle() again, an interval after the call before returned, while it is true: two more calls as a rule. This is a change for a caller that sweeps once and then blocks (Bun's thread pool does, after 100 ms without work): that one call gave back the large pages in the first version of this PR when the last allocation was 100 ms ago; now it never does, because nothing tells it how long ago that was, and the thread's own pages that are not full stay as they are on bun-dev3-v2 until the caller uses _pending or the _start/_end pair (full pages are abandoned and are reached by the sweeps of the other threads of the heap). None of the three workloads below shows a difference from that.

Two earlier shapes of the rule did not survive independent review and are worth naming. Deferring in the handoff only left the inline path purging 58 MB in 3 s of a loop that frees and takes 256 KiB buffers (1.85 page faults per request against 0.06; now 0 discards and 0.07). "Take them at the second sweep in a row that finds the page untouched" never takes them for a caller that sweeps once and then blocks, and an abandoned page that two parked threads sweep one after the other got its two sweeps within a millisecond (with epochs both see the same epoch).

purge_holes_min_interval has two meanings now, both documented at the option: the least time between two sweeps of a thread's heaps in the handoff, and the least length of an epoch, which is how long the free blocks of a large page stay after the page was last allocated from (one to two of them), whoever sweeps and however often. 0 turns the hold off.

Numbers

Bun release build of main (ae7179126de5) with --local-deps=mimalloc; before = bun-dev3-v2 (58460467), clock = the first version of this PR (ae0ad8a6, the stamp is a reading of the coarse clock), epoch = this branch. All three measured again in one session on the same machine (64 cores, busy: load 50 to 110). RSS and anonymous from smaps_rollup, free and resident from a walk of every large page with mincore (a local diagnostic, not in this PR).

at idle	RSS before	clock	epoch	anon before	clock	epoch	free resident before	clock	epoch
typed arrays / strings / Buffers, 1 in 12 kept, idle 2 s	160.3 MB	89.1	88.7	125.3	54.3	53.9	71.6 MB	0.5	0.0
Bun.serve 256 KiB bodies, 40 kept, idle 15 s	77.9 MB	56.7	55.2	38.3	17.1	16.6	21.1 MB	0.0	0.0
same, nothing kept, idle 15 s	51.8 MB	45.9	46.2	15.3	6.5	6.7	8.2 MB	0.0	0.0

The epoch rule gives back what the clock rule gave back. In all three the reading 2 s into the idle is already the final one (free resident 0.0), and at the end of the load, before the idle, free resident is what it is before this PR (20.9 against 21.1 MB, 8.1 against 8.2 MB). Sampled every 50 ms in the first workload, anonymous memory goes 125.4 → 53.8 MB between 105 and 215 ms into the idle (clock: 125.4 → 54.3 between 105 and 210 ms): the last allocations of that workload are some time before it goes idle. (The last row is at 15 s and not at 60 s as in the first version of this text: by 60 s the machine had reclaimed file pages of some of the processes, 39 → 22..36 MB, which says nothing about the allocator. Anonymous at 60 s: 14.0 / 6.4 / 5.8 MB.)

Under sustained load (the same server, 16 connections, 20 s, 4 interleaved rounds; instructions and page faults of the server process from perf stat -p). The machine was at its busiest in the first round, which is the low end of every requests column and the high end of the faults and discarded columns; the other three rounds are in brackets:

	requests	user instr / request	kernel instr / request	page faults / request	discards	discarded under load
before	62.4 k (75.2 .. 95.9 k)	284.2 k .. 284.6 k	118.5 k .. 121.6 k	0.43 .. 1.18	98 .. 117	0.2 MB
clock	61.4 k (83.2 .. 100.3 k)	284.4 k .. 285.2 k	121.1 k .. 126.8 k	0.49 .. 1.71	234 .. 363	177 MB (90 .. 92 MB) = 7.7 (3.9 .. 4.0) MB/s
epoch	55.1 k (90.4 .. 95.7 k)	284.3 k .. 284.7 k	119.9 k .. 126.9 k	0.27 .. 1.97	162 .. 303	115 MB (20 .. 44 MB) = 5.0 (0.9 .. 1.9) MB/s

("discarded under load" is the total of the run less the 22 to 27 MB that go in the 1.5 s pause at its end.) Requests per second and instructions per request are the same for the three within the spread of this machine. The epoch rule discards less under load than the clock rule did, a quarter to a half of it: a page is kept for one to two intervals after it was last allocated from and not for exactly one, so the pages of a size class that are not the first in their queue survive a gap of 100 ms. For comparison, (1) without (2), measured in an earlier session against its own baseline:

	requests	kernel instr / request	page faults / request	discards	discarded
before	80.4 k .. 86.3 k	110.9 k .. 118.3 k	0.71 .. 1.03	92 .. 127	0.9 .. 1.1 MB
(1) without (2)	72.9 k .. 82.1 k	149.4 k .. 155.8 k	7.4 .. 9.3	4237 .. 5202	2.4 .. 3.0 GB

The allocating side. mi_malloc + mi_free of 128 KiB in a loop (every allocation takes the generic path into a large page), mimalloc built with Bun's flags (clang 21, -O3 -march=nehalem, C++, MI_FREE_USE_PAGEMAP), perf stat -e instructions:u, 5 interleaved runs of 20 M pairs each, the runs of one build equal to within 500 instructions:

	instructions per malloc + free	against before
before	197.6
clock	279.6	+82.0
epoch	210.6	+13.0

Of the 13, 8 are the gate (block_size > MI_MEDIUM_MAX_OBJ_SIZE && reserved > 1, compiled without a branch), which the clock version has as well; the stamp itself is mov epoch(%rip), mov, movabs, or, cmp swept_state, je (not taken once per epoch, then a mov to the page). The allocator micro benchmark (25 M malloc + 25 M free of 16..1039 bytes, same flags) executes 2.20735 G instructions in all three, to within 400: nothing is added to the fast paths of mi_malloc and mi_free (alloc.c, free.c and alloc-aligned.c are untouched).

The 30 s of idle after a load cost the process 14.4 M user + 20.3 M kernel instructions before, 15.4 M + 30.4 M with the clock rule and 16.0 M + 29.9 M with the epoch rule: the difference to before is the one-time madvise of 21 MB, and the one or two sweeps more of a park are 0.6 M. When the buffers are used again they fault back in, once. The sweep of a large page walks at most 42 free blocks.

Tests

New test/test-purge-holes-large.c (also run with MIMALLOC_PURGE_HOLES=0), in the style of test-purge-holes.c: the unit of small, medium and large pages; the unit arithmetic; the free blocks of the first, a middle and the largest large-page class are all purged at an idle sweep and, on Linux, not resident after it (mincore), while the survivors keep every byte; reuse after a purge (mi_malloc and mi_zalloc, no block handed out twice, zero where asked, writable all over, the purged blocks come back); frees from another thread into pages with holes, and the last blocks freed from there; mi_realloc shrinking in place and moving between classes next to holes; a page abandoned with free blocks in it, which the sweep of another thread discards, then used and freed from there; mi_heap_destroy and mi_heap_delete with holes (the accounting returns to its start); a double free of a purged block. About the hold, in epochs:

• the park handoff: the first sweep of a park takes nothing of the large pages and does not mark the park as done; the park is swept two or three times, each an interval after the one before, which takes every free block; then it is done and not swept again;
• the inline sweep: nothing of a large page goes at the first sweep after the allocations, nor at a second one right after it (two epochs cannot end within one interval), and mi_on_thread_idle_pending says so; an interval later the pages that were left alone go, and the one that a block was just taken from and put back stays; a page that is allocated from before every sweep keeps its blocks over four more epochs; they go at the first sweep after it was left alone for a whole epoch; a sweep after that has nothing to do;
• an abandoned page that is young is left by the sweep of another thread and taken two epochs later;
• sweepers at the same time: four threads that allocate from a large page of their own and sweep without a pause, next to the thread under test, which does the same and checks (a) from readings of the clock that lie around its readings of the epoch that no epoch is shorter than the interval, to the millisecond, and (b) that a page it allocated from before a sweep keeps its blocks in that sweep. Both fail when the exclusion in _mi_page_purge_holes_epoch_advance is removed and when the start of an epoch is recorded early.

What these cases check about time are lower bounds, which hold on a machine of any load (the first sweep after an allocation leaves the page whenever it comes, so that needs no time at all), and upper bounds only against a reading of the clock from BEFORE the allocation in question; the one liveness check (the epoch does move) runs until it has seen it, for 10 s at the most. CMake runs the file with MIMALLOC_ALLOW_THP=0 (it checks residency with mincore). With the hold rule gutted (no stamp, or no comparison) the park and inline cases fail; with the rate limit of the epoch gutted the "two sweeps within the interval" check fails.

The existing test-purge-holes.c sets purge_holes_min_interval to 0, which turns the hold off (on a 32-bit target its 64 KiB blocks are in a large page; the x86 clang-cl jobs of the CI found that), and no longer expects a 64 KiB block to be in a medium page where it is not (MI_ENABLE_LARGE_PAGES=0 in a debug build: that configuration failed on bun-dev3-v2 as well).

ctest passes in: Release, Debug MI_DEBUG=FULL, ASAN, UBSAN (C++), TSAN, MI_SECURE=ON, MI_SECURE=FULL release and debug, MI_GUARDED=FULL, C++ debug and release, MI_FREE_USE_PAGEMAP debug and release and C++ release, MI_STATS=0 MI_PROFILE=0, MI_PROFILE=OFF, MI_ENABLE_LARGE_PAGES=0 release and debug (Linux x64; 18 configurations). A 32-bit (-m32) compile of the library and the tests has no new warnings; on 32-bit the epoch variable is 32 bits and the time it began a 64-bit atomic like purge_expire.

For the pin in Bun

src/threading/ThreadPool.rs calls mi_on_thread_idle() once after 100 ms without work and then blocks without a timeout, and the Windows event loop calls it from a busy loop only. With this branch the pool threads should either park with mi_on_thread_idle_start() / _end() around the wait, or wake up to call mi_on_thread_idle() again while mi_on_thread_idle_pending() (an interval after the previous call returned, three more times at the most). Without that they keep the free blocks of their own large pages that are not full, as they do today on bun-dev3-v2.