perf(kv-router): serve the extension's Rust heap from a linked-in jemalloc

[PeaBrane]

Summary

Part 1 of a 4-part stack that scales the CRTC KV indexer's write capacity to 64 event workers. This part changes no CRTC code.

glibc malloc capped CRTC writes. Event workers serialized on arena locks (__lll_lock_wait_private), caused by cross-thread frees of producer-allocated events and by realloc growth while extending leaves. On a 96-core host, write capacity stayed flat at 58–67M block ops/s from 8 to 80 event workers. On the host used below, main reaches 45.7M block ops/s at 8 event workers and 50.6M at 64.

This PR serves the Python extension's Rust allocations from a jemalloc linked into the extension:

• The _core cdylib's #[global_allocator] is tikv-jemallocator (jemalloc 5.3.1), behind a default jemalloc feature. Its symbols carry the _rjem_ prefix and stay local to the extension, so Python and C/C++ libraries keep their own allocator.
• DYN_FRONTEND_JEMALLOC=1, or a jemalloc LD_PRELOAD, still moves Python and C/C++ onto a preloaded jemalloc. The Rust heap stays on the extension's jemalloc, so the process then runs two independent instances, each with its own arenas and options.
• On Linux with glibc, background_thread:true is built in. Up to four jemalloc_bg_thd threads return unused pages to the OS, so the allocating threads don't have to. _RJEM_MALLOC_CONF overrides the built-in options when the extension loads, and _RJEM_MALLOC_CONF=background_thread:false turns the threads off.
• jemalloc is built with disable_initial_exec_tls (global-dynamic TLS), so the extension, which is loaded with dlopen, does not use up the static TLS reserve.
• .cargo/config.toml pins jemalloc's page size to 64 KiB on aarch64 Linux, so arm64 wheels start on both 4 KiB and 64 KiB kernels. jemalloc 5.3.1 already defaults to that page size there; the pin protects against the default changing.
• Model teardown trims glibc arenas with malloc_trim, which does not reach the extension's heap. A new register_allocator_trim_hook lets the extension also purge every jemalloc arena.
• The router benches (mooncake_bench, active_sequences_bench, approximate_lru_bench) link the same jemalloc build with the same options, so they measure the shipped allocator. mooncake_bench and approximate_lru_bench also purge jemalloc's arenas when they quiesce the heap. INDEXER_BENCH.md documents the allocator and the purge.
• Docs (frontend configuration, performance tuning):
  • Document DYN_FRONTEND_JEMALLOC and add _RJEM_MALLOC_CONF.
  • Scope DYN_FRONTEND_JEMALLOC, LD_PRELOAD, and MALLOC_CONF to Python and C/C++.
  • Change the recommended MALLOC_CONF to background_thread:true, and advise against the narenas:32,... string.
• The wheel's NOTICES include jemalloc's BSD-2-Clause license.

Trade-off this part leaves for parts 2–3. At 64 event workers writes are 7.8× faster, so under overload, lookups queue behind coverage writers that take each node's state write lock. Overload lookup p99 rises from 7.6 µs to 636 µs (table below). Part 2 stops lookups from freeing detached subtrees, and part 3 takes coverage writes off the write lock.

At equal load, compared with main:

• Lookup p99 is 8–13% higher. This is mostly jemalloc's cost (see Allocator choice).
• Update p99 is 85–97% lower.
• With parts 2–4, keep-up lookup p99 comes back to within about 5% of main.

Measured costs. Against mimalloc, the earlier revision's allocator, in the indexer bench (comparator and details under Allocator choice):

• 2.4–5.7% more indexer CPU. At light load most of it is jemalloc's background purge threads.
• 7–12% higher lookup p99 at keep-up.
• 1.5× update p99 near capacity on the full stack.
• A slower bench load generator under overload: update issue lag p99 is 1.7–13× higher.

Against glibc on main, in the frontend benchmark at saturated concurrency 128, TTFT p50 rises 39% while ITL p50 falls 54% (TTFT p99 is unchanged; at concurrency 16, TTFT p50 falls 11%, indicative).

Memory trade-off. Every process that imports the extension carries its jemalloc. In the frontend benchmark below, against glibc on main:

• The frontend uses less memory: 0.84× at peak and 0.61× after 5 idle minutes.
• The mock worker starts 13% higher and peaks 20% higher. It drops below main only after the background threads purge (0.91× 15 s after load).

Background threads return freed memory on their own schedule (jemalloc's decay time is 10 s by default), so after a burst RSS can stay elevated for several seconds: at c128 the mock worker holds 195 MiB 1 s after the last request and 170 MiB 15 s after it. With _RJEM_MALLOC_CONF=background_thread:false, freed memory is not returned while the process is idle (frontend section below).

Validation

Method for the indexer A/Bs:

• Workload: replay of the Mooncake trace from lib/bench/kv_router/INDEXER_BENCH.md (128 inference workers, duplication factor 20, length factor 4, 128 lookup lanes).
• Host: 2-socket AMD EPYC 7742, with the bench on one socket. Event and lookup issuers run on 9 cores and the indexer on 55, with one logical CPU per physical core and memory on that socket's NUMA node. 64 event workers therefore oversubscribe the indexer's cores.
• Trials: fresh process per trial; arms interleaved by rotating the order each round. No other scheduled job shared the host during any trial, and no trial was excluded or retried.
• Statistics: ratios are of medians, with 95% bootstrap CIs. k/n counts the rounds in which this PR did better.
• Cells: "overload" compresses the trace so that the indexer falls behind, and the achieved rate then measures its capacity; where an arm keeps up anyway, its rate is a lower bound (footnotes say where). "Keep-up" is a 12 s window that both arms sustain.
• Host comparison: this host and placement differ from the hosts of the per-part tables in parts 2–4 and from the binding in INDEXER_BENCH.md, so compare ratios, not absolute rates.

This PR vs main, at a9ba39f538, 10 runs per arm under overload and 6 at keep-up:

Cell	Achieved block ops/s	Lookup p50	Lookup p99	Lookup p99.9	Update p99
Overload, 64 event workers (375 ms)	50.6M → 396M, 7.83× (CI 7.75–7.96, 10/10)	1.33	83.5 (7.6 → 636 µs)	2.28	0.071
Overload, 8 event workers (750 ms)	45.7M → 139M, 3.05× (CI 2.98–3.07, 10/10)¹	1.10	1.38	1.11	0.23
Keep-up, 64 event workers	equal	1.04 (n.s.)	1.13	1.12	0.15
Keep-up, 8 event workers	equal (main lags offered by 0.8%)	1.04	1.08	1.08	0.11

Indexer CPU at 26.7M block ops/s offered (4 event workers, both arms keeping up, 5 runs per arm) drops from 89.0 to 79.2 µs per 1k block ops, 0.89× (CI 0.88–0.90).

¹ This PR's load generator overran the 750 ms window in all 10 trials (784–832 ms), so the cell fails the bench's acceptance rule. The indexer stayed far behind (update p99 about 1.4 s), so the rate is most likely still set by drain, but treat it as indicative. At 64 event workers 1 trial overran; dropping it leaves 7.83×.

Allocator choice. Earlier revisions of this PR used mimalloc. The switch to jemalloc follows review:

• One allocator family. Review objected to carrying both a jemalloc and a mimalloc integration, and showed that with mimalloc the frontend's documented jemalloc memory fix (DYN_FRONTEND_JEMALLOC) no longer reached the Rust heap. With this PR the Rust heap runs jemalloc by default, and with the preload on, the Python and C/C++ allocations do too, configured through _RJEM_MALLOC_CONF and MALLOC_CONF respectively.
• Memory: against glibc on main, the frontend's retained memory falls (frontend section below). The frontend A/B has no mimalloc arm, so this PR makes no memory claim against mimalloc.

The mimalloc figures below come from mimalloc arms in the same indexer runs. Those arms keep mimalloc's default huge-page advice, which the earlier revision turned off in the extension.

What the switch keeps:

• Write capacity within about 2%, apart from one harness-limited cell.
  • On this PR alone at 64 event workers, jemalloc / mimalloc is 1.02 (CI 0.99–1.10, n.s.); both are 7.6–7.8× main. At 8 event workers it is 0.98, in a harness-limited cell (footnote ³).
  • On the full stack at 64 event workers, jemalloc equals mimalloc at 300 ms and is 2% lower at 375 ms, where both keep up; in the harness-limited 187 ms cell it is 7% lower. At 8 event workers it is equal (table below).
  • An earlier bracketed run on the full stack (AMD EPYC 7702P, preloaded jemalloc 5.3.0) agrees: guarded capacity was equal within its 5% resolution at 8 and 32 event workers. Just above that ceiling mimalloc passed more often, so it may keep a few percent more headroom.
• Background threads remove the update tail at moderate load.

  • In the earlier run, the full stack at 26.7M block ops/s with 4 event workers had these update p99 values:

    Allocator	Update p99
    jemalloc with background threads	0.53 ms
    mimalloc with huge pages on (the bench's default)	0.56 ms
    mimalloc with huge pages off (the earlier revision's setting)	2.11 ms
    glibc	3.73 ms
    jemalloc without background threads	4.75 ms

  • In this run, on the full stack at the same load, jemalloc's update p99 is 0.94× mimalloc-with-huge-pages (0.528 vs 0.563 ms, 5/5).

  • The advantage does not hold near capacity (table below).

What it costs, against mimalloc:

• Indexer CPU:

  • +5.6% at 4 event workers while keeping up (CI 1.044–1.062, 0/5), and +5.7% on the full stack. Most of it is the background threads' own CPU; excluding it, the cost is +1.8% (+0.8% on the full stack).
  • +2.4–3.6% under overload at 64 event workers.
  • The earlier run measured +2.5% against mimalloc with huge pages off, without background threads.

• Lookups at keep-up: p50 is 3–9% higher and p99 is 7–12% higher, in every keep-up cell (0/6 or 0/5 rounds better). mimalloc stayed within about 4% of main.

• Update p99 near capacity: 1.53× mimalloc on the full stack at 64 event workers (375 ms, both arms keeping up).

• Event producers. Under overload, the bench's event-issuing threads fall behind with jemalloc:

  • On this PR alone (instrumented runs), the busiest issuing thread is active 0.38 of the time against 0.27 at 64 event workers, and 0.55 against 0.26 at 8.
  • Update issue lag p99 is 1.7–13× higher across the overload cells.
  • With a no-op indexer at 64 event workers, the generator reaches 764M block ops/s with jemalloc, against the full 1.71G offered with mimalloc.

  This was not profiled; extra allocator work and lock blocking in the issue path are both possible. The slower generator is what fails the 8-event-worker cell's generator check above.

Measure	This PR: jemalloc / mimalloc	Full stack: jemalloc / mimalloc
Write capacity, overload, 64 event workers	1.02 (0.99–1.10, n.s.) at 375 ms	1.00 (0.98–1.02, n.s.) at 300 ms; 0.98 (0.97–0.99, lower in 9/10) at 375 ms²
Write capacity, overload, 8 event workers (750 ms)	0.98 (0.96–0.99, lower in 10/10)³	1.01 (0.98–1.03, n.s.)
Indexer CPU, keep-up, 4 event workers	1.056 (1.044–1.062); 1.018 excluding background-thread CPU	1.057 (1.049–1.063); 1.008 excluding it
Indexer CPU, overload, 64 event workers (375 ms, 5 runs)	1.036 (1.023–1.070)	1.024 (1.009–1.035)
Lookup p99, keep-up	1.08–1.12 (three cells)	1.11 (1.07–1.11), 4 event workers
Update p99, keep-up	0.93–1.03 (three cells)	0.94 (0.92–0.96, 5/5), 4 event workers
Update p99, 64 event workers (375 ms)	0.96 (n.s.; neither arm keeps up)	1.53 (1.27–2.03, 0/10; both arms keep up)

² Both stack arms kept up at 375 ms, so this ratio compares drain time. At 187 ms the ratio is 0.93, but the jemalloc stack's load generator overran its window in every trial (and the mimalloc stack's in 5 of 10), so that cell is harness-limited. The 300 ms window was chosen after that run so that the jemalloc generator stays on schedule. It sits below saturation: the mimalloc stack achieved 896M block ops/s at 187 ms and 830M at 300 ms.
³ Harness-limited; see footnote ¹. Two smaller replicates, in which this PR's generator also overran, gave 0.99 (n.s.) and 0.96, 3 runs each.

Frontend end to end. Setup:

• Topology: the repo's dynamo-frontend-benchmark topology, with the frontend pinned to 4 cores in KV-router mode and one mocker with 4 workers.
• Workload: multi-turn chat with 60k-token prompts (48k shared, 12k per user), 4 turns, and 500 output tokens, at concurrency 16 and 128.
• Host: 2-socket AMD EPYC 7742.
• Trials: fresh topology per trial; arm order balanced across rounds, with one seed per round; 4 runs per arm at c128 and 2 at c16; co-tenant-gated windows; every arm built with the same features.
• Arms: this run has no mimalloc arm.
• Statistics: a c128 difference counts only when all 4 rounds agree and the per-arm ranges do not overlap. c16 figures are indicative.
• RSS before load: the settled value 60–80 s after the frontend is ready, before warmup.
• Comparator: jemalloc with background purging, against glibc without periodic malloc_trim. Absolute frontend RSS depends on the tokenizer cache size (1 GiB here).

Comparison	Throughput	Latency	Frontend RSS before load	Frontend peak RSS	Frontend RSS after 5 idle min
This PR vs main, c128	1.02× (n.s.)	p99 0.93×; TTFT p50 1.39×, ITL p50 0.46×⁴	0.55× (347 → 190 MiB)	0.84× (2382 → 1998 MiB)	0.61× (2381 → 1450 MiB)
This PR vs main, c16 (indicative)	1.05×	p99 0.96×, TTFT p50 0.89×	0.49×	0.87×	0.77×
This PR with DYN_FRONTEND_JEMALLOC=1 vs this PR, c128	1.00× (n.s.)	n.s.	1.02×	1.00× (n.s.)	1.01×
Stack top (#15611) vs main, c128	1.02× (n.s.)	p99 0.94×; TTFT p50 1.38×, ITL p50 0.50×	0.55×	0.83×	0.60×

• Correctness: 0 request errors, cancellations, crashes, panics, or OOM kills across 33 runs. Every frontend and mocker exited cleanly on SIGTERM.

• Frontend CPU per request: at c128 the frontend is saturated (3.93–3.96 of 4 cores), so CPU per request tracks throughput: 0.99× (n.s.). Differences under about 3% are not detectable. At c16, 0.94× (indicative).

• Mock worker, which also loads the extension, at c128:

  Point	Ratio vs main	RSS
  Before load	1.13×	98 → 111 MiB
  Peak	1.20×	187 → 225 MiB
  1 s after the last request	1.04× (n.s.)
  15 s after the last request	0.91×
  After 5 idle minutes	0.90×	187 → 168 MiB

  At c16 (indicative) its peak is 1.14× and its RSS 1 s after load is 1.11×. Mock worker CPU per request at c128 is 0.98× (n.s.).

• Background threads. _RJEM_MALLOC_CONF=background_thread:false in the frontend and mocker, against the default, at c128 with 2 runs each (indicative):

  • Throughput is 1.01× and frontend CPU per request 0.98× (both n.s.); latency p99 is 0.98×.
  • Without the threads, freed memory is not returned while idle. After 5 idle minutes the mock worker holds 216 MiB against 172 MiB (1.25×). The frontend settles at 273 MiB against 190 MiB before load (1.44×) and ends idle 3% higher, while its peak is 2% lower.

• Two jemalloc instances (DYN_FRONTEND_JEMALLOC=1, third row). Ubuntu's jemalloc 5.3.0 serves Python and C/C++, and the extension's jemalloc keeps the Rust heap.

  • Throughput and latency match this PR alone. Frontend RSS before load and after 5 idle minutes is 0.6–1.9% higher; the peak is unchanged.
  • The preload ran with its default options, which start no background threads. With the documented MALLOC_CONF=background_thread:true, it adds up to four more.

• Inference engines: vLLM, SGLang, and TensorRT-LLM processes that load the extension are not covered here; they are left to the container e2e CI.

⁴ At c128 the jemalloc arms have higher TTFT and lower ITL in every round: TTFT p50 rises 39% (mean 23%) while ITL p50 falls 54%. TTFT p99 is unchanged, end-to-end latency p50 and mean are n.s., and latency p99 is 7% lower. Why time shifts toward the first token at saturation is not established. At c16, TTFT p50 falls 11% (indicative). Concurrencies between 16 and 128 were not measured.

Extension:

• Binary checks: readelf -d shows no STATIC_TLS flag. The dynamic symbol table exports no allocator entry points and no _rjem_ symbols; every jemalloc (_rjem_) symbol is local.

• test_allocator.py. In fresh interpreters it checks that:

  • a RadixTree's Rust heap bypasses glibc;
  • background threads start;
  • _RJEM_MALLOC_CONF overrides the built-in options;
  • a preloaded jemalloc leaves the Rust heap on the extension's jemalloc;
  • the TLS and symbol checks above hold.

  6 of 6 pass with Ubuntu's libjemalloc 5.3.0 available; without it the preload test skips (5 passed, 1 skipped).

• KV binding tests (test_kv_bindings, test_kv_event_publisher, test_kv_router_config, test_kv_router_stream_errors, test_bindings_install): 41 of 41 pass in a release build with the select-service feature, which the SelectionService tests need. A build without that feature skips those 16.

• Rust tests: cargo test --features ais-forward-pass in lib/bindings/python: 61 passed.

• manylinux_2_28: a minimal cdylib with the same jemalloc features, built in the manylinux_2_28 image, needs no glibc symbol newer than 2.28, has no STATIC_TLS, and starts its background thread.

• arm64: the arm64 build was not exercised. The page-size pin was checked to reach jemalloc's configure for aarch64-unknown-linux-gnu.

Re-check against main's tests. Part 4 (#15611, 41803f6f2d, which contains this PR; release build) and main (e61319d830), built with the same features:

• The CPU-only frontend tests (121) and the router mocker e2e tests (70) give the same result for every test on both: 85 passed and 28 skipped, and 64 passed and 6 skipped.
• The 8 frontend tests that do not pass on main in this CPU-only environment fail the same way on the stack. Two need torch and pass in 3 of 3 runs once it is installed; five disaggregated setups time out, and one metadata check returns 404.
• The allocator and KV binding test results above come from the same build. The mocker replay tests and the race soak were re-run on the stack too (perf(kv-router): cut CRTC writer per-block work #15611).

Notes

• Fork: importing dynamo starts a jemalloc background thread when the extension loads, so the process is multi-threaded from the start. Python 3.12 and later warn on os.fork(), and children forked without exec run without background threads. 200 forks while another thread kept a RadixTree busy completed with no hang or failure.
• TLS in the benches: the linker relaxes jemalloc's TLS in the bench executables to local-exec. The extension keeps global-dynamic TLS (__tls_get_addr calls), so allocator costs in the extension can be somewhat higher than the benches show.
• Platforms: background threads are built in only on Linux with glibc. macOS and musl builds run with jemalloc's defaults; Dynamo ships a wheel for neither.
• kvbm boundary: DistributedRuntime.to_capsule hands kvbm, which keeps glibc, a Weak pointing at memory from _core's allocator. Today the upgrade never succeeds (a separate, older bug), so no memory crosses allocators. A comment now marks the boundary.
• License metadata: the SBOM still lists tikv-jemalloc-sys as MIT OR Apache-2.0, from its Cargo metadata, while the NOTICES carry jemalloc's BSD-2-Clause text.
• Statistics: the indexer A/Bs have 3–10 fresh-process runs per arm (the frontend A/B 2–4), and many comparisons with no multiplicity correction. Treat allocator differences under 3% as tentative unless both the single-part and full-stack pairs show them. Arm order rotated rather than shuffled, so each jemalloc arm usually ran right after its mimalloc counterpart.
• Indexer-bench memory: the bench's RSS is dominated by trace preparation and by the purge before the measured window. These runs therefore support no claim, in either direction, about the indexer's own memory.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: ai-dynamo/dynamo/.coderabbit.yaml
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 7dfbccc2-47e1-4142-a601-258086067776

📥 Commits

Reviewing files that changed from the base of the PR and between 1cbc578 and ae71f77.

⛔ Files ignored due to path filters (3)

• Cargo.lock is excluded by !**/*.lock
• lib/bindings/kvbm/Cargo.lock is excluded by !**/*.lock
• lib/bindings/python/Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (30)

• lib/bench/Cargo.toml
• lib/bench/kv_router/INDEXER_BENCH.md
• lib/bench/kv_router/active_sequences_bench.rs
• lib/bench/kv_router/approximate_lru_bench.rs
• lib/bench/kv_router/mooncake_bench.rs
• lib/bindings/python/Cargo.toml
• lib/bindings/python/rust/lib.rs
• lib/kv-router/Cargo.toml
• lib/kv-router/src/indexer/compressed_radix.rs
• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/AGENTS.md
• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/README.md
• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/block_lookup.rs
• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/children.rs
• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/coverage.rs
• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/dump.rs
• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/matches.rs
• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/mod.rs
• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/node.rs
• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/remove.rs
• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/repair.rs
• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/state.rs
• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/store.rs
• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/sync_impl.rs
• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/tests.rs
• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/types.rs
• lib/kv-router/src/lib.rs
• lib/kv-router/src/lookup_update.rs
• lib/kv-router/tests/crtc_thread_exit.rs
• lib/llm/src/discovery.rs
• lib/llm/src/discovery/allocator.rs

💤 Files with no reviewable changes (2)

• lib/kv-router/src/lib.rs
• lib/kv-router/src/lookup_update.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

Walkthrough

The pull request adds mimalloc configuration and trim hooks for Python bindings and router benchmarks. It also changes the compressed radix tree to track worker coverage by reusable slots, protect child access with epochs, and update store, lookup, removal, repair, and dump operations to use those mechanisms.

Changes

Mimalloc setup and benchmark alignment

Layer / File(s)	Summary
Configure mimalloc and register trim hooks lib/bindings/python/Cargo.toml, lib/bindings/python/rust/lib.rs, lib/llm/src/discovery.rs, lib/llm/src/discovery/allocator.rs	The Python binding enables mimalloc by default and installs it as the global allocator when enabled. It registers a trim hook, which the Linux GNU allocator trim callback invokes after malloc_trim.
Align router benchmarks with mimalloc lib/bench/Cargo.toml, lib/bench/kv_router/*	The router benchmarks use mimalloc and call mimalloc collection during Linux heap quiescence. The benchmark guide discusses allocator comparisons and event-worker scaling.

Compressed radix-tree coverage and reclamation

Layer / File(s)	Summary
Add slot coverage and lookup primitives lib/kv-router/Cargo.toml, lib/kv-router/src/indexer/compressed_radix.rs, lib/kv-router/src/indexer/concurrent_radix_tree_compressed/{AGENTS.md,README.md,block_lookup.rs,coverage.rs,state.rs,types.rs}, lib/kv-router/src/{lib.rs,lookup_update.rs}	The tree adds slot-based coverage, cutoff state, slot mappings, and a BlockLookup map. Related types and documentation use the new coverage model.
Protect child access with epochs lib/kv-router/src/indexer/concurrent_radix_tree_compressed/{children.rs,node.rs,mod.rs,sync_impl.rs}, lib/kv-router/tests/crtc_thread_exit.rs	Child snapshots and detached nodes use epoch-protected access and deferred reclamation. Tree teardown and event processing drain retired state.
Use slots across store, lookup, and dump paths lib/kv-router/src/indexer/concurrent_radix_tree_compressed/{node.rs,store.rs,matches.rs,dump.rs,mod.rs,sync_impl.rs,README.md}	Store and lookup paths use slots to track coverage and resolve workers. Split updates and tree dumps use slot-table coverage information.
Coordinate removal, repair, and slot reuse lib/kv-router/src/indexer/concurrent_radix_tree_compressed/{remove.rs,repair.rs,tests.rs,README.md}	Removal sweeps slot coverage and coordinates slot release. Lookup repair redirects entries still pointing to stale nodes. Tests cover concurrent removal, slot reuse, lookup results, and tree dumps.

Priority: ➖ Normal

Estimated code review effort: 5 (Critical) | ~150 minutes

Merge Risk: ⚪ Minimal · up to ae71f

This change makes mimalloc the allocator and reworks CRTC coverage and reclamation for write scaling. The review found no concrete defects. The PR does not show benchmark results for the preferred 8-event-worker setup or a keep-up read-path replay; maintainers should confirm those results before relying on the change at default settings.

🚥 Pre-merge checks | ✅ 2 | ❌ 3

❌ Failed checks (3 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 52.13% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 305 functions across 22 files. (6 skipped…	Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check	⚠️ Warning	The title says the extension uses jemalloc, but the changes and objectives describe mimalloc integration and major CRTC updates. It therefore misidentifies the allocator and does not summarize the mai…	Retitle the pull request to describe the CRTC scaling changes and the mimalloc integration, rather than jemalloc.
Description check	⚠️ Warning	The description is detailed, but it conflicts with the changes: it says the PR uses jemalloc and changes no CRTC code, while the summary shows mimalloc integration and extensive CRTC changes. It also …	Revise the description to match this pull request’s actual mimalloc and CRTC changes, update any benchmarks and validation claims that do not apply, use the required template sections, and add a Related Issues reference such as Closes #XXXX…

✅ Passed checks (2 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 52.13% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 305 functions across 22 files. (6 skipped: 6 unsupported.)

Full details: Title check

Explanation

The title says the extension uses jemalloc, but the changes and objectives describe mimalloc integration and major CRTC updates. It therefore misidentifies the allocator and does not summarize the main changes.

Full details: Description check

Explanation

The description is detailed, but it conflicts with the changes: it says the PR uses jemalloc and changes no CRTC code, while the summary shows mimalloc integration and extensive CRTC changes. It also omits the required Related Issues reference and the template headings.

Resolution

Revise the description to match this pull request’s actual mimalloc and CRTC changes, update any benchmarks and validation claims that do not apply, use the required template sections, and add a Related Issues reference such as Closes #XXXX or Relates to #XXXX.

• Fix all pre-merge checks with AI

---

_{Comment @coderabbitai help to get the list of available commands.}

[devin-ai-integration]

❌ Dynamo PR CI failed — run 37599990823 (attempt 1) on ca51e42427

Gate checks: ❌ backend-status-check · ✅ deploy-status-check · ❌ dynamo-status-check

Framework	Build
vLLM	⏹️ 1
SGLang	⏹️ 1
TRT-LLM	⏹️ 1

Other	Jobs
dynamo-runtime	⏹️ 1
dynamo-sidecar	⏹️ 1
frontend	❌ 1
planner	⏹️ 1
triton-runtime	⏹️ 1

Failure details

1 job failed: the frontend EPP image step hit its 60-minute timeout with no compile or build error. The 7 cancelled build jobs are not fail-fast cancels; each hit its own job time limit (1h0m or 1h30m, per its annotation).

❌ frontend / Build multi-arch cpu: `Build EPP Image (For Frontend Only)` timed out after 60 minutes

Job: frontend / Build multi-arch cpu · Failed step: Build EPP Image (For Frontend Only) · Logs: gh run view --job 112721894591 -R ai-dynamo/dynamo --log-failed

09:22:01 + make -C deploy/inference-gateway/ext-proc image-multiarch-push IMAGE_REPO=***/dynamo-epp ...
...
#58 [linux/arm64 builder 11/13] COPY --from=dynamo deploy/inference-gateway/sidecar/ ...
#58 DONE 0.1s            <- ~23 min with no output before the next step
...
#60 [linux/arm64 builder 13/13] RUN ... cargo build ...
#60 DONE 352.1s
#67 exporting to image
#67 exporting manifest list sha256:7be33e00a5ae... done
10:21:46 #67 DONE 1.6s
10:22:13 ##[error]The action 'Build EPP Image (For Frontend Only)' has timed out after 60 minutes.

The EPP image built for both amd64 and arm64 (Rust compile done, sccache hit rate 100%, image exported) and was killed 27 s after the export, right at the 60-minute step limit. There is no error in the build itself. Most of the time went to long silent gaps in the BuildKit builder (about 23 min after #58 on arm64, plus two gaps of about 6 min on trivial COPY steps). This looks like builder or infrastructure slowness, and since the other build jobs hit their own time limits too, a rerun of failed jobs is likely enough.

For agents

{"pr": 15606, "run_id": 37599990823, "run_attempt": 1, "head_sha": "ca51e42427a88bc9a6ca1d61bed491a3a9ba836e", "failures": [{"job": "frontend / Build multi-arch cpu", "job_id": 112721894591, "failed_step": "Build EPP Image (For Frontend Only)", "signature": "step timed out after 60 minutes (EPP multi-arch buildx build/push)", "tests": [], "log_cmd": "gh run view --job 112721894591 -R ai-dynamo/dynamo --log-failed"}], "cancelled_by_timeout": ["vllm-runtime / Build multi-arch cuda13.0", "dynamo-sidecar / Build multi-arch cpu", "triton-runtime / Build multi-arch cpu", "trtllm-runtime / Build multi-arch cuda13.1", "sglang-runtime / Build multi-arch cuda13.0", "planner / Build multi-arch cpu", "dynamo-runtime / image / Build multi-arch cuda13.0"]}

_{Posted automatically by Devin for run 37599990823. Updated on every full-CI run of this PR.}

[PeaBrane]

@coderabbitai review

[coderabbitai]

⚠️ Action not completed

No files to review.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[dmitry-tokarev-nv]

Approve at 4c3e7e280f6a8ccc334ccd930bc50cc753961ade. The bar holds with one P2 (inline) and one P3, and the open items below do not block.

• [P2] Inline at lib/bindings/python/rust/lib.rs:4-6: with jemalloc preloaded, the frontend's Rust heap goes to mimalloc, so the documented jemalloc fix for frontend memory no longer works.
• [P3] lib/bindings/python/rust/lib.rs:1621: to_capsule gives kvbm._core a Rust Weak<DistributedRuntime> from this extension. The two extensions now use different Rust allocators. If upgrade() ever succeeds and KVBM drops the last reference, glibc frees memory that mimalloc allocated. This cannot happen today only because upgrade() always fails, on main too. Please note the constraint at to_capsule.
• Open, not graded: the new test_allocator_thp.py runs only in dynamo-runtime / test / sequential (amd64 and arm64). That job did not run on this head, because the dynamo-runtime image build stopped at the 1-hour job limit. The test passed in the head's vllm-runtime-test image.
• Open, not graded: xpu-ci-vllm was skipped, so this PR's CI built no XPU wheel with the default mimalloc feature.

What I read, and what I measured on the CI-built images of the head and the merge base.

I read all lines of both commits: the 340 diff lines of ee55ac52 and the 181 of 4c3e7e28. I also read all of allocator.rs, the libmimalloc-sys 0.1.49 build script, and the TLS, THP and mi_collect code of the bundled mimalloc. Of the bench sources I read only the changed lines. I did not judge the bench tables. I leave them to the performance team.

New crates

All three checksums match a download from crates.io.

crate	version	license	checksum
cty	0.2.2	MIT OR Apache-2.0	b365fabc…
libmimalloc-sys	0.1.49	MIT	6a45a52f…
mimalloc	0.1.52	MIT	2d4139bb…

libmimalloc-sys bundles mimalloc C 3.3.2 (MI_MALLOC_VERSION 30302) and builds v3 unless the v2 feature is set. Its build.rs passes -ftls-model=local-dynamic with local_dynamic_tls, and -ftls-model=initial-exec without it. cargo-deny printed bans ok, licenses ok in rust-tests for . and lib/bindings/python. The CI-built image of ee55ac52 lists all three crates in /legal/NOTICES-Rust.txt and osrb-deps.csv, so no checked-in list needs a change. The second commit changes no Cargo file.

Built _core

From the CI images. Both head commits give the same numbers:

build	STATIC_TLS	TLS relocations	allocator exports (nm -D)
head x86_64	no	38 R_X86_64_DTPMOD64	none
merge base x86_64	no	38 R_X86_64_DTPMOD64	none
head aarch64	no	68 R_AARCH64_TLSDESC	none
merge base aarch64	no	63 R_AARCH64_TLSDESC	none

The head TLS segment grows by 0x28 bytes, the size of mimalloc's block, and mi_collect calls __tls_get_addr. Control: the same mimalloc C built with -ftls-model=initial-exec has STATIC_TLS and 5 R_X86_64_TPOFF64 on x86_64. On aarch64 it has 3 R_AARCH64_TLS_TPREL64 and no STATIC_TLS flag. On aarch64, use the relocation type, not the flag.

Static TLS in a real process

I loaded torch, CUDA and vllm first. Then the head _core imports on x86_64, in the head vllm-runtime image. It also imports on aarch64, in an arm64 vllm-runtime-test image on CPU. On x86_64 this holds also with GLIBC_TUNABLES=glibc.rtld.nns=1:glibc.rtld.optional_static_tls=0. On aarch64 that setting stops torch itself (libc10.so: cannot allocate memory in static TLS block). Failing control: an initial-exec library of 2048 bytes (x86_64) or 1024 bytes (aarch64) fails with cannot allocate memory in static TLS block.

Transparent huge pages

In the runs below, mimalloc advises 1024 MiB for huge pages in the frontend and none in the dynamo.vllm worker. The new test passes with the head _core. It fails with the _core of ee55ac52 (advised_mib: 1024) and with the merge-base _core (advised_mib: 0), so both of its asserts can fail.

Memory

These numbers are for 4c3e7e28, in its CI-built vllm-runtime image, with the same setup as the inline comment. RSS in MiB, glibc, one value per run:

process	_core	start	after load	after 60 s idle	after worker teardown
frontend	merge base	302	469	458	244
frontend	head	195, 189	703, 723	682, 414	370, 388
dynamo.vllm worker	merge base	1500	1517	1517	-
dynamo.vllm worker	head	1497, 1484	1525, 1511	1525, 1511	-

The vLLM engine-core process does not load _core. It is the control, at 2262 to 2276 MiB after load in all 8 runs. At ee55ac52, before the huge-page commit, the worker held 1617 to 1628 MiB after load. The merge-base glibc frontend varies between images: 469 MiB after load here, 545 and 552 MiB in the ee55ac52 image. Without huge pages (MIMALLOC_ALLOW_THP=0), the head frontend holds 395 MiB after load. So huge pages cost it about 310 to 330 MiB here. The PR body estimates about 120 MiB for its own workload.

No limit in this repository is near these numbers. The operator sets no default memory for frontend or worker pods. The Helm charts run no process that loads _core. Of the 78 frontend services in the DGD manifests of this repository, 70 set no memory and 8 set limits of 32Gi to 100Gi. Not measured: SGLang and TensorRT-LLM workers, and kv_state_agent in examples/backends/vllm/deploy/kvcr/agg-memory-service.yaml (4Gi limit with a 2 GB shared-memory pool).

Allocator boundary

I searched the first-party crates and the 734 third-party crates of the production feature set. C buffers are copied and then freed with C free (nixl-sys, video-rs, ffmpeg-next). Rust boxes that C holds are freed by Rust callbacks (zmq, ffmpeg-next). _core has no DLPack or numpy hand-off. The only Rust object that crosses into another extension is the capsule in the P3.

DYN_FRONTEND_JEMALLOC=1

The head frontend starts with jemalloc mapped and serves 1200 of 1200 requests. The trim runs on a dedicated thread in allocator.rs. It starts 100 ms after the last model teardown guard drops. It runs malloc_trim(0) and then the new hook, mi_collect(true). Each run logged one trim after the worker exited. jemalloc defines no malloc_trim, so the call trims glibc's arenas. A Rust probe with the same trim wiring and 8 idle worker threads: the hook releases 366 MiB of freed mimalloc memory down to 31 to 35 MiB (glibc with malloc_trim: 320 to 0.7 MiB).

CI

On 4c3e7e28, Pre Merge is green (run 37392668489). In run 37392671301, the vLLM, SGLang and TensorRT-LLM image builds, compliance audits and runtime tests are green on amd64 and arm64. The frontend, planner and dynamo-runtime image builds stopped at the 1-hour job limit with no build error, so dynamo-status-check and backend-status-check are red. The same timeouts stopped attempts 1 to 3 of run 37237854280 on ee55ac52, and its attempt 4 was green. The deploy tests were skipped on both heads.

Merge with main

main at afd07588a4 changed Cargo.lock and lib/bench/Cargo.toml since the merge base. A local merge with the head is clean, and both lock files pass cargo tree --locked. Control: the same command fails after I delete one line from a merged lock.

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-01a115af-cd67-731b-8974-f37046304c5b.docs.buildwithfern.com/dynamo/dev

[dmitry-tokarev-nv]

Approve at ca51e42427a88bc9a6ca1d61bed491a3a9ba836e. Our P2 and P3 from the last round are fixed, and the bar holds with one new P3.

• [P2, fixed] r1001 thread at lib/bindings/python/rust/lib.rs:4-6: the frontend's Rust heap now returns memory with or without DYN_FRONTEND_JEMALLOC=1. The numbers are in that thread.
• [P3, fixed] lib/bindings/python/rust/lib.rs:1589-1592: the new comment at to_capsule names the allocator boundary. kvbm's upgrade() still fails on this head, so no memory crosses allocators.
• [P3] lib/bindings/python/tests/test_allocator.py:220: the arm64 linker sets no STATIC_TLS flag, so this test passes for an initial-exec build that fails to load. Please also fail on R_X86_64_TPOFF64 and R_AARCH64_TLS_TPREL64 relocations from readelf -r.
• Not run on this head: CI built the wheels for both architectures, but every image job stopped at its time limit. So dynamo-runtime / test / sequential, which collects test_allocator.py, and the engine runtime tests did not run.

What I read, built and ran.

• I read all 846 lines of the three-dot diff: 18 files, +513/-25, as the files API reports. I also read the tikv-jemalloc-sys build script and the NOTICES code in container/compliance.
• The 7 new commits sit on 4c3e7e28 without a force push, and the merge base is still e61319d8.
• No CI image of this head exists. So I built _core for this head, the merge base and the r1001 head on the GPU box. I used maturin 1.15.0 and CI's feature list without media-ffmpeg. I ran them in the CI-built vllm-runtime image of 3cff862bc8, whose _core sources and lock file equal the merge base. "CI-built merge-base _core" below means that image's own _core.
• I did not judge the bench and frontend performance tables in the description.

New crates, build flags and the wheel NOTICES.

crate	version	checksum	declared license
tikv-jemalloc-sys	0.7.1+5.3.1-0-g81034ce1f1373e37dc865038e1bc8eeecf559ce8	1a2825c7…	MIT/Apache-2.0
tikv-jemallocator	0.7.0	249f09e4…	MIT/Apache-2.0

• Both lock files add these two packages and remove none. Against the r1001 head, they also drop cty 0.2.2, libmimalloc-sys 0.1.49 and mimalloc 0.1.52.
• Both checksums match the crate files from crates.io. tikv-jemalloc-sys vendors jemalloc 5.3.1 (ChangeLog entry of Apr 13, 2026), with its BSD-2-Clause license in jemalloc/COPYING.
• The build script ran configure with --with-malloc-conf=background_thread:true (feature background_threads), --disable-initial-exec-tls (feature disable_initial_exec_tls), --with-jemalloc-prefix=_rjem_, --with-private-namespace=_rjem_ and --disable-stats.
• On aarch64, the .cargo/config.toml entry adds --with-lg-page=16, and the library reports a 65536-byte page. With the value changed to 14, it reports 16384. Without the entry, configure's own aarch64 default still gives 65536. On x86_64, the entry has no effect, and the probe library is byte-identical with the mutant.
• The wheel_builder.Dockerfile change copies jemalloc/COPYING into the license folder of tikv-jemalloc-sys as jemalloc-COPYING. I ran the harvest and bundle_wheel_notices as the Dockerfile does, on a wheel of this head that maturin 1.15.0 built.
• The wheel's SBOM names the crate 0.7.1+5.3.1-0-g81034ce…, which matches the folder name. THIRD-PARTY-RUST-LICENSES.txt then carries jemalloc's BSD-2-Clause text once. With the harvest of the base Dockerfile, it does not.
• In CI, the image jobs of this head logged the harvest and wheel NOTICES bundled into 2 wheel(s) for both architectures before they stopped.
• In Pre Merge, cargo-deny 0.19.0 printed bans ok, licenses ok for . and lib/bindings/python.

Built artifact: ELF checks, and loading after torch and vLLM.

x86_64, this head's _core:

• readelf -d shows no STATIC_TLS flag. The dynamic TLS relocations are 39 R_X86_64_DTPMOD64, against 38 in the merge base.
• The jemalloc objects use only R_X86_64_TLSGD, R_X86_64_TLSLD and R_X86_64_DTPOFF32, which are the global-dynamic and local-dynamic models.
• nm -D lists 29 defined symbols: three PyInit_* and 26 crc_fast_*. It lists no malloc, free, mallctl, je_* or _rjem_*. All 584 _rjem_ symbols are local.
• Positive control: the same search on the image's libjemalloc.so.2 lists malloc, free, mallctl and more.

aarch64: no CI artifact of this head exists. So I built a probe library with the same two crates, features and .cargo/config.toml (Debian 12, GNU ld 2.40). It has no STATIC_TLS flag, 6 R_AARCH64_TLSDESC, only TLSDESC relocations in the jemalloc objects, and no exports.

Loading after torch, CUDA and vllm:

• In the x86_64 vllm-runtime image with a GPU, this head's _core loads and starts one jemalloc_bg_thd thread. The probe loads too. Both also load with GLIBC_TUNABLES=glibc.rtld.nns=1:glibc.rtld.optional_static_tls=0.
• In an arm64 vllm-runtime-test image on CPU, the probe loads. It reports a 65536-byte page and opt.background_thread true.
• Failing control on both architectures: the probe built without disable_initial_exec_tls fails with cannot allocate memory in static TLS block.

The new P3: the static TLS test cannot fail on arm64.

The initial-exec probe on aarch64 has 1 R_AARCH64_TLS_TPREL64 and no STATIC_TLS flag, but it fails to load after torch and vLLM. None of the 307 libraries that load in the arm64 engine image carries the flag. On x86_64, the same probe carries the flag and 1 R_X86_64_TPOFF64. Three x86_64 _core builds have neither relocation: this head, the merge base, and the CI-built merge-base _core. So the extra assert passes on them.

Tests, fork and the kvbm capsule.

• test_allocator.py ran under the repository's pytest settings in the vllm-runtime image. With this head's _core, 6 of 6 pass. With the CI-built merge-base _core, 3 fail: glibc growth 23 MiB, 0 background threads, and preloaded-jemalloc growth 27 MiB. So the tests catch a lost allocator.
• approximate_lru_bench, which Pre Merge does not build, passes cargo check --locked with the approximate-lru feature.
• After import dynamo._core, the process has one more thread, jemalloc_bg_thd. os.fork() then warns This process (pid=7) is multi-threaded, use of fork() may lead to deadlocks in the child., as the docs say. A test that forks after the import still passes under the repository's filterwarnings = error.
• For the CI-built merge-base _core and this head, kvbm's extract_distributed_runtime_from_obj fails with runtime is no longer alive (weak ref upgrade failed).

Merge with main, and CI on this head.

• git merge-tree --write-tree with main at f787ba7568 is clean. Since the merge base, main changed both lock files and three other files of this PR.
• On that merge, cargo tree --locked passes for the root and the bindings workspaces. Control: it fails after I remove one dependency line from the merged bindings lock.
• Pre Merge passed on the merge with 548a010548, with cargo test --locked and clippy with --locked.
• PR run 37599990823 stopped. The frontend job failed in Build EPP Image (For Frontend Only) at its 60-minute step limit, 27 s after the image export. The other image jobs were cancelled at their time limits after their wheel stages passed. No test job ran, so backend-status-check and dynamo-status-check are red.
• The PR runs of the stacked PRs #15609, #15610 and #15611 started in the same minute and were cancelled too. Nothing in the logs points at this diff.
• Passed on this head: Pre Merge, DCO, copyright, codeowners, Lint PR, Docs link check, Fern Docs and CodeQL. PR-XPU ran only its guard job.

[PeaBrane]

I'm closing this PR to separate the allocator from the CRTC stack.

• The stack. perf(kv-router): epoch-reclaimed CRTC child maps and borrowed reads #15609 → perf(kv-router): dense rank slots with atomic coverage bitsets #15610 → perf(kv-router): cut CRTC writer per-block work #15611 are now rebased directly onto main and change no allocator, so the default stays glibc malloc. The test-only race soak in test(kv-router): add an opt-in CRTC race soak #15614 stays a separate draft.
• jemalloc stays opt-in.
  • The frontend uses DYN_FRONTEND_JEMALLOC=1.
  • feat(components): add a DYN_JEMALLOC preload opt-in for the router and indexer #15880 adds a generic DYN_JEMALLOC=1 for the router (including --serve-indexer) and the standalone indexer.
  • A preloaded jemalloc also serves the extension's Rust allocations, so the indexer does not need a linked allocator.
• Docs. The docs part of this PR moved to docs(frontend): document DYN_FRONTEND_JEMALLOC and MALLOC_CONF guidance #15884. It documents DYN_FRONTEND_JEMALLOC and recommends MALLOC_CONF=background_thread:true instead of the narenas:32,... string.
• Results. The stack's descriptions now say which allocator each result used. The allocator-matched end-to-end checks are still running.

The measurements in this description stay here for reference: jemalloc against glibc on main in the indexer bench and the frontend benchmark, and jemalloc against mimalloc.