perf(kv-router): port #15611's CRTC writer per-block work onto main

[jthomson04]

Summary

Ports the writer-path changes of #15611 (PeaBrane) onto main, without the two layers it was stacked on:

• perf(kv-router): epoch-reclaimed CRTC child maps and borrowed reads #15609 (epoch-reclaimed child maps)
• perf(kv-router): dense rank slots with atomic coverage bitsets #15610 (slot ids and atomic coverage bitsets)

In Rudy's per-PR e2e breakdown, #15611 carried nearly all of the stack's gain. The lower layers add a lot of mechanism for gains that only show up under heavy overload. Builds on #16032 (merged), whose redirect-only lookup semantics this PR relies on.

Commits:

1. CRTC-private node state with an edge position index. CrtcNodeState replaces the compressed_radix::NodeState that CRTC shared with RadixTree. RadixTree is unchanged.
   • Same worker-keyed coverage, plus an EdgeIndex. Edges of 16 blocks or fewer keep no index and scan from the tail. Longer edges keep an open-addressing u32 position table.
   • A split truncates the prefix's table in O(1), and the split coverage logic moves into the state unchanged.
2. LaneLookup. Per-lane block lookups move to an open-addressing BlockLookup that prefetches a batch's slots.
   • Entries name nodes by lane-local id. The lane holds one Arc per distinct node, so storing or evicting a block no longer touches the node's shared refcount.
   • Stale-leaf cleanup still sees exactly one strong reference per lane that names a node.
   • lookup_update.rs is removed; fix(kv-router): stop CRTC lookup rewrites from clearing live coverage #16032's redirect rule is kept.
3. Store parent validation inside the coverage check. parent_coverage returns Covered, Uncovered or Missing. A missing parent repairs once and continues from the resolved node.
   • Anchor nodes carry a flag instead of a per-store anchor_nodes probe; debug builds assert it matches the map.
4. Eviction runs under one exclusive gate.
   • A run is the following hashes whose lane entries name the node, which is fix(kv-router): stop CRTC lookup rewrites from clearing live coverage #16032's grouping rule.
   • The run is scanned under an upgradable state read using position_near. It upgrades to the write lock only when coverage changes.
   • A stale run head repairs once toward the head and retries on the resolved node. The lifecycle check runs once per run.
5. README.

Correctness and locking

• No new overcount path. Readers are unchanged, including the documented equal-size skip. Coverage still only shrinks through remove_worker_at_pos under the state write lock.
• Lock change. What changes is that removal now detects a stale run head under the node's exclusive shape gate, instead of probing it under a state read first. Before/after microbench numbers are below.
• One intentional EdgeIndex difference. If a split prefix repeats a hash that also appears in its suffix, the prefix keeps its own last position of that hash. The old map dropped the hash, while a scan finds it.
• perf(kv-router): cut CRTC writer per-block work #15611's random_hole_streams_never_overcount test isn't ported as-is. It lets every rank evict mid-chain, so on main it fails only through the documented equal-size skip. With the skip disabled locally, 200 seeds had zero overcounts. Instead, fix(kv-router): stop CRTC lookup rewrites from clearing live coverage #16032's two-lane randomized test now also checks LaneLookup's hold invariants after every event.

Validation

• cargo test -p dynamo-kv-router --all-targets --features metrics passes.
• --features bench CRTC tests pass.
• cargo test -p dynamo-bench --test mooncake_trace passes.
• cargo clippy --all-targets -D warnings is clean for kv-router (default and bench,metrics) and dynamo-bench.
• cargo fmt --check is clean. Every commit builds and passes clippy and the CRTC tests on its own.

End to end (tyche, GB200 Grace)

Setup: 2,048 mock workers at speedup 10, AgentX at concurrency 6,144, one frontend per NUMA domain (72 CPUs each), QUIC response plane, KV routing with replica sync, block size 16. Both arms were built and run in one allocation on the same five nodes:

• one discarded rehearsal;
• then one timing run and one profile run per arm, ordered main timing, port timing, port profile, main profile;
• 30 s settle and a 90 s measurement;
• profiles use perf record -e cycles:u -F 99 --call-graph dwarf on both frontends from +40 s to +55 s.

The workload stores only and never evicts, so e2e covers the store and lookup half of this change.

CPU attributed to CRTC in the frontends, as inclusive cycles:u share times measured user CPU, both frontends combined:

	main	port	change	2σ sampling
CRTC, µs per applied KV event	2.764	2.540	−8.1%	±4.2%
CRTC apply_event, µs/event	2.288	1.983	−13.3%	±4.7%
CRTC reads (find_matches), µs/event	0.289	0.315	+9.2%	±12.4%
KV indexer total, µs/event	7.081	6.792	−4.1%	±2.6%
CRTC, ms of frontend CPU per request	0.321	0.298	−7.1%

Timing runs (no profiler) showed no regression beyond run-to-run noise, which is about ±3–6% here:

• 4,960 → 5,091 req/s;
• frontend CPU 24.87 → 24.61 ms/req;
• ITL p50 0.89 → 0.84 ms;
• TTFT p50 39 → 45 ms;
• frontend RSS 32.5 → 33.0 GiB mean.

Main's CRTC cost matches an earlier independent run of this setup (2.76 µs/event).

The e2e arms put #16032 and this PR on e61319d830. That's because current main's mocker could not start 2,048 workers in this harness; the issue is unrelated and being followed up separately. Apart from #16032, main has no CRTC changes since that commit. The port arm's non-test code is byte-identical to this PR. The main arm is #16032 as of c8714cdb8c; #16032's later f71cc95af1 only moves the lane-local naming check in removal grouping out of the node lock, and groups the same hashes.

Microbench (mooncake_bench, per lib/bench/kv_router/INDEXER_BENCH.md)

Setup:

• full Mooncake trace, 128 workers, duplication 20, length factor 4, 8 event workers;
• one Grace node (144 cores), 20 interleaved fresh-process runs per arm per load, all generator-valid;
• built from the actual stacked heads;
• the corpus includes 860K remove events (95.7M removed blocks), so removal is exercised.

Load	Metric	main	port	port/main (95% CI)
750 ms overloaded	achieved_block_ops_per_sec	62.3M	69.3M	1.112 (1.091–1.122)
12 s keep-up (20/20 kept up)	update_accepted_to_finished p50	104.5 µs	58.1 µs	0.556 (0.541–0.567)
	update_accepted_to_finished p99	2,020 µs	1,426 µs	0.706 (0.679–0.729)
	query_service p50	3.42 µs	3.42 µs	1.000 (0.991–1.009)
	query_service p99	12.19 µs	12.70 µs	1.042 (1.017–1.065)

Write capacity is up 11% and update latency is down 44% (p50) and 29% (p99). Lookup p50 is unchanged. Lookup p99 shows a small tail increase (+4% at keep-up, +7% overloaded); reader code is unchanged, so this is noted rather than explained.

Indexer memory (mooncake_bench)

Method:

• Each build carries a local probe patch, not part of this PR. After the replay has drained, it records the glibc live heap (mallinfo2 in-use plus mmapped) just before and just after dropping the indexer.
• Nothing else is freed between the two probes, so the difference is the indexer's own heap.
• One Grace node, 3 fresh-process keep-up runs per build. Runs of each build agreed to within 0.1%.

Workload	#16032	this PR	change
Mooncake replay per INDEXER_BENCH.md (block size 128, 128 workers, with evictions)	2,846 MiB	1,840 MiB	−35.3%
Store-only, block size 16, 2,048 workers (--num-gpu-blocks 32000000, trace duplication 16)	9,950 MiB	7,882 MiB	−20.8%
Same, with the frontend's jemalloc 5.3.0 preloaded (resident memory released at drop, decay 0)	11,632 MiB	9,309 MiB	−20.0%

• Where the saving comes from. Most of it comes from the first commit (EdgeIndex): −32.9% and −20.1% on its own. LaneLookup adds the rest; the last two commits don't change memory.
• Against the full stack. Measured the same way, the full stack (perf(kv-router): epoch-reclaimed CRTC child maps and borrowed reads #15609 → perf(kv-router): cut CRTC writer per-block work #15611) gives −36.9%, −20.9% and −19.7%. perf(kv-router): epoch-reclaimed CRTC child maps and borrowed reads #15609 and perf(kv-router): dense rank slots with atomic coverage bitsets #15610 add at most 1.5% between them. The remaining 1.5% gap at block size 128 matches perf(kv-router): dense rank slots with atomic coverage bitsets #15610's coverage bitsets, which this PR doesn't include.
• End to end. In the tyche run above, the indexer is a small share of frontend memory. So combined frontend RSS at the same number of applied KV events (30M per frontend) was 3% lower: 41.1 → 40.0 GiB. The mean RSS in the timing runs isn't comparable between arms, because the port applied 3.7% more events in its window.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Performance
  • Improved block and edge lookup efficiency, particularly for longer sequences, while keeping lookup storage bounded.
  • Reduced unnecessary updates during removals and repaired stale lookup entries when encountered.
• Reliability
  • Improved handling of parent coverage and concurrent changes, helping preserve correct results during inserts, splits, and removals.
• Testing
  • Added randomized and targeted checks for lookup behavior, coverage changes, stale entries, and concurrent tree operations.

[devin-ai-integration]

✅ Dynamo PR CI passed — run 38291821485 (attempt 1) on 0ab9fde996

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

_{Posted automatically by Devin for run 38291821485. Updated on every full-CI run of this PR.}

[jthomson04]

/ok to test 0ab9fde

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: ai-dynamo/dynamo/.coderabbit.yaml
• Review profile: CHILL
• Plan: Enterprise
• Run ID: d3ebf50d-6df8-4099-9925-8bf635f7d055

---

📥 Commits

Reviewing files that changed from the base of the PR and between d7caf74 and 0ab9fde.

---

📒 Files selected for processing (15)

• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/README.md
• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/block_lookup.rs
• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/edge_index.rs
• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/lane_lookup.rs
• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/mod.rs
• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/node.rs
• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/remove.rs
• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/repair.rs
• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/state.rs
• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/store.rs
• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/sync_impl.rs
• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/tests.rs
• lib/kv-router/src/indexer/concurrent_radix_tree_compressed/types.rs
• lib/kv-router/src/lib.rs
• lib/kv-router/src/lookup_update.rs

---

💤 Files with no reviewable changes (2)

• lib/kv-router/src/lib.rs
• lib/kv-router/src/lookup_update.rs

---

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

---

Walkthrough

The compressed radix tree now uses indexed edge positions and lane-local block-to-node lookups. Store-parent resolution, removal, and stale-entry repair use the new lookup and node-state APIs.

Changes

Compressed radix tree lookup

Layer / File(s)	Summary
Indexed edge and node state lib/kv-router/src/indexer/concurrent_radix_tree_compressed/edge_index.rs, state.rs, node.rs, types.rs, README.md	EdgeIndex indexes eligible edge positions. CrtcNodeState manages edge hashes and worker coverage, while Node delegates state operations and reports parent coverage.
Lane-local lookup and tree integration lib/kv-router/src/indexer/concurrent_radix_tree_compressed/block_lookup.rs, lane_lookup.rs, mod.rs, sync_impl.rs, tests.rs, types.rs, lib/kv-router/src/lib.rs, lib/kv-router/src/lookup_update.rs, README.md	BlockLookup and LaneLookup replace per-worker hash maps. Tree lookup updates, anchor resolution, split handling, worker statistics, and lookup tests use the new API. The old lookup-update helpers are removed.
Store, removal, and stale-entry repair lib/kv-router/src/indexer/concurrent_radix_tree_compressed/{node.rs,remove.rs,repair.rs,store.rs,tests.rs,types.rs,README.md}	Store-parent resolution distinguishes covered, uncovered, and missing hashes, and repairs stale lookups. Removal processes same-node hash runs and retries after repair. Tests cover stale entries, parent errors, and anchor behavior.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: ⚪ Minimal · up to 0ab9f

This change redesigns the compressed radix tree's edge indexing and per-lane lookup tables. The author reports passing tests and Clippy, and no concrete defect was established at the current head. The remaining risk is ordinary rollout risk for a performance-oriented rewrite, so the change appears ready to merge.

Pre-merge checks | 4 | 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 47.50% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 160 functions across 12 files. (1 skipped…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check		The title clearly and concisely identifies the main change: porting the CRTC writer per-block work from #15611 onto main.
Description check		The description is detailed and covers the implementation, validation, benchmarks, and reviewer starting points. It does not include the required "Related Issues" section or an explicit issue-link dir…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

---

Full details: Docstring Coverage

Explanation

Docstring coverage is 47.50% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 160 functions across 12 files. (1 skipped: 1 unsupported.)

---

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Commit to this branch
• Create a new PR

---

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}