test(kv-router): differential harness, race soak and memory bench for indexer backends

[PeaBrane]

Status (2026-10-09). John Thomson is taking this over. Draft, not for merge as is: it is the shared correctness and memory gate for the follow-up indexer redesign exploration (branches rupei/crtc-volume-sweep, rupei/kv-arena-seqlock and rupei/kv-arena-lanes, which stack on this branch). Background and the decision rule are in the handoff gist's followups-2026-10-09.md.

Summary

One harness that any SyncIndexer backend plugs into with a single macro line, so CRTC and candidate replacements are gated by the same checks.

• Differential test against a set-semantics reference (lib/kv-router/src/indexer/harness/).
  • The reference is written from Dynamo's event semantics: each rank holds a set of blocks. It gives a held score (the most a backend may report) and a reachable score (the least, once removal holes are allowed).
  • Above the held score is an overcount and fails. Between the two is a hole undercount and is allowed. Below the reachable score is an unexplained undercount and fails under HARNESS_STRICT=1 (the default).
  • Event streams are engine-valid and cover 2 to 8 ranks, per-rank Cleared, remove_worker and remove_worker_dp_rank, tail-first, head-first and shuffled multi-hash removals, partial re-stores, chained stores, shared decode extensions, edges longer than 16 blocks, and workers returning under the same or a fresh id.
  • Three drivers: a serial lane checked after every operation, and ThreadPoolIndexer with 1 and 4 pipelined lanes checked at quiescent checkpoints.
  • Also counts hash mismatches (detailed vs plain lookups and the scored tail) and valid operations the backend rejects, and shrinks the earliest overcount to a minimal repro, including overcounts that only appear under concurrency.
• Race soak (soak.rs): the test(kv-router): add an opt-in CRTC race soak #15614 soak adapted to drive any backend through its own SyncIndexer::worker lanes, keeping its overcount, hash-mismatch, parity and apply-error checks, strict and chaos modes, and SOAK_* knobs.
• Self-test (selftest.rs): a wrapper backend that drops every removal must fail both the differential and the strict soak.
• Memory bench (lib/bench/kv_router/indexer_memory.rs): bytes per (rank, block) membership over a replay through a counting global allocator, averaged over 100 evenly spaced points, plus final, after-sweep and peak values. --sweeps 12 runs the backend's cleanup at CRTC's five-minute cadence over a 60-minute trace. --features indexer-memory-system-alloc counts over the system allocator instead of jemalloc.

Plugging in a backend:

impl HarnessBackend for MyBackend { fn harness_new() -> Self { Self::new() } }
// in harness/mod.rs
harness_suite!(my_backend, crate::indexer::my_backend::MyBackend);

Optional hooks: harness_match_details (hash-mismatch check), harness_rank_slot (recycled-slot counting in the soak) and harness_structure_size.

Outside the new files this touches one line each in indexer/mod.rs and concurrent_radix_tree_compressed/mod.rs, adds harness_impl.rs for CRTC, and adds the bench entry and feature in lib/bench/Cargo.toml.

Validation

• Finds a real bug. On the base (the perf(kv-router): cut CRTC writer per-block work #15611 stack), the differential finds the CRTC grouped-removal overcount and shrinks it to 5 events. known_grouped_removal_overcount replays it, and the 6-event form scores 3 for a true 2 under all three drivers. The fix is going onto perf(kv-router): cut CRTC writer per-block work #15611.
• Clean on the fix. With the grouped-removal fix applied: 1000 seeds in each of the three drivers, zero failures of any kind.
• Self-test. The removal-dropping wrapper fails both the differential and the strict soak, as required.

cargo test -p dynamo-kv-router --release --lib harness::crtc:: -- --nocapture
HARNESS_SEEDS=1000 cargo test -p dynamo-kv-router --release --lib harness::crtc::differential -- --nocapture
SOAK_SECS=60 cargo test -p dynamo-kv-router --release --lib harness::crtc::race_soak_strict -- --ignored --nocapture
cargo test -p dynamo-kv-router --release --lib harness::selftest -- --nocapture

Base: rupei/campaign-smg-ideas-20261009, which is the #15611 stack plus the linked-in jemalloc from closed #15606. The memory bench's default counting allocator wraps jemalloc, hence that base.

[devin-ai-integration]

✅ Dynamo PR CI passed — run 37987268462 (attempt 1) on 0ba054de71

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

_{Posted automatically by Devin for run 37987268462. Updated on every full-CI run of this PR.}