feat(kv-router): arena index under CRTC's lane and epoch model (redesign branch C)

[PeaBrane]

Status (2026-10-09). John Thomson is taking this over. Exploratory draft, not for merge as is. This is "Branch C" of the follow-up indexer redesign, the proposed direction; Branch B (#16035) is its SMG-style sibling. The decision rule: C becomes canonical if it is within noise of B or better and passes the same gates; B wins only at ≥~10% better sustained throughput; either must beat improved CRTC (rupei/crtc-volume-sweep). Details are in the handoff gist's followups-2026-10-09.md; the full spec is spec-C-arena-lanes.md. Outcome: the parked canonical arena branch. The gate passed and the microbench is below. Neither arena branch beats the improved CRTC (rupei/crtc-volume-sweep) by enough to justify a new stack now; if the arena direction resumes, it resumes from this branch (with B's optimistic reads for reads under write load), after a DEP.

Summary

A new SyncIndexer backend, indexer::arena_c::ArenaIndexC, hosted by ThreadPoolIndexer and selectable in mooncake_bench as arena-c. CRTC is unchanged. It takes the arena's storage and keeps Dynamo's concurrency model.

• Storage (design ideas adapted from Shepherd Model Gateway's chain index, feat(kv_index): the chain index, a lossless KV-event relay with state snapshots, and liveness-aware cache routing smg-project/smg#2814, Apache-2.0; clean-room implementation, written without reading SMG's source):
  • runs in a segmented slab named by u32 ids, each a window into a shared hash array in a u32-addressed word arena with size-class free lists;
  • children at any offset, keyed by (offset, head) in open-addressing tables (claims to 3/4 load, rebuilds to 3/8, same-key tombstone reuse);
  • splits only past 16 partial holders, sharing the array and leaving forwarding records; no splits on divergence;
  • per-rank lane maps with 16-byte (run, offset) slots, resolved through DEAD, forwarding and the external-hash check, with path compression. This replaces CRTC's LaneLookup holds, repair.rs and EdgeIndex.
• Concurrency is CRTC's: sticky lanes, a per-run shape gate plus state lock, plan-then-validate writers with an end-child re-probe for appends, and readers under the state read lock, with the root table read lock-free under the epoch pin. There is no seqlock and no generation check on the read path: every id, array, table and chunk is reused only through a per-lane batch deferred to crossbeam-epoch.
• Reclamation: eager unlink with try-lock, pending retries when idle, and a volume-triggered sweep, with the five-minute timer as backstop.
• Tests: unit tests per spec invariant, the CRTC race tests ported, the shared harness suite, the shared KvIndexerInterface matrix with an arena_c variant, an adapted thread-exit test, and loom models with negative controls in the standalone lib/kv-router/loom-arena-c workspace.
• Bench: mooncake_bench arena-c --num-event-workers N, the indexer_memory arena-c backend, and a local mooncake-system-alloc feature that builds the bench without jemalloc.

Validation

Correctness gate (independent reviewer): pass.

• Differential (test(kv-router): differential harness, race soak and memory bench for indexer backends #16021 harness): 2000 seeds in serial, 1-lane and 4-lane pool modes, at two document lengths and with a low split cap: 0 overcounts, 0 unexplained undercounts. known_grouped_removal_overcount and the harness self-test pass.
• Race soaks: 106 strict and chaos runs on a 2-socket server, all with 0 overcounts, 0 hash mismatches, 0 parity mismatches and 0 strict-mode apply errors.
• Mutation check: 6 of 8 injected overcount or lost-update bugs were caught (the gate required 2); the 2 misses point to test gaps, not code defects.
• Suites: 88 arena_c tests, the thread-exit test, and 4 loom models with negative controls pass.
• Microbench (2x AMD EPYC 7742, 10 interleaved fresh-process rounds per cell, co-tenant gated; ratios against the current stack head A):
  • jemalloc write capacity: 0.77x at 8 event threads, 0.55x at 64 (socket 0), 0.39x at 64 (both sockets).
  • Keep-up read p50 / p99: 0.87x / 0.44x, the best read latency of any arm. Keep-up update p50 / p99: 1.86x / 1.62x.
  • Read p99 at 64 threads: 7.2x / 8.6x, because reads take state locks under heavy write load.
  • glibc write capacity: 0.67x at 8 threads, 0.26x at 64.
  • Bytes per membership, time-averaged: 92.8 (A 480.5, improved CRTC 68.7).
  • Against Branch B directly, C wins at the production-like point on every jemalloc metric (keep-up read p50, read p99, update p50 and p99, and 8-thread write capacity).
• Next steps if resumed: optimistic version-validated reads for reads under write load; merge the copied slot registry and rank map back into modules shared with CRTC; add anchors, approximate LRU and HashLifecycle; add a unit test for the append version check.

[devin-ai-integration]

⏹️ Dynamo PR CI cancelled — run 38001611155 (attempt 1) on 70319b9f83

Gate checks: ❌ backend-status-check · ✅ deploy-status-check · ❌ dynamo-status-check

Other	Jobs
dynamo-runtime	⏹️ 1
planner	⏹️ 1

Failure details

No job failed; the 2 cancelled jobs are not fail-fast or manual cancels: both multi-arch image builds hit the 1h job timeout (The job has exceeded the maximum execution time of 1h0m0s) while still building, with no build error in the log before the timeout. The two gate checks fail only because of these.

⏰ dynamo-runtime / image / Build multi-arch cuda13.0: job exceeded 1h0m0s timeout

Job: dynamo-runtime / image / Build multi-arch cuda13.0 · Failed step: Build and Push Test Image · Logs: gh run view --job 114060878898 -R ai-dynamo/dynamo --log-failed

#128 [linux/amd64 runtime_wheel_builder  5/13] RUN ... maturin build --release --features "kv-indexer,slot-tracker,select-service,mm-routing,ais-forward-pass,request-trace-s3,nvtx" ...
#157 [linux/amd64 runtime 14/19] RUN ... uv pip install /opt/dynamo/wheelhouse/ai_dynamo_runtime*.whl ...
##[error]The operation was canceled.
Annotation: The job has exceeded the maximum execution time of 1h0m0s

The multi-arch runtime image build (step started 22:54, cancelled 23:54 UTC) was still running buildkit stages when the 60-minute job timeout hit; no compile or Docker error precedes the cancel. A rerun of failed jobs is likely enough if the build completes faster with a warm cache.

⏰ planner / Build multi-arch cpu: job exceeded 1h0m0s timeout

Job: planner / Build multi-arch cpu · Failed step: Build and Push Image · Logs: gh run view --job 114060878627 -R ai-dynamo/dynamo --log-failed

#158 53.64 Successfully built /opt/dynamo/dist/nixl/nixl_cu13-1.3.2-cp312-cp312-linux_aarch64.whl
#158 DONE 53.7s
#159 [linux/arm64 runtime_wheel_builder  7/15] COPY deploy/inference-gateway/ext-proc/ ...
#160 [linux/arm64 wheel_builder  4/11] COPY .cargo/ /opt/dynamo/.cargo/
#161 [linux/arm64 runtime_wheel_builder  8/15] COPY deploy/inference-gateway/sidecar/ ...
##[error]The operation was canceled.
Annotation: The job has exceeded the maximum execution time of 1h0m0s

The arm64 half of the multi-arch planner image build (ffmpeg/libvpx, NIXL and runtime wheel stages) was still in progress when the 60-minute job timeout hit; no build error precedes the cancel. A rerun of failed jobs is likely enough if the build completes faster with a warm cache.

For agents

{"pr": 16036, "run_id": 38001611155, "run_attempt": 1, "head_sha": "70319b9f83bd7d21e7e1391b921222348eecf13f",
 "failures": [
  {"job": "dynamo-runtime / image / Build multi-arch cuda13.0", "job_id": 114060878898, "failed_step": "Build and Push Test Image", "signature": "job exceeded maximum execution time of 1h0m0s (cancelled)", "tests": [], "log_cmd": "gh run view --job 114060878898 -R ai-dynamo/dynamo --log-failed"},
  {"job": "planner / Build multi-arch cpu", "job_id": 114060878627, "failed_step": "Build and Push Image", "signature": "job exceeded maximum execution time of 1h0m0s (cancelled)", "tests": [], "log_cmd": "gh run view --job 114060878627 -R ai-dynamo/dynamo --log-failed"}
 ]}

_{Posted automatically by Devin for run 38001611155. Updated on every full-CI run of this PR.}