refactor(kv_index)!: the chain index is the only KV index, the positional indexer goes

[slin1237]

Description

#2814 shipped the chain index beside the positional indexer behind --kv-index, the positional one left as the default so that a revert stayed one flag away while the chain index ran its gateway soak. The soak is in, so the chain index becomes the only KV index and the positional path is deleted. This pull request is the revert unit: reverting it restores the positional path exactly as #2814 shipped it. One commit from the removal series at 2ef056d9 on main fad0f55, heads in the Provenance line at the end.

Problem

Since #2814 the gateway has carried two KV indexes with one switch between them.

• The positional indexer stayed the default as an escape hatch while the chain index ran its gateway soak.
• Two apply paths, two metric sets and two exactness harnesses had to stay green for one routing decision.
• The --kv-indexer-ttl-secs and --kv-indexer-max-entries prune bounds applied to the positional indexer only.
• The soak evidence is in, below under Test Plan, so the escape hatch has no remaining reader.

Solution

The chain index is the only index behind the gateway, and the switch goes with the code it selected.

• --kv-index, the kv_index config field, KvIndexKind and its run alias are removed from the CLI, the YAML and the Python launcher.
• The gateway always builds the chain index, KvEventMonitor::new() takes nothing, and the monitor's kind, jump-size and prune machinery goes.
• The positional indexer, crates/kv_index/src/event_tree.rs at 3,245 lines, goes with PositionalIndexer, PruneStats, WorkerBlockMap and the ContentSeq trait.
• Its exactness harness tests/exactness_positional.rs goes too, the chain index's own covering the same corpora.
• --kv-indexer-ttl-secs and --kv-indexer-max-entries, public since feat(kv_index): bound the positional indexer with TTL and capacity pruning #2095, stay as deprecated no-ops that warn at startup, their config fields kept so existing YAML loads.
• The hash vocabulary its module hosted moves to block.rs with its tests.
• The benches lose their positional arms: mooncake_replay's backend and --jump-size, match_insert's group, throughput_bench now on ShardedChainIndex.
• The radix-tree differential oracle and the decision bench run on the chain index, and the READMEs keep the positional design as history.
• The kv_index crate's public surface shrinks, so it goes from 1.5.0 to 2.0.0 and the workspace pin follows.
• The chain path is untouched beyond wiring: 19 lines of module docs and an import path in chain_index.rs, 4 in sharded.rs, 2 in lane_map.rs.

Changes

42 files against main fad0f55, 670 insertions and 4,931 deletions, by area.

• crates/kv_index, 15 files: the positional indexer and its harness deleted, the hash vocabulary moved, the benches' positional arms gone, the crate at 2.0.0.
• model_gateway, 16 files: the switch, the backend enum, the monitor's kind and prune machinery and the positional path's metrics removed, the prune flags deprecated.
• crates/radix_tree and its benchmark workflow, 5 files: the differential oracle and the bench on the chain index.
• bindings/python, 4 files: the kv_index argument and the launcher flag removed.
• Cargo.toml and Cargo.lock: the crate's version pin.

Test Plan

• Gate on 2ef056d9 with 1.98.0: nightly fmt clean, clippy -D warnings over the workspace's targets and the two CI feature configurations without OpenCV and nixl.
• An all-features clippy pass over smg, smg-python, kv-index, smg-radix-tree, engine-servicer and mock-worker stands in for the rest.
• kv-index: lib 211, churn_gate 2, concurrency_chain 1, exactness_chain 6 and split_counters 1 passed.
• The 85 positional unit tests and the 6-test positional exactness suite left with the code, block.rs adds 6.
• engine-servicer 173, mock-worker 65, smg-radix-tree 21 passed and 3 ignored on the ChainIndex oracle, the differential fuzz run included.
• smg lib 2,229 passed and 5 ignored, routing_tests 138, api_tests 109, pushed_loads_test 10, k8s_discovery_test 21.
• pre-commit on the changed files, the Python extension rebuilt and its bindings tests 337 passed and 4 skipped, the leak grep clean.
• CI: pending.
• Behaviour-neutral basis from feat(kv_index): the chain index, a lossless KV-event relay with state snapshots, and liveness-aware cache routing #2814: in the gateway the chain index matches the positional path's hit rate, lookup p99 fifty times lower, less memory.
• feat(kv_index): the chain index, a lossless KV-event relay with state snapshots, and liveness-aware cache routing #2814's two-backend comparison fed both indexes the same engine streams through the monitor's apply path, every lookup equal to the reference's, the held blocks matching.
• That harness stays as chain against reference, model_gateway/src/worker/kv_index_backend/exactness.rs and crates/kv_index/tests/exactness_chain.rs.
• The admission, apply and cache-aware tests that ran on the positional index pass on the chain index with their expectations unchanged.
• Mock fleet of 8 workers at 3×: hit/oracle 0.996 against 0.994, in-gateway lookup p50 / p99 3.1 / 7.9 against 25.5 / 438 µs.
• Event apply per batch 18.8 / 230 against 39 / 722 µs, gateway RSS peak / final 676 / 443 against 711 / 598 MB.
• TTFT p50 / p99 181 / 3,430 against 178 / 5,840 ms, goodput and reuse inside the unlocked noise.
• Soak s6 on the chain index at 8 workers served 7 of 7 windows at hit/oracle 0.994, per-minute minimum 0.952.
• Its idle allocated memory was 99 MB against the positional soak's 212 MB.
• Soak s7 at 128 workers served 94 of 94 windows over twelve hours, idle allocated 202 to 204 MB, RSS 366 to 378 MB flat.
• Soak s8 at six hours served 47 of 47 windows at hit/oracle 0.996, per-minute minimum 0.945, 0 preemptions against s6's 284.
• Its median TTFT p50 / p99 read 144 / 1,352 ms, below s6's first six hours.
• Reverting this pull request restores the positional path: feat(kv_index): the chain index, a lossless KV-event relay with state snapshots, and liveness-aware cache routing #2814 shipped both indexes and this commit removes one.

Provenance

perf/kv-index-chain-only at 19754b0, one commit from the removal series head 2ef056d9 on main fad0f55 (four commits behind main 833f798, which touch no shared file and auto-merge clean, not chased), the removed code being the positional path #2814 merged there.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}

[claude]

🟡 Nit: This doc was carried over from event_tree.rs, where it described PositionalIndexer::intern_worker. It now links to ChainIndex::intern_worker, and it doesn't hold there. The chain index recycles ids (registry.free.pop() / release_worker, chain_index.rs:444/479) and refuses a new worker once names.len() >= max_workers. The gateway sets that limit to CHAIN_INDEX_MAX_WORKERS = 1024 per model (kv_index_backend.rs:28).

This PR makes the chain index the only path, so 1,024 workers streaming events for one model at the same time is now a hard limit for every cache-aware deployment. The positional index had no practical limit. When an operator crosses it, subscription_loop logs error = "worker id space exhausted (u32::MAX workers interned)" (subscription.rs:166). That points them at a u32 overflow when the cause is the per-index slot limit.

Suggest rewording both the doc and the Display text to describe the slot limit, e.g.:

/// Error returned by [`ChainIndex::intern_worker`](crate::ChainIndex::intern_worker) when every
/// worker slot of the index is taken (`max_workers`, 1,024 per model in the gateway). Ids are
/// handed back when a worker is removed, so this bounds the workers interned at once, not the
/// churn over the index's lifetime.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct WorkerIdExhausted;

impl fmt::Display for WorkerIdExhausted {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        write!(f, "every worker slot of the index is taken")
    }
}

It may also be worth putting the new per-model limit in the PR description or release notes, since the soak only covered 8 and 128 workers.