Skip to content

feat(kv-router): remote primary indexer for the selection service - #15

Closed
ishandhanani wants to merge 1 commit into
idhanani/p0-selection-approx-indexerfrom
idhanani/p0-selection-remote-indexer
Closed

ishandhanani wants to merge 1 commit into
idhanani/p0-selection-approx-indexerfrom
idhanani/p0-selection-remote-indexer

Conversation

@ishandhanani

Copy link
Copy Markdown
Owner

Phase 0 (selection-core parity), item 4b. Stacked on #14.

Summary

Ports the frontend Indexer::Remote shape (lib/llm/src/kv_router/indexer/mod.rs) into the runtime-free services backend, over HTTP instead of the Dynamo request plane. A selection service can serve its primary KV index from a standalone indexer, so several selector replicas (or a sidecar-backed deployment with no etcd/NATS) share one index and workers publish KV events to exactly one place.

  • PrimaryRetention::Remote(Arc<RemoteIndexerTransport>), IndexerPolicy::with_remote_indexer(url) (validates scheme, trims trailing /). Rejected with use_kv_events=false: the standalone indexer has no routing-decision write path, so an approximate primary cannot be remote. A predict-on-route side indexer still layers locally.
  • Indexer::Remote { primary: RemoteIndexerClient, approx }. Lookups call the standalone indexer's new POST /query_tiered_by_hash, which returns { block_size, tiered: WireTieredMatchDetails } (lossless device scores + lower-tier hits) rather than the Mooncake summary /query_by_hash renders. Local KV events are dropped with a trace log, remove_worker* only clears the side indexer, dump_events is empty (peers recover from the indexer), and transport/status/body failures map to KvRouterError::IndexerOffline. supports_router_hint_chain_retention() is false for remote (matches the frontend).
  • SelectionCore.listens_for_kv_events = use_kv_events && !remote: with a remote primary, workers become schedulable without kv_events_endpoint(s), no ZMQ listener is registered or deregistered, and indexer_peers recovery is skipped with a warning.
  • Plumbing: SelectionServiceBuilder::remote_indexer(url), SelectionServiceConfig::remote_indexer_url, --remote-indexer-url on python -m dynamo.select_service, and remote_indexer_url= on the Python SelectionService constructor.
  • create_indexer_with_policy now takes the partition key (the remote client is per partition).
  • Docs: standalone-selection.md (CLI row, indexing-modes row, remote semantics) and standalone-indexer.md (new endpoint).

Validation

  • cargo test -p dynamo-kv-router --features standalone-selection --lib: 1067 passed. New: query_tiered_by_hash_returns_wire_tiered_matches (server round trip + 404 for unknown partition), remote_indexer_queries_the_standalone_service_and_merges_side_scores (live axum indexer on an ephemeral port: remote device + host-pinned hits, local side merge, dropped local events, empty dump, unknown partition and aborted server -> IndexerOffline), remote_indexer_rejects_approximate_primary_and_bad_urls, remote_indexer_serves_selection_without_local_kv_listeners (core admits workers without kv endpoints under use_kv_events=true and routes on remote cache credit).
  • cargo check --features select-service in lib/bindings/python: clean (3m16s).
  • cargo clippy --features standalone-selection --all-targets: only the pre-existing registry.rs dead-code warning. cargo fmt and docs_lint.py clean.
  • Also folded into feat(kv-router): approximate and predict-on-route indexing in the selection service #14 (amended): its 2-thread side-indexer test now flushes Concurrent indexers, removing a race.

Port the frontend Indexer::Remote variant into the runtime-free services
backend. A selection service can now serve its primary KV index from a
standalone indexer over HTTP instead of subscribing to worker ZMQ events
itself, which is how a sidecar-backed deployment shares one index across
selector replicas without any Dynamo request plane.

- PrimaryRetention::Remote(RemoteIndexerTransport) and IndexerPolicy::
  with_remote_indexer(url). Rejected with use_kv_events=false: the standalone
  indexer has no routing-decision write path. A local predict-on-route side
  indexer still layers over the remote primary.
- Indexer::Remote { primary: RemoteIndexerClient, approx }: lookups go to the
  indexer's new POST /query_tiered_by_hash, which returns the lossless wire
  TieredMatchDetails (device scores plus lower-tier hits) rather than the
  Mooncake summary /query_by_hash renders. Local KV events are dropped,
  remove_worker only touches the side indexer, dump_events is empty, and a
  transport or status failure maps to KvRouterError::IndexerOffline.
- SelectionCore no longer requires kv_events endpoints, registers ZMQ
  listeners, or deregisters them when the primary is remote
  (listens_for_kv_events = use_kv_events && !remote). Peer recovery is skipped.
- Plumbing: SelectionServiceBuilder::remote_indexer, SelectionServiceConfig::
  remote_indexer_url, --remote-indexer-url on the select_service CLI, and
  remote_indexer_url= on the Python SelectionService constructor.

Tests: server round trip for /query_tiered_by_hash; remote indexer against a
live standalone indexer (remote device and lower-tier hits, side merge,
dropped local events, unknown partition and unreachable service -> offline);
SelectionCore with a remote policy admits workers without kv_events endpoints
and routes on remote cache credit.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani

Copy link
Copy Markdown
Owner Author

Superseded by the upstream-shaped staging stack: continued as #46 (rebased onto ai-dynamo/main, see #33).

This branch had an error being deployed

1 failed deployment
external_collaborator — ed406f25 Deployed Sep 3, 2026 by ishandhanani via ok-to-test #29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation feat

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant