Repository navigation
feat(kv-router): remote primary indexer for the selection service - #15
Closed
ishandhanani wants to merge 1 commit into
Closed
ishandhanani wants to merge 1 commit into
ishandhanani wants to merge 1 commit into
Conversation
Port the frontend Indexer::Remote variant into the runtime-free services
backend. A selection service can now serve its primary KV index from a
standalone indexer over HTTP instead of subscribing to worker ZMQ events
itself, which is how a sidecar-backed deployment shares one index across
selector replicas without any Dynamo request plane.
- PrimaryRetention::Remote(RemoteIndexerTransport) and IndexerPolicy::
with_remote_indexer(url). Rejected with use_kv_events=false: the standalone
indexer has no routing-decision write path. A local predict-on-route side
indexer still layers over the remote primary.
- Indexer::Remote { primary: RemoteIndexerClient, approx }: lookups go to the
indexer's new POST /query_tiered_by_hash, which returns the lossless wire
TieredMatchDetails (device scores plus lower-tier hits) rather than the
Mooncake summary /query_by_hash renders. Local KV events are dropped,
remove_worker only touches the side indexer, dump_events is empty, and a
transport or status failure maps to KvRouterError::IndexerOffline.
- SelectionCore no longer requires kv_events endpoints, registers ZMQ
listeners, or deregisters them when the primary is remote
(listens_for_kv_events = use_kv_events && !remote). Peer recovery is skipped.
- Plumbing: SelectionServiceBuilder::remote_indexer, SelectionServiceConfig::
remote_indexer_url, --remote-indexer-url on the select_service CLI, and
remote_indexer_url= on the Python SelectionService constructor.
Tests: server round trip for /query_tiered_by_hash; remote indexer against a
live standalone indexer (remote device and lower-tier hits, side merge,
dropped local events, unknown partition and unreachable service -> offline);
SelectionCore with a remote policy admits workers without kv_events endpoints
and routes on remote cache credit.
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ishandhanani
had a problem deploying
to
external_collaborator
September 3, 2026 06:59 — with
GitHub Actions
Failure
This was referenced Sep 3, 2026
Owner
Author
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 0 (selection-core parity), item 4b. Stacked on #14.
Summary
Ports the frontend
Indexer::Remoteshape (lib/llm/src/kv_router/indexer/mod.rs) into the runtime-free services backend, over HTTP instead of the Dynamo request plane. A selection service can serve its primary KV index from a standalone indexer, so several selector replicas (or a sidecar-backed deployment with no etcd/NATS) share one index and workers publish KV events to exactly one place.PrimaryRetention::Remote(Arc<RemoteIndexerTransport>),IndexerPolicy::with_remote_indexer(url)(validates scheme, trims trailing/). Rejected withuse_kv_events=false: the standalone indexer has no routing-decision write path, so an approximate primary cannot be remote. A predict-on-route side indexer still layers locally.Indexer::Remote { primary: RemoteIndexerClient, approx }. Lookups call the standalone indexer's newPOST /query_tiered_by_hash, which returns{ block_size, tiered: WireTieredMatchDetails }(lossless device scores + lower-tier hits) rather than the Mooncake summary/query_by_hashrenders. Local KV events are dropped with a trace log,remove_worker*only clears the side indexer,dump_eventsis empty (peers recover from the indexer), and transport/status/body failures map toKvRouterError::IndexerOffline.supports_router_hint_chain_retention()is false for remote (matches the frontend).SelectionCore.listens_for_kv_events = use_kv_events && !remote: with a remote primary, workers become schedulable withoutkv_events_endpoint(s), no ZMQ listener is registered or deregistered, andindexer_peersrecovery is skipped with a warning.SelectionServiceBuilder::remote_indexer(url),SelectionServiceConfig::remote_indexer_url,--remote-indexer-urlonpython -m dynamo.select_service, andremote_indexer_url=on the PythonSelectionServiceconstructor.create_indexer_with_policynow takes the partition key (the remote client is per partition).standalone-selection.md(CLI row, indexing-modes row, remote semantics) andstandalone-indexer.md(new endpoint).Validation
cargo test -p dynamo-kv-router --features standalone-selection --lib: 1067 passed. New:query_tiered_by_hash_returns_wire_tiered_matches(server round trip + 404 for unknown partition),remote_indexer_queries_the_standalone_service_and_merges_side_scores(live axum indexer on an ephemeral port: remote device + host-pinned hits, local side merge, dropped local events, empty dump, unknown partition and aborted server ->IndexerOffline),remote_indexer_rejects_approximate_primary_and_bad_urls,remote_indexer_serves_selection_without_local_kv_listeners(core admits workers without kv endpoints underuse_kv_events=trueand routes on remote cache credit).cargo check --features select-serviceinlib/bindings/python: clean (3m16s).cargo clippy --features standalone-selection --all-targets: only the pre-existingregistry.rsdead-code warning.cargo fmtanddocs_lint.pyclean.Concurrentindexers, removing a race.