feat(kv-router): add standalone selector peer sync

docs/components/router/README.md

@@ -64,5 +64,6 @@ For basic model registration without KV routing, use `--router-mode round-robin`
 - **[Router Examples](router-examples.md)**: Python API usage, K8s examples, and custom routing patterns
 - **[Router Testing](router-testing.md)**: Test layers from Rust unit tests to fixture-backed replay and full process E2E
 - **[Standalone Indexer](standalone-indexer.md)**: Run the KV indexer as a separate service for independent scaling
+- **[Standalone Selection Service](standalone-selection.md)**: Expose KV-aware selection and reservation accounting over HTTP
 - **[Standalone Slot Tracker](standalone-slot-tracker.md)**: Run active-request load accounting as a separate HTTP service
 - **[Router Design](../../design-docs/router-design.md)**: Architecture details, algorithms, and event transport modes

docs/components/router/router-guide.md

@@ -179,4 +179,6 @@ Disaggregated mode is activated automatically when prefill workers register alon
 - **[Router Examples](router-examples.md)**: Python API usage, K8s examples, and custom routing patterns
 - **[Router Testing](router-testing.md)**: Recommended test layers for non-trivial router changes
 - **[Standalone Indexer](standalone-indexer.md)**: Run the KV indexer as a separate service
+- **[Standalone Selection Service](standalone-selection.md)**: Select workers and account for reservations without forwarding requests
+- **[Standalone Slot Tracker](standalone-slot-tracker.md)**: Run active-request accounting as a separate service
 - **[KV Event Replay — Dynamo vs vLLM](kv-event-replay-comparison.md)**: Gap detection and replay behavior

docs/components/router/standalone-selection.md

@@ -0,0 +1,251 @@
+---
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+title: Standalone Selection Service
+subtitle: Select workers and account for reservations without forwarding inference requests
+---
+
+## Overview
+
+The standalone selection service (`python -m dynamo.select_service`) exposes the
+KV router's worker selection and active-load accounting over HTTP. It does not
+forward model requests or own response streams. External runtimes such as Ray
+register their worker catalog, request a selection, contact the selected worker,
+and report the reservation lifecycle.
+
+The service combines:
+
+- KV overlap indexing from worker ZMQ events.
+- KV-aware and load-aware worker selection.
+- Explicit or atomic selection and reservation.
+- Best-effort active-load synchronization between selector replicas.
+- Startup KV index recovery from another selector or standalone indexer.
+
+## Build And Launch
+
+Build the Python bindings with the `select-service` feature:
+
+```bash
+cd lib/bindings/python
+VIRTUAL_ENV=../../../.venv ../../../.venv/bin/maturin develop --uv --features select-service
+```
+
+Launch the service from the repository root:
+
+```bash
+.venv/bin/python -m dynamo.select_service --port 8092
+```
+
+The service binds to `0.0.0.0` and does not provide authentication. Run it on a
+trusted internal network or place it behind an appropriate network policy.
+
+### CLI
+
+| Flag | Default | Description |
+|------|---------|-------------|
+| `--port` | `8092` | HTTP server port. |
+| `--threads` | `4` | KV indexer worker threads. |
+| `--indexer-peers` | none | Comma-separated HTTP URLs used for startup KV recovery through `/dump`. |
+| `--replica-sync-bind` | none | Local ZMQ PUB endpoint for active-load lifecycle events. |
+| `--replica-sync-advertise` | none | Externally reachable form of the local replica-sync endpoint. Requires `--replica-sync-bind`. |
+| `--replica-sync-peers` | none | Comma-separated ZMQ PUB endpoints for selector peers. Requires `--replica-sync-bind`. |
+
+Router scheduling behavior continues to use the standard Dynamo router
+environment configuration.
+
+## Worker Registration
+
+Every selector replica must receive the same worker catalog before it serves
+selection traffic. Replica traffic never creates workers.
+
+```http
+POST /workers
+Content-Type: application/json
+
+{
+  "worker_id": 1,
+  "model_name": "model",
+  "tenant_id": "default",
+  "endpoint": "http://worker:8000",
+  "block_size": 16,
+  "data_parallel_start_rank": 0,
+  "data_parallel_size": 2,
+  "kv_events_endpoints": {
+    "0": "tcp://worker:5557",
+    "1": "tcp://worker:5558"
+  },
+  "replay_endpoint": "tcp://worker:5560"
+}
+```
+
+`POST /workers` returns `201`. `PATCH /workers/{worker_id}` updates supplied
+fields, `DELETE /workers/{worker_id}` removes the worker, and `GET /workers`
+lists catalog state. `model_name` and `tenant_id` scope all selection, indexer,
+and load state; both default to `"default"` when omitted.
+
+`GET /health` is process liveness. `GET /ready` returns `200` only after at
+least one worker is schedulable, otherwise `503` with lifecycle details.
+
+## Selection API
+
+### `POST /select`
+
+Select a worker without booking active load:
+
+```json
+{
+  "selection_id": "select-123",
+  "model_name": "model",
+  "tenant_id": "default",
+  "token_ids": [1, 2, 3, 4]
+}
+```
+
+### `POST /select_and_reserve`
+
+Select and atomically book load in the receiving selector process. Supply a
+globally unique `reservation_id`, or allow the service to generate one:
+
+```json
+{
+  "selection_id": "select-123",
+  "reservation_id": "request-123",
+  "model_name": "model",
+  "tenant_id": "default",
+  "token_ids": [1, 2, 3, 4]
+}
+```
+
+Both endpoints return the same selection shape:
+
+```json
+{
+  "selection_id": "select-123",
+  "reservation_id": "request-123",
+  "model_name": "model",
+  "tenant_id": "default",
+  "worker_id": 1,
+  "dp_rank": 0,
+  "endpoint": "http://worker:8000",
+  "block_size": 16,
+  "overlap": {
+    "longest_matched": 128,
+    "gpu": 64,
+    "dp": {"0": 64, "1": 32},
+    "cpu": 96,
+    "disk": 128
+  },
+  "effective_prefill_tokens": 384
+}
+```
+
+`selection_id` and `reservation_id` are omitted when absent. All `overlap`
+values are matched token counts. `gpu`, `cpu`, and `disk` use the cumulative
+Mooncake tier semantics documented in the standalone indexer's
+[per-instance tier breakdown](standalone-indexer.md#per-instance-tier-breakdown).
+A zero-overlap response includes the selected `dp_rank` with value `0`.
+
+The overlap summary is raw observability. `effective_prefill_tokens` is the
+authoritative weighted prefill-load value computed by the same cache-credit
+formula used for scheduler booking. It is not derived from `longest_matched`.
+
+The previous public fields `cached_tokens` and `effective_overlap_blocks` have
+been removed. Their values remain internal scheduler inputs.
+
+## Ray Select-Then-Reserve Flow
+
+Ray can keep model invocation separate from selector admission:
+
+1. Call `POST /select`.
+2. Send the request to the returned `endpoint` and `dp_rank`.
+3. Call `POST /reservations` with a globally unique reservation ID, selected
+   worker identity, the same prompt representation, and the returned
+   `effective_prefill_tokens`.
+4. Report prefill completion and request completion through the lifecycle API.
+
+```http
+POST /reservations
+Content-Type: application/json
+
+{
+  "reservation_id": "request-123",
+  "model_name": "model",
+  "tenant_id": "default",
+  "worker_id": 1,
+  "dp_rank": 0,
+  "token_ids": [1, 2, 3, 4],
+  "effective_prefill_tokens": 4
+}
+```
+
+When supplied, `effective_prefill_tokens` is authoritative and directly enables
+prefill-load tracking. It must not exceed the normalized input sequence length.
+When omitted, existing router configuration controls prefill tracking. The
+reservation API does not accept or derive accounting from overlap fields.
+
+## Reservation Lifecycle
+
+```http
+POST /reservations/{reservation_id}/prefill_complete
+POST /reservations/{reservation_id}/output_block
+DELETE /reservations/{reservation_id}
+```
+
+`prefill_complete` clears active prefill load. `output_block` updates only the
+receiving selector's local decode-block accounting and accepts an optional
+`decay_fraction` in `[0.0, 1.0]`. `DELETE` frees the reservation.
+
+**NOTE:** Output-block updates are intentionally not replica-synchronized.
+They can occur at high frequency, and broadcasting them would consume
+disproportionate network bandwidth.
+
+## Peer Planes
+
+The selector has two independent peer configurations:
+
+| Plane | Transport | Flags | Purpose |
+|-------|-----------|-------|---------|
+| Indexer recovery | HTTP | `--indexer-peers` | Fetch a compatible `/dump` during startup and replay KV events into the local indexer. |
+| Replica synchronization | ZMQ | `--replica-sync-bind`, `--replica-sync-advertise`, `--replica-sync-peers` | Share admission, prefill-complete, and free events by model and tenant. |
+
+Example:
+
+```bash
+.venv/bin/python -m dynamo.select_service \
+  --port 8092 \
+  --indexer-peers http://selector-b:8092 \
+  --replica-sync-bind 'tcp://*:9092' \
+  --replica-sync-advertise 'tcp://selector-a:9092' \
+  --replica-sync-peers 'tcp://selector-b:9092'
+```
+
+Configure the reverse peer direction on selector B for bidirectional lifecycle
+synchronization. `GET /dump` exposes the selector's current indexer snapshot in
+the same recovery format as the standalone indexer.
+
+## Consistency Invariants
+
+- Replica synchronization is bounded and best-effort. Delays, reordering,
+  dropped events, and temporary active-load divergence are accepted.
+- There is no sequencing, acknowledgement, replay, backpressure, or
+  resynchronization for replica lifecycle events.
+- Unknown worker, model, tenant, DP-rank, and block-size events are dropped.
+  Register the same worker catalog on every selector before routing traffic.
+- Admission, prefill-complete, and free are synchronized. Output-block growth
+  remains local to avoid excessive network bandwidth.
+- Startup recovery waits for recovered events to be submitted to the indexer,
+  not for complete processing. Early selections may temporarily miss recovered
+  KV state.
+- `/select` followed by `/reservations` provides eventual, not atomic,
+  cross-replica admission. Use `/select_and_reserve` for atomic local booking.
+- Reservation IDs must be globally unique. Existing conflict and retry
+  semantics are unchanged; no idempotency ledger is added.
+
+## Inspection APIs
+
+- `GET /loads` returns active-load snapshots, optionally filtered by
+  `model_name` and `tenant_id`.
+- `POST /potential_loads` estimates worker load for a prompt without selection.
+- `POST /overlap_scores` returns per-worker/per-rank tiered overlap rows.
+- `GET /dump` returns the compatible indexer recovery snapshot.
+

docs/components/router/standalone-slot-tracker.md

@@ -74,6 +74,10 @@ after peer registration. There is no acknowledgement, replay, snapshot, or gap r
 that accepted `/add`; replica synchronization provides advisory peer state rather than
 cross-replica lifecycle ownership.
 
+**NOTE:** Output-block updates remain local and are intentionally not replica-synchronized.
+They can occur at high frequency, and broadcasting them would consume disproportionate
+network bandwidth.
+
 Peers may also be managed dynamically:
 
 ```http

docs/index.yml

@@ -372,6 +372,8 @@ navigation:
             path: components/router/router-examples.md
           - page: Standalone Indexer
             path: components/router/standalone-indexer.md
+          - page: Standalone Selection Service
+            path: components/router/standalone-selection.md
           - page: Standalone Slot Tracker
             path: components/router/standalone-slot-tracker.md
           - page: KV Event Replay — Dynamo vs vLLM

lib/bindings/python/rust/llm/kv.rs

@@ -200,7 +200,7 @@ where
 }
 
 #[cfg(feature = "select-service")]
-#[derive(Parser)]
+#[derive(Debug, Parser)]
 #[command(
     name = "python -m dynamo.select_service",
     about = "Runtime-free Dynamo worker selection service"
@@ -213,6 +213,61 @@ struct SelectServiceCli {
     /// Number of KV indexer worker threads
     #[arg(long, default_value_t = 4)]
     threads: usize,
+
+    /// Comma-separated selector/indexer HTTP URLs used for startup KV recovery
+    #[arg(long, value_delimiter = ',')]
+    indexer_peers: Vec<String>,
+
+    /// ZMQ PUB endpoint for active-load replica events
+    #[arg(long)]
+    replica_sync_bind: Option<String>,
+
+    /// Externally reachable local active-load replica endpoint
+    #[arg(long, requires = "replica_sync_bind")]
+    replica_sync_advertise: Option<String>,
+
+    /// Comma-separated ZMQ PUB endpoints for peer selectors
+    #[arg(long, value_delimiter = ',', requires = "replica_sync_bind")]
+    replica_sync_peers: Vec<String>,
+}
+
+#[cfg(all(test, feature = "select-service"))]
+mod select_service_cli_tests {
+    use super::*;
+
+    #[test]
+    fn parses_selector_peer_planes() {
+        let cli = SelectServiceCli::try_parse_from([
+            "dynamo.select_service",
+            "--indexer-peers",
+            "http://indexer-a:8092,http://indexer-b:8092",
+            "--replica-sync-bind",
+            "tcp://0.0.0.0:9000",
+            "--replica-sync-advertise",
+            "tcp://selector-a:9000",
+            "--replica-sync-peers",
+            "tcp://selector-b:9000,tcp://selector-c:9000",
+        ])
+        .unwrap();
+
+        assert_eq!(cli.indexer_peers.len(), 2);
+        assert_eq!(cli.replica_sync_peers.len(), 2);
+    }
+
+    #[test]
+    fn replica_peers_require_bind_endpoint() {
+        let error = SelectServiceCli::try_parse_from([
+            "dynamo.select_service",
+            "--replica-sync-peers",
+            "tcp://selector-b:9000",
+        ])
+        .unwrap_err();
+
+        assert_eq!(
+            error.kind(),
+            clap::error::ErrorKind::MissingRequiredArgument
+        );
+    }
 }
 
 pub fn run_select_service_cli<I, T>(args: I) -> anyhow::Result<()>
@@ -233,6 +288,10 @@ where
         rt.block_on(selection::run_server(SelectionServiceConfig {
             port: cli.port,
             threads: cli.threads,
+            indexer_peers: cli.indexer_peers,
+            replica_sync_bind: cli.replica_sync_bind,
+            replica_sync_advertise: cli.replica_sync_advertise,
+            replica_sync_peers: cli.replica_sync_peers,
             kv_router_config: kv_router_config_from_dynamo_env(),
         }))
     }

lib/kv-router/src/protocols.rs

@@ -641,6 +641,8 @@ pub enum ActiveSequenceEventData {
         #[serde(default)]
         prefill_load_hint: Option<PrefillLoadHint>,
     },
+    // NOTE: Output-block growth is intentionally not a replica-sync event. It can occur
+    // at high frequency, and broadcasting it would consume disproportionate network bandwidth.
     Free,
     MarkPrefillCompleted,
 }

lib/kv-router/src/scheduling/local.rs

@@ -368,12 +368,8 @@ where
             let by_worker = effective_cached_tokens
                 .iter()
                 .map(|(worker, cached_tokens)| {
-                    let delta = isl_tokens.checked_sub(*cached_tokens).unwrap_or_else(|| {
-                        tracing::error!(
-                            "prefill_tokens < 0 with ISL {isl_tokens} < cached_tokens {cached_tokens}, returning 0"
-                        );
-                        0
-                    });
+                    let delta =
+                        super::prefill_load::effective_prefill_tokens(isl_tokens, *cached_tokens);
                     (*worker, delta)
                 })
                 .collect();