feat(router): add runtime router policy management

docs/fern/pages/developer-guide/knowledge-base/modular-components/router/configuration-and-tuning.md

@@ -584,6 +584,128 @@ Use `--router-prefill-load-model ais` when you want prompt-side load tracking to
 
 Use `--router-queue-policy wspt` when your workload has a mix of short and long requests and you want to minimize average TTFT. Use the default `fcfs` when you want to minimize tail TTFT.
 
+## Runtime Policy Updates
+
+**Experimental.** `/router_policy` groups worker-selection parameters and model-level busy
+thresholds on the frontend HTTP port (8000 by default).
+
+| Resource | Methods | Scope |
+| --- | --- | --- |
+| `/router_policy` | GET, PATCH | One concrete router instance, with a revision |
+| `/router_policy/busy_threshold` | GET, POST | All existing WorkerSets for a model in this frontend |
+
+### Worker Selection
+
+The frontend admin API can update the built-in default policy's scoring parameters without
+replacing the router's queue, KV index, or active request bookings.
+The update replaces the scorer and picker between routing decisions. Requests that already
+have a selected worker keep that worker; subsequent selections, including queued requests,
+use the new parameters.
+
+`GET /router_policy` lists frontend-local KV router instances. Use `?model=<name>` to filter
+by the model name returned in the response. Each entry includes `router_id`, `model`,
+`namespace`, `component`, `endpoint`, `worker_type`, and `configuration`. Configuration contains
+a `revision` and the complete resolved `parameters`. A null configuration means that the
+selected custom policy does not support runtime updates. Non-KV routing paths are not listed.
+Routers that have stopped or do not respond within five seconds are omitted; healthy routers
+remain available in the listing.
+
+```bash
+curl --get http://localhost:8000/router_policy --data-urlencode model=qwen
+```
+
+Use a returned `router_id` and revision in `PATCH /router_policy`:
+
+```json
+{
+  "router_id": "<router_id from GET>",
+  "expected_revision": 0,
+  "parameters": {
+    "overlap_score_credit": 1.5,
+    "prefill_load_scale": 2.0,
+    "router_temperature": 0.1
+  }
+}
+```
+
+The default policy accepts the following finite, non-negative parameters:
+`overlap_score_credit`, `overlap_score_credit_decay`, `prefill_load_scale`,
+`decode_active_request_weight`, `host_cache_hit_weight`, `disk_cache_hit_weight`,
+`shared_cache_multiplier`, and `router_temperature`. These change policy scoring only;
+host-owned load accounting and queue settings remain unchanged.
+
+Omitted parameters retain their current values. A parameter set to `null` restores that
+parameter's resolved startup value, including explicit policy YAML overrides. To restore all
+startup parameters, send `reset: true` instead of `parameters`. Every successful update or
+reset advances the revision. The response returns the target metadata and applied configuration.
+
+Updates must preserve the worker signals available when the pipeline was constructed. For
+example, a policy initialized without cache inputs cannot enable cache-dependent scoring at
+runtime. Rebuild that router with cache inputs enabled first. Temporarily disabling a scoring
+term does not prevent restoring its startup value later.
+
+Any nonzero device, host, disk, or shared-cache scoring weight requires cache inputs. A
+router that started with those inputs can set device credit to zero while keeping another
+cache tier's scoring weight active. A cache-free router rejects attempts to enable any of
+these weights, without advancing its revision.
+
+Legacy load-only configurations inherit zero effective cache-scoring weights when device
+credit is zero; explicit policy parameters can enable individual tiers at startup. Plain
+disaggregated decode resolves all four cache-scoring weights to zero. Runtime snapshots and
+resets use these effective values; host-owned cache accounting settings are unchanged.
+
+| Status | Meaning |
+| --- | --- |
+| `200` | Query succeeded, or the complete update was applied. |
+| `400` | Invalid JSON/query, empty patch, or conflicting reset/parameters. |
+| `404` | The router instance ID is unknown or has expired. |
+| `409` | `expected_revision` does not match. Query the current configuration before retrying. |
+| `413` | Request body exceeds 16 KiB. |
+| `415` | Request is not sent as `application/json`. |
+| `422` | Unknown request fields, missing revision, unsupported policy, unknown parameter, invalid value, or a change requiring unavailable worker signals. No parameters are changed. |
+| `503` | The scheduler targeted by a PATCH stopped or did not respond within five seconds. Query its state before retrying an update. |
+
+### Busy Thresholds
+
+`GET /router_policy/busy_threshold` lists the model-level thresholds. Send a model name and
+one or more threshold fields with `POST /router_policy/busy_threshold` to update them:
+
+```bash
+curl -X POST http://localhost:8000/router_policy/busy_threshold \
+  -H 'Content-Type: application/json' \
+  -d '{"model":"qwen","active_decode_blocks_threshold":0.85,"active_prefill_tokens_threshold":1000,"active_prefill_tokens_threshold_frac":0.8}'
+```
+
+| Field | Meaning |
+| --- | --- |
+| `active_decode_blocks_threshold` | KV block utilization threshold, from 0 to 1 |
+| `active_prefill_tokens_threshold` | Absolute active-prefill token threshold, a non-negative integer |
+| `active_prefill_tokens_threshold_frac` | Active-prefill threshold as a finite, non-negative fraction of `max_num_batched_tokens` |
+
+The response returns the model name and current values of all three fields. Omitted or null
+fields retain their current values; a body containing only `model` queries that model. A write
+to an undiscovered model returns 404. Invalid threshold ranges return 400, and invalid field
+types return 422, without changing the thresholds.
+
+The legacy `GET` and `POST /busy_threshold` routes remain compatibility aliases with the same
+handlers and state. Writes through either path are visible through the other. These updates
+retain their existing model scope and do not use the worker-selection revision or support
+reset. Updating thresholds and selection parameters is not one atomic transaction.
+
+### Scope and Access
+
+The frontend admin API remains enabled by default on the HTTP port, with no separate admin
+authentication. `DYN_DISABLE_FRONTEND_ADMIN_API=1` disables `/router_policy`, its busy-threshold
+subresource, and the legacy `/busy_threshold` routes. Restrict access to these management
+routes at the deployment's gateway or network boundary.
+
+Updates are process-local and not persisted. A rebuilt router receives a new ID and starts
+from its configured startup values. A controller must address each frontend replica directly,
+retain the desired settings, and reapply them to new router instances; a load-balanced request
+does not update every replica. The standalone Python router does not expose this HTTP API.
+Rust hosts can use `KvRouter::policy_control()` or `LocalScheduler::policy_control()` to access
+the same actor-serialized control handle.
+
 ## Prometheus Metrics
 
 The router exposes Prometheus metrics on the frontend's HTTP port (default 8000) at `/metrics`:

docs/fern/pages/developer-guide/knowledge-base/modular-components/router/custom-worker-selection.mdx

@@ -607,4 +607,42 @@ worker_selection:
         router_temperature: 0.0
 ```
 
-The provider also accepts `overlap_score_credit_decay`, `decode_active_request_weight`, `host_cache_hit_weight`, `disk_cache_hit_weight`, and `shared_cache_multiplier`. Omitted parameters inherit router configuration. If no shared-cache multiplier is set, the default policy uses `0.5` when shared cache is enabled. Values must be finite and non-negative. Per-request score and temperature overrides do not apply to this policy. The builtin skips cache inputs at zero overlap credit and for plain disaggregated decode.
+The provider also accepts `overlap_score_credit_decay`, `decode_active_request_weight`,
+`host_cache_hit_weight`, `disk_cache_hit_weight`, and `shared_cache_multiplier`. Values must be
+finite and non-negative. Per-request score and temperature overrides do not apply to this policy.
+
+Omitted parameters normally inherit router configuration. When the resolved
+`overlap_score_credit` is nonzero, an omitted shared-cache multiplier uses the router's value,
+or `0.5` if shared cache is enabled and no value is configured. When device credit is zero,
+omitted host, disk, and shared-cache weights instead resolve to zero.
+
+For cache-capable roles, any nonzero cache-scoring weight requires cache inputs. An explicit
+nonzero host, disk, or shared-cache weight therefore enables cache inputs even when
+`overlap_score_credit: 0`. This changes startup behavior for existing policy YAML that
+previously used zero device credit to suppress all cache inputs despite explicit nonzero tier
+weights. Set all four cache-scoring weights to zero to keep such a policy load-only. Plain
+disaggregated decode always resolves all four weights to zero and skips cache inputs.
+
+## Runtime Configuration
+
+Policies can opt in to process-local configuration updates with
+`WorkerSelectionPolicy::with_runtime_config(initial_parameters, builder)`. The initial JSON
+object defines the patchable parameter names and their resolved reset values. The builder
+receives a complete parameter object and must validate it before returning a replacement policy.
+It runs synchronously on the scheduler actor: do not perform I/O or mutate shared policy state
+while constructing a replacement. Return `RuntimePolicyError::InvalidParameters` for invalid
+values. Policies that do not opt in remain startup-only.
+
+Use the scheduler's `policy_control()` handle to read or update an opted-in policy.
+The [runtime policy API](configuration-and-tuning.md#runtime-policy-updates) defines revision,
+patch, and reset semantics.
+
+Updates run between selection calls. A replacement cannot add worker-input groups beyond those
+declared at startup, change the worker label, or change exclusive-affinity behavior. Queues,
+worker discovery, cache indexes, and host accounting are retained. Scorer/picker-local scratch
+state is rebuilt. A cancelled control request still waiting in the actor queue is skipped.
+
+The [simple-filter-score-pick example](https://github.com/ai-dynamo/dynamo/blob/main/examples/router/custom-policy-example/simple-filter-score-pick/src/lib.rs)
+opts in for its `min_device_overlap_blocks` parameter. The built-in default policy opts in for
+its scoring weights and temperature. See [Runtime Policy Updates](configuration-and-tuning.md#runtime-policy-updates)
+for the frontend HTTP contract, scope, and persistence limits.

examples/router/custom-policy-example/README.md

@@ -376,3 +376,11 @@ python3 -m dynamo.mocker \
 ```
 
 Send the same `curl` request from a fourth terminal. The frontend log records separate prefill and decode selections. Each worker set runs its own filter, scorer, and picker.
+
+## Runtime Parameters
+
+The `simple-filter-score-pick` provider supports runtime updates to
+`min_device_overlap_blocks`; the other examples remain startup-only. See the
+[runtime policy API](https://github.com/ai-dynamo/dynamo/blob/main/docs/fern/pages/developer-guide/knowledge-base/modular-components/router/configuration-and-tuning.md#runtime-policy-updates).
+A positive threshold requires usable device-overlap data. Cold or replay requests without
+that data, including in the Mocker walkthrough, can have no eligible workers.

examples/router/custom-policy-example/simple-filter-score-pick/Cargo.toml

@@ -14,3 +14,4 @@ description = "Filter, scorer, and picker custom worker-selection policy example
 [dependencies]
 dynamo-kv-router = { workspace = true, features = ["standalone-selection"] }
 serde.workspace = true
+serde_json.workspace = true

examples/router/custom-policy-example/simple-filter-score-pick/src/lib.rs

@@ -15,7 +15,7 @@ use std::sync::Arc;
 use dynamo_kv_router::KvRouterConfig;
 use dynamo_kv_router::plugins::RouterPluginRegistry;
 use dynamo_kv_router::plugins::worker_selection::{
-    WorkerFilter, WorkerSelectionPolicy, WorkerSelectionPolicyFactory,
+    RuntimePolicyError, WorkerFilter, WorkerSelectionPolicy, WorkerSelectionPolicyFactory,
     WorkerSelectionPolicyParameters, WorkerSelectionPolicyProviderError,
     WorkerSelectionPolicyRegistryError,
 };
@@ -49,20 +49,52 @@ fn provider(
 
     Ok(Arc::new(
         move |config: &KvRouterConfig, worker_type, _partition| {
-            let filters: Vec<Box<dyn WorkerFilter>> = vec![Box::new(MinimumDeviceOverlapFilter {
-                min_device_overlap_blocks,
-            })];
-            WorkerSelectionPolicy::new_with_filters(
-                config.clone(),
-                worker_type.as_str(),
-                filters,
-                vec![Box::new(ActiveRequestsScorer)],
-                Box::new(RequestAwarePicker),
+            let policy = build_policy(config, worker_type.as_str(), min_device_overlap_blocks);
+            let config = config.clone();
+            policy.with_runtime_config(
+                [(
+                    "min_device_overlap_blocks".into(),
+                    min_device_overlap_blocks.into(),
+                )]
+                .into_iter()
+                .collect(),
+                Arc::new(move |values| {
+                    let parameters: Parameters = serde_json::from_value(serde_json::Value::Object(
+                        values,
+                    ))
+                    .map_err(|error| RuntimePolicyError::InvalidParameters(error.to_string()))?;
+                    validate_min_device_overlap_blocks(parameters.min_device_overlap_blocks)
+                        .map_err(|error| {
+                            RuntimePolicyError::InvalidParameters(error.to_string())
+                        })?;
+                    Ok(build_policy(
+                        &config,
+                        worker_type.as_str(),
+                        parameters.min_device_overlap_blocks,
+                    ))
+                }),
             )
         },
     ))
 }
 
+fn build_policy(
+    config: &KvRouterConfig,
+    worker_label: &'static str,
+    min_device_overlap_blocks: f64,
+) -> WorkerSelectionPolicy {
+    let filters: Vec<Box<dyn WorkerFilter>> = vec![Box::new(MinimumDeviceOverlapFilter {
+        min_device_overlap_blocks,
+    })];
+    WorkerSelectionPolicy::new_with_filters(
+        config.clone(),
+        worker_label,
+        filters,
+        vec![Box::new(ActiveRequestsScorer)],
+        Box::new(RequestAwarePicker),
+    )
+}
+
 /// Register the `simple-filter-score-pick` policy type.
 pub fn register(
     registry: &mut RouterPluginRegistry,

lib/kv-router/src/plugins/worker_selection.rs

@@ -6,6 +6,12 @@
 mod config;
 mod context;
 mod inputs;
+mod runtime;
+
+pub(crate) use runtime::RuntimePolicyState;
+pub use runtime::{
+    RuntimePolicyBuilder, RuntimePolicyConfig, RuntimePolicyError, RuntimePolicyUpdate,
+};
 
 pub use context::WorkerSelectionContext;
 pub(crate) use inputs::{CacheSnapshot, CandidateData, WorkerCacheData};