Skip to content
Original file line number Diff line number Diff line change
Expand Up @@ -84,9 +84,10 @@ for busy rejection.
3. Recalculates the busy set when load or runtime configuration changes.
4. Publishes the current busy set to the router.

A `POST /busy_threshold` update changes the stored threshold configuration. It does not synchronously
recompute every worker. The next worker-load or runtime-configuration update triggers reevaluation, so
a new threshold can take a short time to change routing decisions.
A [`POST /busy_threshold`](../../../../reference/components/frontend-configuration.mdx#admin)
update changes the stored threshold configuration. It does not synchronously recompute every worker.
The next worker-load or runtime-configuration update triggers reevaluation, so a new threshold can
take a short time to change routing decisions.

## Rejection Path

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -587,7 +587,8 @@ Use `--router-queue-policy wspt` when your workload has a mix of short and long
## Runtime Policy Updates

**Experimental.** `/router_policy` groups worker-selection parameters and model-level busy
thresholds on the frontend HTTP port (8000 by default).
thresholds. It is served on the frontend admin listener (`127.0.0.1:8002` by default), which is
off unless `DYN_ENABLE_FRONTEND_ADMIN_API=1` is set.

| Resource | Methods | Scope |
| --- | --- | --- |
Expand All @@ -611,7 +612,7 @@ Routers that have stopped or do not respond within five seconds are omitted; hea
remain available in the listing.

```bash
curl --get http://localhost:8000/router_policy --data-urlencode model=qwen
curl --get http://localhost:8002/router_policy --data-urlencode model=qwen
```

Use a returned `router_id` and revision in `PATCH /router_policy`:
Expand Down Expand Up @@ -672,7 +673,7 @@ resets use these effective values; host-owned cache accounting settings are unch
one or more threshold fields with `POST /router_policy/busy_threshold` to update them:

```bash
curl -X POST http://localhost:8000/router_policy/busy_threshold \
curl -X POST http://localhost:8002/router_policy/busy_threshold \
-H 'Content-Type: application/json' \
-d '{"model":"qwen","active_decode_blocks_threshold":0.85,"active_prefill_tokens_threshold":1000,"active_prefill_tokens_threshold_frac":0.8}'
```
Expand All @@ -695,15 +696,18 @@ reset. Updating thresholds and selection parameters is not one atomic transactio

### Scope and Access

The frontend admin API remains enabled by default on the HTTP port, with no separate admin
authentication. `DYN_DISABLE_FRONTEND_ADMIN_API=1` disables `/router_policy`, its busy-threshold
subresource, and the legacy `/busy_threshold` routes. Restrict access to these management
routes at the deployment's gateway or network boundary.
The admin listener has no authentication and is never served on the inference port. If you
widen its bind address with `DYN_FRONTEND_ADMIN_HOST`, restrict the admin port at the network
boundary. See [Admin](../../../../reference/components/frontend-configuration.mdx#admin) for the
listener settings.

Updates are process-local and not persisted. A rebuilt router receives a new ID and starts
from its configured startup values. A controller must address each frontend replica directly,
Comment thread
jthomson04 marked this conversation as resolved.
Resolved
retain the desired settings, and reapply them to new router instances; a load-balanced request
does not update every replica. The standalone Python router does not expose this HTTP API.
does not update every replica. With the default loopback bind, the admin port is reachable only
from inside the pod or through `kubectl port-forward`. An out-of-pod controller needs
`DYN_FRONTEND_ADMIN_HOST` set to a reachable address, such as the pod IP, and a network policy
that limits the admin port to that controller. The standalone Python router does not expose this HTTP API.
Rust hosts can use `KvRouter::policy_control()` or `LocalScheduler::policy_control()` to access
the same actor-serialized control handle.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -140,4 +140,4 @@ Backend KV event publishing is independent of the frontend's `--no-router-kv-eve

The CLI arg `--router-ttl-secs` controls local cache prediction lifetime when the router operates without receiving events from workers and the approximate cache policy is `ttl`. The experimental `lru` policy is local to one router replica and does not synchronize releases with other replicas. A missing or zero per-rank `total_kv_blocks` value clears that rank's LRU state and pins it to TTL until the worker rank is removed and re-registered. When workers publish KV events and the router is configured to consume them, the exact event index becomes primary and the selected approximate primary policy is inactive. An explicitly configured predicted side index remains active and is always TTL-based. Worker publication alone does not disable approximate indexing.

`--router-queue-threshold` and the busy thresholds (`--active-decode-blocks-threshold`, `--active-prefill-tokens-threshold`, `--active-prefill-tokens-threshold-frac`) serve different purposes. Busy thresholds reject a worker entirely from the candidate set when it exceeds a utilization limit. In contrast, `--router-queue-threshold` defers the entire routing decision while every eligible worker exceeds `threshold * max_num_batched_tokens`, so the request is routed with the freshest load metrics once capacity is available. `nvext.agent_hints.strict_priority` selects the primary pending-queue tier, and `nvext.agent_hints.priority` adjusts the configured policy score within that tier. The busy thresholds can be updated at runtime without restarting the frontend via the `/busy_threshold` HTTP endpoint. For the eligibility and backpressure distinction, see [Router Filtering](worker-filtering.md). For rejection behavior details, see [Request Rejection](../../../../kubernetes/fault-tolerance/request-rejection.md) and [Priority Scheduling](../../../../use-cases/agents/priority-scheduling.md).
`--router-queue-threshold` and the busy thresholds (`--active-decode-blocks-threshold`, `--active-prefill-tokens-threshold`, `--active-prefill-tokens-threshold-frac`) serve different purposes. Busy thresholds reject a worker entirely from the candidate set when it exceeds a utilization limit. In contrast, `--router-queue-threshold` defers the entire routing decision while every eligible worker exceeds `threshold * max_num_batched_tokens`, so the request is routed with the freshest load metrics once capacity is available. `nvext.agent_hints.strict_priority` selects the primary pending-queue tier, and `nvext.agent_hints.priority` adjusts the configured policy score within that tier. The busy thresholds can be updated at runtime without restarting the frontend via the [`/busy_threshold` admin endpoint](../../../../reference/components/frontend-configuration.mdx#admin). For the eligibility and backpressure distinction, see [Router Filtering](worker-filtering.md). For rejection behavior details, see [Request Rejection](../../../../kubernetes/fault-tolerance/request-rejection.md) and [Priority Scheduling](../../../../use-cases/agents/priority-scheduling.md).
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Busy thresholds are worker filters. They are configured separately from the queu
- `--active-prefill-tokens-threshold`
- `--active-prefill-tokens-threshold-frac`

When a threshold is exceeded, worker monitoring reports that worker as overloaded. The scheduler then excludes that worker from normal candidate selection. The thresholds can be updated at runtime through the `/busy_threshold` HTTP endpoint.
When a threshold is exceeded, worker monitoring reports that worker as overloaded. The scheduler then excludes that worker from normal candidate selection. The thresholds can be updated at runtime through the [`/busy_threshold` admin endpoint](../../../../reference/components/frontend-configuration.mdx#admin).

Busy filtering preserves error classification:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -198,10 +198,13 @@ reachable.
target individual
workers. This drops `request.nvext` at handler entry and ignores the
routing-override headers.
- **Admin API.** The frontend's HTTP admin API (for example,
`GET`/`POST /busy_threshold`) is enabled by default. If the deployer does not need to
change runtime tunables through it, set `DYN_DISABLE_FRONTEND_ADMIN_API=1`.
Inference, metrics, models, health, and liveness routes are unaffected.
- **Admin API.** The frontend's HTTP admin API (`/router_policy` and `/busy_threshold`),
which changes routing for every client, is disabled by default.
`DYN_ENABLE_FRONTEND_ADMIN_API=1` starts it on its own listener, bound to `127.0.0.1:8002`
and never served on the inference port. It uses the frontend's TLS settings but has no
authentication. If you widen the bind with
`DYN_FRONTEND_ADMIN_HOST`, restrict the admin port to trusted operators with a network
policy or firewall.
- **Metrics endpoint.** The `/metrics` endpoint is intended for scraping by
trusted monitoring systems; scope it to your observability stack rather than
exposing it to untrusted networks.
Expand Down
31 changes: 24 additions & 7 deletions docs/fern/pages/kubernetes/fault-tolerance/request-rejection.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,17 +83,34 @@ for the complete flag, environment-variable, and validation reference.
<Step title="Adjust thresholds at runtime">

Optional. Use the Frontend admin API to change thresholds for a discovered model without redeploying.
The API is available by default. Set `DYN_DISABLE_FRONTEND_ADMIN_API` to a truthy value to turn it
off, which unregisters both `busy_threshold` routes so they return 404.
The API is off by default. Set `DYN_ENABLE_FRONTEND_ADMIN_API=1` in the Frontend's environment to
Comment thread
jthomson04 marked this conversation as resolved.
Resolved
serve it on a separate admin port, bound to `127.0.0.1:8002` inside each Frontend pod; the inference
port does not serve it. Add it to the Frontend `main` container from the previous step:

```yaml
- name: main
env:
- name: DYN_ENABLE_FRONTEND_ADMIN_API
value: "1"
```

`kubectl port-forward` connects through the pod's loopback interface, so you don't need to widen
the bind address. Updates apply to one Frontend process. List the Frontend pods, then port-forward
to one:

```bash
kubectl port-forward svc/<deployment-name>-frontend 8000:8000 -n ${NAMESPACE}
kubectl get pods -n ${NAMESPACE} \
-l nvidia.com/dynamo-graph-deployment-name=<deployment-name>,nvidia.com/dynamo-component-type=frontend \
-o name
kubectl port-forward pod/<frontend-pod> 8000:8000 8002:8002 -n ${NAMESPACE}
```

Repeat the port-forward and the `curl` commands below for each Frontend pod you want to change.

Set one or more thresholds:

```bash
curl -X POST http://localhost:8000/busy_threshold \
curl -X POST http://localhost:8002/busy_threshold \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-0.6B",
Expand All @@ -105,7 +122,7 @@ curl -X POST http://localhost:8000/busy_threshold \
Read the stored thresholds:

```bash
curl http://localhost:8000/busy_threshold
curl http://localhost:8002/busy_threshold
```

A numeric value enables only its corresponding check. The router applies the new configuration when
Expand All @@ -120,7 +137,7 @@ decision immediately. This endpoint does not enable `--router-mode kv` or
Inspect the configured thresholds and worker-load metrics:

```bash
curl -s http://localhost:8000/busy_threshold
curl -s http://localhost:8002/busy_threshold
curl -s http://localhost:8000/metrics \
| grep -E 'dynamo_frontend_worker_active_(decode_blocks|prefill_tokens)'
```
Expand All @@ -142,7 +159,7 @@ For all metric fields and labels, see

If decode-block load does not produce HTTP 529 responses:

1. Confirm that `GET /busy_threshold` shows a numeric `active_decode_blocks_threshold` for the model.
1. Confirm that `GET /busy_threshold` on the admin port (`8002`) shows a numeric `active_decode_blocks_threshold` for the model.
2. Confirm that the Frontend started with `--router-mode kv`.
3. For long-output workloads, confirm that the Frontend started with `--router-track-output-blocks`.
4. Check worker availability on the same Frontend before diagnosing missing load telemetry:
Expand Down
38 changes: 31 additions & 7 deletions docs/fern/pages/reference/components/frontend-configuration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -861,8 +861,37 @@ accepted fields, limits, errors, and rolling upgrades.
| `GET` | `/metrics` | Prometheus metrics |
| `GET` | `/openapi.json` | OpenAPI specification |
Comment thread
jthomson04 marked this conversation as resolved.
Resolved
| `GET` | `/docs` | Swagger UI |
| `POST` | `/busy_threshold` | Set independently enabled busy thresholds for a discovered model (disabled when `DYN_DISABLE_FRONTEND_ADMIN_API` is truthy) |
| `GET` | `/busy_threshold` | Get stored busy thresholds (disabled when `DYN_DISABLE_FRONTEND_ADMIN_API` is truthy) |

### Admin

Admin routes are off by default and are served on a separate listener (`127.0.0.1:8002` by
default), never on the inference port. They have no authentication; see
[Secure Deployment Guidelines](../../developer-guide/security/secure-deployment-guidelines.md#frontend-extensions-and-admin-api).
When frontend TLS is configured, the admin listener uses the same certificate and client CA
settings, so call it with `https://`. The inference port's `/openapi.json` does not list these
routes; `generate-frontend-openapi` does.

| Method | Path | Description |
|--------|------|-------------|
| `GET`, `PATCH` | `/router_policy` | Get or update worker-selection parameters of a router instance |
| `GET`, `POST` | `/router_policy/busy_threshold` | Get or set busy thresholds for a discovered model |
| `POST` | `/busy_threshold` | Set independently enabled busy thresholds for a discovered model |
| `GET` | `/busy_threshold` | Get stored busy thresholds |

<ParamField path="DYN_ENABLE_FRONTEND_ADMIN_API" type="boolean" default="false">
When truthy (`1`, `true`, `yes`, or `on`), the frontend starts the admin listener and serves the
routes above.
</ParamField>

<ParamField path="DYN_FRONTEND_ADMIN_HOST" type="string" default="127.0.0.1">
Address the admin listener binds to. If you widen it, restrict the admin port at the network
boundary.
</ParamField>

<ParamField path="DYN_FRONTEND_ADMIN_PORT" type="integer" default="8002">
Comment thread
jthomson04 marked this conversation as resolved.
Resolved
Port of the admin listener. When the admin API is enabled and the port cannot be bound, the
frontend fails to start. A value that is not a valid port number falls back to `8002`.
</ParamField>

`POST /busy_threshold` accepts `model` plus any of `active_decode_blocks_threshold`,
`active_prefill_tokens_threshold`, and `active_prefill_tokens_threshold_frac`. A numeric field enables
Expand Down Expand Up @@ -908,11 +937,6 @@ are disabled by default. Truthy values are `1`, `true`, `yes`, or `on`, case-ins
these endpoints do not use `x-tenant-id` or `x-dynamo-*` routing headers in either mode.
</ParamField>

<ParamField path="DYN_DISABLE_FRONTEND_ADMIN_API" type="boolean" default="false">
When truthy, `GET /busy_threshold` and `POST /busy_threshold` are not registered and return 404.
Inference, metrics, models, health, and liveness routes are unaffected.
</ParamField>

### Endpoint path customization

All endpoint paths can be overridden via environment variables:
Expand Down
4 changes: 4 additions & 0 deletions docs/fern/pages/reference/general/releases/deprecations.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,10 @@ Upgrading? Pick the version you run today to see the dependency migration and th

> **Migrate:** Update launch commands, imports, dashboards, alerts, and saved JSON together. Upgrade KV state agents and the routers that read their events together, because a router cannot decode state-agent events from the other version. Regenerate version-3 `dc_ckf_consumer_bench` corpora as version 4 before loading them.

<span className="dynref-badge dynref-badge--blue">Behavioral</span> **Frontend admin API is off by default and moves to a separate loopback port** ([#17165](https://github.com/ai-dynamo/dynamo/pull/17165)). `/router_policy`, `/router_policy/busy_threshold`, and `/busy_threshold` are no longer served on the inference port, and `DYN_DISABLE_FRONTEND_ADMIN_API` is removed.

> **Migrate:** Set `DYN_ENABLE_FRONTEND_ADMIN_API=1` on frontends that need these routes and call port `8002` (`DYN_FRONTEND_ADMIN_PORT`). The listener binds to `127.0.0.1` (`DYN_FRONTEND_ADMIN_HOST`), so on Kubernetes use `kubectl port-forward pod/<frontend-pod> 8002:8002`. Remove `DYN_DISABLE_FRONTEND_ADMIN_API`; it is ignored.

<a id="v160"></a>

## v1.6.0 - Oct 22, 2026
Expand Down
1 change: 1 addition & 0 deletions lib/llm/src/bin/generate_frontend_openapi.rs
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ fn generate_openapi() -> anyhow::Result<()> {
.enable_responses_endpoints(true)
.enable_anthropic_endpoints(true)
.enable_batch_endpoints(true)
.enable_admin_api(true)
.build()
.context("failed to build HttpService for OpenAPI generation")?;

Expand Down
Loading
Loading