Skip to content
Open
6 changes: 6 additions & 0 deletions docs/fern/pages/cli/kv-aware-routing/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,8 @@ worker_selection:
instances:
- name: dynamo-two-tier-cost-fn
type: dynamo-two-tier-cost-fn
parameters:
respect_soft_affinity: true
```

Then point the frontend at it:
Expand All @@ -109,6 +111,10 @@ python3 -m dynamo.frontend \
To compare against the built-in selector, restart with
`DYN_ROUTER_WORKER_SELECTION_POLICY=default` — no config change needed.

`respect_soft_affinity: true` keeps an eligible soft session-affinity target as the exclusive
candidate. The default is `false`, which leaves the target advisory. Ineligible soft targets fall
back to normal two-tier selection; hard affinity and explicit worker targets remain exact.

For the available policy types and per-stage prefill/decode selection, see
[Worker-Selection Policies](../../developer-guide/knowledge-base/modular-components/router/configuration-and-tuning.md#worker-selection-policies).

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -146,20 +146,28 @@ experimental router exactly. Add any subset to tune it:
cache_threshold: 0.5
balance_abs_threshold: 32
balance_rel_threshold: 1.1
respect_soft_affinity: true
```

| Parameter | Default | Meaning |
|---|---|---|
| `cache_threshold` | `0.5` | Fraction of the request's blocks that must be device-resident on the best worker before the cache tier applies. Compared strictly. Must be in `[0.0, 1.0]`. |
| `balance_abs_threshold` | `32` | Minimum active-request spread before the load tier applies. |
| `balance_rel_threshold` | `1.1` | Minimum ratio of largest to smallest active-request count before the load tier applies. Must be at least `1.0`. |
| `respect_soft_affinity` | `false` | When `true`, an eligible soft session-affinity target is the only candidate passed to the two-tier picker. When `false`, the target remains advisory and the picker can select another eligible worker. |

Both load gates must hold before the load tier displaces the cache tier. Parameters are validated at
startup, so an out-of-range value or an unknown key fails the process immediately, naming the key,
rather than being silently ignored. It selects the least-loaded worker once the active-request spread is greater than 32 and the
largest count is more than 1.1 times the smallest; otherwise it prefers the worker holding the
largest device-KV overlap when that overlap covers more than 50% of the request's blocks.

`respect_soft_affinity` affects soft affinity only. A target is eligible when it remains discovered,
available, below the overload threshold, allowed by the request's worker allowlist, and compatible
with its routing constraints. If the target is ineligible, the policy falls back to normal two-tier
selection. Hard session affinity and explicit request targets remain exact constraints for either
value.

#### Override the Selection

`DYN_ROUTER_WORKER_SELECTION_POLICY` overrides every stage. `--router-prefill-policy` and
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -412,7 +412,7 @@ Request context and worker identity are always available. Dynamo calculates opti
| `block_size()` | Tokens in one KV block |
| `tracks_prefill_tokens()` | Whether the request contributes to prefill-load tracking |
| `session_context()` | Optional session metadata described below |
| `affinity_target()` | Optional advisory worker and data-parallel rank from soft session affinity |
| `affinity_target()` | Optional worker and data-parallel rank from soft session affinity |
| `expected_output_tokens()` | Optional expected output length |
| `priority_jump()` | Scheduler priority boost. Queue policies treat negative values as zero |
| `strict_priority()` | Strict integer priority. The queue orders larger values first |
Expand All @@ -433,7 +433,7 @@ The [custom policy examples](https://github.com/ai-dynamo/dynamo/blob/main/examp

### Soft Session Affinity

Set `--router-session-affinity-mode soft` with a session-affinity TTL to let a custom policy influence an existing session binding. The current binding enters the normal selection pipeline as `affinity_target()`. The policy still receives the full host-eligible candidate set, subject to its own filters, and may select another worker. Dynamo rebinds the session after dispatch returns a response stream.
Set `--router-session-affinity-mode soft` with a session-affinity TTL to let a custom policy influence an existing session binding. The current binding enters the normal selection pipeline as `affinity_target()`. By default, the policy receives the full host-eligible candidate set, subject to its own filters, and may select another worker. Dynamo rebinds the session after dispatch returns a response stream.

```mermaid
flowchart LR
Expand All @@ -445,6 +445,24 @@ flowchart LR

Match both fields when a policy wants to recognize the target. A target with `dp_rank: None` matches every rank on its worker; a populated rank matches only that worker-rank pair. The target can be absent from the candidate table when the worker is unavailable or a custom filter rejects it.

To retain an eligible target without duplicating affinity logic in the picker, opt into scheduler
candidate narrowing when you build the policy:

```rust
WorkerSelectionPolicy::new(
config.clone(),
worker_type.as_str(),
scorers,
Box::new(LowestCostPicker),
)
.with_exclusive_affinity_target(true)
```

The default is `false`. When enabled, Dynamo checks discovery, availability, overload, request
allowlists, routing constraints, and the target's data-parallel rank before narrowing. An ineligible
target leaves the normal candidate set intact. Policy filters still run after narrowing, so rejecting
the sole narrowed candidate returns the policy's usual all-filtered error.

Hard affinity remains the default. A hard binding and every explicit request target take the exact-target path instead of advisory custom selection. A selection, setup, or dispatch failure before a response stream leaves the previous soft binding intact. An error or cancellation after the stream is returned does not roll back a completed rebind.

The [custom policy examples](https://github.com/ai-dynamo/dynamo/blob/main/examples/router/custom-policy-example/README.md) include an overload-aware soft pinning policy that retains the advisory target until its active-request count exceeds a configured threshold, then selects the least-loaded alternative. Its two-Mocker walkthrough holds the first request open and verifies an `A -> B -> B` worker sequence: overload moves the second request to B, and the third request retains B after load drains, proving that the plugin-selected dispatch updated the binding.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,10 @@ SPDX-License-Identifier: Apache-2.0

The `soft-pin-repin` policy keeps a session on its current worker until that worker exceeds an active-request threshold. It then selects the least-loaded alternative and lets Dynamo store the selected worker as the new soft binding.

The policy implements only `WorkerPicker`. It requests `WorkerInputs::LOAD`; Dynamo continues to own worker discovery, eligibility, reservations, accounting, dispatch, and session binding state.
The policy implements only `WorkerPicker`. It requests `WorkerInputs::LOAD` and explicitly keeps
`with_exclusive_affinity_target(false)` so the picker can compare the current soft target with every
eligible alternative. Dynamo continues to own worker discovery, eligibility, reservations,
accounting, dispatch, and session binding state.

## Policy Behavior

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,8 @@ fn provider(
max_active_requests,
}),
)
// This policy needs every eligible candidate so it can decide when to repin.
.with_exclusive_affinity_target(false)
},
))
}
Expand Down
5 changes: 3 additions & 2 deletions lib/kv-router/src/plugins/worker_selection.rs
Original file line number Diff line number Diff line change
Expand Up @@ -191,8 +191,9 @@ impl WorkerSelectionContext<'_> {

/// Return the session-affinity target resolved by the request host.
///
/// The default selector treats an eligible target as exclusive. Custom policies receive it as
/// advisory context; it may be absent from their candidate set when unavailable or filtered.
/// Custom policies receive the target as advisory context by default and may opt into exclusive
/// handling through [`WorkerSelectionPolicy::with_exclusive_affinity_target`]. An ineligible
/// target does not narrow the candidate set and may be absent from it.
pub fn affinity_target(&self) -> Option<WorkerAffinityTarget> {
self.request.affinity_target
}
Expand Down
Loading
Loading