feat(kv-router): enforce hard session affinity in the scheduler for every policy

[ishandhanani]

Summary

Session affinity (binding a session to the worker that served it so its prefix cache is reused) is implemented twice around one shared table: the frontend (lib/llm/src/session_affinity, kv_router/routing_host*) and the selection core used by the standalone selection service and the EPP (services/selection/core). This stack moves the session layer into the selection core and makes every host use it. This first PR defines the one rule for how a binding constrains worker selection and removes the per-policy switch it replaces.

Problem. The frontend pins a Hard binding as an exact pinned_worker. The core passed the binding to policies as an advisory affinity_target and narrowed selection only when the selector reported uses_exclusive_affinity_target() (the default selector). Under Hard, a custom policy could select another worker and have its booking rejected at commit (ai-dynamo#15661).

Commits

1. fix(kv-router): keep custom selection policies on hard session bindings: byte-identical port of fix(kv-router): keep custom selection policies on hard session bindings ai-dynamo/dynamo#15661 (still open upstream). Its factory wrapper is replaced by commit 3; its second fix (drop a bound hold when a policy filter rejects the bound worker) and both tests stay. If fix(kv-router): keep custom selection policies on hard session bindings ai-dynamo/dynamo#15661 merges first, this commit drops on rebase and the PR becomes two commits.
2. refactor(kv-router): carry affinity strength on scheduling requests: ScheduleRequest::affinity_target and SchedulingRequest::affinity_target become affinity: Option<AffinityRequirement { target, strength: Hard | Soft }>. The core derives the strength from the table's mode for a session binding; an explicit wire affinity_target and a classifier-supplied target are Soft. 34 struct-literal sites updated mechanically. No selection change.
3. feat(kv-router): enforce hard session affinity in the scheduler for every policy: the pin rule below; removes WorkerSelector::uses_exclusive_affinity_target, WorkerSelectionPolicy::with_exclusive_affinity, and fix(kv-router): keep custom selection policies on hard session bindings ai-dynamo/dynamo#15661's wrapper.

The pin rule

Applied by the scheduler queue when it selects a worker (scheduling/queue.rs):

• Hard: selection is limited to the bound worker for every policy while that worker is eligible (known, inside the caller's allowed set, available, not overloaded). An ineligible target selects normally; the host's commit then re-binds if the worker departed or rejects if it is live. Queue admission is not narrowed: a hard-bound request waits in the router only when every eligible worker is busy, as any request does, and otherwise goes to its bound worker and queues there. (The admission-narrowing experiment reverted on fix(kv-router): keep custom selection policies on hard session bindings ai-dynamo/dynamo#15661 is not reintroduced.)
• Soft: the target reaches the policy as WorkerSelectionContext::affinity_target(). The builtin default policy keeps the bound worker while it is a candidate, now inside its picker (ranked and worker-only, with and without sampling), so the worker it picks is unchanged. Custom policies keep receiving the full candidate set. A classifier's worker_selection_target override is Soft and cannot replace or clear a Hard binding.
• Explicit pinned_worker: unchanged. Exact, its own queue lane, and it still waits in the router for a busy pinned worker (test_blocked_pinned_lane_does_not_block_other_worker). A pin and a managed session binding never coexist on one request (the core rejects a disagreeing explicit target before selection), so the pin rule does not special-case pins.

Where to start. scheduling/queue.rs::select_worker_for_request (the rule, one if), then router-plugins/builtin/src/default/picker.rs (the Soft preference), then scheduling/types.rs (AffinityRequirement).

Reconciliation with ai-dynamo#15139. That PR adds respect_soft_affinity inside the two-tier picker and records uses_exclusive_affinity_target() as a default-selector compatibility hook. This PR removes the hook. Soft affinity reaches every picker the same way, so ai-dynamo#15139's picker-side option composes unchanged, and Hard needs no per-policy handling at all.

Behavior changes

• Selection core under Hard: every policy is limited to an eligible bound worker. Before fix(kv-router): keep custom selection policies on hard session bindings ai-dynamo/dynamo#15661 only the default selector was; fix(kv-router): keep custom selection policies on hard session bindings ai-dynamo/dynamo#15661's wrapper reached the same result per policy.
• Selection core under Hard with a classifier plugin that sets a worker_selection_target (the thunderagent classifier does): the binding now wins. Before, the classifier's target replaced it and the request was rejected at commit.
• Frontend Soft mode: the worker and rank picked are unchanged, but the Soft preference now applies inside the picker instead of narrowing the candidate set before scoring. Three observable side effects: the scorer's min_prefill reference is fleet-wide instead of the bound worker's (only matters with overlap_score_credit_decay > 0, default off); the frontend's non-max-overlap observer (observe_non_max_overlap_selection, a debug log and a counter) now fires for a Soft request kept on a bound worker that is not the max-overlap worker; max_raw_cached_tokens is computed fleet-wide. Hard on the frontend is still an exact pin here; the next PRs move it onto the shared resolver.
• Decisions with defaults in this PR: Hard falls back to normal selection when the bound worker is ineligible (departed, outside the caller's allowed set, unavailable, or shed by the overload detector), and the host's commit decides what that means for the binding. An explicit pinned_worker keeps waiting in the router for a busy worker while a Hard binding queues on its worker. Both are design decisions; flagging them for sign-off.

Hosts

Host	How it reaches the rule
Selection service / standalone EPP	Session binding from the partition table, strength = table mode (Hard today)
Frontend KV	Hard as exact pinned_worker (unchanged here); Soft as a Soft requirement
Classifier plugins	worker_selection_target override is Soft

Validation

Run with RUSTC_WRAPPER= and a per-worktree CARGO_TARGET_DIR.

Command	Result
cargo test -p dynamo-kv-router --features standalone-selection --lib	1273 passed, 2 ignored (1267 at base: upstream's 1266 plus the ai-dynamo#15529 config test; +4 scheduler tests, +2 eligibility tests)
cargo test -p dynamo-custom-policy-builtin	48 + 5 + 1 + 2 + 15 + 4 passed (15 in default_policy.rs, 2 new Soft picker tests)
cargo test -p dynamo-mocker	252 passed
cargo test -p dynamo-llm --lib kv_router	408 passed
cargo test -p dynamo-llm --lib session_affinity	49 passed
cargo test -p dynamo-ext-proc	160 + 1 passed
cargo clippy -p dynamo-kv-router --features standalone-selection,bench --all-targets -- -D warnings	clean
cargo clippy -p dynamo-llm -p dynamo-ext-proc -p dynamo-custom-policy-builtin -p dynamo-mocker --all-targets -- -D warnings	clean
cargo fmt --all -- --check	clean

New scheduler tests (scheduling/queue.rs): hard_affinity_limits_a_custom_picker_to_the_eligible_target, hard_affinity_limits_the_default_selector_to_an_eligible_target, hard_affinity_survives_a_classifier_target, hard_affinity_admission_stays_fleet_wide (admission fleet-wide, waits only when every worker is busy, and does not hold back a later unbound request). The test-only reference selector narrows only for Soft (it has no picker stage), so the queue tests exercise the queue's rule, not the shim. Eligibility (scheduling/filter.rs): routing_eligibility_ranked_affinity_target_yields_exact_rank, affinity_target_is_eligible_requires_a_schedulable_target (rank out of range, unknown, outside the allowed set, overloaded, unavailable). Builtin: soft_affinity_keeps_the_bound_worker_while_it_is_a_candidate, soft_affinity_limits_sampling_to_the_bound_worker. Removed assertions on the deleted hook only; no test deleted.

Benchmark (cargo bench -p dynamo-kv-router --bench worker_selection --features bench, idle machine, criterion medians, 35 cases). The bench drives the selector directly, so the queue-level rule is not on its path; the test-only reference selector's Soft narrowing is. Baseline is commit 1 run twice (the second run measures the noise floor: median 1.9%, p90 11%); the before/after builtin picker is not on this bench (the builtin crate is not a dev-dependency), its allocation claim is by inspection (Copy types, picker-owned scratch vectors).

benchmark	baseline (rerun)	after
default_worker_selection/direct/temperature_0/2	78.56 ns	76.68 ns
default_worker_selection/direct/temperature_0/32	632.53 ns	620.37 ns
default_worker_selection/direct/temperature_0/2048	42.374 µs	42.892 µs
default_worker_selection/direct/temperature_0/10000	270.99 µs	274.53 µs
default_worker_selection/direct/temperature_0.7/2	105.61 ns	106.05 ns
default_worker_selection/direct/temperature_0.7/2048	51.961 µs	48.282 µs
default_worker_selection/filtered_prefill_baseline/temperature_0/2048	61.832 µs	61.615 µs
custom_worker_selection/10_000/custom_no_filter	217.83 µs	221.69 µs
custom_worker_selection/10_000/custom_keep_all_filter	353.03 µs	354.29 µs
default_policy_wrapper/policy/1024	19.852 µs	20.081 µs

Median absolute delta against the baseline rerun is 1.2%. Two cases move more than 5%: direct/temperature_0.7/2048 (-7.1%, the sampling path reads the candidate slice directly again instead of a scratch copy) and the untouched worker_projection pair (-8% and -15% against the rerun, +0% against the first run: noise in that group). An earlier revision of this PR had a real +7% on direct/temperature_0/2; it came from a strength-agnostic narrowing block in the test-only reference-selector path and is gone (76.68 ns vs 78.56 ns).

Related Issues

• fix(kv-router): keep custom selection policies on hard session bindings ai-dynamo/dynamo#15661 (ported here), feat(router): configure two-tier soft affinity ai-dynamo/dynamo#15139 (reconciled above).
• Linear ticket: to be added (please supply the ID).