[karen-sy]

Overview

Router modes such as round-robin and least-loaded are "bespoke" code embedded into the core runtime / llm crates today. Worker-selection plugins allow for custom routing modes, but forces the SelectionCore/scheduler/indexer to be involved. So today, there is no way to port built-in routing-mode's such as round-robin/random/least-loaded -- which today run on a specialized path with no SelectionCore involvement -- into the plugins without sacrificing these very simple algorithms' latencies.

This PR makes plugins (1) performant/thread-safe to share across requests and (2) lets a plugin that needs no KV state run without the scheduler, with no lock and no per-request allocation.

The next PR in the stack uses this extended plugin capability to turn the built-in router modes into plugins.

Key Changes

Core logic: +423 / −88 lines (excluding tests, docs and comments). Most need review:

• plugins/worker_selection.rs: plugin methods take &self, and the traits require Send + Sync.
• selector/hosted.rs: HostedWorkerPicker::select and HostedSelectionRequest.
• worker_selection/inputs.rs: WorkerInputView builds rows lazily, and adds ACTIVE_REQUESTS.

New:

• Plugins without the KV scheduler. WorkerSelectionPolicy::into_hosted_picker() returns a HostedWorkerPicker for a policy that has only a picker, declares no KV inputs and has no runtime config. Otherwise it returns the policy unchanged. The next PR in the stack calls it to run the router modes.
  • Why: a mode like round-robin needs no KV state. Sending it through the scheduler would add a hop to the scheduler actor and a row per worker on every request, costs today's bespoke code doesn't pay.
  • Note: Plugins without KV scheduler must be a picker-only policy. A policy with filters or scorers stays on the KV path; a picker can compute its own scores instead.
• ACTIVE_REQUESTS input, replacing OCCUPANCY. Each worker's in-flight request count, which a picker reads with active_requests(row) on either path. On the KV path it equals LOAD's active_requests(). OCCUPANCY stays as a deprecated alias until v1.9.
  • Why: least-loaded and power-of-two need this count to become plugins. OCCUPANCY was a flag only dynamo-llm's built-in selector used, and a plugin had no way to read the count. The rename from Occupancy to Active_requests is optional, but it feels like "Occupancy" (request count) sounds too similar to "Load" (token+block usage)..
• WorkerInputView::len() and is_empty(). A picker can choose by index without the host building rows. candidates() still works, building rows on first call.
  • Why: a picker could learn how many workers it had only from candidates(), which on the hosted path means building a row per worker on every request. Random and round-robin need only the count, and least-loaded and power-of-two need the count plus active_requests(row).

Changed:

• Breaking: keep, score and pick take &self instead of &mut self, and the traits require Send + Sync. One plugin instance can then serve every request with no lock, and the compiler checks its thread safety. A plugin that keeps state between calls uses atomics or a lock. The KV path is unchanged: one scheduler still calls each instance serially.
• The default policy keeps per-call state out of self: a stack local in the scorer and a thread-local buffer in the picker. It allocates nothing new.
• Docs: concurrency rules for plugin authors, the ACTIVE_REQUESTS input, and a deprecation entry for OCCUPANCY.

Validation

• Unit tests for the hosted entry point:
  • which policies can run without the scheduler;
  • selection that builds no rows;
  • in-flight counts read only when declared;
  • the error cases: no workers, missing counts, a bad row, block size 0.
• ACTIVE_REQUESTS on the KV path matches LOAD's active-request count, including for a worker with no load data.
• The default picker's thread-local buffer is safe across threads and between roles, and matches the seeded reference. The existing default-policy parity tests pass unchanged.
• Benchmarks against main (criterion; both built with the same settings, and main run twice to measure run-to-run noise):
  • allocations: all 67 counts the benchmarks print are identical to main;
  • default-policy selection (default_selection): 7.2 µs, 119.7 µs and 745.1 µs at 64, 1024 and 4096 workers, against 7.2–7.3 µs, 119.3–119.6 µs and 745.8–749.3 µs on main;
  • KV selection path (worker_selection, selection_core): within run-to-run noise. One shared-cache group looked slower in the first run; three alternating reruns against main put it at −2% to +4%.

Issues and Ongoing Work

• Closes DYN-5249
• Related: DEP #17607, per-role router policies.

Pull Request Checklist

• I used Dynamo's /pr-checklist skill.

Summary by CodeRabbit

• New Features
  • Added hosted worker selection for eligible custom policies, with support for selecting from worker IDs and using active-request counts.
  • Custom worker filters, scorers, and pickers can now be shared across threads.
• Documentation
  • Expanded guidance on concurrent worker-selection callbacks and hosted selection.
  • Documented OCCUPANCY as deprecated in favor of ACTIVE_REQUESTS, with removal planned for version 1.9.

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-01a1a29f-d16e-7293-b481-c5c2ef021f4f.docs.buildwithfern.com/dynamo/dev

[coderabbitai]

No actionable comments were generated in the recent review.

Recent review info

Run configuration

• Source mode: Auto
• Repository: ai-dynamo/dynamo
• Branch: karenc/router-modes-1-hosted-plugins
• File: .coderabbit.yaml
• Commit: a3d9797
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Enterprise
• Run ID: e7560281-6ffe-46ab-9df7-5eed25e21ea5

Commits

Reviewing files that changed from the base of the PR and between c10c879 and a3d9797.

Files selected for processing: 39

• deploy/inference-gateway/ext-proc/src/selector.rs
• docs/fern/pages/developer-guide/knowledge-base/modular-components/router/custom-worker-selection.mdx
• docs/fern/pages/reference/general/releases/deprecations.mdx
• examples/router/custom-policy-example/disagg-filter-score-pick/src/filter.rs
• examples/router/custom-policy-example/disagg-filter-score-pick/src/picker.rs
• examples/router/custom-policy-example/disagg-filter-score-pick/src/scorer.rs
• examples/router/custom-policy-example/simple-filter-score-pick/src/filter.rs
• examples/router/custom-policy-example/simple-filter-score-pick/src/picker.rs
• examples/router/custom-policy-example/simple-filter-score-pick/src/scorer.rs
• examples/router/custom-policy-example/simple-stacked-score-pick/src/picker.rs
• examples/router/custom-policy-example/simple-stacked-score-pick/src/scorer/active_requests.rs
• examples/router/custom-policy-example/simple-stacked-score-pick/src/scorer/uncached_blocks.rs
• examples/router/custom-policy-example/soft-pin-repin/src/lib.rs
• lib/kv-router/benches/worker_selection.rs
• lib/kv-router/src/plugins/registry.rs
• lib/kv-router/src/plugins/worker_selection.rs
• lib/kv-router/src/plugins/worker_selection/inputs.rs
• lib/kv-router/src/scheduling/queue.rs
• lib/kv-router/src/scheduling/selector/hosted.rs
• lib/kv-router/src/scheduling/selector/mod.rs
• lib/kv-router/src/scheduling/selector/policy.rs
• lib/kv-router/src/scheduling/types.rs
• lib/kv-router/src/services/selection/core/tests.rs
• lib/kv-router/src/services/selection/tests.rs
• lib/kv-router/tests/plugin_api_compat.rs
• lib/llm/src/kv_router.rs
• lib/llm/src/kv_router/routing_host.rs
• lib/llm/src/kv_router/routing_host/builder.rs
• lib/llm/src/kv_router/routing_host/builtin.rs
• lib/llm/src/kv_router/routing_host/tests.rs
• lib/router-plugins/builtin/benches/default_selection.rs
• lib/router-plugins/builtin/src/default/picker.rs
• lib/router-plugins/builtin/src/default/scorer.rs
• lib/router-plugins/builtin/src/thunderagent/worker_selection/mod.rs
• lib/router-plugins/builtin/src/two_tier_cost_fn.rs
• lib/router-plugins/builtin/tests/batch_scoring.rs
• lib/router-plugins/builtin/tests/cache_inputs.rs
• lib/router-plugins/builtin/tests/default_lifecycle.rs
• lib/router-plugins/builtin/tests/default_policy.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

Walkthrough

Worker-selection callbacks now use shared references and require thread-safe implementations. The router adds hosted picker-only selection and an ACTIVE_REQUESTS input. Built-in pickers, scorers, examples, tests, and documentation are updated for the callback and input changes.

Changes

Worker selection

Layer / File(s)	Summary
Shared callback contract and implementations lib/kv-router/src/plugins/worker_selection.rs, lib/kv-router/src/scheduling/selector/policy.rs, lib/router-plugins/builtin/src/default/*, lib/router-plugins/builtin/tests/*, examples/router/custom-policy-example/*, lib/kv-router/benches/*, lib/llm/src/kv_router.rs, deploy/inference-gateway/ext-proc/src/selector.rs, docs/fern/pages/developer-guide/knowledge-base/modular-components/router/custom-worker-selection.mdx	Callback traits require Send + Sync, and callback methods take &self. Implementations and test policies are updated. DefaultPicker uses thread-local scratch buffers, and DefaultScorer passes prepared request data through scoring calls.
Hosted picker and input view lib/kv-router/src/plugins/worker_selection/inputs.rs, lib/kv-router/src/scheduling/selector/*, lib/kv-router/src/scheduling/types.rs, lib/kv-router/tests/plugin_api_compat.rs	WorkerInputView supports hosted worker IDs and active-request lookups. Eligible picker-only policies can become HostedWorkerPicker instances. Tests cover input access, selection results, and error cases.
Active-request routing and migration lib/llm/src/kv_router/routing_host/*, docs/fern/pages/developer-guide/knowledge-base/modular-components/router/custom-worker-selection.mdx, docs/fern/pages/reference/general/releases/deprecations.mdx	Built-in routing uses ACTIVE_REQUESTS where it previously used OCCUPANCY. The guide and release notes document ACTIVE_REQUESTS and the deprecated OCCUPANCY alias.

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: Low · up to a3d97

A custom picker that selects by worker capacity could choose a different worker when run through hosted selection. This is a bounded compatibility risk to address before adopting that path for such pickers.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 15.71% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Title check		The title "feat(router): make worker-selection plugins support host-path routing modes" matches the main change. The PR makes worker-selection plugins shareable through &self receivers and Send + Sync…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Description check		The description includes all required sections: Overview, Key Changes, Validation, Issues and Ongoing Work, and the pull request checklist. It clearly explains the implementation, breaking API changes…

Full details: Title check

Explanation

The title "feat(router): make worker-selection plugins support host-path routing modes" matches the main change. The PR makes worker-selection plugins shareable through &self receivers and Send + Sync bounds, and adds a hosted path (HostedWorkerPicker, into_hosted_picker) that runs picker-only policies without the KV scheduler. The title names the hosted routing path, but it does not mention the breaking &self/Send + Sync change or the ACTIVE_REQUESTS rename, which are also central to the changeset. That makes the title partially specific, but it is still related to the main change.

Full details: Description check

Explanation

The description includes all required sections: Overview, Key Changes, Validation, Issues and Ongoing Work, and the pull request checklist. It clearly explains the implementation, breaking API changes, validation coverage, and linked work. The checklist item remains unchecked, but the description is otherwise complete.

Full details: Docstring Coverage

Explanation

Docstring coverage is 15.71% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 70 functions across 37 files. (2 skipped: 2 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[devin-ai-integration]

✅ Dynamo PR CI passed — run 41687980363 (attempt 1) on add51b71b0

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

_{Posted automatically by Devin for run 41687980363. Updated on every full-CI run of this PR.}

[ishandhanani]

Read the full diff, ran cargo clippy -D warnings (kv-router, router-policies, routing-algorithms, runtime) and the kv-router / router-policies test suites at the stack head (17713 head, which carries this PR). All green. No correctness defects found; the hosted path, the &self change and the ACTIVE_REQUESTS wiring hold up. Comments below are on cost and API surface on the new hosted path.

[ishandhanani]

Round 1 AI review

Automated review (round 1 of up to 2 before a human review). Findings may be wrong; reply on a thread to dispute one.

This PR changes the worker-selection plugin callbacks (keep, score, pick) to take &self and requires the traits to be Send + Sync. It adds HostedWorkerPicker, so a picker-only policy can run without the KV scheduler, and replaces OCCUPANCY with ACTIVE_REQUESTS, keeping OCCUPANCY as a deprecated alias. I traced into_hosted_picker eligibility, the lazy WorkerInputView::hosted rows, the KV-path active_requests closure, the default picker's thread-local buffers, the default scorer's per-call PreparedRequest and the WorkerSelectionContext field copy, and found no correctness defect. The KV path still projects loads before every selection, so ACTIVE_REQUESTS and LOAD's active_requests() agree. Two small items remain: the release ledger does not record the breaking trait change, and the hosted input contract is not checked.

New findings: 1 minor, 1 nit

Nits (1)

• lib/kv-router/src/scheduling/selector/hosted.rs:98 [nit] Hosted id ordering is a public contract but never checked: WorkerInputView tells plugin authors that hosted rows are "selectable workers in ascending worker-ID order", and select documents ids as ascending without duplicates. Nothing checks this. A picker that keeps state by index, such as the round-robin cursor planned for #17713, would silently skew or repeat workers if a host passes an unsorted or duplicated slice, and no test would fail. A debug_assert! costs nothing in release builds and catches a broken host in the routing-host tests.

The next AI round runs after new commits are pushed.

[ishandhanani]

Round 2 AI review

Automated review (round 2 of up to 2 before a human review). Findings may be wrong; reply on a thread to dispute one.

Since round 1, the only change is a merge from main. It brings in sticky_min_load.rs, which this PR now adapts to &self by wrapping the existing picker in a parking_lot::Mutex. The wrapper is correct. The policy declares LOAD, so it never runs on the hosted path, and on the KV path the single scheduler calls it one request at a time, so the lock is never contended. The tests still call the inner StickyMinLoadPicker directly. One minor documentation gap from round 1 remains: the release ledger does not record the breaking change to the plugin trait signatures.

New findings: none
Earlier findings: 0 fixed, 0 no longer apply, 1 still open

• still open [DESIGN] Ledger records the OCCUPANCY rename but not the breaking trait change

AI review is done; a human reviewer takes it from here.

[ishandhanani]

lgtm - address rest of my comments