refactor(router): move default worker selection into builtin plugin

[ishandhanani]

Summary

Move the default worker-selection policy into dynamo-custom-policy-builtin, alongside the two-tier policy. The default now uses the same public filter, score, and pick API as custom policies. Routing hosts install it automatically; plugin authors do not need to register or depend on the builtin crate themselves.

Refs #14778.

How it works

flowchart LR
    H[Host: snapshot and eligible workers] --> F[Filters]
    F --> S[Score: costs for all candidates]
    S --> P[Pick: choose one candidate]
    P --> B[Host: validate and book capacity]

Loading

• Score receives the request context and all remaining candidates, then writes one cost per candidate. The default computes cache credit, load cost, decay, and preferred-taint cost here. Request-wide calculations happen once inside the scorer; there is no public prepare callback.
• Pick receives candidates with their summed costs and returns a row. The default handles minimum cost, ties, and temperature sampling. Unseeded sampling reads candidates directly; deterministic replay retains its sorted buffer.
• The host still owns eligibility, cache/load snapshots, accounting, reservations, cancellation, and completion. Peer updates reach the sequence tracker asynchronously. Plugins read the current snapshot; they do not synchronize trackers or wait for peers.

API and configuration

• Each component declares WorkerInputs for the data it needs. Cache, load, and preferred-taint access is gated independently for that component. Combined CACHE requirements determine whether the host starts cache indexing. The builtin omits CACHE at zero overlap credit and for plain disaggregated decode, preserving load-only accounting.
• Cache inputs contain worker overlap estimates, tier information, and borrowed shared-cache hits. There is no separate context.cache(). Exact prompt length remains available through context.prompt_tokens().
• Existing top-level router settings still configure the default. Named policy instances can configure dynamo-default-cost-fn through plugin parameters. Per-request overrides no longer replace its scoring weights or temperature.
• All in-tree plugins, examples, and affected documentation use the updated API. No compatibility shim is added.

Validation

• 416 targeted tests passed, covering load-only accounting, cache input gating, conditional decode, booking/release, and seeded/unseeded parity.
• All-target Clippy with warnings denied, formatting, and changed-file pre-commit passed.
• Fixed the Python test-link failure by returning a Rust error from input validation and constructing ValueError at the Python boundary. Both plain cargo test --locked --no-run and cargo test --locked pass without extra linker flags.

Performance

Selection reuses candidate and cost buffers, computes request-wide values once per batch, and skips unused shared-cache scans. Warmed selection checks recorded zero allocations.

The short sampling comparison was inconclusive: 0.3% faster to 3.1% slower on a noisy shared host. The accepted stacked-scorer overhead remains a follow-up. End-to-end serving impact has not been measured.

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-01a0ca5a-5770-74bf-9fc2-4079c790a4eb.docs.buildwithfern.com/dynamo/dev

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[ishandhanani]

/ok to test 1c6a01b

[ishandhanani]

/ok to test 93c7415

[ishandhanani]

/ok to test efe5625

[ishandhanani]

/ok to test 03cd7cf

[ishandhanani]

/ok to test 140a13d

[ishandhanani]

/ok to test 9f796fe

[ishandhanani]

/ok to test 7dc4bb2

[jthomson04]

Two findings from the source review of feb8b34. I did not run tests or benchmarks, or inspect CI.

[nv-tusharma]

Not safe to merge yet.

Blocker: missing crate version

Cargo.toml line 58 defines the new workspace dependency without a version:

dynamo-custom-policy-builtin = { path = "lib/router-plugins/builtin" }

Publishable crates use this dependency. cargo package fails with:

dependency `dynamo-custom-policy-builtin` does not specify a version

Suggested fix:

dynamo-custom-policy-builtin = {
  path = "lib/router-plugins/builtin",
  version = "1.6.0"
}

The release process must publish this crate before its consumers.

Minor: stale workflow comment

pre-merge.yml line 543 describes a build without worker-selection policies.

The builtin policy dependency is now unconditional. Update the comment to mention only the optional policy catalog.

Validation

The reviewer confirmed:

• Locked offline Cargo metadata passes.
• Python and KVBM binding metadata passes.
• Current Rust tests and Clippy checks pass.
• Frontend multi-architecture builds pass.
• No new third-party versions enter the root lockfile.
• Pre-merge CI lacks a cargo package check.

Recommendation: add the workspace version, then package every affected publishable crate.

[dmitry-tokarev-nv]

Round 2 review at 1084384acf

The P1 from round 1 is fixed, and I proved it by building both sides of the commit that fixed it. One P2 and two P3 items remain, all non-blocking. Approving.

What I verified this round, and where I stopped

Scope. The pull request file list from gh api --paginate repos/ai-dynamo/dynamo/pulls/14785/files gives 83 files. A local three-dot diff against the merge base 25a8159d2d gives the same 83 files and the same names, with 4106 insertions and 1168 deletions.

Push shape. My round-1 head 1dca8375be is an ancestor of 1084384acf, so the 15 new commits are an append with no rewrite. There are no merge commits in the range. All 15 git patch-id --stable values are distinct. None of them matches any of the 16 patch identifiers already in 25a8159d2d..1dca8375be. The range is new content, not a replay. The timeline records 5 head_ref_force_pushed events, all dated 2026-09-18 and 2026-09-19, which is before round 1.

The P1 is fixed. Detail is on the lib/bindings/python/rust/llm/entrypoint.rs thread. 7101ccae97 is the commit. Its parent exits 101 with five undefined CPython symbols, and the commit itself exits 0.

CI on this head. All four rust-tests jobs ran and passed. Two of them, rust-tests (.) and rust-tests (lib/bindings/kvbm), were cancelled on both heads I reviewed in round 1. rust-tests (lib/bindings/python) and rust-tests (lib/runtime/examples) also passed. All four rust-clippy jobs passed. The workspace test suite has now run on this branch.

The #15138 overlap grew. Neither head is an ancestor of the other. git merge-tree --write-tree between 1084384acf and ef6999b061 reports 7 content conflicts across 18 overlapping files, up from 5 conflicts across 16 files at round 1. The conflicts are Cargo.lock, custom-worker-selection.mdx, lib/kv-router/src/plugins/worker_selection.rs, lib/kv-router/src/scheduling/selector/mod.rs, lib/kv-router/src/scheduling/selector/policy.rs, lib/router-plugins/builtin/Cargo.toml, and lib/router-plugins/builtin/src/lib.rs. Neither branch assumes the other landed. Whichever merges second owns the resolution.

What I retracted. I opened this round suspecting that the CACHE gating in d83839ce7b was inconsistent, because scorer.rs:135 keeps the full overlap credit when a plain-decode policy tracks prefill tokens, while scorer.rs:236 denies CACHE for that policy in every case. I dropped it. lib/router-plugins/builtin/tests/default_policy.rs:239 asserts inputs.contains(WorkerInputs::CACHE) == (role != WorkerType::Decode || conditional_disagg_enabled) over 4 roles and 2 settings, so the contract is deliberate and pinned. The pull request body states it as well. What is left is a redundant condition, not a defect.

I also dropped a suspicion that the picker changes in feb8b346c7 and 3bf711fe4d alter the choice. The host rejects an empty candidate set at lib/kv-router/src/scheduling/selector/mod.rs:384 before pick runs, a single candidate takes no draw at zero temperature, and lib/kv-router/src/scheduling/selector/policy.rs:224 rejects non-finite costs before the picker sees them.

Where I stopped. I did not compare selection end to end between the merge base and this head. I therefore have no measurement that the chosen worker is identical for every input shape. I did not establish the linker-level mechanism behind the original P1. I did not establish the crate publish set, because this repository holds no publish tooling.