[ishandhanani]

Part of #16543 (PR 4 of 5)

Summary

In KV mode, busy-worker rejection ran on a separate path: router load snapshots went to the worker monitor, the monitor compared them against the busy thresholds, and its overloaded set fed back into selection. This PR has the KV router's queue apply the thresholds itself, on the same rows and check as everything else.

• Routing the thresholds does not mute the worker monitor. The queue judges the load this router booked; the monitor keeps the decode threshold and applies it only to worker-published load (kv_used_blocks, or active_decode_blocks when a worker reports it), the signals the queue cannot see (generated tokens, requests from other frontends, prefill workers that track no active blocks); scheduler snapshots of booked load neither feed that state nor move its latch once routed. Selection skips a worker that either check marks busy.
• The reject thresholds are judged in the pass that already projects each rank's load for the request: the registry keeps each rank's reported capacity beside its load (ActiveSequencesMultiWorker::set_worker_capacities, called when LocalScheduler reconciles worker configs), the projection pass hands every rank's projection and capacity to a ProjectedLoadObserver, and the queue's observer counts ranks past the thresholds per worker (BusyThresholds::has_room, the check classifiers and the class-queue gate use). RoutingEligibility consults only that count, so a worker whose ranks are all past the thresholds is skipped, no catalog is scanned, no set is merged with the host's overloaded set, and a pinned request judges one worker. While no rank is full the count is empty and selection does no extra work. If no eligible worker remains, the request gets AllEligibleWorkersOverloaded (529), as before.
• Thresholds stay live. POST /busy_threshold updates the same handle, and the queue reads it at each selection.
• Once a KV router takes over the thresholds, the queue owns the booked-load checks and the worker monitor keeps only the decode check on worker-published load; both feed selection, which skips a worker either marks busy. Request-path overload leases (worker-reported overload) still feed the client's overloaded set the same way. Non-KV routing is unchanged.
• Rejection is host-owned eligibility, not a WorkerFilter in the selection policy: a custom policy replaces the filter list, and "all filtered" maps to a different error than overload.
• Request classifiers can read both threshold sets that apply to a request and run the same has_room the router runs: queue_thresholds() (the class the request resolves to now, including an override), queue_thresholds_for(name) (another family or class, resolved with this request's uncached tokens), and reject_thresholds() (router-wide, read when classification started). The two sets stay separate because they mean different things: the queue waits on class thresholds and selection rejects on router-wide ones. Selection plugins do not get them yet; that waits for a consumer such as a placement filter.

Behavior differences in KV mode: the check runs synchronously at selection on fresh booked load, where the monitor applied it asynchronously after each load snapshot. The monitor's decode latch still clears only once every signal it judges is under the threshold; once routed it judges worker-published load alone, so a stale booked snapshot can no longer hold it.

Validation

• reject_thresholds_shed_full_workers_and_reject_when_none_remain: a worker holding 6 of 8 KV blocks is skipped, and the request is rejected once both are full. Fails with the check disabled.
• classifiers_read_class_and_reject_thresholds: a classifier reads its class's thresholds, another family's for this request's bucket, None for an unknown class, the router-wide ones, and the new class's thresholds after set_policy_class.
• llm: routed_busy_thresholds_stay_live_and_hand_booked_load_to_the_queue (routing marks the monitor routed; a /busy_threshold update reaches the routed handle; an encode context does not route), routed_monitor_keeps_only_the_engine_decode_check, engine_only_ignores_scheduler_snapshots_and_keeps_worker_published_load, and engine_only_latch_is_held_and_released_by_worker_published_load_alone.
• Mocker e2e (tests/router/test_router_e2e_with_mockers.py): 7 passed: test_mocker_kv_router_overload_529 (decode-blocks, prefill-tokens), test_mocker_disagg_router_overload_529 (both), threshold_none_disables_rejection, busy_threshold_endpoint (both planes). With the queue-side check disabled, all 4 overload-529 cases fail.
• cargo test -p dynamo-kv-router --features standalone-indexer --lib: 1181 passed, including reject_thresholds_follow_the_live_provider_on_prefill_tokens (prefill thresholds, provider flipped at runtime) and the shed test now pinning to the full worker. cargo test -p dynamo-llm --lib -- kv_router:: discovery:: busy_threshold: 696 passed, including load_threshold_handle_merges_updates_and_shares_state and busy_thresholds_from_load_threshold_config. cargo clippy --workspace --all-targets -- -D warnings: clean.
• Review round: the reject-threshold provider is a constructor argument of LocalScheduler::new / SchedulerQueue::new (no OnceLock, no hidden setter); one RoutableThresholds owns the live handle and the routed switch, built before the sender and the monitor; impl From<LoadThresholdConfig> for BusyThresholds names the conversion; the profile is shared through one Arc; uncached_tokens comes from QueueSnapshot; LoadThresholdHandle is an ArcSwap, so the per-request reads on worker threads are wait-free and a /busy_threshold write never stalls them.

Stack

1. feat(router): expose worker inputs to request classifiers #16616 feat(router): expose worker inputs to request classifiers
2. refactor(router): judge queue capacity on the shared LOAD row #16617 refactor(router): judge queue capacity on the shared LOAD row
3. feat(router): add a decode busy threshold to policy-class queues #16618 feat(router): add a decode busy threshold to policy-class queues
4. feat(router): enforce busy-reject thresholds in the KV router queue #16619 feat(router): enforce busy-reject thresholds in the KV router queue (this PR)
5. refactor(router): unify classifier inputs; deprecate absolute busy thresholds #16620 refactor(router): rename classifier contexts; deprecate absolute busy thresholds

Summary by CodeRabbit

• New Features

  • In KV router mode, worker availability is checked against current thresholds when selecting a worker. Workers whose ranks are all busy are skipped, and requests are rejected if no eligible worker remains.
  • Custom rejection thresholds can now guide worker selection alongside existing overload checks.

• Documentation

  • Clarified when busy-threshold updates take effect and how queue and rejection thresholds apply to requests and workers.

• Reject-threshold cost on the selection path (selection_core, no_hints_reject_off/on, bench-only constructor installing a provider that sheds nothing; six alternating rounds pinned to one core, medians, per select-and-reserve; A = reject check evaluated per worker in RoutingEligibility, B = judged in the load projection pass): 512 workers 68.1/76.9 us -> 68.1/72.4 us (off/on; configured-threshold cost +12.9% -> +6.2%), 4096 workers 474.6/583.7 us -> 476.4/527.8 us (+23.0% -> +10.8%); thresholds-on alone -5.8% and -9.6%. With no thresholds configured the two are equal within noise. The remaining cost of configured thresholds is the has_room arithmetic per rank inside the pass; it no longer re-reads each rank's load.

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-01a17be8-6064-75c8-8d22-ea7a0c0ffa8a.docs.buildwithfern.com/dynamo/dev

[devin-ai-integration]

✅ Dynamo PR CI passed — run 40457088709 (attempt 1) on 7ae4f08499

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

_{Posted automatically by Devin for run 40457088709. Updated on every full-CI run of this PR.}

[ishandhanani]

The direction is right: the queue is the only place with fresh booked load, and exposing both threshold sets on the plugin contexts closes the DEP's "plugins cannot see either version" gap. The mocker e2e coverage of the 529 cases is the right gate.

The main inline is the per-selection cost of the reject scan when thresholds are configured, measured with the selection_core bench. The rest is consolidation: the same two-field struct defined twice, a field set after construction, and small duplications.

[jthomson04]

One blocking concern: routing the thresholds drops the engine-reported kv_used_blocks decode signal. One simplification on the provider wiring.

[jthomson04]

One note on the shared lock this adds to the per-request path.

[jthomson04]

Stale docs after the kv_used_blocks split, plus an indent nit.

[jthomson04]

Lazy reject check and routed monitor verified on ab65570.