You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
perf(router): probe worker availability with an Fx set - #18008
Selection checks the frontend's available-worker set once per worker on every admission. That happens in candidate preparation, best_cached_tokens, the queue gate, and the scheduler actor's candidate collection when ranks are full. The set comes from Client::available_instance_ids() as a std HashSet, so with 2,048 workers each admission pays for thousands of SipHash probes. This PR converts the provider's set to an FxHashSet once per distinct set and probes that instead. The runtime hands out the same Arc until membership changes, so the conversion runs only when membership changes.
Busy thresholds on, scheduler actor (kcycles per admission)
451
377
−16%
Baseline, run_selection callers (kcycles per admission)
521
471
−10%
Baseline saturation / admission p50 at 4.6k req/s
12.1k / 0.60 ms
12.1k / 0.58 ms
flat
Setup: Tyche Grace node. Both arms ran at once, one per socket, swapping sockets between the two rounds, replaying the same corpus (equal corpus_digest). The cycle counts come from window-bracketed perf profiles. Busy thresholds: --active-decode-blocks-threshold 0.25 --active-prefill-tokens-threshold-frac 1.0. Bench: #17959.
Key Changes
AvailableWorkerIds (scheduling/types.rs) wraps the WorkerAvailabilityProvider and caches the last converted set against its source Arc. Holding the source keeps its address from being reused, so a pointer match means the same set. A per-request set, such as a LoRA filter's, is converted per request, at about the cost of building it.
SchedulerQueue wraps the provider once and shares it between the handle and the actor. RoutingEligibility and PreparedAdmission take the Fx set (WorkerIdSet).
any_eligible_worker_rank still walks the smaller of the allowed and available sets when one is sparse. Each set type now has its own branch.
The runtime client (RoutingInstances) now keeps its routable-ID set's Arc across rebuilds that leave membership unchanged (overload updates, expiry, reconciliation), as it already did for the overloaded set. Without this, each overload update forced a new conversion and a full compare in place of a pointer compare.
Validation
cargo test -p dynamo-kv-router --lib (1,164 passed, including a new test for the conversion cache).
cargo test -p dynamo-router-policies --test default_policy, cargo test -p dynamo-runtime --lib component::client (including a new test that overload updates keep the availability snapshot), and the lib/llm availability test.
No actionable comments were generated in the recent review.
Walkthrough
The scheduler now represents available workers with WorkerIdSet and a shared, cached AvailableWorkerIds snapshot. Eligibility filtering and queue processing use these types to obtain and evaluate available workers.
Updates eligibility APIs to accept WorkerIdSet. Sparse rank enumeration compares allowed and available set capacities and scans the smaller set when it is below the threshold.
Mixed LoRA and ordinary traffic may lose some of the intended admission-performance benefit, but worker selection remains correct. This is mergeable with awareness of that performance limitation.
Pre-merge checks: 4 · 1
Failed checks: 1 warning
Check name
Status
Explanation
Resolution
Docstring Coverage
Docstring coverage is 41.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta…
Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed checks: 4 passed
Check name
Status
Explanation
Title check
The title "perf(router): probe worker availability with an Fx set" matches the main change. The changeset replaces the std HashSet availability checks with an FxHashSet-based WorkerIdSet, cached throu…
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Description check
The description covers the Overview, Key Changes, and Validation sections, and it gives detailed benchmark data and a clear rationale for the change. It has no Issues and Ongoing Work list entry beyon…
Full details: Title check
Explanation
The title "perf(router): probe worker availability with an Fx set" matches the main change. The changeset replaces the std HashSet availability checks with an FxHashSet-based WorkerIdSet, cached through AvailableWorkerIds, and the PR objectives describe the same change. The title is specific and follows a conventional commit style.
Full details: Description check
Explanation
The description covers the Overview, Key Changes, and Validation sections, and it gives detailed benchmark data and a clear rationale for the change. It has no Issues and Ongoing Work list entry beyond a related reference (#17959), and it does not mention closing an issue. The template's "Closes #XXXX" line is absent. The Pull Request Checklist item ("I used Dynamo's /pr-checklist skill") is also not shown as checked. These gaps are non-critical, so the description is mostly complete.
Full details: Docstring Coverage
Explanation
Docstring coverage is 41.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 12 functions across 4 files.
Resolution
Write docstrings for the functions missing them to satisfy the coverage threshold.
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Reviewing files that changed from the base of the PR and between de49ef3 and eb426f4.
Files selected for processing: 4
lib/kv-router/src/scheduling/filter.rs
lib/kv-router/src/scheduling/queue.rs
lib/kv-router/src/scheduling/types.rs
lib/llm/src/kv_router.rs
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
4 jobs failed, all with the same 2 planner unit tests in test_live_retune_frontend_config.py: the frontend/worker argparser rejects --router-mode power-of-two (valid choices: round-robin, random, kv, direct, least-loaded, device-aware-weighted). Both architectures fail identically, so this is not an infrastructure problem.
❌ planner / CPU Test cpu, amd64: invalid choice 'power-of-two' for --router-mode
Job: planner / CPU Test cpu, amd64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 126400381798 -R ai-dynamo/dynamo --log-failed
_______________ test_effective_argv[command7-args7-power_of_two] _______________
E AssertionError: P4: cannot parse the frontend args: argument --router-mode: invalid choice: 'power-of-two' (choose from 'round-robin', 'random', 'kv', 'direct', 'least-loaded', 'device-aware-weighted')
E assert False
______ test_worker_router_mode_matches_the_real_worker_parser[args3-env3] ______
E argparse.ArgumentError: argument --router-mode: invalid choice: 'power-of-two' (choose from 'round-robin', 'random', 'kv', 'direct', 'least-loaded', 'device-aware-weighted')
E SystemExit: 2
FAILED components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_effective_argv[command7-args7-power_of_two]
FAILED components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_worker_router_mode_matches_the_real_worker_parser[args3-env3]
============ 2 failed, 4170 passed, 30 skipped in 435.37s (0:07:15) ============
Two tests parametrized with router mode power_of_two fail because the real frontend and worker parsers no longer accept power-of-two as a --router-mode choice (failed on 3 auto-retries each). The test cases and the parser's allowed router modes disagree.
❌ planner / CPU Test cpu, arm64: same 2 tests, invalid choice 'power-of-two'
Job: planner / CPU Test cpu, arm64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 126400381759 -R ai-dynamo/dynamo --log-failed
FAILED components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_effective_argv[command7-args7-power_of_two] - AssertionError: P4: cannot parse the frontend args: argument --router-mode: invalid choice: 'power-of-two' (...)
FAILED components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_worker_router_mode_matches_the_real_worker_parser[args3-env3] - SystemExit: 2
== 2 failed, 7778 passed, 170 skipped, 2854 deselected in 1588.81s (0:26:28) ===
##[error]A pytest stage did not record a passing result. See this step's log for the stage.
The runtime suite picks up the same 2 planner tests and fails on the same --router-mode power-of-two parse error; nothing else failed. The "custom container implementation failed" lines after it just report the non-zero pytest exit.
❌ dynamo-runtime / test / sequential cuda13.0, arm64: same 2 planner tests
Selection checks the provider's available-worker set once per worker on every
admission: candidate preparation, best cached tokens, the queue gate, and the
actor's candidate collection. The set is a std HashSet, so each check paid for
SipHash. Convert the provider's set to an FxHashSet once per distinct set (the
frontend's provider returns the same Arc until membership changes) and probe
that instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…ate policy tests
RoutingInstances rebuilt the routable-ID set on every overload update,
expiry and reconciliation, even with membership unchanged, so each one made
the scheduler convert the available set again and fall back from a pointer
compare to a full compare of prepared rows. Share the set across rebuilds
that leave the routable IDs unchanged, as the overloaded set already is.
Build the builtin policy tests' availability sets as WorkerIdSet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Third upstream merge for the split v2 PR: the five commits past
53015f9 that landed while the second merge was validated. Four
conflicts, each placed in the owner crate, upstream behavior whole, the
stack's moves intact, no shims.
- ai-dynamo#18008 probe worker availability with an Fx set: WorkerIdSet and the
per-set AvailableWorkerIds cache in the kernel's types.rs, the listed-
worker walk in the kernel's filter.rs, the actor's cached availability in
queue.rs (auto-merged); WorkerIdSet exported to hosts through
dynamo_scheduler::host (lib/llm's LoRA availability filter) and to
plugins through dynamo_kv_router::plugins::prelude (the builtin
default_policy test stays prelude-only).
- ai-dynamo#18084 CRTC stale-leaf sweep trigger scaled with tree size:
lib/router/kv-indexer concurrent_radix_tree_compressed (auto-merged).
- ai-dynamo#17900 shared fixed-lane bench primitive: lib/bench (auto-merged).
- ai-dynamo#17947 planner priority pools, ai-dynamo#17625 beam tool-call finalization:
outside the router, auto-merged.
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Selection checks the frontend's available-worker set once per worker on every admission. That happens in candidate preparation,
best_cached_tokens, the queue gate, and the scheduler actor's candidate collection when ranks are full. The set comes fromClient::available_instance_ids()as a stdHashSet, so with 2,048 workers each admission pays for thousands of SipHash probes. This PR converts the provider's set to anFxHashSetonce per distinct set and probes that instead. The runtime hands out the sameArcuntil membership changes, so the conversion runs only when membership changes.selection_replay_bench(AgentX workload, 2,048 workers)run_selectioncallers (kcycles per admission)Setup: Tyche Grace node. Both arms ran at once, one per socket, swapping sockets between the two rounds, replaying the same corpus (equal
corpus_digest). The cycle counts come from window-bracketedperfprofiles. Busy thresholds:--active-decode-blocks-threshold 0.25 --active-prefill-tokens-threshold-frac 1.0. Bench: #17959.Key Changes
AvailableWorkerIds(scheduling/types.rs) wraps theWorkerAvailabilityProviderand caches the last converted set against its sourceArc. Holding the source keeps its address from being reused, so a pointer match means the same set. A per-request set, such as a LoRA filter's, is converted per request, at about the cost of building it.SchedulerQueuewraps the provider once and shares it between the handle and the actor.RoutingEligibilityandPreparedAdmissiontake the Fx set (WorkerIdSet).any_eligible_worker_rankstill walks the smaller of the allowed and available sets when one is sparse. Each set type now has its own branch.RoutingInstances) now keeps its routable-ID set'sArcacross rebuilds that leave membership unchanged (overload updates, expiry, reconciliation), as it already did for the overloaded set. Without this, each overload update forced a new conversion and a full compare in place of a pointer compare.Validation
cargo test -p dynamo-kv-router --lib(1,164 passed, including a new test for the conversion cache).cargo test -p dynamo-router-policies --test default_policy,cargo test -p dynamo-runtime --lib component::client(including a new test that overload updates keep the availability snapshot), and the lib/llm availability test.cargo clippy --all-targets -p dynamo-kv-router -p dynamo-router-policies -p dynamo-worker-selection-policy-catalog -p dynamo-llm -p dynamo-runtime -- -D warnings.Issues and Ongoing Work
selection_replay_bench).🤖 Generated with Claude Code