Skip to content

docs(stage-router): mark capable_first experimental, and warn at startup - #363

Merged
sabhatinas merged 1 commit into
mainfrom
sabhatinas/capable-first-experimental
Aug 11, 2026
Merged

docs(stage-router): mark capable_first experimental, and warn at startup#363
sabhatinas merged 1 commit into
mainfrom
sabhatinas/capable-first-experimental

Conversation

@sabhatinas

@sabhatinas sabhatinas commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Every published threshold and routing result comes from efficient_first runs. capable_first works and the server accepts it, but it has not been benchmarked — so there is no calibrated confidence_threshold for it and no measured accuracy or cost to set expectations against.

Docs now lead with efficient_first and carry a warning admonition; the TOML schema table marks capable_first experimental. The server also logs a warning at startup when a stage_router route selects it, so the caveat reaches operators who never read the page:

WARN switchyard_server::config: stage_router route stage uses picker "capable_first",
which is experimental: published thresholds and routing results all come from
"efficient_first", so there is no calibrated confidence_threshold for it and no
measured accuracy or cost. Use "efficient_first" unless you are running your own
calibration.

Verified the warning fires for capable_first and stays silent for efficient_first (--dry-run on both). cargo test -p switchyard-server, clippy, cargo fmt --check and mkdocs build --strict all pass.

Summary by CodeRabbit

  • New Features

    • Added a server startup warning when the experimental capable_first routing picker is selected.
  • Documentation

    • Clarified that efficient_first is the recommended picker.
    • Documented that published thresholds and routing results are based on efficient_first.
    • Noted that capable_first is unbenchmarked and lacks calibrated metrics.

Every published threshold and routing result comes from efficient_first
runs. capable_first works and the server accepts it, but it has not been
benchmarked, so there is no calibrated confidence_threshold for it and no
measured accuracy or cost to set expectations against.

The docs now list efficient_first first and carry a warning admonition
saying so, and the TOML schema table marks capable_first experimental. The
server logs a warning at startup when a stage_router route selects it, so
the caveat reaches operators who never read the page.

Signed-off-by: Sabhatina Selvam <sabhatinas@nvidia.com>
@sabhatinas
sabhatinas requested a review from a team as a code owner August 11, 2026 18:56
@github-actions

Copy link
Copy Markdown
PR Preview Action v1.8.1

🚀 View preview at
https://NVIDIA-NeMo.github.io/Switchyard/pr-preview/pr-363/

Built to branch gh-pages at 2026-08-11 18:58 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

@ayushag-nv ayushag-nv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good

@coderabbitai

coderabbitai Bot commented Aug 11, 2026

Copy link
Copy Markdown

Review Change Stack

Walkthrough

The server now warns when a stage_router uses capable_first. The reference and routing documentation identify efficient_first as the calibrated option and describe capable_first as experimental and unbenchmarked.

Changes

Stage router picker guidance

Layer / File(s) Summary
Picker warning and documentation
crates/switchyard-server/src/config.rs, docs/reference/toml_schema.md, docs/routing_algorithms/stage_router_routing.md
build_algorithm warns when PickerMode::CapableFirst is configured. The documentation lists efficient_first first and describes capable_first as experimental, unbenchmarked, and lacking calibrated thresholds and metrics.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Poem

I’m a rabbit by the route,
Hopping warnings into place.
“Choose efficient_first,” I toot,
While capable_first joins the chase.
Docs now guide each picker’s race.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main changes: marking capable_first as experimental and adding a startup warning.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/routing_algorithms/stage_router_routing.md`:
- Around line 70-85: Update the nearby documentation sentence referring to the
“default capable_first picker” so it instead says “the capable_first picker.”
Preserve the surrounding explanation and align the wording with the required
picker configuration and efficient_first recommendation.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2afd415b-ab96-417e-9508-8d070a80c51e

📥 Commits

Reviewing files that changed from the base of the PR and between 1e5767b and 137e12e.

📒 Files selected for processing (3)
  • crates/switchyard-server/src/config.rs
  • docs/reference/toml_schema.md
  • docs/routing_algorithms/stage_router_routing.md

Comment thread docs/routing_algorithms/stage_router_routing.md
@sabhatinas
sabhatinas merged commit 22b78d9 into main Aug 11, 2026
19 checks passed
@sabhatinas
sabhatinas deleted the sabhatinas/capable-first-experimental branch August 11, 2026 19:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants