docs(routing): update routing internals & enterprise config - #2462
Merged
Merged
Conversation
Bring the routing docs in line with the current implementation: - Replace the stale weight percentages (uptime 50/throughput 20/price 20/latency 10) with the actual default relative weights (price 0.6, uptime 0.5, throughput 0.05, latency 0.025, cache 0.2, imagePrice 1.0) and explain the ratio-based normalized scoring. - Document the time-decayed 60-minute metrics window (1m ×10, 5m ×3, rest ×1) instead of the inaccurate "last 5 minutes" snapshot. - Add the previously undocumented provider-priority mechanism. - Note that the uptime-penalty, exploration-rate, and low-uptime fallback thresholds are configurable. - Add a "Per-Project Routing Configuration (Enterprise)" section documenting that all routing values can be customized per project on the Enterprise plan via Project Settings -> Routing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Contributor
WalkthroughThis PR updates the Smart Routing documentation to describe a new weighted scoring algorithm replacing the older "last 5 minutes" model, adds explicit default weights and normalization mechanics, documents 60-minute time-decayed metrics aggregation, and introduces cache weighting for large prompts. It also documents provider priorities, epsilon-greedy exploration, and adds a new Enterprise section for per-project routing configuration. ChangesRouting Algorithm and Enterprise Configuration
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
Updates the Routing documentation to match the current routing implementation and adds documentation for Enterprise per-project routing overrides.
Changes:
- Replaces outdated fixed-percentage scoring descriptions with the current normalized relative-weight scoring model (including cache and image pricing behavior).
- Corrects the described metrics window to the current time-decayed 60-minute aggregation.
- Documents provider priority and adds an Enterprise “Per-Project Routing Configuration” section outlining configurable groups and defaults.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| - **Throughput (20%)** - Favors providers with higher tokens per second generation speed | ||
| - **Price (20%)** - Considers cost efficiency while maintaining quality | ||
| - **Latency (10%)** - Considers time to first token (only applied for streaming requests) | ||
| Each factor has a **relative weight**. The factors are scored as ratios against the best provider in the candidate set (e.g. a provider that is twice as expensive as the cheapest scores `1.0` on price), and each ratio is multiplied by its weight divided by the sum of all active weights. The provider with the lowest (best) total score wins. |
Comment on lines
+64
to
+66
| - The most recent **1 minute** is weighted **10×** | ||
| - The most recent **5 minutes** are weighted **3×** | ||
| - The remainder of the 60-minute window is weighted **1×** |
|
|
||
| **Provider Priority**: | ||
|
|
||
| Each provider has a **priority** value (default `1`) that nudges routing toward or away from it independently of live metrics: |
| | **Timeouts** | Per-request time limits (end-to-end, streaming, non-streaming). Capped at the infrastructure defaults — an override can only lower them | `gatewayMs 1,500,000`, `streamingMs 1,200,000`, `plainMs 600,000` | | ||
| | **History** | The metrics window and the time-decay tier boundaries and weights | `windowMinutes 60` (max 120), `tier1Minutes 1`, `tier2Minutes 5`, `tier1Weight 10`, `tier2Weight 3`, `tier3Weight 1` | | ||
| | **Sticky** | Stable-provider preference: on/off, TTL, hard-switch uptime floor, soft-switch score margin | `enabled true`, `ttlSeconds 3600`, `uptimeThreshold 85`, `scoreMargin 0.15` | | ||
| | **Provider priorities** | Per-provider priority multipliers; set a provider to `0` to disable it for that project | `1` for every provider | |
| | **Weights** | Relative importance of each scoring factor | `price 0.6`, `imagePrice 1.0`, `uptime 0.5`, `throughput 0.05`, `latency 0.025`, `cache 0.2` | | ||
| | **Thresholds** | Cache prompt-size threshold, uptime-penalty threshold, exploration rate, and the assumed defaults used when no metrics exist | `cachePromptTokens 5000`, `uptimePenalty 95`, `defaultUptime 100`, `defaultLatency 1000`, `defaultThroughput 50`, `explorationRate 0.01` | | ||
| | **Retry** | Max cross-provider fallback attempts and the low-uptime reroute threshold | `maxRetries 2`, `lowUptimeFallbackThreshold 90` | | ||
| | **Timeouts** | Per-request time limits (end-to-end, streaming, non-streaming). Capped at the infrastructure defaults — an override can only lower them | `gatewayMs 1,500,000`, `streamingMs 1,200,000`, `plainMs 600,000` | |
steebchen
enabled auto-merge
May 31, 2026 10:07
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
apps/docs/content/features/routing.mdx (1)
64-66: 💤 Low valueConsider clarifying the time-decay tier boundaries.
The phrasing "most recent 1 minute" and "most recent 5 minutes" could be interpreted as overlapping ranges. Consider making it explicit whether these are exclusive tiers or cumulative:
If the tiers are exclusive (most likely):
- Minutes 0–1: weighted 10×
- Minutes 1–5: weighted 3×
- Minutes 5–60: weighted 1×
♻️ Suggested clarification
-Provider metrics (uptime, throughput, latency) are not a flat "last N minutes" snapshot. They are aggregated over a rolling **60-minute window** with a time-decay weighting so very recent behavior dominates while older data still contributes: - -- The most recent **1 minute** is weighted **10×** -- The most recent **5 minutes** are weighted **3×** -- The remainder of the 60-minute window is weighted **1×** +Provider metrics (uptime, throughput, latency) are not a flat "last N minutes" snapshot. They are aggregated over a rolling **60-minute window** with a time-decay weighting so very recent behavior dominates while older data still contributes: + +- **0–1 minutes ago**: weighted **10×** +- **1–5 minutes ago**: weighted **3×** +- **5–60 minutes ago**: weighted **1×**🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/docs/content/features/routing.mdx` around lines 64 - 66, Update the phrasing around the time-decay tiers to explicitly show exclusive boundaries so readers don’t interpret them as overlapping; replace the ambiguous lines "most recent 1 minute", "most recent 5 minutes", "remainder of the 60-minute window" with explicit ranges such as "Minutes 0–1: weighted 10×", "Minutes 1–5: weighted 3×", and "Minutes 5–60: weighted 1×" (referencing the existing phrases "most recent 1 minute", "most recent 5 minutes", and "60-minute window" in the current text).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/docs/content/features/routing.mdx`:
- Line 41: Update the routing docs text to reflect how priceScore is actually
computed: state that priceScore = (providerPrice / minPrice) - 1 so the cheapest
provider scores 0 and a provider twice as expensive scores 1.0, and explicitly
mention the “-1 offset” rather than describing it as a raw ratio; reference the
terms priceScore and minPrice used in the implementation for clarity.
---
Nitpick comments:
In `@apps/docs/content/features/routing.mdx`:
- Around line 64-66: Update the phrasing around the time-decay tiers to
explicitly show exclusive boundaries so readers don’t interpret them as
overlapping; replace the ambiguous lines "most recent 1 minute", "most recent 5
minutes", "remainder of the 60-minute window" with explicit ranges such as
"Minutes 0–1: weighted 10×", "Minutes 1–5: weighted 3×", and "Minutes 5–60:
weighted 1×" (referencing the existing phrases "most recent 1 minute", "most
recent 5 minutes", and "60-minute window" in the current text).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: ddeab870-cffa-40d5-86a3-76f04c0acb25
📒 Files selected for processing (1)
apps/docs/content/features/routing.mdx
| - **Throughput (20%)** - Favors providers with higher tokens per second generation speed | ||
| - **Price (20%)** - Considers cost efficiency while maintaining quality | ||
| - **Latency (10%)** - Considers time to first token (only applied for streaming requests) | ||
| Each factor has a **relative weight**. The factors are scored as ratios against the best provider in the candidate set (e.g. a provider that is twice as expensive as the cheapest scores `1.0` on price), and each ratio is multiplied by its weight divided by the sum of all active weights. The provider with the lowest (best) total score wins. |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Search for the price ratio calculation in the routing logic
rg -nP -C5 --type=ts 'price.*ratio|score.*price' --glob '*get-cheapest-from-available-providers.ts' --glob '*routing*.ts'Repository: theopenco/llmgateway
Length of output: 2465
🏁 Script executed:
#!/bin/bash
set -euo pipefail
FILE="packages/actions/src/get-cheapest-from-available-providers.ts"
# 1) Find likely scoring/ratio variables
rg -n "score|ratio|normalize|normalized|bestPrice|minPrice|priceScore|priceWeight|effectivePriceWeight|uptime|timeDecay|decay" "$FILE"
# 2) Pull the sections around providerScores construction (where scoring is likely computed)
# Use a broader context window around "providerScores" occurrences
rg -n "providerScores" "$FILE" -C20
# 3) Find explicit price normalization/comparison
rg -n "best.*price|min.*price|price.*(best|min)|cheapest.*price|price.*ratio" "$FILE" -C10
# 4) If weights are used to compute a total score, locate "totalScore" / "weighted" math
rg -n "totalScore|weighted|weights\.|sum.*weights|active weights|dominates|dominant" "$FILE" -C20Repository: theopenco/llmgateway
Length of output: 23995
Correct the wording for the “price ratio” example in routing docs: the implementation computes priceScore as (providerPrice / minPrice) - 1 (where minPrice is the cheapest provider), so a provider that is 2x as expensive yields priceScore = 1.0—the example at apps/docs/content/features/routing.mdx line 41 is numerically consistent. Update the text to clarify the -1 offset (cheapest scores 0, 2x scores 1.0) rather than implying a raw ratio.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/docs/content/features/routing.mdx` at line 41, Update the routing docs
text to reflect how priceScore is actually computed: state that priceScore =
(providerPrice / minPrice) - 1 so the cheapest provider scores 0 and a provider
twice as expensive scores 1.0, and explicitly mention the “-1 offset” rather
than describing it as a raw ratio; reference the terms priceScore and minPrice
used in the implementation for clarity.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The routing docs (
apps/docs/content/features/routing.mdx) had drifted from the actual implementation. This brings them back in line and documents the Enterprise per-project routing configuration.What changed
price 0.6,uptime 0.5,throughput 0.05,latency 0.025,cache 0.2, andimagePrice 1.0(replaces price for image models). Added a table and an explanation of the ratio-based normalized scoring. Source:packages/shared/src/routing-config.ts,packages/actions/src/get-cheapest-from-available-providers.ts.packages/db/src/provider-metrics-history.ts.1,0disables a provider) — previously undocumented.apps/gateway/src/lib/routing-config-loader.ts(gated onorgPlan === "enterprise"), UI atapps/ui/.../settings/routing.Verification
pnpm format✅pnpm build✅ (17/17, docs MDX compiles)🤖 Generated with Claude Code
Summary by CodeRabbit