Skip to content

refactor(combo): decompose combo.ts and enforce the live latency budget #53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
LMPrado-DZ23 merged 6 commits into from
Sep 20, 2026
Merged

refactor(combo): decompose combo.ts and enforce the live latency budget #53

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
7c8335b
refactor(combo): move the auto-strategy candidate builder out of comb…
zodyprado-web Sep 19, 2026
7dbeb73
Merge remote-tracking branch 'origin/release/v3.8.55' into refactor/c…
zodyprado-web Sep 20, 2026
d315f07
refactor(combo): split the candidate builder into one function per si…
zodyprado-web Sep 20, 2026
2585b4a
feat(routing): enforce the request latency budget on live traffic
zodyprado-web Sep 20, 2026
a037ecc
Merge remote-tracking branch 'origin/release/v3.8.55' into refactor/c…
zodyprado-web Sep 20, 2026
764872d
test(routing): assert the family-scoped quota fetch, not combo.ts's s…
zodyprado-web Sep 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion config/quality/file-size-baseline.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -425,7 +425,7 @@
"open-sse/mcp-server/server.ts": 1572,
"open-sse/services/accountFallback.ts": 2467,
"open-sse/services/adobeFireflyBrowserLogin.ts": 1401,
"open-sse/services/combo.ts": 4080,
"open-sse/services/combo.ts": 3779,
"open-sse/translator/response/openai-responses.ts": 1466,
"open-sse/utils/cursorAgentProtobuf.ts": 1547,
"open-sse/utils/proxyFetch.ts": 1271,
Expand Down
45 changes: 26 additions & 19 deletions docs/routing/ROUTING_CONTRACT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: "Routing Contract and Route Explanation"
version: 3.8.54
lastUpdated: 2026-09-18
lastUpdated: 2026-09-19
---

# Routing Contract and Route Explanation
Expand All @@ -17,7 +17,7 @@ The types live in `src/shared/contracts/routing.ts` and are shared by `src/` and
| Type | Purpose |
| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `RoutingRequest` | What is routed: `requestId`, `model`, `protocol`, optional `capabilities`, `workspaceId`, `policyId`, `stream`, `budget` |
| `RoutingBudget` | Optional `maxCost` (USD) and `maxLatencyMs`; read by previews and by the `attemptPolicy.ts` library, not by live traffic (see Guarantees) |
| `RoutingBudget` | Optional `maxCost` (USD) and `maxLatencyMs`; previews apply both. Live traffic applies `maxLatencyMs` only, from `X-OmniRoute-Latency-Budget` (see Guarantees) |
| `RoutingCandidate` | One provider/model: `score`, weighted `factors`, `eligible`, `exclusionReasons`, `quota`, `circuit`, estimated cost and latency |
| `RoutingDecision` | `decisionId`, `requestId`, `selected`, `candidates`, `policyVersion`, `generatedAt`, `liveRequestExecuted`, `selectionMode`, `strategy`, optional `omittedCandidates` |
| `ProviderAttempt` | One upstream call: provider, model, attempt number, start time, duration, status, `outcome` |
Expand All @@ -27,17 +27,18 @@ Candidates never carry connection or account identifiers, prompts or credentials

## Guarantees

| Guarantee | How it holds |
| ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Preview uses the live selection algorithm | Live selection and preview both run `selectProviderWithTrace()` in `open-sse/services/autoCombo/engine.ts` |
| Preview calls no provider, changes no state | The preview passes `previewSelectionDeps()`: clones of the self-healing and rotation state, exploration off |
| Policy version on every decision | `computeRoutingPolicyVersion()` hashes combo name, candidate pool, weights, mode pack, budget, exploration rate and router strategy (`rp_` + 16 hex characters) |
| Every exclusion has a reason | `hardExclusionReasons()` and the engine trace in `open-sse/services/autoCombo/routingDecision.ts` |
| Unknown quota is not exhausted | `quota: "unknown"` stays eligible and is scored neutral; only a quota cutoff produces `quota_exhausted` |
| No blind retry of permanent errors | Live: the combo loops retry the same target only on 408, 429, 500, 502, 503 and 504 (`isRetryableAttemptStatus()` in `open-sse/services/routing/attemptPolicy.ts`) |
| Auto combo failover stays in the cost cap | Live, `rules` router strategy only: `orderTargetsByCostBudget()` drops (`strict`) or moves last (`cheapest`) targets whose estimated 1K-token request cost exceeds `budgetCap`. Explicit router strategies (`cost`, `latency`, `lkgp`, ...) ignore `budgetCap` |
| Deterministic circuit breaker | `src/shared/utils/circuitBreaker.ts` accepts an injected clock; `peekState()` reads the effective state without changing it |
| End-to-end correlation | Live decisions are recorded under the request id the client receives (`x-request-id`) and under their decision id |
| Guarantee | How it holds |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Preview uses the live selection algorithm | Live selection and preview both run `selectProviderWithTrace()` in `open-sse/services/autoCombo/engine.ts` |
| Preview calls no provider, changes no state | The preview passes `previewSelectionDeps()`: clones of the self-healing and rotation state, exploration off |
| Policy version on every decision | `computeRoutingPolicyVersion()` hashes combo name, candidate pool, weights, mode pack, budget, exploration rate and router strategy (`rp_` + 16 hex characters) |
| Every exclusion has a reason | `hardExclusionReasons()` and the engine trace in `open-sse/services/autoCombo/routingDecision.ts` |
| Unknown quota is not exhausted | `quota: "unknown"` stays eligible and is scored neutral; only a quota cutoff produces `quota_exhausted` |
| No blind retry of permanent errors | Live: the combo loops retry the same target only on 408, 429, 500, 502, 503 and 504 (`isRetryableAttemptStatus()` in `open-sse/services/routing/attemptPolicy.ts`) |
| Auto combo failover stays in the cost cap | Live, `rules` router strategy only: `orderTargetsByCostBudget()` drops (`strict`) or moves last (`cheapest`) targets whose estimated 1K-token request cost exceeds `budgetCap`. Explicit router strategies (`cost`, `latency`, `lkgp`, ...) ignore `budgetCap` |
| Opt-in live latency budget | Live, every router strategy, only when the request sends `X-OmniRoute-Latency-Budget: <ms>`: a candidate whose `estimatedLatencyMs` (its p95 estimate) exceeds the budget is excluded from selection and dropped from the failover chain, and is reported with `latency_over_budget`. If nothing fits, the request gets a 503 instead of an over-budget answer. Without the header routing is unchanged |
| Deterministic circuit breaker | `src/shared/utils/circuitBreaker.ts` accepts an injected clock; `peekState()` reads the effective state without changing it |
| End-to-end correlation | Live decisions are recorded under the request id the client receives (`x-request-id`) and under their decision id |

## Previewing a decision

Expand Down Expand Up @@ -101,13 +102,19 @@ contains prompts, credentials or connection identifiers.

- Decisions are recorded for the `auto` combo strategy; other combo strategies keep the combo
decision trace (`/api/usage/combo-trace/{id}`).
- Live candidates do not yet distinguish unknown quota from a known value, because the candidate
builder lives in a file at its size cap; preview candidates do.
- Live traffic has no per-request `RoutingBudget` input. The only live cost limit is the auto
combo's `budgetCap` on the `rules` path, checked per attempt against a 1K-token estimate; there
is no cumulative spend check and no live latency budget.
- Live candidates do not yet distinguish unknown quota from a known value: the live candidate
builder (`open-sse/services/combo/autoCandidates.ts`) does not set `quotaKnown` yet; preview
candidates do.
- Live traffic has no per-request `maxCost` input. The only live cost limit is the auto combo's
`budgetCap` on the `rules` path, checked per attempt against a 1K-token estimate; there is no
cumulative spend check.
- The live latency budget is checked per attempt against each candidate's p95 latency estimate
(`exceedsLatencyBudget()` in `attemptPolicy.ts`), not against the time the request has already
spent: a failover attempt that fits the budget on its own is still tried even if earlier attempts
used part of it. A candidate with no latency estimate is not excluded. Only `auto` combos read it.
- `classifyAttemptOutcome()`, `isPermanentAttemptOutcome()`, `canRetrySameCandidate()`,
`checkFailoverBudget()` and `planNextAttempt()` are a tested library with no live caller yet.
`checkFailoverBudget()` and `planNextAttempt()` are a tested library with no live caller yet;
of that module, live traffic calls only `isRetryableAttemptStatus()` and `exceedsLatencyBudget()`.
Previews apply `maxCost` and `maxLatencyMs` as exclusions.
- Decisions are held in memory and are lost on restart. A stored live decision keeps at most 40
candidates (the selected one always), full factors only for the selected candidate and the 10
Expand Down
28 changes: 22 additions & 6 deletions open-sse/services/autoCombo/requestControls.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,13 @@
* X-OmniRoute-Mode: fast | balanced | quality | <raw mode-pack name> (#6024/#6025)
* X-OmniRoute-Budget: <max USD per request> (#6023)
* X-OmniRoute-Budget-Fallback: cheapest | strict (#3470)
* X-OmniRoute-Latency-Budget: <max estimated latency per attempt, ms>
*
* All resolvers are pure so they can be unit-tested and reused by the entry
* handler (src/sse/handlers/chat.ts) and the combo router (open-sse/services/combo.ts).
* The resolved values feed the auto-combo engine's existing `config.modePack` /
* `config.budgetCap` / `config.budgetFallback` inputs.
* `config.budgetCap` / `config.budgetFallback` inputs, and the latency budget feeds
* `RoutingBudget.maxLatencyMs` for the request.
*/

import { MODE_PACKS } from "./modePacks";
Expand Down Expand Up @@ -71,11 +73,20 @@ export function resolveRequestModePack(input: unknown): RequestModePack {
*/
export function parseRequestBudgetCap(input: unknown): number | undefined {
const n =
typeof input === "number"
? input
: typeof input === "string"
? Number(input.trim())
: NaN;
typeof input === "number" ? input : typeof input === "string" ? Number(input.trim()) : NaN;
if (!Number.isFinite(n) || n <= 0) return undefined;
return n;
}

/**
* Parse the `X-OmniRoute-Latency-Budget` header into a hard per-request latency ceiling in
* milliseconds (`RoutingBudget.maxLatencyMs`). Only a finite, strictly-positive duration is
* accepted; anything else returns `undefined`, and the request then routes with no latency budget
* at all — there is no stored combo-level latency budget to fall back to.
*/
export function parseRequestLatencyBudgetMs(input: unknown): number | undefined {
const n =
typeof input === "number" ? input : typeof input === "string" ? Number(input.trim()) : NaN;
if (!Number.isFinite(n) || n <= 0) return undefined;
return n;
}
Expand All @@ -101,6 +112,8 @@ export interface PerRequestAutoControls {
mode?: string;
budgetCap?: number;
budgetFallback?: RequestBudgetFallback;
/** `RoutingBudget.maxLatencyMs` for this request; absent means no latency budget. */
latencyBudgetMs?: number;
}

/**
Expand All @@ -116,14 +129,17 @@ export function resolveRequestAutoControls(headers: {
const modeHeader = headers.get("x-omniroute-mode")?.trim() || null;
const budgetHeader = headers.get("x-omniroute-budget")?.trim() || null;
const budgetFallbackHeader = headers.get("x-omniroute-budget-fallback")?.trim() || null;
const latencyBudgetHeader = headers.get("x-omniroute-latency-budget")?.trim() || null;

const mode = resolveRequestModePack(modeHeader);
const budgetCap = parseRequestBudgetCap(budgetHeader);
const budgetFallback = parseRequestBudgetFallback(budgetFallbackHeader);
const latencyBudgetMs = parseRequestLatencyBudgetMs(latencyBudgetHeader);

return {
...(mode.override && modeHeader ? { mode: modeHeader } : {}),
...(budgetCap !== undefined ? { budgetCap } : {}),
...(budgetFallback !== undefined ? { budgetFallback } : {}),
...(latencyBudgetMs !== undefined ? { latencyBudgetMs } : {}),
};
}
4 changes: 2 additions & 2 deletions open-sse/services/autoCombo/routingDecision.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ import type {
RoutingQuotaState,
RoutingRequest,
} from "@/shared/contracts/routing";
import { exceedsLatencyBudget } from "../routing/attemptPolicy";
import type { ProviderCandidate, ScoredProvider, ScoringWeights } from "./scoring";
import {
BudgetExceededError,
Expand Down Expand Up @@ -98,8 +99,7 @@ export function hardExclusionReasons(
if (maxCost !== undefined && estimateAutoRequestCostUsd(candidate.costPer1MTokens) > maxCost) {
reasons.push("cost_over_budget");
}
const maxLatencyMs = request.budget?.maxLatencyMs;
if (maxLatencyMs !== undefined && candidate.p95LatencyMs > maxLatencyMs) {
if (exceedsLatencyBudget(candidate.p95LatencyMs, request.budget?.maxLatencyMs)) {
reasons.push("latency_over_budget");
}
return reasons;
Expand Down
Loading
Loading