Audit fixes: bounded decision store, strategy budget order, SLO recovery, read-only metrics

docs/ops/MONITORING_GUIDE.md

@@ -1,7 +1,7 @@
 ---
 title: "Monitoring & Observability Guide"
 version: 3.8.50
-lastUpdated: 2026-08-13
+lastUpdated: 2026-09-18
 ---
 
 # Monitoring & Observability Guide
@@ -421,6 +421,13 @@ previous alert state (no flapping on low traffic). Payloads never contain prompt
 responses, API keys, connection ids or account ids. Subscribe a webhook to these events
 (or `*`) in the dashboard. Alerts are off by default: set `slo.alertsEnabled = true` to start
 evaluation, so existing `*` subscribers do not receive new events after an upgrade.
+Turning alerts off forgets the remembered alert state, so turning them back on never replays a
+transition that happened while they were off. A tick is skipped while the previous one is still
+dispatching webhooks, so slow endpoints cannot cause duplicate or interleaved alerts.
+
+The alert loop and `GET /api/metrics` read circuit breakers without changing them: an `OPEN`
+breaker whose cooldown has elapsed is reported as `HALF_OPEN`, but a scrape never transitions
+it, persists it or grants its half-open probe; only live traffic does.
 
 ---
 
@@ -499,6 +506,9 @@ Enforced by `open-sse/services/routing/metricLabels.ts`:
 - Bounded values: first 64 providers, 100 models, 16 strategies, 16 engines are kept;
   later distinct values collapse into `other`. Each family is additionally capped at
   2000 series. Combo names, connection ids, request ids and finish reasons are never labels.
+  The fixed values `redacted`, `unknown` and `other` never take a slot, and only a model
+  that served a successful response takes a model slot: failed requests for model ids a
+  client made up are counted under `other` unless the model is already tracked.
 - Values are restricted to `[A-Za-z0-9._:/-]`, truncated to 80 chars, and replaced by
   `redacted` when they look like a credential (API-key prefixes such as `sk-`, `ghp_`),
   an opaque token (32+ alphanumerics), an e-mail address or a UUID.
@@ -528,6 +538,12 @@ defaults by `src/lib/monitoring/sloSettings.ts`. Evaluated by
 `failed` excludes `cancelled` and `guardrail_blocked` (client or policy decisions).
 Rate limits and timeouts count against availability but not against `error_rate`.
 
+For `provider_recovery`, an ongoing open episode (breaker still `OPEN` or `HALF_OPEN`) counts
+only while that breaker fails, or opens, inside the window. A breaker only leaves `HALF_OPEN`
+when traffic probes it, so a provider that stopped receiving traffic would otherwise report a
+breach forever. Such an idle open breaker adds no sample; it still shows in
+`omniroute_circuit_breakers` and `omniroute_circuit_breaker_open`.
+
 `GET /api/telemetry/summary` `errorRate` (percent) uses the same definition —
 failed / (success + failed) routed requests in the requested window (clamped to 1–60 min).
 

docs/routing/ROUTING_CONTRACT.md

@@ -1,7 +1,7 @@
 ---
 title: "Routing Contract and Route Explanation"
 version: 3.8.54
-lastUpdated: 2026-09-14
+lastUpdated: 2026-09-18
 ---
 
 # Routing Contract and Route Explanation
@@ -17,40 +17,42 @@ The types live in `src/shared/contracts/routing.ts` and are shared by `src/` and
 | Type                     | Purpose                                                                                                                                                                 |
 | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | `RoutingRequest`         | What is routed: `requestId`, `model`, `protocol`, optional `capabilities`, `workspaceId`, `policyId`, `stream`, `budget`                                                |
-| `RoutingBudget`          | Optional `maxCost` (USD) and `maxLatencyMs`, applied to the decision and to every failover attempt                                                                      |
+| `RoutingBudget`          | Optional `maxCost` (USD) and `maxLatencyMs`; read by previews and by the `attemptPolicy.ts` library, not by live traffic (see Guarantees)                               |
 | `RoutingCandidate`       | One provider/model: `score`, weighted `factors`, `eligible`, `exclusionReasons`, `quota`, `circuit`, estimated cost and latency                                         |
-| `RoutingDecision`        | `decisionId`, `requestId`, `selected`, all `candidates`, `policyVersion`, `generatedAt`, `liveRequestExecuted`, `selectionMode`                                         |
+| `RoutingDecision`        | `decisionId`, `requestId`, `selected`, `candidates`, `policyVersion`, `generatedAt`, `liveRequestExecuted`, `selectionMode`, `strategy`, optional `omittedCandidates`   |
 | `ProviderAttempt`        | One upstream call: provider, model, attempt number, start time, duration, status, `outcome`                                                                             |
 | `RoutingExclusionReason` | `not_in_candidate_pool`, `model_not_found`, `capability_missing`, `quota_exhausted`, `circuit_open`, `self_healing_excluded`, `cost_over_budget`, `latency_over_budget` |
 
 Candidates never carry connection or account identifiers, prompts or credentials.
 
 ## Guarantees
 
-| Guarantee                                   | How it holds                                                                                                                                                             |
-| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| Preview uses the live selection algorithm   | Live selection and preview both run `selectProviderWithTrace()` in `open-sse/services/autoCombo/engine.ts`                                                               |
-| Preview calls no provider, changes no state | The preview passes `previewSelectionDeps()`: clones of the self-healing and rotation state, exploration off                                                              |
-| Policy version on every decision            | `computeRoutingPolicyVersion()` hashes combo name, candidate pool, weights, mode pack, budget, exploration rate and router strategy (`rp_` + 16 hex characters)          |
-| Every exclusion has a reason                | `hardExclusionReasons()` and the engine trace in `open-sse/services/autoCombo/routingDecision.ts`                                                                        |
-| Unknown quota is not exhausted              | `quota: "unknown"` stays eligible and is scored neutral; only a quota cutoff produces `quota_exhausted`                                                                  |
-| No blind retry of permanent errors          | `open-sse/services/routing/attemptPolicy.ts`: authentication, model not found, invalid request and exhausted quota are permanent and never retried on the same candidate |
-| Failover respects the budget                | `checkFailoverBudget()` and `planNextAttempt()`; the live auto strategy keeps its failover chain inside the cost cap (`orderTargetsByCostBudget()`)                      |
-| Deterministic circuit breaker               | `src/shared/utils/circuitBreaker.ts` accepts an injected clock; `peekState()` reads the effective state without changing it                                              |
-| End-to-end correlation                      | Live decisions are recorded under the request id the client receives (`x-request-id`) and under their decision id                                                        |
+| Guarantee                                   | How it holds                                                                                                                                                                                                                                                   |
+| ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Preview uses the live selection algorithm   | Live selection and preview both run `selectProviderWithTrace()` in `open-sse/services/autoCombo/engine.ts`                                                                                                                                                     |
+| Preview calls no provider, changes no state | The preview passes `previewSelectionDeps()`: clones of the self-healing and rotation state, exploration off                                                                                                                                                    |
+| Policy version on every decision            | `computeRoutingPolicyVersion()` hashes combo name, candidate pool, weights, mode pack, budget, exploration rate and router strategy (`rp_` + 16 hex characters)                                                                                                |
+| Every exclusion has a reason                | `hardExclusionReasons()` and the engine trace in `open-sse/services/autoCombo/routingDecision.ts`                                                                                                                                                              |
+| Unknown quota is not exhausted              | `quota: "unknown"` stays eligible and is scored neutral; only a quota cutoff produces `quota_exhausted`                                                                                                                                                        |
+| No blind retry of permanent errors          | Live: the combo loops retry the same target only on 408, 429, 500, 502, 503 and 504 (`isRetryableAttemptStatus()` in `open-sse/services/routing/attemptPolicy.ts`)                                                                                             |
+| Auto combo failover stays in the cost cap   | Live, `rules` router strategy only: `orderTargetsByCostBudget()` drops (`strict`) or moves last (`cheapest`) targets whose estimated 1K-token request cost exceeds `budgetCap`. Explicit router strategies (`cost`, `latency`, `lkgp`, ...) ignore `budgetCap` |
+| Deterministic circuit breaker               | `src/shared/utils/circuitBreaker.ts` accepts an injected clock; `peekState()` reads the effective state without changing it                                                                                                                                    |
+| End-to-end correlation                      | Live decisions are recorded under the request id the client receives (`x-request-id`) and under their decision id                                                                                                                                              |
 
 ## Previewing a decision
 
 `POST /api/omniroute/route/preview` requires management authentication and never calls a
-provider.
+provider. The examples on this page read `OMNIROUTE_MANAGE_KEY`, an OmniRoute API key with the
+`manage` scope (a dashboard session works too), as in the
+[API use cases](../reference/API_USE_CASES.md).
 
 - `{ "candidates": [...] }` keeps the original adaptive what-if ranking and its response shape.
 - `{ "engine": "auto", "request"?, "policy"?, "candidates": [...] }` runs the live auto-combo engine
   on the supplied candidates and returns the full decision.
 
 ```bash
 curl -s -X POST http://localhost:20128/api/omniroute/route/preview \
-  -H "Authorization: Bearer $OMNIROUTE_TOKEN" \
+  -H "Authorization: Bearer $OMNIROUTE_MANAGE_KEY" \
   -H "Content-Type: application/json" \
   -d '{
     "engine": "auto",
@@ -65,14 +67,18 @@ curl -s -X POST http://localhost:20128/api/omniroute/route/preview \
 
 The answer contains `selected`, `candidates` and `decision` (with `policyVersion` and
 `liveRequestExecuted: false`), plus the `x-request-id` and `x-omniroute-decision-id` headers. A
-candidate without `quotaRemaining` is reported with `quota: "unknown"`.
+candidate without `quotaRemaining` is reported with `quota: "unknown"`. The top-level `selected`
+is only the provider id, kept for compatibility; `decision.selected` has the provider and the
+model. An invalid body gets a 400 whose `error` string lists the invalid fields.
 
 ## Explaining a live request
 
 1. The v1 chat completions, responses and messages routes run inside the request id the
    authorization pipeline stamps, so the router records its decision under that id.
-2. Non-streaming responses carry `X-OmniRoute-Decision-Id` and `X-OmniRoute-Policy-Version`.
-   Streaming responses are sent before routing finishes; use the request id instead.
+2. Responses carry `X-OmniRoute-Decision-Id` and `X-OmniRoute-Policy-Version` when a decision
+   was recorded, streaming responses included (the target is chosen before the stream starts).
+   The headers are absent when the request was not routed by an `auto` combo or when the response
+   headers cannot be changed; the request id lookup below still works.
 3. `GET /api/omniroute/route/decisions/{id}` returns a decision by decision id or request id.
    Decisions are kept in memory for 30 minutes; unknown and malformed ids both return 404.
 4. `GET /api/usage/route-explain/{id}` adds `routingDecision` to the call-log explanation when a
@@ -81,7 +87,7 @@ candidate without `quotaRemaining` is reported with `quota: "unknown"`.
 
 ```bash
 curl -s http://localhost:20128/api/omniroute/route/decisions/<request-id> \
-  -H "Authorization: Bearer $OMNIROUTE_TOKEN"
+  -H "Authorization: Bearer $OMNIROUTE_MANAGE_KEY"
 ```
 
 ## Diagnostic mode
@@ -97,6 +103,13 @@ contains prompts, credentials or connection identifiers.
   decision trace (`/api/usage/combo-trace/{id}`).
 - Live candidates do not yet distinguish unknown quota from a known value, because the candidate
   builder lives in a file at its size cap; preview candidates do.
-- There is no live latency budget input; latency budgets apply to previews and to
-  `planNextAttempt()`.
-- Decisions are held in memory and are lost on restart.
+- Live traffic has no per-request `RoutingBudget` input. The only live cost limit is the auto
+  combo's `budgetCap` on the `rules` path, checked per attempt against a 1K-token estimate; there
+  is no cumulative spend check and no live latency budget.
+- `classifyAttemptOutcome()`, `isPermanentAttemptOutcome()`, `canRetrySameCandidate()`,
+  `checkFailoverBudget()` and `planNextAttempt()` are a tested library with no live caller yet.
+  Previews apply `maxCost` and `maxLatencyMs` as exclusions.
+- Decisions are held in memory and are lost on restart. A stored live decision keeps at most 40
+  candidates (the selected one always), full factors only for the selected candidate and the 10
+  best, and reports the rest as `omittedCandidates`. The store also has a 32 MB size budget, so
+  under heavy traffic a decision can be evicted before its 30 minutes are up.

open-sse/services/autoCombo/routingDecision.ts

@@ -203,6 +203,50 @@ function selectionModeOf(input: BuildRoutingDecisionInput): RoutingDecision["sel
   return describeRotation(input.outcome.trace.eligible);
 }
 
+interface DecisionEntry {
+  key: string;
+  candidate: DecisionCandidateInput;
+  routing: RoutingCandidate;
+}
+
+function isStrategyPick(candidate: DecisionCandidateInput, pick: StrategySelection): boolean {
+  if (candidate.provider !== pick.provider || candidate.model !== pick.model) return false;
+  return !pick.connectionId || (candidate.connectionId ?? "") === pick.connectionId;
+}
+
+/**
+ * The candidate an explicit router strategy picked. The strategy chooses among the routable
+ * candidates on its own rules, so the scoring pass run only to explain the decision does not get
+ * to exclude its pick: a pick without a hard exclusion is reported eligible. A pick without a
+ * connection id matches the best-ranked candidate with its provider and model.
+ */
+function strategyPickEntry(
+  entries: DecisionEntry[],
+  input: BuildRoutingDecisionInput,
+  pick: StrategySelection
+): DecisionEntry | undefined {
+  const entry = entries.find(
+    (candidateEntry) =>
+      isStrategyPick(candidateEntry.candidate, pick) &&
+      hardExclusionReasons(candidateEntry.candidate, input.request).length === 0
+  );
+  if (entry && !entry.routing.eligible) {
+    entry.routing = { ...entry.routing, eligible: true, exclusionReasons: [] };
+  }
+  return entry;
+}
+
+function selectedEntry(
+  entries: DecisionEntry[],
+  input: BuildRoutingDecisionInput
+): DecisionEntry | undefined {
+  if (input.budgetExceeded) return undefined;
+  if (input.strategySelection) return strategyPickEntry(entries, input, input.strategySelection);
+  const chosen = input.outcome?.selection;
+  if (!chosen) return undefined;
+  return entries.find((entry) => entry.key === candidateKey(chosen) && entry.routing.eligible);
+}
+
 /** Turn a selection into the shared decision contract. Pure apart from the clock. */
 export function buildRoutingDecision(
   input: BuildRoutingDecisionInput,
@@ -213,20 +257,16 @@ export function buildRoutingDecision(
     if (!scoredByKey.has(candidateKey(scored))) scoredByKey.set(candidateKey(scored), scored);
   }
   const eligibleKeys = new Set((input.outcome?.trace.eligible ?? []).map(candidateKey));
-  const entries = input.candidates.map((candidate) => ({
+  const entries: DecisionEntry[] = input.candidates.map((candidate) => ({
     key: candidateKey(candidate),
+    candidate,
     routing: toRoutingCandidate(candidate, input, scoredByKey, eligibleKeys),
   }));
   entries.sort(
     (a, b) =>
       Number(b.routing.eligible) - Number(a.routing.eligible) || b.routing.score - a.routing.score
   );
-  const chosen = input.budgetExceeded
-    ? undefined
-    : (input.strategySelection ?? input.outcome?.selection);
-  const selected = chosen
-    ? entries.find((entry) => entry.key === candidateKey(chosen) && entry.routing.eligible)
-    : undefined;
+  const selected = selectedEntry(entries, input);
   return {
     decisionId: clock.newDecisionId(),
     requestId: input.request.requestId,

open-sse/services/combo/autoRoutingDecision.ts

@@ -3,8 +3,9 @@
  *
  * The scoring engine's pick is recorded as a `RoutingDecision` (candidates, scores, exclusion
  * reasons, policy version) under the request id, so a live request can be explained after it ran.
- * The failover chain is kept inside the request cost budget: the budget cap applied to the first
- * pick also applies to every later attempt.
+ * On the scoring-engine ("rules") path the failover chain is kept inside the request cost budget:
+ * the budget cap applied to the first pick also applies to every later attempt. Explicit router
+ * strategies ignore the budget cap, as they did before decisions were recorded.
  */
 import type { RoutingDecision } from "@/shared/contracts/routing";
 import { getRequestId } from "@/shared/utils/requestId";
@@ -128,7 +129,9 @@ export function recordExplicitStrategyDecision(
 }
 
 /**
- * Keep the failover chain inside the request cost budget. With `budgetFallback: "strict"`,
+ * Keep the failover chain of a scoring-engine ("rules") selection inside the request cost budget.
+ * The estimate is per attempt (1K tokens at the candidate's price), not cumulative spend. With
+ * `budgetFallback: "strict"`,
  * targets whose estimated request cost exceeds `budgetCap` are dropped (unless that would leave
  * nothing, in which case the engine has already refused the request); otherwise they move behind
  * the in-budget targets. Targets without a known price are treated as in budget.