From e60b4df56020af4bc3be8a0731387892145ec224 Mon Sep 17 00:00:00 2001 From: Max Date: Tue, 25 Aug 2026 18:26:32 +0200 Subject: [PATCH] fix(docs): Auto-Combo factor count is read from DEFAULT_WEIGHTS MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The engine was documented as 6-, 9-, 13- and 14-factor in seventeen places while `DEFAULT_WEIGHTS` declares fifteen factors. Two of the fifteen sit at weight 0 by default, which is where some of the lower counts came from, but they are computed for every candidate and `cacheAffinity` gates prompt-cache deduplication outside the score — they are declared factors that do not vote by default, and `AUTO-COMBO.md` now says so. `check:docs-counts` grows a strict check that reads the count from `scoring.ts` and validates every `N-factor` claim in the README, `AGENTS.md`, the guides, the diagrams and the code comments. Adding a factor without updating the prose now fails. `CHANGELOG.md` is deliberately out of scope: its old entries record what was true when they were written. The gate fails on the pre-change tree and passes after, and the counting helper is covered for zero-weight factors, comment noise and unreadable sources. --- AGENTS.md | 4 +- .../fixes/0001-scoring-factor-count.md | 1 + docs/guides/FEATURES.md | 2 +- docs/guides/FREE_PROVIDER_RANKINGS.md | 4 +- docs/guides/TIERS.md | 2 +- docs/routing/AUTO-COMBO.md | 13 +++-- open-sse/services/autoCombo/routerStrategy.ts | 7 ++- open-sse/services/autoCombo/scoring.ts | 2 +- open-sse/services/taskAwareRouter.ts | 2 +- scripts/check/check-docs-counts-sync.mjs | 49 +++++++++++++++++++ tests/integration/combo-matrix/auto.test.ts | 2 +- tests/unit/check-docs-counts-sync.test.ts | 45 +++++++++++++++++ tests/unit/lkgp-enabled-context-11181.test.ts | 2 +- 13 files changed, 114 insertions(+), 21 deletions(-) create mode 100644 changelog.d/fixes/0001-scoring-factor-count.md diff --git a/AGENTS.md b/AGENTS.md index 7691a5ad97e2..d12bcf12bad1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -83,7 +83,7 @@ Client → /v1/chat/completions (Next.js route) API routes follow a consistent pattern: `Route → CORS preflight → Zod body validation → Optional auth (extractApiKey/isValidApiKey) → API key policy enforcement → Handler delegation (open-sse)`. No global Next.js middleware — interception is route-specific. -**Combo routing** (`open-sse/services/combo.ts`): 19 public strategies (priority, weighted, fill-first, round-robin, p2c, random, least-used, cost-optimized, reset-aware, reset-window, headroom, strict-random, auto, lkgp, context-optimized, cache-optimized, context-relay, fusion, pipeline). Each target calls `handleSingleModel()` which wraps `handleChatCore()` with per-target error handling and circuit breaker checks. The `fusion` strategy is the exception: it fans out to a panel of models in parallel, then a judge model synthesizes one final answer (`open-sse/services/fusion.ts`). See `docs/routing/AUTO-COMBO.md` for the 14-factor Auto-Combo scoring + the full strategy table and `docs/architecture/RESILIENCE_GUIDE.md` for the 3 resilience layers. +**Combo routing** (`open-sse/services/combo.ts`): 19 public strategies (priority, weighted, fill-first, round-robin, p2c, random, least-used, cost-optimized, reset-aware, reset-window, headroom, strict-random, auto, lkgp, context-optimized, cache-optimized, context-relay, fusion, pipeline). Each target calls `handleSingleModel()` which wraps `handleChatCore()` with per-target error handling and circuit breaker checks. The `fusion` strategy is the exception: it fans out to a panel of models in parallel, then a judge model synthesizes one final answer (`open-sse/services/fusion.ts`). See `docs/routing/AUTO-COMBO.md` for the 15-factor Auto-Combo scoring + the full strategy table and `docs/architecture/RESILIENCE_GUIDE.md` for the 3 resilience layers. --- @@ -411,7 +411,7 @@ For any non-trivial change, read the matching deep-dive first: | Repo navigation | `docs/architecture/REPOSITORY_MAP.md` | | Architecture | `docs/architecture/ARCHITECTURE.md` | | Engineering reference | `docs/architecture/CODEBASE_DOCUMENTATION.md` | -| Auto-Combo (14-factor scoring, 19 strategies) | `docs/routing/AUTO-COMBO.md` | +| Auto-Combo (15-factor scoring, 19 strategies) | `docs/routing/AUTO-COMBO.md` | | Resilience (3 mechanisms) | `docs/architecture/RESILIENCE_GUIDE.md` | | Reasoning replay | `docs/routing/REASONING_REPLAY.md` | | Skills framework | `docs/frameworks/SKILLS.md` | diff --git a/changelog.d/fixes/0001-scoring-factor-count.md b/changelog.d/fixes/0001-scoring-factor-count.md new file mode 100644 index 000000000000..5f6ff4ce4e84 --- /dev/null +++ b/changelog.d/fixes/0001-scoring-factor-count.md @@ -0,0 +1 @@ +- **fix(docs):** the Auto-Combo engine is described with the number of factors it actually declares. Four different counts were in circulation — 6, 9, 13 and 14 across `AGENTS.md`, four guides and six code comments — while `DEFAULT_WEIGHTS` declares 15. `check:docs-counts` now reads that number from the source, so adding a factor without updating the prose fails CI ([#0001](https://github.com/diegosouzapw/OmniRoute/pull/0001)) diff --git a/docs/guides/FEATURES.md b/docs/guides/FEATURES.md index d398c902d1d4..f70d74df239d 100644 --- a/docs/guides/FEATURES.md +++ b/docs/guides/FEATURES.md @@ -18,7 +18,7 @@ Visual guide to every section of the OmniRoute dashboard. The v3.7.x → v3.8.0 cycle added zero-config auto routing, new providers, OAuth flows, deeper resilience, and a much richer CLI experience. Headline features below — full details further in the document and in linked specs. -- 🤖 **Auto Combo / Zero-config auto-routing** — use prefixes `auto/coding`, `auto/fast`, `auto/cheap`, `auto/offline`, `auto/smart`, `auto/lkgp`. Backed by a 13-factor scoring engine and 4 curated **mode packs** (ship-fast, cost-saver, quality-first, offline-friendly) +- 🤖 **Auto Combo / Zero-config auto-routing** — use prefixes `auto/coding`, `auto/fast`, `auto/cheap`, `auto/offline`, `auto/smart`, `auto/lkgp`. Backed by a 15-factor scoring engine and 4 curated **mode packs** (ship-fast, cost-saver, quality-first, offline-friendly) - 🆕 **Command Code provider** (#2199) — first-class registration with model catalog and quota tracking - 🆕 **Z.AI provider** — new free-tier provider with quota labels - 🎬 **KIE media expansion** — extended catalog including video generation models diff --git a/docs/guides/FREE_PROVIDER_RANKINGS.md b/docs/guides/FREE_PROVIDER_RANKINGS.md index 79d2ba6ed804..7215e8a88c3b 100644 --- a/docs/guides/FREE_PROVIDER_RANKINGS.md +++ b/docs/guides/FREE_PROVIDER_RANKINGS.md @@ -238,7 +238,7 @@ provider that ships at least one strong free model. `user_override → arena_elo → models_dev_tier → static table`). So after you connect the top free providers, routing with `model: "auto"` (e.g. `auto/coding`) will automatically prefer the higher-quality free models per request. See - [Auto-Combo](../routing/AUTO-COMBO.md) for the full 13-factor scoring. + [Auto-Combo](../routing/AUTO-COMBO.md) for the full 15-factor scoring. --- @@ -248,7 +248,7 @@ provider that ships at least one strong free model. providers, no credit card required. - [Free Tiers directory](../reference/FREE_TIERS.md) — full catalog of free providers and their limits. -- [Auto-Combo](../routing/AUTO-COMBO.md) — the 13-factor routing engine that consumes the +- [Auto-Combo](../routing/AUTO-COMBO.md) — the 15-factor routing engine that consumes the same Arena ELO task-fitness data. - [Environment variables](../reference/ENVIRONMENT.md) — `ARENA_ELO_SYNC_ENABLED` / `ARENA_ELO_SYNC_INTERVAL` reference. diff --git a/docs/guides/TIERS.md b/docs/guides/TIERS.md index c79b012b1219..e0c392d967f4 100644 --- a/docs/guides/TIERS.md +++ b/docs/guides/TIERS.md @@ -69,7 +69,7 @@ rate limits — circuit breaker recovers them on backoff. Dashboard → **Tiers** → assign your providers. Defaults (from `tierDefaults.json`) are sensible; edit when you have specific subscriptions to prioritize or providers to exclude. -Auto-Combo's 13-factor scoring also considers tier. See +Auto-Combo's 15-factor scoring also considers tier. See [`docs/routing/AUTO-COMBO.md`](../routing/AUTO-COMBO.md). ## Telemetry diff --git a/docs/routing/AUTO-COMBO.md b/docs/routing/AUTO-COMBO.md index eacd1a700b37..6e3d598181e7 100644 --- a/docs/routing/AUTO-COMBO.md +++ b/docs/routing/AUTO-COMBO.md @@ -183,7 +183,7 @@ See [#7992](https://github.com/diegosouzapw/OmniRoute/issues/7992) and [#7111](h ## How It Works (Persisted Auto-Combos) -The Auto-Combo Engine dynamically selects the best provider/model for each request using a **15-factor scoring function** (defined in `open-sse/services/autoCombo/scoring.ts` → `DEFAULT_WEIGHTS`). The default weights sum to `1.0`; custom weights are renormalized by `normalizeScoringWeights()`. +The Auto-Combo Engine dynamically selects the best provider/model for each request using a **15-factor scoring function** (defined in `open-sse/services/autoCombo/scoring.ts` → `DEFAULT_WEIGHTS`). The default weights sum to `1.0`; custom weights are renormalized by `normalizeScoringWeights()`. Two of the fifteen — `cacheAffinity` and `resetWindowAffinity` — carry a default weight of `0`: they are still computed for every candidate, and `cacheAffinity` gates prompt-cache deduplication outside the score, so they are declared factors that simply do not vote by default. ![Auto-Combo 15-factor scoring](../diagrams/exported/auto-combo-12factor.svg) @@ -279,7 +279,7 @@ OmniRoute's combo engine supports **19 routing strategies** (declared in `src/sh | `reset-window` | Prefer targets whose quota window resets soonest | | `headroom` | Pick the target with the most remaining quota headroom | | `strict-random` | Random without deduplication of repeats | -| `auto` | Use Auto Combo scoring (9-factor) — **recommended** | +| `auto` | Use Auto Combo scoring (15-factor) — **recommended** | | `lkgp` | Last-Known-Good Path (sticky route to last successful target) | | `context-optimized` | Pick target with best fit for current context size | | `cache-optimized` | Reorder targets by prompt-cache affinity — the connection likeliest to already hold this request's cached prefix is tried first (`open-sse/services/combo/promptCacheAffinity.ts`, #8008) | @@ -368,7 +368,7 @@ The Auto Combo engine doesn't require pre-defined combos. Instead, `open-sse/ser 3. Cross-references with `getProviderRegistry()` for model availability + pricing 4. For each tuple `(provider, model, connection)`, builds a `VirtualAutoComboCandidate` 5. Picks `connection.defaultModel` (or the registry's first model) as the dispatch target -6. Scores each candidate using the 9-factor `scorePool()` and the variant's weight pack +6. Scores each candidate using the 15-factor `scorePool()` and the variant's weight pack 7. Returns the resulting in-memory `AutoComboConfig` for `handleComboChat()` — never persisted to DB This means **adding a new provider with `auto/*` enabled automatically expands the candidate pool** — no manual combo editing needed. The virtual combo is rebuilt per request, so newly-added or newly-healthy connections are picked up immediately. @@ -427,7 +427,7 @@ Each strategy picks one provider from the candidate pool, given a `RoutingContex (task type, tool/vision hints, token estimate, optional SLA policy, optional last-known-good provider). -#### 1. `rules` (default) — 6-factor weighted scoring +#### 1. `rules` (default) — 15-factor weighted scoring Wraps the existing scoring engine. Filters out `OPEN` circuit-breaker candidates, then runs `scorePool()` with the current task type and `getTaskFitness()`, @@ -436,8 +436,7 @@ picking the top-scoring provider. ```ts class RulesStrategyImpl implements RouterStrategy { readonly name = "rules"; - readonly description = - "6-factor weighted scoring: quota, health, cost, latency, taskFit, stability"; + readonly description = "15-factor weighted scoring (see DEFAULT_WEIGHTS)"; select(pool, context) { const eligible = pool.filter((c) => c.circuitBreakerState !== "OPEN"); @@ -733,7 +732,7 @@ intentionally excluded from CI because they require live credentials and VPS acc | File | Purpose | | :-------------------------------------------------------- | :------------------------------------------------------------------------- | -| `open-sse/services/autoCombo/scoring.ts` | 9-factor scoring function, `DEFAULT_WEIGHTS`, pool norm | +| `open-sse/services/autoCombo/scoring.ts` | 15-factor scoring function, `DEFAULT_WEIGHTS`, pool norm | | `open-sse/services/autoCombo/taskFitness.ts` | Model × task fitness lookup | | `open-sse/services/autoCombo/engine.ts` | Selection logic, bandit, budget cap | | `open-sse/services/autoCombo/selfHealing.ts` | Exclusion, probes, incident mode | diff --git a/open-sse/services/autoCombo/routerStrategy.ts b/open-sse/services/autoCombo/routerStrategy.ts index 2bc0c59edbb0..fdadc815b719 100644 --- a/open-sse/services/autoCombo/routerStrategy.ts +++ b/open-sse/services/autoCombo/routerStrategy.ts @@ -3,7 +3,7 @@ * * Inspired by ClawRouter commit 14c83c258 "refactor: extract routing into pluggable RouterStrategy system". * Provides a RouterStrategy interface and built-in implementations: - * - RulesStrategy (default): wraps the existing 6-factor scoring engine + * - RulesStrategy (default): wraps the existing 15-factor scoring engine * - CostStrategy: always picks cheapest available model * - LatencyStrategy: prioritizes low p95 latency with reliability weighting * - SLAStrategy: prefers candidates that satisfy latency/error/cost SLOs @@ -50,7 +50,7 @@ export interface RouterStrategy { select(pool: ProviderCandidate[], context: RoutingContext): RoutingDecision; } -// ── RulesStrategy: wraps 6-factor scoring engine ──────────────────────────── +// ── RulesStrategy: wraps 15-factor scoring engine ─────────────────────────── function toSpeedCandidate(c: ProviderCandidate): SpeedCandidate { return { @@ -84,8 +84,7 @@ function toSpeedCandidate(c: ProviderCandidate): SpeedCandidate { class RulesStrategyImpl implements RouterStrategy { readonly name = "rules"; - readonly description = - "6-factor weighted scoring: quota, health, cost, latency, taskFit, stability"; + readonly description = "15-factor weighted scoring (see DEFAULT_WEIGHTS)"; select(pool: ProviderCandidate[], context: RoutingContext): RoutingDecision { const eligible = pool.filter((c) => c.circuitBreakerState !== "OPEN"); diff --git a/open-sse/services/autoCombo/scoring.ts b/open-sse/services/autoCombo/scoring.ts index 4c939501c727..1c3ce2b07b49 100644 --- a/open-sse/services/autoCombo/scoring.ts +++ b/open-sse/services/autoCombo/scoring.ts @@ -26,7 +26,7 @@ export interface ScoringFactors { /** * Feedback-driven quality signal [0,1] from the routing-event quality tracker * (open-sse/services/routing/quality.ts). Optional so cold candidates with no - * observed events default to neutral (1.0) and are never penalized. + * observed events default to neutral (0.5) and are never penalized. */ quality?: number; } diff --git a/open-sse/services/taskAwareRouter.ts b/open-sse/services/taskAwareRouter.ts index e1ba402b3f6c..06f9efd584f2 100644 --- a/open-sse/services/taskAwareRouter.ts +++ b/open-sse/services/taskAwareRouter.ts @@ -174,7 +174,7 @@ const TASK_PATTERNS: Record = { * - The list rotted. Those ids aged out by a generation or two, and every model * release made them staler. Naming an intent instead removes the maintenance. * - It bypassed the router. `applyTaskAwareRouting` overwrites `body.model`, so a - * literal target skipped auto-combo's 13-factor scoring (quota, circuit-breaker + * literal target skipped auto-combo's 15-factor scoring (quota, circuit-breaker * health, cost, latency, stability), connection cooldown and model lockout — and * hard-failed for any operator who simply had no connection for that provider. * diff --git a/scripts/check/check-docs-counts-sync.mjs b/scripts/check/check-docs-counts-sync.mjs index 04419c99a911..895e7620e654 100644 --- a/scripts/check/check-docs-counts-sync.mjs +++ b/scripts/check/check-docs-counts-sync.mjs @@ -66,6 +66,25 @@ function countRoutingStrategies() { return (m[1].match(/"[^"]+"/g) || []).length; } +// PURE: count the factors the Auto-Combo scorer actually declares. +// The engine is described as "N-factor" in prose, in code comments and in the +// strategy descriptions; N is `DEFAULT_WEIGHTS`, and nothing else. Two of those +// factors sit at weight 0 by default — they are still computed and consumed +// (`cacheAffinity > 0` gates prompt-cache dedup), so they count as declared. +export function parseScoringFactors(sourceText) { + if (!sourceText) return 0; + const m = sourceText.match(/DEFAULT_WEIGHTS[^=]*=\s*\{([\s\S]*?)\n\};/); + if (!m) return 0; + const body = m[1].replace(/\/\*[\s\S]*?\*\//g, "").replace(/\/\/[^\n]*/g, ""); + return (body.match(/^\s*([A-Za-z_$][\w$]*)\s*:/gm) || []).length; +} + +function countScoringFactors() { + const file = path.join(ROOT, "open-sse", "services", "autoCombo", "scoring.ts"); + if (!fs.existsSync(file)) return 0; + return parseScoringFactors(fs.readFileSync(file, "utf8")); +} + // PURE: parse the canonical provider total out of the auto-generated catalog text. export function parseProviderTotal(referenceText) { if (!referenceText) return 0; @@ -479,6 +498,36 @@ export function buildChecks() { strict: false, files: ["docs/architecture/ARCHITECTURE.md", "docs/architecture/CODEBASE_DOCUMENTATION.md"], }, + { + // The Auto-Combo engine is advertised as "N-factor" in a dozen places, in + // prose and in code comments alike, and N had drifted to five different + // values (6, 9, 12, 13, 14) against a code that declares 15. The number + // now comes from `DEFAULT_WEIGHTS`; adding a factor without touching the + // prose fails here. `CHANGELOG.md` is deliberately out of scope: its old + // entries record what was true when they were written. + label: "Auto-Combo scoring factors count", + actual: countScoringFactors(), + docKey: "scoring factors", + strict: true, + files: [ + "README.md", + "AGENTS.md", + "docs/routing/AUTO-COMBO.md", + "docs/guides/TIERS.md", + "docs/guides/FEATURES.md", + "docs/guides/FREE_PROVIDER_RANKINGS.md", + "docs/diagrams/strategies-grid.svg", + "docs/diagrams/auto-combo-12factor.mmd", + "open-sse/services/autoCombo/routerStrategy.ts", + "open-sse/services/taskAwareRouter.ts", + "tests/unit/lkgp-enabled-context-11181.test.ts", + "tests/integration/combo-matrix/auto.test.ts", + ], + validate: makeNumberClaimValidator(countScoringFactors(), { + what: "scoring factors", + pattern: /(\d+)[- ]factors?\b/gi, + }), + }, { label: "Routing strategies count", actual: countRoutingStrategies(), diff --git a/tests/integration/combo-matrix/auto.test.ts b/tests/integration/combo-matrix/auto.test.ts index ebc1362128d9..6a9b80918a2a 100644 --- a/tests/integration/combo-matrix/auto.test.ts +++ b/tests/integration/combo-matrix/auto.test.ts @@ -9,7 +9,7 @@ // - combo.ts fetches getLKGP(combo.name, combo.id) = getLKGP("auto", "auto") → key "auto:auto". // - LKGPStrategy: if a LKGP record exists for that provider AND the provider is in the // candidate pool, it pins that provider deterministically (no randomness). -// - Without a LKGP record, falls back to "rules" 9-factor scorer. +// - Without a LKGP record, falls back to "rules" 15-factor scorer. // // Test 1 strategy: single-request, deterministic. // We seed a LKGP record pointing to "openai" via settingsDb.setLKGP("auto","auto","openai"). diff --git a/tests/unit/check-docs-counts-sync.test.ts b/tests/unit/check-docs-counts-sync.test.ts index 6fb605cab943..def3a89e9a6c 100644 --- a/tests/unit/check-docs-counts-sync.test.ts +++ b/tests/unit/check-docs-counts-sync.test.ts @@ -280,3 +280,48 @@ test("SVG gate ignores coordinates, sizes and unrelated small counts", () => { const r = checkSvg(noise, SVG_EXPECTED); assert.equal(r.ok, true, `coordinates/attrs must never register claims: ${r.detail}`); }); + +// --- Auto-Combo scoring factors ------------------------------------------------------ +// The engine was described as 6-, 9-, 12-, 13- and 14-factor across the repo while +// `DEFAULT_WEIGHTS` declared 15. The count is now read from the source of truth. + +import { parseScoringFactors } from "../../scripts/check/check-docs-counts-sync.mjs"; + +const parseFactors = parseScoringFactors as (sourceText: string) => number; + +const WEIGHTS_SOURCE = ` +export const DEFAULT_WEIGHTS: ScoringWeights = { + quota: 0.1429, + health: 0.1605, + // A comment naming a decoy: fake: 0.5 + cacheAffinity: 0, + /* block comment with another decoy: alsoFake: 0.2 */ + quality: 0.03, +}; +`; + +test("counts every factor DEFAULT_WEIGHTS declares, comments included as noise", () => { + assert.equal(parseFactors(WEIGHTS_SOURCE), 4); +}); + +test("a zero-weight factor still counts as declared", () => { + // `cacheAffinity` and `resetWindowAffinity` sit at 0 but are computed, and + // `cacheAffinity` gates prompt-cache dedup outside the score. + assert.ok(WEIGHTS_SOURCE.includes("cacheAffinity: 0")); + assert.equal(parseFactors(WEIGHTS_SOURCE.replace("cacheAffinity: 0,", "")), 3); +}); + +test("returns 0 rather than a wrong number when the source cannot be read", () => { + assert.equal(parseFactors(""), 0); + assert.equal(parseFactors("export const SOMETHING_ELSE = { a: 1 };"), 0); +}); + +test("scoring factor validator flags every stale count the repo carried", () => { + const v = makeValidator(15, { what: "scoring factors", pattern: /(\d+)[- ]factors?\b/gi }); + assert.equal(v("a **15-factor** scoring function").ok, true); + assert.equal(v("scores every candidate on **15 factors**").ok, true); + assert.equal(v("a doc with no claim at all").ok, true); + for (const stale of ["6-factor", "9-factor", "12-factor", "13-factor", "14-factor"]) { + assert.equal(v(`the ${stale} scoring engine`).ok, false, `expected ${stale} to be flagged`); + } +}); diff --git a/tests/unit/lkgp-enabled-context-11181.test.ts b/tests/unit/lkgp-enabled-context-11181.test.ts index 04950c2c2a51..3ef204f2fa59 100644 --- a/tests/unit/lkgp-enabled-context-11181.test.ts +++ b/tests/unit/lkgp-enabled-context-11181.test.ts @@ -127,7 +127,7 @@ test("#11181 — a persisted lkgpEnabled:false makes the lkgp strategy delegate lkgpEnabled: false, }); - // The whole point of the toggle: the LKGP pin must be ignored and the 6-factor + // The whole point of the toggle: the LKGP pin must be ignored and the 15-factor // rules scorer must pick the winner instead. assert.doesNotMatch( selection,