Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ Client → /v1/chat/completions (Next.js route)

API routes follow a consistent pattern: `Route → CORS preflight → Zod body validation → Optional auth (extractApiKey/isValidApiKey) → API key policy enforcement → Handler delegation (open-sse)`. No global Next.js middleware — interception is route-specific.

**Combo routing** (`open-sse/services/combo.ts`): 15 strategies (priority, weighted, fill-first, round-robin, P2C, random, least-used, cost-optimized, reset-aware, reset-window, strict-random, auto, lkgp, context-optimized, context-relay). Each target calls `handleSingleModel()` which wraps `handleChatCore()` with per-target error handling and circuit breaker checks. See `docs/routing/AUTO-COMBO.md` for the 9-factor Auto-Combo scoring and `docs/architecture/RESILIENCE_GUIDE.md` for the 3 resilience layers.
**Combo routing** (`open-sse/services/combo.ts`): 17 strategies (priority, weighted, fill-first, round-robin, P2C, random, least-used, cost-optimized, reset-aware, reset-window, headroom, strict-random, auto, lkgp, context-optimized, context-relay, fusion). Each target calls `handleSingleModel()` which wraps `handleChatCore()` with per-target error handling and circuit breaker checks. The `fusion` strategy is the exception: it fans out to a panel of models in parallel, then a judge model synthesizes one final answer (`open-sse/services/fusion.ts`). See `docs/routing/AUTO-COMBO.md` for the 9-factor Auto-Combo scoring + the full strategy table and `docs/architecture/RESILIENCE_GUIDE.md` for the 3 resilience layers.

---

Expand Down Expand Up @@ -332,7 +332,7 @@ For any non-trivial change, read the matching deep-dive first:
| Repo navigation | `docs/architecture/REPOSITORY_MAP.md` |
| Architecture | `docs/architecture/ARCHITECTURE.md` |
| Engineering reference | `docs/architecture/CODEBASE_DOCUMENTATION.md` |
| Auto-Combo (9-factor scoring, 15 strategies) | `docs/routing/AUTO-COMBO.md` |
| Auto-Combo (9-factor scoring, 17 strategies) | `docs/routing/AUTO-COMBO.md` |
| Resilience (3 mechanisms) | `docs/architecture/RESILIENCE_GUIDE.md` |
| Reasoning replay | `docs/routing/REASONING_REPLAY.md` |
| Skills framework | `docs/frameworks/SKILLS.md` |
Expand Down
10 changes: 6 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
[![50+ Free](https://img.shields.io/badge/50%2B-Free_Tiers-00B894?style=for-the-badge)](#-231-ai-providers--50-free)
[![1.6B Free Tokens/mo](https://img.shields.io/badge/1.6B-Free_Tokens%2Fmo-00B894?style=for-the-badge)](docs/reference/FREE_TIERS.md)
[![Token Savings](https://img.shields.io/badge/up_to_95%25-Token_Savings-E17055?style=for-the-badge)](#%EF%B8%8F-save-1595-tokens--automatically)
[![15 Strategies](https://img.shields.io/badge/15-Routing_Strategies-0984E3?style=for-the-badge)](#-combos--the-flagship)
[![17 Strategies](https://img.shields.io/badge/17-Routing_Strategies-0984E3?style=for-the-badge)](#-combos--the-flagship)
[![$0 to start](https://img.shields.io/badge/%240-To_Start-FDCB6E?style=for-the-badge&logoColor=black)](#-quick-start)

<br/>
Expand Down Expand Up @@ -182,7 +182,7 @@
▼
┌──────────────────────────────────────────────────────────┐
│ OmniRoute — Smart Router │
│ RTK + Caveman compression · 15 routing strategies │
│ RTK + Caveman compression · 17 routing strategies │
│ Circuit breakers · TLS stealth · MCP · A2A · Guardrails │
└─────────────────────────┬──────────────────────────────────┘
┌─────────────┬────┴────────┬─────────────┐
Expand Down Expand Up @@ -220,7 +220,7 @@ No combo to create. Set your model to `auto` (or a variant) and OmniRoute builds

##

### 🔀 Or build your own — 15 routing strategies
### 🔀 Or build your own — 17 routing strategies

| Goal | Strategy / combo |
| --------------------------------------- | -------------------------------------------------- |
Expand All @@ -229,6 +229,8 @@ No combo to create. Set your model to `auto` (or a variant) and OmniRoute builds
| 💸 Always cheapest viable model | `cost-optimized` · `auto/cheap` |
| 🧠 Hand off long context between models | `context-relay` · `context-optimized` |
| 🎲 Randomized / privacy routing | `random` · `strict-random` |
| 🧬 Fan out to a panel + judge synthesis | `fusion` |
| 📊 Route by remaining quota headroom | `reset-window` · `headroom` |
| 🤖 Just make it smart | `auto` (9-factor scoring) · `lkgp` · `reset-aware` |

<sub>The Auto-Combo engine scores every candidate on **9 factors** (health, quota, cost, latency, success rate, freshness…) — see [`docs/routing/AUTO-COMBO.md`](docs/routing/AUTO-COMBO.md).</sub>
Expand Down Expand Up @@ -266,7 +268,7 @@ Result: 4 layers of fallback = zero downtime
| -------------------------------------- | ----------------------------------------------------------- | ------------- |
| 🌐 Providers | **231** | 20–100 |
| 🆓 Free providers | **50+ (11 free forever)** | 1–5 |
| 🔀 Routing strategies | **15** (priority, weighted, cost-optimized, context-relay…) | 1–3 |
| 🔀 Routing strategies | **17** (priority, weighted, cost-optimized, context-relay, fusion…) | 1–3 |
| 🗜️ Token compression | **RTK + Caveman stacked (15–95%)** | None / 20–40% |
| 🧰 Built-in MCP server | **87 tools, 3 transports, 30 scopes** | Rare |
| 🤝 A2A agent protocol | **6 skills, JSON-RPC 2.0** | None |
Expand Down
2 changes: 1 addition & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ Pluggable subsystems exposed to clients, agents, and operators.

Combo routing, scoring, and replay.

- [AUTO-COMBO.md](routing/AUTO-COMBO.md) — Auto-Combo (9-factor scoring, 15 strategies).
- [AUTO-COMBO.md](routing/AUTO-COMBO.md) — Auto-Combo (9-factor scoring, 17 strategies).
- [REASONING_REPLAY.md](routing/REASONING_REPLAY.md) — reasoning replay flow.

## security/
Expand Down
8 changes: 5 additions & 3 deletions docs/architecture/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -365,9 +365,11 @@ relying on a static combo definition. It powers the `auto/*` model prefix family

Key capabilities:

- **14 routing strategies** (priority, weighted, fill-first, round-robin, P2C, random,
least-used, cost-optimized, strict-random, **auto**, lkgp, context-optimized,
context-relay, plus a fallback path) — auto is the headline addition in v3.8.0.
- **17 routing strategies** (priority, weighted, fill-first, round-robin, P2C, random,
least-used, cost-optimized, reset-aware, reset-window, headroom, strict-random,
**auto**, lkgp, context-optimized, context-relay, **fusion**, plus a fallback path) —
auto is the headline addition in v3.8.0; `fusion` (panel fan-out + judge synthesis,
`open-sse/services/fusion.ts`) is new in v3.8.36.
- **9-factor scoring**: cost, latency p95, success rate, quota headroom, lockout
proximity, breaker state, recent failures, model availability, and tag affinity.
- **Virtual factory** materializes ephemeral combos when no matching named combo
Expand Down
2 changes: 1 addition & 1 deletion docs/architecture/RESILIENCE_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,7 @@ Bounded by `comboCooldownWait` (`enabled`, `maxWaitMs` 5s, `maxAttempts` 2,

## Other Resilience Features

- **14 routing strategies** (priority, weighted, round-robin, context-relay, fill-first, p2c, random, least-used, cost-optimized, reset-aware, strict-random, auto, lkgp, context-optimized) — see [AUTO-COMBO.md](../routing/AUTO-COMBO.md).
- **17 routing strategies** (priority, weighted, round-robin, context-relay, fill-first, p2c, random, least-used, cost-optimized, reset-aware, reset-window, headroom, strict-random, auto, lkgp, context-optimized, fusion) — see [AUTO-COMBO.md](../routing/AUTO-COMBO.md).
- **Reset-aware routing** (v3.8.0) — prioritizes connections by quota reset time.
- **Background mode degradation** — Responses API `background: true` degraded to sync with warning.
- **Dynamic tool limit detection** — backs off providers when tool count limits hit.
Expand Down
2 changes: 1 addition & 1 deletion docs/comparison/OMNIROUTE_VS_ALTERNATIVES.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Objective feature comparison vs popular open-source AI routers.
| **Providers** | **207+** | ~100 | ~50 | ~30 |
| **Self-hostable** | ✅ | ✅ | ❌ | ⚠ paid |
| **OAuth providers (Claude, Codex, Copilot, etc.)** | **15+** | partial | ❌ | ❌ |
| **Auto-fallback combos** | **15 strategies** | priority-based | tier-based | weighted |
| **Auto-fallback combos** | **17 strategies** | priority-based | tier-based | weighted |
| **Tier 1/2/3 fallback (subscription→cheap→free)** | ✅ + UI | manual | n/a | manual |
| **Token compression** | RTK (47 filters) + Caveman | none | none | none |
| **Built-in MCP server** | ✅ 87 tools, 30 scopes | ❌ | ❌ | ❌ |
Expand Down
2 changes: 1 addition & 1 deletion docs/diagrams/request-pipeline.mmd
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ flowchart LR
Cache -->|yes| Return["Return cached"]
Cache -->|no| Rate["Rate limit<br/>(per-key, per-IP)"]
Rate --> Combo{"Combo<br/>target?"}
Combo -->|combo| Resolve["resolveComboTargets<br/>(14 strategies)"]
Combo -->|combo| Resolve["resolveComboTargets<br/>(17 strategies)"]
Resolve --> Single["handleSingleModel<br/>(per target)"]
Combo -->|single| Single
Single --> Translate["translateRequest<br/>(OpenAI↔Claude↔Gemini)"]
Expand Down
4 changes: 3 additions & 1 deletion docs/frameworks/OPEN_SSE_ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -226,7 +226,7 @@ export async function handleComboChat(body, comboId): Promise<ChatResult> {
}
```

Supports **15 routing strategies** (see `src/shared/constants/routingStrategies.ts`):
Supports **17 routing strategies** (see `src/shared/constants/routingStrategies.ts`):

| Strategy | Behavior |
|----------|----------|
Expand All @@ -241,10 +241,12 @@ Supports **15 routing strategies** (see `src/shared/constants/routingStrategies.
| `cost-optimized` | Cheapest healthy target first |
| `reset-aware` | Aware of provider reset windows |
| `reset-window` | Reset window-based routing |
| `headroom` | Most remaining quota headroom first |
| `strict-random` | Truly uniform (no quality weighting) |
| `auto` | Use 9-factor scoring (`autoCombo/`) |
| `lkgp` | Last known good provider first |
| `context-optimized` | Best for long-context requests |
| `fusion` | Fan out to a panel in parallel, then synthesize via a judge (`fusion.ts`) |

### base.ts (1170 LOC)

Expand Down
2 changes: 1 addition & 1 deletion docs/guides/FEATURES.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ OpenRouter connections can store a per-connection `preset` in Advanced Settings.

## 🎨 Combos

Create model routing combos with 14 strategies: priority, weighted, fill-first, round-robin, p2c (power-of-two-choices), random, least-used, cost-optimized, reset-aware, strict-random, auto, lkgp (last-known-good-provider), context-optimized, and **context-relay**. Each combo chains multiple models with automatic fallback and includes quick templates and readiness checks.
Create model routing combos with 17 strategies: priority, weighted, fill-first, round-robin, p2c (power-of-two-choices), random, least-used, cost-optimized, reset-aware, reset-window, headroom, strict-random, auto, lkgp (last-known-good-provider), context-optimized, context-relay, and **fusion** (fan out to a panel of models in parallel, then synthesize one answer via a judge). Each combo chains multiple models with automatic fallback and includes quick templates and readiness checks.

Recent combo improvements:

Expand Down
3 changes: 3 additions & 0 deletions docs/openapi.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6672,10 +6672,13 @@ components:
- least-used
- cost-optimized
- reset-aware
- reset-window
- headroom
- strict-random
- auto
- lkgp
- context-optimized
- fusion
default: priority
nodes:
type: array
Expand Down
68 changes: 65 additions & 3 deletions docs/routing/AUTO-COMBO.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ Notes:

## All Routing Strategies

OmniRoute's combo engine supports **14 routing strategies** (declared in `src/shared/constants/routingStrategies.ts` → `ROUTING_STRATEGY_VALUES`). The Auto Combo engine itself is exposed under the `auto` strategy; the others are available for persisted combos.
OmniRoute's combo engine supports **17 routing strategies** (declared in `src/shared/constants/routingStrategies.ts` → `ROUTING_STRATEGY_VALUES`). The Auto Combo engine itself is exposed under the `auto` strategy; the others are available for persisted combos.

| Strategy | Description |
| :------------------ | :----------------------------------------------------------------- |
Expand All @@ -161,12 +161,74 @@ OmniRoute's combo engine supports **14 routing strategies** (declared in `src/sh
| `least-used` | Pick target with lowest current load |
| `cost-optimized` | Minimize $ per request given catalog pricing |
| `reset-aware` ⭐ | Prioritize by quota reset time — short reset windows ranked higher |
| `reset-window` | Prefer targets whose quota window resets soonest |
| `headroom` | Pick the target with the most remaining quota headroom |
| `strict-random` | Random without deduplication of repeats |
| `auto` | Use Auto Combo scoring (9-factor) — **recommended** |
| `lkgp` | Last-Known-Good Path (sticky route to last successful target) |
| `context-optimized` | Pick target with best fit for current context size |
| `fusion` 🧬 | Fan out to a panel of models in parallel, then synthesize one answer via a judge (see below) |

⭐ = New in v3.8.0
⭐ = New in v3.8.0 · 🧬 = New in v3.8.36

## Fusion Strategy

`fusion` is the one strategy that does **not** pick a single target. It fans the prompt
out to **every panel model in parallel**, then a configurable **judge model** synthesizes
a single final answer from all panel responses. Ported from upstream `decolua/9router`
(OpenRouter's Fusion design); implementation in `open-sse/services/fusion.ts`.

How it works:

1. **Fan-out** — the prompt is sent to every panel model at once, forced non-streaming
with tools stripped (the judge needs complete prose to synthesize).
2. **Quorum-grace collection** — as soon as `minPanel` answers arrive, a short grace
timer starts for the stragglers, then fusion proceeds with whatever was collected.
This caps the slowest model's penalty on wall time, bounded by a hard timeout.
3. **Judge synthesis** — panel answers are anonymized (`Source 1`, `Source 2`, … — so
the judge weighs substance, not model brand) and handed to the judge, which analyzes
consensus / contradictions / partial coverage / unique insights / blind spots, then
writes **one** authoritative answer. The judge call keeps the client's original
`stream` flag + tools, so streaming and downstream tool use still work.
4. **Graceful degradation** — 0 panel answers → `503`; exactly 1 survivor → that answer
is returned directly (nothing to fuse); a single-model panel answers directly.

### Configuration

Configured on the combo's `config` blob (no schema migration — it reuses the existing
`combos` table):

| Field | Type | Default | Purpose |
| :--------------------------------- | :------- | :-------- | :----------------------------------------------------------- |
| `config.judgeModel` | `string` | first panel model | Model that synthesizes the final answer |
| `config.fusionTuning.minPanel` | `number` | `2` | Successful answers required before the grace timer starts (clamped to `[2, panelSize]`) |
| `config.fusionTuning.stragglerGraceMs` | `number` | `8000` | How long to wait for laggards once quorum is reached |
| `config.fusionTuning.panelHardTimeoutMs` | `number` | `90000` | Absolute cap so one hung model can't stall the request |

Defaults live in `FUSION_DEFAULTS` (`open-sse/services/fusion.ts`).

### Example

```bash
curl -X POST http://localhost:20128/api/combos \
-H "Authorization: Bearer <key>" \
-H "Content-Type: application/json" \
-d '{
"name": "fusion-panel",
"strategy": "fusion",
"targets": [
{ "model": "cc/claude-opus-4-7" },
{ "model": "cx/gpt-5.5" },
{ "model": "glm/glm-5.1" }
],
"config": {
"judgeModel": "cc/claude-opus-4-7",
"fusionTuning": { "minPanel": 2, "stragglerGraceMs": 8000, "panelHardTimeoutMs": 90000 }
}
}'
```

Then call it like any combo: `{"model":"fusion-panel","messages":[...]}`.

## Virtual Auto-Combo Factory

Expand Down Expand Up @@ -519,5 +581,5 @@ See `docs/marketing/TIERS.md` for tier definitions and provider classification.
| `open-sse/services/autoCombo/autoPrefix.ts` | `auto/` prefix parser + 6 variants |
| `open-sse/services/autoCombo/virtualFactory.ts` | Builds in-memory `AutoComboConfig` from live connections |
| `open-sse/services/autoCombo/providerRegistryAccessor.ts` | Test hook for mocking provider registry |
| `src/shared/constants/routingStrategies.ts` | `ROUTING_STRATEGY_VALUES` (14 strategies) |
| `src/shared/constants/routingStrategies.ts` | `ROUTING_STRATEGY_VALUES` (17 strategies) |
| `src/sse/handlers/chat.ts` | Integration: auto-prefix short-circuit |
Loading