[WIP] Add FastAPI OpenAI-compatible LLM gateway with routing by Copilot · Pull Request #1 · cheaper-inference/OmniRoute

Copilot · 2026-02-13T12:38:35Z

Original prompt

Bootstrap OmniRoute: Python 3.12 + FastAPI OpenAI-compatible LLM gateway. Implement /v1/chat/completions and /health. Add provider abstraction (OpenAI, Anthropic stub), config-driven routing + weighted load balancing + retries/fallback. Include Dockerfile, docker-compose (optional Redis), pytest, ruff, mypy, GitHub Actions CI, and a solid README + config examples. Add .github/copilot-instructions.md and AGENTS.md with build/test commands and conventions.

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

- Generate "Account cheaper-inference#1", "Account cheaper-inference#2" names for providers that use non-email identifiers - Only use email as display name when it's a valid email (contains @) - Hide email field in UI when value is not a real email address

…ance, robustness ## Critical Fixes - #1: Server readiness — waitForServer() polls before loading window - #2: Restart timeout — 5s + SIGKILL prevents IPC handler from hanging - #3: changePort — now stops/restarts server on new port ## Important Fixes - #4: Tray cleanup — destroy old Tray before recreating - #5: IPC emission — server-status & port-changed events - #6: Disposer pattern — replaces removeAllListeners - #7: useSyncExternalStore — eliminates 5x re-renders ## Minor: #8-#16 (dead code, CSP, platform titlebar, types, errors, version) Tests: 76 / 15 suites (was 64/9)

feat(#255): add Ollama Cloud provider (api.ollama.com/v1) - New registry entry 'ollama-cloud' (alias: ollamacloud) - OpenAI-compatible, Bearer API key auth - 7 pre-populated models: Gemma 3 27B, Llama 3.3 70B, Qwen3 72B, Devstral 24B, DeepSeek R2 671B, Phi 4 14B, Mistral Small 3.2 24B - passthroughModels: true for dynamic model names fix(#258): security hardening — all 9 findings addressed - [CRITICAL] db-backups/export: add isAuthRequired + isAuthenticated guard - [CRITICAL] db-backups/import: add isAuthRequired + isAuthenticated guard - [HIGH] oauth/cursor/auto-import: add isAuthRequired + isAuthenticated guard - [HIGH] oauth/kiro/auto-import: add isAuthRequired + isAuthenticated guard - [LOW x4] oauth/[provider]/[action]: replace === with safeEqual() using crypto.timingSafeEqual() for CWE-208 (timing-oracle attacks) on email and workspaceId comparisons across all 3 handler branches - [FALSE POSITIVE] Finding #1 (package.json 'reset-password' binary name) is not a hardcoded credential — no change needed Tests: 590/594 pass (4 pre-existing failures in combo-circuit-breaker)

TLS / Certificate Validation (#50) - src/mitm/server.ts: rejectUnauthorized now defaults to true (opt-out via MITM_DISABLE_TLS_VERIFY=1 env var only) Path Injection (#44, #41-#49) - src/mitm/server.ts: safeLogPath() guards log filenames inside LOG_DIR - src/lib/db/backup.ts: path.resolve() anchor + sep/slash guard on backupId - src/shared/services/backupService.ts: safePath() helper anchors toolId and backupId within BACKUP_DIR (prevents path traversal) - src/app/api/cli-tools/codex-profiles/route.ts: safeProfilePath() helper anchors profileId within PROFILES_DIR Prototype Pollution (#18, #19, #20) - src/lib/usage/usageHistory.ts: byModel/byAccount initialized with Object.create(null); assignments guarded with hasOwnProperty checks Missing Workflow Permissions (#1-#8, #53) - .github/workflows/ci.yml: added global permissions: contents: read Dependabot CVE-2026-0540 (#16) - dompurify updated to ^3.3.2 (fixes XSS vulnerability)

@kfiramar

Thanks @kfiramar! 🎉 Critical security fix — different startup paths were generating different `STORAGE_ENCRYPTION_KEY` values over the same SQLite database, causing `Unsupported state or unable to authenticate data` for all stored tokens. Improvements added on top: - Normalized `overridePath?.trim()` in `electron/main.js` to match `bootstrap-env.mjs` (addresses kilo-code-bot warning #1) - Added explanatory comment documenting the `preferredEnv` merge order intent in Electron startup (addresses kilo-code-bot warning #3) 4 commits + 113-line test file. The fail-closed behaviour (refusing to mint a new key when encrypted rows exist) is an excellent safeguard. Merged!

- FIX #1: Add null check for cred.password (prevent undefined access) - FIX #2: Prioritize actual credentials over hardcoded account patterns - FIX #3: Convert CommonJS require() to ES imports for consistency - FIX #4: Move to App Router, add credential metadata response, document maintainer integration Additional improvements: - Better TypeScript error typing with optional chaining - Improved error messages for missing dependencies - Added maintainer TODO for provider system integration - Proper Next.js App Router format (route.ts) All bot warnings resolved. Ready for maintainer review.

@kfiramar

…v-key-loading Thanks @kfiramar! 🎉 Critical security fix — different startup paths were generating different `STORAGE_ENCRYPTION_KEY` values over the same SQLite database, causing `Unsupported state or unable to authenticate data` for all stored tokens. Improvements added on top: - Normalized `overridePath?.trim()` in `electron/main.js` to match `bootstrap-env.mjs` (addresses kilo-code-bot warning cheaper-inference#1) - Added explanatory comment documenting the `preferredEnv` merge order intent in Electron startup (addresses kilo-code-bot warning cheaper-inference#3) 4 commits + 113-line test file. The fail-closed behaviour (refusing to mint a new key when encrypted rows exist) is an excellent safeguard. Merged!

- FIX cheaper-inference#1: Add null check for cred.password (prevent undefined access) - FIX cheaper-inference#2: Prioritize actual credentials over hardcoded account patterns - FIX cheaper-inference#3: Convert CommonJS require() to ES imports for consistency - FIX cheaper-inference#4: Move to App Router, add credential metadata response, document maintainer integration Additional improvements: - Better TypeScript error typing with optional chaining - Improved error messages for missing dependencies - Added maintainer TODO for provider system integration - Proper Next.js App Router format (route.ts) All bot warnings resolved. Ready for maintainer review.

Fly.io Launch config files

…CLI tools Two critical bugs prevented the Dashboard CLI Tools feature from working: Bug #1: keyId stripped by Zod validation - Zod's safeParse() stripped the keyId field before backend could read it - The real API key was never resolved from DB and written to settings.json - Fix: Extract keyId from rawBody BEFORE validation in all 6 CLI settings routes - Affected: claude, codex, droid, cline, openclaw, kilo settings routes Bug #2: Key matching logic compared full key vs masked key - Frontend compared full key (sk-31c4ea6a...) against masked key (sk-31c4****dfa6) - They never matched, so the dropdown never showed the correct selected key - Fix: Mask the token from file before comparing against the masked API keys list - Affected: ClaudeToolCard.tsx Also fixes model mapping being scrambled when key resolution failed. Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

Round of fixes addressing the gemini-code-assist and chatgpt-codex review comments on the initial PR. ## High priority - **PoW solver no longer blocks the event loop** (gemini #1, #2). The 100k prekey solver and 500k proof-of-work solver were synchronous SHA3-512 loops that pinned a CPU core for tens to hundreds of milliseconds per request. Both are now async and `await`-yield to the event loop every 1000 iterations via setImmediate, so concurrent requests and I/O still get scheduled. Wall time is approximately the same; what changes is fairness, not throughput. - **Real upstream streaming for stream=true requests** (codex cheaper-inference#6). The conv call now passes `stream: true` through to the TLS client when the caller asked for streaming. The TLS client uses tls-client-node's streamOutputPath primitive to write the response body to a temp file as it arrives, and we tail that file as a ReadableStream so clients see chunks in real time instead of getting one buffered burst at the end. Also peeks the first 256 bytes — if the response starts with `{` it's almost certainly a JSON error envelope, so we wait for the full body and surface as a non-streaming error response. ## Medium priority - **Per-cookie device id** (gemini cheaper-inference#3). Replaced the single process-wide DEVICE_ID with a per-cookie SHA-256-derived UUID that's stable across requests for one connection but unique per cookie. This matches how the browser's persistent oai-did cookie behaves and avoids cross-account fingerprint sharing. Cache is bounded to 200 entries with FIFO eviction. - **Removed dead conv-cache code** (gemini cheaper-inference#4). The convCache / convLookup / convStore trio (~70 LOC) was unused — conversationId is hard-pinned to null because Temporary Chat conversation_ids 404 on reuse. Deleted entirely; the comment explains why we don't persist. - **No more console.log in the conv 4xx path** (gemini cheaper-inference#5). Replaced with log?.warn so it respects the application's logging configuration. - **Bound the warmup cache** (codex cheaper-inference#7). The (cookie, accessToken) -> timestamp map was unbounded; long-running multi-user deployments with rotating tokens would grow it forever. Now capped at 200 entries with FIFO eviction (Map iteration order = insertion order). - **Honor abort signals in TLS fetch** (codex cheaper-inference#8). tlsFetchChatGpt now checks options.signal before issuing the upstream call, after the call returns, and the streaming body listens for abort to stop tailing the temp file. tls-client-node's koffi binding can't cancel an in-flight request mid-call, but we no longer process / re-emit a response that the caller has already given up on. ## Tests All 27 chatgpt-web tests still pass; updated several to find calls by URL via findIndex rather than hardcoded indices, since the warmup sequence (/me, /conversations, /models) and two-stage Sentinel (prepare + chat-requirements) shifted positional offsets. Manually verified end-to-end: - Non-streaming completions - Streaming completions (real-time chunks; SSE [DONE] terminator) - Multi-turn with full history each turn (memory preserved correctly) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

dlynch90

LGTM — all 15 CodeRabbit findings triaged and resolved in 177231d.

Initial plan

e8a99dc

Copilot AI assigned Copilot and diegosouzapw Feb 13, 2026

Copilot started work on behalf of diegosouzapw February 13, 2026 12:38

View session

diegosouzapw closed this Feb 13, 2026

Copilot AI requested a review from diegosouzapw February 13, 2026 12:43

Copilot stopped work on behalf of diegosouzapw due to an error February 13, 2026 12:43
The session was cancelled by the user.

diegosouzapw deleted the copilot/bootstrap-omniroute-gateway branch February 18, 2026 03:03

abhinavjnu mentioned this pull request Mar 23, 2026

feat: Add Zed IDE OAuth credential import support #550

Closed

6 tasks

inklov3 mentioned this pull request Mar 27, 2026

[BUG] OpenCode Zen provider "Import from /models" still not working #667

Closed

jjj124 mentioned this pull request Mar 27, 2026

[BUG] Tool-call corruption in Claude Code CLI → OmniRoute → Codex translation path #678

Closed

oyi77 mentioned this pull request Apr 1, 2026

[FEATURE] CLIProxyAPI Integration & OmniRoute Version Manager #902

Closed

22 tasks done

diegosouzapw pushed a commit that referenced this pull request Apr 3, 2026

Merge pull request #1 from xiaoge1688/flyio-new-files

3f82b05

Fly.io Launch config files

kilo-code-bot Bot mentioned this pull request Apr 14, 2026

[Bug] Codex account works in direct mode but fails when used in Combo (priority strategy) #1234

Closed

diegosouzapw mentioned this pull request Apr 14, 2026

[Bug] Combo fallback skips secondary Codex account after rate-limit on Codex #1 #1233

Closed

benzntech mentioned this pull request Apr 15, 2026

fix(cli-tools): resolve API key resolution and model mapping bugs #1263

Merged

dhaern mentioned this pull request Apr 16, 2026

[Feature] Add Meta Muse Spark (meta.ai) Web Subscription Provider #1308

Closed

ether-btc mentioned this pull request Apr 20, 2026

[Security] Code Audit Findings — Command Injection, Crypto, and Auth Gaps #1465

Closed

payne0420 mentioned this pull request Apr 25, 2026

feat: add ChatGPT Web (Plus/Pro) session provider #1593

Closed

11 tasks done

muhammedminhal mentioned this pull request Sep 21, 2026

feat(sse): add cascade routing backend , semantic-entropy quality gate over cheap models #14305

Closed

Squaregarden mentioned this pull request Sep 21, 2026

fix(auth): OAuth device-flow onboarding broken: wrong poll route, missing state, no copilot->github alias, gemini absent, codex/kiro 404/405, github/* catalog mapping #14298

Closed

sxh313 mentioned this pull request Sep 21, 2026

fix(providers): carry tool-call history into the gemini-web prompt #14374

Merged

diegosouzapw mentioned this pull request Sep 22, 2026

fix(backend): semantic-cache cluster — hash omits tools/response_format (poisoning), embedding client SSRF with stored key, cleartext embedding key/Redis URL echoed by GET #14484

Closed

valvevan163 mentioned this pull request Sep 23, 2026

fix(): Opencode free models no longer working #14313

Closed

torch817 mentioned this pull request Sep 26, 2026

fix(antigravity): resolve HTTP 400 toolConfig and HTTP 403 x-goog-user-project errors #14909

Closed

skygunner mentioned this pull request Sep 27, 2026

fix(backend): Context guard rejects Codex CLI at ~50k real tokens: chars/4 estimator over-counts ~1MB of tool schemas 5-6x, checked against a generic 128k fallback limit (model catalog miss → combo-min) #14931

Closed

afonsoft mentioned this pull request Sep 28, 2026

fix(auth): bound the per-connection refresh mutex and rotation lane (#14970) #15002

Merged

4 tasks done

diegosouzapw mentioned this pull request Sep 29, 2026

Release v3.8.51 #11442

Merged

This was referenced Sep 30, 2026

fix(api): persist comboAutoPromoteEnabled in settings schema #15237

Merged

fix(resilience): priority combo try-order overridden by session stickiness #15241

Closed

skygunner mentioned this pull request Oct 3, 2026

fix(combo): keep declared priority order ahead of session stickiness #15458

Merged

dlynch90 approved these changes Oct 3, 2026

View reviewed changes

ragnar-claude mentioned this pull request Oct 5, 2026

fix(backend): Virtual auto/<category>:<tier> routes ignore weights (RulesStrategy), :subscription fails open to metered, and unpriced models score as near-free #15546

Closed

Rushifakami mentioned this pull request Oct 5, 2026

fix(providers): connection reports active when validated with a model that always 403s, and /api/models availability contradicts the live catalog #15585

Closed

This was referenced Oct 8, 2026

fix(auto-combo): honour routing weights in RulesStrategy and pin auto/claude-* to the Claude family (#15546, #15675) #15813

Merged

fix(auto): subscription fail-closed, neutral unpriced cost, thrifty rung order (#15546) #15852

Merged

jukmisael mentioned this pull request Oct 8, 2026

[BUG] OpenCode free provider returns 403 for all API requests through OmniRoute #13935

Closed

M3stryX mentioned this pull request Oct 10, 2026

fix(routing): count the first abandoned combo target as a fallback #16250

Merged

5 tasks done

opensource-elearning mentioned this pull request Oct 11, 2026

perf(sse): defer two conditional chatCore imports to first use #16346

Merged

ak22021990-jpg mentioned this pull request Oct 12, 2026

fix(claude): extended thinking is generated and metered but never reaches reasoning_content on the claude providerId #16407

Closed

MRQC-st mentioned this pull request Oct 14, 2026

fix(providers): OpenCode Free Tier 403 Errors When Used Standalone (without OpenCode CLI) #16328

Closed

castocolina mentioned this pull request Oct 17, 2026

fix(auth): kimi-coding device-flow grants without refresh_token condemn the connection; re-auth duplicates it #16984

Closed

diegosouzapw mentioned this pull request Oct 19, 2026

Release v3.8.52 #15083

Merged

yelsoft16 mentioned this pull request Oct 20, 2026

fix(sse): streamed tool_calls arguments are duplicated when credentialRedactionEnabled=true #17291

Open

diegosouzapw mentioned this pull request Oct 22, 2026

fix(dashboard): F4 sweep — leftovers, a11y and screen fixes deferred from the lotes (KEA-7886) #17914

Merged

juniorhillory-creator mentioned this pull request Oct 23, 2026

Graphify server config #18023

Closed

1 of 7 tasks

diegosouzapw mentioned this pull request Oct 25, 2026

Workflow #17407

Closed

7 tasks done

IamBread2007 mentioned this pull request Oct 26, 2026

fix(providers): pair orphaned tool_use blocks on the Antigravity Claude path #18106

Open

diegosouzapw mentioned this pull request Oct 26, 2026

fix(ci): main-green companion — 12 GB build heap + hosted-suite validator (#17350/#17851) and byte-pinned mirror tests (#17250) #18252

Merged

Navigation Menu

Repository navigation

[WIP] Add FastAPI OpenAI-compatible LLM gateway with routing - #1

[WIP] Add FastAPI OpenAI-compatible LLM gateway with routing#1
diegosouzapw with Copilot wants to merge 1 commit into
mainfrom
copilot/bootstrap-omniroute-gateway

Copilot AI commented Feb 13, 2026 •

edited

Loading

Uh oh!

dlynch90 left a comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Copilot AI commented Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dlynch90 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Feb 13, 2026 •

edited

Loading