Repository navigation
feat: nginx/njs endpoint-aware model routing (no OmniRoute code changes) - #1
devin-ai-integration[bot] wants to merge 3 commits into
Conversation
Adds njs module and nginx config for transparent endpoint-based provider routing without modifying OmniRoute code: - /v1/responses → main/ prefix → KFC (responses provider) - /v1/chat/completions → old/ prefix → KFC-Old (chat provider) The njs handler reads body.model, strips/rewrites the prefix based on which endpoint the client called, sets X-Route-Model header via internalRedirect. OmniRoute's resolveRoutingModel() picks up the header for routing. Original request body is never modified, SSE streaming is fully preserved. Non-KFC models (gemini-*) are skipped by the rewriter and resolved via OmniRoute aliases as before. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| location /v1 { | ||
| proxy_pass http://127.0.0.1:20131; | ||
| proxy_set_header Host $host; | ||
| proxy_set_header X-Real-IP $remote_addr; | ||
| proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; | ||
| proxy_set_header X-Forwarded-Proto $scheme; | ||
|
|
||
| proxy_buffering off; | ||
| proxy_cache off; | ||
| proxy_read_timeout 86400; | ||
|
|
||
| # CORS | ||
| add_header 'Access-Control-Allow-Origin' '*' always; | ||
| add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS, PUT, DELETE, PATCH' always; | ||
| add_header 'Access-Control-Allow-Headers' 'Authorization, Content-Type, Accept, Origin, User-Agent, X-Requested-With' always; | ||
| add_header 'Access-Control-Expose-Headers' 'Content-Length, Content-Range' always; | ||
|
|
||
| if ($request_method = 'OPTIONS') { | ||
| add_header 'Access-Control-Allow-Origin' '*'; | ||
| add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS, PUT, DELETE, PATCH'; | ||
| add_header 'Access-Control-Allow-Headers' 'Authorization, Content-Type, Accept, Origin, User-Agent, X-Requested-With'; | ||
| add_header 'Access-Control-Max-Age' 1728000; | ||
| add_header 'Content-Type' 'text/plain; charset=utf-8'; | ||
| add_header 'Content-Length' 0; | ||
| return 204; | ||
| } | ||
| } |
There was a problem hiding this comment.
🔍 Fallback-location /v1 не фильтрует клиентский заголовок X-Route-Model
Блок location /v1 (omnirouter.conf:72-98) проксирует запросы к OmniRoute без установки proxy_set_header X-Route-Model. В отличие от /_omni_proxy/ (строка 51), где заголовок явно устанавливается из $route_model, здесь nginx пропускает оригинальный клиентский заголовок X-Route-Model как есть. Это означает, что клиент может напрямую отправить X-Route-Model: any/model на любой /v1/* эндпоинт (кроме /v1/responses и /v1/chat/completions) и переопределить маршрутизацию. Функция resolveRoutingModel() в src/sse/handlers/resolveRoutingModel.ts:15-16 примет этот заголовок. Это может быть намеренным (документировано в PR cheaper-inference#4863), но стоит убедиться, что это не создаёт нежелательный обход логики маршрутизации для эндпоинтов, которые не проходят через njs.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
This is intentional and not a regression. X-Route-Model is a documented OmniRoute feature (PR cheaper-inference#4863) — it's designed to let proxies/callers override routing. The /v1 fallback location handles endpoints like /v1/models, /v1/embeddings etc. that don't need model prefix rewriting. Clients who send X-Route-Model directly are using the feature as designed, and the header still passes through enforceApiKeyPolicy so it can't bypass per-key allowlists.
Note: this OmniRoute PR is a duplicate — the canonical PR was merged in ai-service (cheaper-inference#140).
Summary
Adds an nginx/njs layer that transparently routes requests to the correct KFC provider based on which OpenAI API endpoint the client calls — without modifying OmniRoute code or the request body.
Problem: OmniRoute doesn't allow two aliases with the same name pointing to different providers. Clients switching between
/v1/responsesand/v1/chat/completionshad to manually change model prefixes.Solution: njs handler in
contrib/nginx/endpoint_routing.js:Key design:
handleRequest(r)readsr.requestText, extractsbody.model, callsstripPrefix()+prefixForEndpoint(r.uri), stores result in$route_model(js_var), thenr.internalRedirect("/_omni_proxy" + r.uri). The internal proxy location addsproxy_set_header X-Route-Model $route_model— OmniRoute's existingresolveRoutingModel()picks it up. Original body untouched → SSE streaming preserved.Non-KFC models (e.g.
gemini-*, any model with unknown provider prefix likegemini-cli/) are skipped byshouldSkipModel()and resolved via OmniRoute aliases as before.Deployed & tested on production — all 5 routing scenarios verified. 24 KFC/KFC-Old aliases removed from DB (9 gemini aliases kept).
Link to Devin session: https://app.devin.ai/sessions/0a727cac8ccc444e8356683bb6bed778
Requested by: @fyodoregor