fix(resilience): route chat by per-connection synced model inventory (#11089) by pacocartones · Pull Request #11186 · diegosouzapw/OmniRoute

pacocartones · 2026-08-23T00:25:14Z

Thanks @yourspraveen for the report — it was precise enough to reproduce from the description alone.

Problem. When one self-hosted provider has several connections pointing at different hosts, each host keeps its own synced model inventory, but connection selection only consulted the manual excludedModels denylist — so a request could be routed to a host that never synced the model, surfacing as a spurious model-not-found.

Fix. Filter candidate connections by each connection's synced inventory, pinning the request to the host that actually advertises the model.

Note on the path in the issue

The issue cites src/lib/providerModels/syncedEndpointRouting.ts as prior art. That file doesn't exist on this branch yet — it ships inside the still-open #11088, which is based on main and touches the images/embeddings paths. Building on it would not compile here.

So the logic went where the actual decision is made instead:

src/sse/services/auth.ts — loadAdvertisedModelsForSelfHostedConnections(), one extra predicate in the candidate filter, and a modelNotAdvertised log branch alongside the existing excluded branch.
src/domain/connectionModelRules.ts — isModelAdvertisedByConnection(), matching the same candidate semantics as the existing denylist helper.

It reuses getSyncedAvailableModelsByConnection (src/lib/db/models.ts), which already had five consumers; connection selection simply wasn't one of them.

It fails open in three places — empty inventory, malformed persisted row, or a DB read that throws all mean "unknown", never "this host lacks the model". A partial read must never silently shrink the connection pool. Scoped to SELF_HOSTED_CHAT_PROVIDER_IDS, so hosted providers don't pay an extra DB read.

Evidence

RED — src/ reverted to the parent commit, new test file kept:

tests 5 | pass 3 | fail 2
AssertionError: jetson never synced flux2-klein:9b and must not be selected for it

GREEN — with the fix: tests 5 | pass 5 | fail 0.

Regression sweep — 199 tests, 196 pass. The 3 failures (session-affinity-generic-7274, provider-health-matrix, a Codex virtual-children case) were each confirmed failing at unmodified release/v3.8.50 too. Net: 0 regressions, +5 new passing tests.

Heads-up: release/v3.8.50 is currently red for unrelated reasons (README migration counts, stale eslint suppressions, an incomplete Hack Club removal), so CI here may show pre-existing failures that this branch doesn't cause.

Context

#11098 also nominally claims this issue. For maintainer awareness it is currently CONFLICTING, based on main rather than release/v3.8.50, and spans 3431 files / +583818 lines. Noting it only so the overlap isn't a surprise.

4 files, +287/-1. Rebased on release/v3.8.50 @ 92ef3c71e.

Closes #11089

…iegosouzapw#11089) Multi-host self-hosted providers (ollama-local, lm-studio, vllm, ...) keep a per-connection synced inventory, but getProviderCredentials only ever consulted the manual excludedModels denylist. A request for a model that only one host advertises could therefore be routed to a host that never had it, producing a spurious model-not-found instead of pinning to the host that does. Adds isModelAdvertisedByConnection to connectionModelRules (literal id match, same candidate semantics as the denylist, fails open on an empty inventory) and an additional predicate in the availableConnections filter, scoped to SELF_HOSTED_CHAT_PROVIDER_IDS.

pacocartones · 2026-08-23T00:43:56Z

@diegosouzapw CI note: the red checks here are pre-existing on release/v3.8.50, not from this PR.

Unit Tests fast-path (1-4/4) - the same 4 shards fail identically on the other open PRs against this base (feat(routing): pre-compile model prefix regexes in cloudCodeThinking service #11187, fix(codex): prefer max_context_window over context_window as the usable input limit #11179, fix(sse): default single-target incompatible reasoning to drop for agentic replay (#10959) #11178). I also reverted just this PR's src/ back to the base and re-ran the shard-1 failures locally: openrouter-free-model-credits-exhausted (3 cases) and source-scanner-guards still fail, tests 8 | pass 4 | fail 4. Same result with and without this change.
No new ESLint warnings - fails with There are suppressions left that do not occur anymore, a stale-suppression prune issue. No warning is attributed to either file this PR touches.

The 5 new tests did run and pass inside CI on shard 1:

ok #11089 selects only the host whose synced inventory advertises the model
ok #11089 does not preemptively fail over to a host lacking the model when the owner is cooling
ok #11089 keeps the connection that does advertise the model selectable
ok #11089 a model advertised by every host leaves both connections eligible
ok #11089 fails open when the provider has no synced inventory at all

Green here: Build, Docs Gates, Vitest, Change Classification, Merge integrity (changelog), semgrep, dast-smoke. Happy to rebase once the base-reds are drained.

@ggdayup

… confirmed) (diegosouzapw#11114) Validated on the combined batch board over tip 92ef3c7: static gates clean (changelog, file-size, complexity 2624<=2774, cognitive 1182<=1223, dead-code 411<=416), typecheck:core clean, focused tests green. Companion to diegosouzapw#11113 (consumer side): tokenrouter joins BUILTIN_PROVIDERS_SYSTEM_MUST_BE_FIRST — memory-system-first-6135 suite green. Live-confirmed 400 class documented in the body. Thank you @ggdayup!

@maxmad64bis

…zapw#11162) Validated on the combined batch board over tip 92ef3c7: static gates clean (changelog, file-size, complexity 2624<=2774, cognitive 1182<=1223, dead-code 411<=416), typecheck:core clean, focused tests green. Combo without models is now refused at the schema boundary (API 400), the CLI flags it, and openapi.yaml matches the real contract (phantom props removed). combo-* suites + cli-combo-create-models green on the board. Closes diegosouzapw#10954. Thank you @maxmad64bis!

@maxmad64bis

…e) (diegosouzapw#11158) Validated on the combined batch board over tip 92ef3c7: static gates clean (changelog, file-size, complexity 2624<=2774, cognitive 1182<=1223, dead-code 411<=416), typecheck:core clean, focused tests green. Empty-envelope 400 (no error field, empty content, finish_reason null) now rotates/retries instead of propagating as success; 200/streaming path never buffered; real-error 400s untouched. account-rotation + new rotation suite 34/34 on the board. Thank you @maxmad64bis!

@ggdayup

…stem message (diegosouzapw#11113) Validated on the combined batch board: purify-system-first suite 4/4, typecheck clean. Pre-merge: file-size baseline gained a frozen entry for contextManager.ts at 1001 (+1, this PR's merge-into-leading-system branch) with a dated annotation — the gate caps unlisted files at 1000. Producer side of the live-confirmed TokenRouter 400 class: no internal path emits a mid-array system message anymore. Thank you @ggdayup — the call-log evidence made this airtight!

@maxmad64bis

…ly output (diegosouzapw#11151) Merged after conflict resolution against the tip's diegosouzapw#11109 (per-call tool_call tracking): scanOpenAiSseText keeps the per-call finish_reason special-case AND gains reasoningText + literal finishReason; canContinue uses the in-flight predicate with the new reasoning-only-clean-stop escape. One integration fix on the branch: the PR's hallucinatedEmptyStop referenced emittedToolCall, which diegosouzapw#11109 had renamed — the branch now tracks emittedSawToolCall at the emitted level (any tool_call delta, complete or not), preserving the PR's don't-recover-after-tool-calls intent. Chain suites green: stream-continuation-wiring + stream-continuation + stream-recovery-toolcall 29/29. Thank you @maxmad64bis!

@maxmad64bis

…f concatenating it raw (diegosouzapw#11152) Merged after sibling diegosouzapw#11151 landed: streamRecovery.ts auto-merged byte-identical to the validated combined board; the test-file conflict (both PRs added suites at the same anchor) resolved keeping all 11 tests — diegosouzapw#11151's four clean-stop cases plus this PR's three threshold cases, with the PR's updated partial-tail fixture for the pre-existing overlap test. Full chain green: 32/32 (wiring + continuation + toolcall regression). The documented 8-char overlap threshold ends the silent mid-word gluing. Thank you @maxmad64bis!

diegosouzapw#11190) * feat(api): structured ?format=json for the self-service usage endpoint GET /api/usage/om-usage already let any key read its own usage — personal daily/weekly USD limits and the provider quota snapshot — but only as text/plain, which a UI cannot parse safely. OmniCopilot issue diegosouzapw#8 asks exactly for this surface. Adds ?format=json, returning the ApiKeyUsageLimitStatus + UsageSnapshot the text is rendered from. Text and JSON share the same collectors (collectUsageSnapshots, getApiKeyUsageLimitStatus), so the two can never disagree about a number. The response is a discriminated union: a key without allowUsageCommand (403) or an invalid key (401) returns { allowed:false, error:{message} }, distinct from allowed:true with empty sections — the state a panel must render as "nothing learned yet", not a refusal. Text form unchanged; without ?format the contract is untouched. The endpoint was previously missing from API_REFERENCE.md; it now has a section documenting both forms, the allowUsageCommand gate, and the self-service auth model (caller's own key, not requireManagementAuth). Regression guards in tests/unit/usage-command-json-format.test.ts (4 tests: json shape, text default preserved, structured 403, sanitized 401 with no stack trace). Existing internal-usage-command suite still 12/12. * chore(changelog): correct the fragment to the real PR number (diegosouzapw#11190) --------- Co-authored-by: Xiangzhe <bakryun0718@proton.me>

…-usage json (diegosouzapw#11192) * feat(api): structured ?format=json for the self-service usage endpoint GET /api/usage/om-usage already let any key read its own usage — personal daily/weekly USD limits and the provider quota snapshot — but only as text/plain, which a UI cannot parse safely. OmniCopilot issue diegosouzapw#8 asks exactly for this surface. Adds ?format=json, returning the ApiKeyUsageLimitStatus + UsageSnapshot the text is rendered from. Text and JSON share the same collectors (collectUsageSnapshots, getApiKeyUsageLimitStatus), so the two can never disagree about a number. The response is a discriminated union: a key without allowUsageCommand (403) or an invalid key (401) returns { allowed:false, error:{message} }, distinct from allowed:true with empty sections — the state a panel must render as "nothing learned yet", not a refusal. Text form unchanged; without ?format the contract is untouched. The endpoint was previously missing from API_REFERENCE.md; it now has a section documenting both forms, the allowUsageCommand gate, and the self-service auth model (caller's own key, not requireManagementAuth). Regression guards in tests/unit/usage-command-json-format.test.ts (4 tests: json shape, text default preserved, structured 403, sanitized 401 with no stack trace). Existing internal-usage-command suite still 12/12. * chore(changelog): correct the fragment to the real PR number (diegosouzapw#11190) * feat(api): return every connection's snapshot under providers[] in om-usage json Closes diegosouzapw#11191. buildUsageCommandJson picked a single snapshot via selectUsageSnapshot, so a panel could only ever show one provider. The collector already had them all — the single-pick is a presentation choice for a terminal. The JSON form now also returns the full UsageSnapshot[] alongside the selected provider, so a UI can render Codex / Claude / OpenCode side by side. The text form is untouched. --------- Co-authored-by: Xiangzhe <bakryun0718@proton.me>

… links through the shortener (diegosouzapw#11196) - New CheaperInferenceSponsorBanner on the dashboard home, same size/shape as KimiSponsorBanner, no version gate (durable partnership). Uses the cheaperinference ProviderIcon and the brand green (#31f889) with the dark ink CTA (contrast, per colors.ts token). - CTA points at https://link.omniroute.online/cheaper — the branded short link — so clicks land in our Kutt metrics. - VscodeCopilotBanner CTA now points at https://link.omniroute.online/vsx instead of the raw Marketplace URL, for the same reason. - i18n strings in en + pt (en is the namespace-level fallback for the other 41 locales). - Tests: new cheaperInferenceSponsorBanner.test.tsx (render, CTA href, dismiss persistence); vscodeCopilotBanner.test.tsx updated to the new CTA URL. Co-authored-by: Xiangzhe <bakryun0718@proton.me>

@asorourx

… response quality validation (diegosouzapw#11036) SSE comment lines (OpenRouter keep-alives) and leading whitespace no longer fail the combo quality gate's JSON fallback. First contribution — clean minimal fix with test. Thank you @asorourx, welcome aboard!

@jackjinke

…#11041) Compaction-V2 output now counts as real model output (no synthetic response.failed after response.completed), the Codex SSE filter handles CRLF framing, and terminal detection runs before scan-state bounding. 88/88 stream/readiness suites on the board. Thank you @jackjinke!

@RaviTharuma

Validated on the combined batch board + this branch alone: chat-body-admission + authz/pipeline 65/65, file-size gate green with a dated frozen entry (chatBodyAdmission 1005→1009 — the +4 lease/drain wiring lines, owner-authorized rebaseline). trackRequest was never called, so SIGTERM waitForDrain saw zero in-flight and killed live SSE; leases now hold the drain counter for the stream's lifetime, and the 503 carries Retry-After. Closes diegosouzapw#11015. Thank you @RaviTharuma!

@HouMinXi

… retrieve tool (diegosouzapw#11084) Validated on the combined batch board + this branch: ccr-non-mcp-full-prompt-loss + ccr-retrieval-ramp 20/20, file-size gate green with the ccr/index listing (1024, dated annotation — owner-authorized). The callerSupportsCcrRetrieve gate now skips the whole engine for callers whose tools[] cannot reach omniroute_ccr_retrieve — no more 15KB prompt arriving upstream as 112 tokens. Production-measured root cause, textbook TDD. Thank you @HouMinXi!

@xz-dev

Merged after conflict resolution in modelMetadataRegistry.ts: the tip's effortTiers chain (declared efforts → declared tiers → undefined-if-thinking-declared → codex extension) now carries this PR's GLM guard as the final-fallback override — GLM-family models without a provider-declared contract get the authoritative empty tier list instead of generic OpenAI tiers. GLM/ZCode suites 40/40 on the resolved branch. Closes diegosouzapw#10962. Thank you @xz-dev!

@jackjinke

…iegosouzapw#10949, diegosouzapw#10959) (diegosouzapw#10961) Validated on the combined batch board + this branch: 231/231 across chatcore-translation-paths, reasoning-cache, strip-reasoning-blobs, and both Responses translator suites. Pre-merge: propagated the diegosouzapw#11110/diegosouzapw#11129 summary:[] defaults into five assertions here (each commented with its PR) — without it this branch red against the tip, and as a bonus the merge drains the 4 reasoning reds that were live on the tip from those merges. Plaintext now wins over a coexisting opaque companion; opaque-only drops cleanly for plaintext targets; combos keep explicit Skip. Fixes diegosouzapw#10949 and diegosouzapw#10959. Thank you @jackjinke!

@jonlwheat2-gif

…egosouzapw#10644) (diegosouzapw#10987) Merged after count reconciliation: the branch's regenerated docs claimed 57 free forever / 157 migrations from its older base; gate-verified values on the current tip are 56 free forever (Logfare carries a Free badge via gateways.ts freeNote but has no freeModelCatalog per-model entries, so the live-code counter stays at 56) and 159 migrations — the README/SVGs now match the check:docs-counts output exactly. provider-consistency OK at 267 registry / 349 canonical. logfare-registry 4/4, icon + KNOWN_PNGS + discovery-set membership all verified present. Closes diegosouzapw#10644. Thank you @jonlwheat2-gif!

@backryun

…egosouzapw#10964) Merged after conflict triage: the six base-red repair files (vi.json, opencode.ts JSDoc, context-manager test, the three webhook dispatcher tests, the uncloseai orphan-test rename) were already drained on the tip by today's diegosouzapw#11130/diegosouzapw#11157/diegosouzapw#11160/diegosouzapw#11113 — those hunks resolved to the tip shape. What lands is the production-fix set: GLM transport-aware Anthropic headers, Claude Code-compatible model-listing rejection, combo live-test single-probe, zero-cost Auto-Combo interval normalization, recovery-clearing union handling, LLMLingua real-path compare, macOS netstat PID discovery, AI Horde R2 strict public-host validation. Sweep of every touched test file: 243/243 green; typecheck + file-size clean. (guide-settings-route's 4 reds reproduce on the pure tip — pre-existing drift from diegosouzapw#11079, not from here.) Thank you @backryun!

@rqzbeh

…ocess-spawning endpoints (diegosouzapw#11189) Validated on the combined batch board (gates + typecheck clean) and this branch: security-route-guard-tiers green. Regression coverage for the Hard Rule diegosouzapw#15/diegosouzapw#17 contract — Tier 1 process-spawning prefixes (/api/services/, /api/mcp/, /api/cli-tools/runtime/) must stay LOCAL_ONLY before any auth check. Conflict with the tip was only stale provider-count docs. Thank you @rqzbeh!

@rqzbeh

…ed labels (diegosouzapw#11188) Validated on the combined batch board + this branch: dashboard-ux-operability green. Unmapped custom quota keys now render as title-cased labels instead of raw snake_case. Conflict with the tip was only stale provider-count docs. Thank you @rqzbeh!

@rqzbeh

…login (diegosouzapw#11143) (diegosouzapw#11175) Validated on the combined batch board + this branch: login-11143 green. Full document navigation after login guarantees the auth_token cookie is committed before any RSC prefetch fires — no more 307 back to /login. Conflict with the tip was only stale provider-count docs. Fixes diegosouzapw#11143. Thank you @rqzbeh!

@rqzbeh

…fication (diegosouzapw#10443) (diegosouzapw#11177) Validated on the combined batch board + this branch: antigravity-dynamic-session-id + proxy-fetch-dns-retry green; file-size gate green with the proxyFetch 1244 frozen entry (dated annotation for the +5 retry-classification lines, owner-authorized). Static per-account sessionId unpinning ends the concurrent-turn 429s and EmptyStreamError drops on the Hermes→Antigravity path; EAI_AGAIN/ENOTFOUND/ETIMEDOUT now classified retryable. Conflict with the tip was only stale provider-count docs. Resolves the remaining diegosouzapw#10443 root causes. Thank you @rqzbeh!

@rqzbeh

…ormance optimizations (A, B, C, D) (diegosouzapw#11182) Validated on the combined batch board + this branch: perf-a-b-c-d + account-fallback-service 93/93, gates + typecheck clean. Owner approved the full bundle including item A (async proxy-log batching, 1s/100-item flush with an unref'd timer — reviewed the implementation: flush helper exists for shutdown wiring, buffered logs are the accepted tradeoff). B (lazy modals), C (O(1) alias maps), D (pre-compiled regex — this is also the entire content of diegosouzapw#11187, being closed as subsumed) ride along. Conflict with the tip was only stale provider-count docs. Thank you @rqzbeh!

@pacocartones

…diegosouzapw#9763) (diegosouzapw#11086) Validated on the combined batch board over tip c92bd40: static gates clean (changelog, file-size 158 frozen, complexity 2626<=2774, cognitive 1183<=1223, dead-code 409<=416), typecheck:core clean, 70 focused tests green (PR suites 49/49 + auth/combo neighbors 21/21). Operator-configured positive minTime floor now survives the plenty-of-headroom relaxation (resolveMinTime instead of a hard 0). Fixes diegosouzapw#9763. Thank you @pacocartones!

@pacocartones

…er (diegosouzapw#10071) (diegosouzapw#11185) Validated on the combined batch board over tip c92bd40: static gates clean (changelog, file-size 158 frozen, complexity 2626<=2774, cognitive 1183<=1223, dead-code 409<=416), typecheck:core clean, 70 focused tests green (PR suites 49/49 + auth/combo neighbors 21/21). Five g4f-* entries re-flagged hasFree:false with the live-probed 402 evidence (proof-of-work wall, member key still works); the two dissenting providers deliberately untouched, matching the chutes/aimlapi/yi precedent. Fixes diegosouzapw#10071. Thank you @pacocartones and @chirag127 for the capture!

@pacocartones

…gle works (diegosouzapw#11193) Validated on the combined batch board over tip c92bd40: static gates clean (changelog, file-size 158 frozen, complexity 2626<=2774, cognitive 1183<=1223, dead-code 409<=416), typecheck:core clean, 70 focused tests green (PR suites 49/49 + auth/combo neighbors 21/21). lkgpEnabled finally reaches the RoutingContext literal — the Settings→Routing LKGP toggle was persisted but unreachable (context.lkgpEnabled always undefined). Scope discipline noted and appreciated: the applyStrategyOrdering dependency change stays out. Fixes diegosouzapw#11181. Thank you @pacocartones!

@pacocartones

…* pool (diegosouzapw#11198) Validated on the combined batch board over tip c92bd40: static gates clean (changelog, file-size 158 frozen, complexity 2626<=2774, cognitive 1183<=1223, dead-code 409<=416), typecheck:core clean, 70 focused tests green (PR suites 49/49 + auth/combo neighbors 21/21). Keyless custom-compatible connections stay in auto/* pools — the credential filter now recognizes a registry-free keyless endpoint instead of dropping the connection before pool construction. Fixes diegosouzapw#11180. Thank you @pacocartones!

@pacocartones

…gosouzapw#11199) Validated on the combined batch board over tip c92bd40: static gates clean (changelog, file-size 158 frozen, complexity 2626<=2774, cognitive 1183<=1223, dead-code 409<=416), typecheck:core clean, 70 focused tests green (PR suites 49/49 + auth/combo neighbors 21/21). opencode-go joins FLAT_RATE_SUBSCRIPTION_PROVIDER_IDS — cost analytics stop pricing a flat 0 subscription at metered aggregator rates (3.35 reported vs 0 actual). Same pattern as diegosouzapw#10774. Fixes diegosouzapw#11149. Thank you @pacocartones!

…11186)

@pacocartones

…iegosouzapw#11089) (diegosouzapw#11186) Validated on the combined batch board + this branch: synced-inventory + auth neighbor suites 16/16; file-size gate green with the auth.ts 3337 frozen entry (dated annotation for the +77 inventory-filter lines at the credential-selection chokepoint, owner-authorized). Chat routing now pins to the connection whose synced inventory actually advertises the model — no more spurious model-not-found on multi-host self-hosted setups. Scoping call (not building on the still-open diegosouzapw#11088) was the right one. Fixes diegosouzapw#11089. Thank you @pacocartones and @yourspraveen for the precise report!

pacocartones requested a review from diegosouzapw as a code owner August 23, 2026 00:25

ggdayup and others added 27 commits August 22, 2026 21:51

hartmark added 2 commits August 23, 2026 01:25

board diegosouzapw#11186

100b60a

chore(quality): freeze auth.ts at 3337 with annotation (diegosouzapw#…

1722c83

…11186)

diegosouzapw merged commit 5853e22 into diegosouzapw:release/v3.8.50 Aug 23, 2026
0 of 3 checks passed

This was referenced Aug 23, 2026

fix(chat): respect per-connection model inventory in chat routing (#11089) #11217

Closed

fix(chat): respect per-connection model inventory in chat routing (#11089)Fix/chat routing model inventory #11098

Closed

Navigation Menu

Uh oh!

fix(resilience): route chat by per-connection synced model inventory (#11089) - #11186

fix(resilience): route chat by per-connection synced model inventory (#11089)#11186
diegosouzapw merged 30 commits into
diegosouzapw:release/v3.8.50from
pacocartones:fix/11089-chat-routing-synced-inventory

pacocartones commented Aug 23, 2026

Uh oh!

pacocartones commented Aug 23, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants

Uh oh!

Conversation

pacocartones commented Aug 23, 2026

Note on the path in the issue

Evidence

Context

Uh oh!

pacocartones commented Aug 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants

pacocartones commented Aug 23, 2026 •

edited

Loading