Conversation
Enhance provider selection logic to consider uptime and latency from the last 5 minutes in addition to price. - Add getProviderMetrics functions in @llmgateway/db to fetch uptime/latency from model_provider_mapping_history - Update getCheapestFromAvailableProviders to accept optional metrics and use weighted scoring algorithm: * 40% price * 30% uptime * 30% latency - Integrate metrics into both auto routing and regular model routing in the gateway - Falls back to price-only selection when metrics unavailable
github-actions
Bot
changed the title
WalkthroughGateway now fetches recent provider-model runtime metrics and supplies a metrics map to provider selection; selection scores providers by price, uptime and latency, filters unstable/experimental providers, and returns routing metadata that is persisted to logs and passed downstream. Changes
Sequence Diagram(s)sequenceDiagram
participant Chat as Gateway Chat
participant DB as Metrics DB
participant Selector as Provider Selector
Chat->>DB: getProviderMetricsForCombinations(combinations, 5)
DB-->>Chat: metricsMap {"modelId:providerId" → ProviderMetrics}
Chat->>Selector: getCheapestFromAvailableProviders(providers, selectedModel, metricsMap)
rect rgb(245,252,250)
Note over Selector: Filter unstable/experimental providers\nNormalize price, uptime, latency\nCompute composite score = 0.2*price + 0.5*(1-uptime_norm) + 0.3*latency_norm
end
Selector-->>Chat: ProviderSelectionResult { provider, metadata }
Chat->>Chat: createLogEntry(..., routingMetadata = metadata)
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes
Possibly related PRs
Suggested reviewers
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
apps/gateway/src/chat/chat.ts(3 hunks)packages/db/src/index.ts(1 hunks)packages/db/src/provider-metrics.ts(1 hunks)packages/models/src/get-cheapest-from-available-providers.ts(2 hunks)
🧰 Additional context used
📓 Path-based instructions (5)
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (AGENTS.md)
Always use top-level import; never use require() or dynamic import()
Files:
packages/db/src/index.tspackages/db/src/provider-metrics.tsapps/gateway/src/chat/chat.tspackages/models/src/get-cheapest-from-available-providers.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx}: Never useanyoras anyin this TypeScript project unless absolutely necessary
Always use top-levelimport; do not userequireor dynamicimport()
Files:
packages/db/src/index.tspackages/db/src/provider-metrics.tsapps/gateway/src/chat/chat.tspackages/models/src/get-cheapest-from-available-providers.ts
packages/db/**/*.ts
📄 CodeRabbit inference engine (CLAUDE.md)
Use Drizzle ORM with the latest object syntax for database schema and code
Files:
packages/db/src/index.tspackages/db/src/provider-metrics.ts
{apps/{api,gateway}/**/*.ts,packages/db/**/*.ts}
📄 CodeRabbit inference engine (CLAUDE.md)
For read operations, use
db().query.<table>.findMany()ordb().query.<table>.findFirst()
Files:
packages/db/src/index.tspackages/db/src/provider-metrics.tsapps/gateway/src/chat/chat.ts
apps/{gateway,api}/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
apps/{gateway,api}/**/*.ts: Use Hono for HTTP routing in Gateway and API services
Use Zod schemas for request/response validation in server routes
Files:
apps/gateway/src/chat/chat.ts
🧠 Learnings (1)
📚 Learning: 2025-10-20T21:33:03.287Z
Learnt from: CR
Repo: theopenco/llmgateway PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-10-20T21:33:03.287Z
Learning: Applies to packages/db/**/*.ts : Use Drizzle ORM with the latest object syntax for database schema and code
Applied to files:
packages/db/src/index.tspackages/db/src/provider-metrics.ts
🧬 Code graph analysis (3)
packages/db/src/provider-metrics.ts (3)
packages/models/src/get-cheapest-from-available-providers.ts (1)
ProviderMetrics(4-10)packages/db/src/db.ts (1)
db(21-25)packages/db/src/schema.ts (1)
modelProviderMappingHistory(679-726)
apps/gateway/src/chat/chat.ts (2)
packages/db/src/provider-metrics.ts (1)
getProviderMetricsForCombinations(81-145)packages/models/src/get-cheapest-from-available-providers.ts (1)
getCheapestFromAvailableProviders(38-155)
packages/models/src/get-cheapest-from-available-providers.ts (3)
packages/db/src/provider-metrics.ts (1)
ProviderMetrics(6-12)packages/models/src/types.ts (2)
AvailableModelProvider(265-268)ModelWithPricing(253-262)packages/models/src/models.ts (1)
ProviderModelMapping(24-121)
🪛 GitHub Actions: ci
packages/models/src/get-cheapest-from-available-providers.ts
[error] 91-91: TS2339: Property 'id' does not exist on type 'ModelWithPricing'.
[error] 1-1: Command 'tsc && resolve-tspaths' failed during @llmgateway/models build.
🪛 GitHub Actions: e2e
packages/models/src/get-cheapest-from-available-providers.ts
[error] 91-91: TypeScript error TS2339: Property 'id' does not exist on type 'ModelWithPricing'.
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
packages/db/src/provider-metrics.ts (1)
81-145: Consider using idiomatic Drizzle ORM syntax for the OR conditions.The OR condition building (lines 92-97) uses raw SQL templates. For better type safety and readability, consider using Drizzle's
or()andeq()functions:+import { and, eq, gte, or, sql, sum } from "drizzle-orm"; ... -const conditions = combinations.map((combo) => - and( - sql`${modelProviderMappingHistory.modelId} = ${combo.modelId}`, - sql`${modelProviderMappingHistory.providerId} = ${combo.providerId}`, - ), -); +const conditions = combinations.map((combo) => + and( + eq(modelProviderMappingHistory.modelId, combo.modelId), + eq(modelProviderMappingHistory.providerId, combo.providerId), + ), +); const results = await db .select({...}) .from(modelProviderMappingHistory) .where( and( gte(modelProviderMappingHistory.minuteTimestamp, fiveMinutesAgo), - sql`(${sql.join(conditions, sql` OR `)})`, + or(...conditions), ), )
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
packages/db/src/provider-metrics.ts(1 hunks)packages/models/src/get-cheapest-from-available-providers.ts(2 hunks)
🧰 Additional context used
📓 Path-based instructions (4)
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (AGENTS.md)
Always use top-level import; never use require() or dynamic import()
Files:
packages/models/src/get-cheapest-from-available-providers.tspackages/db/src/provider-metrics.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx}: Never useanyoras anyin this TypeScript project unless absolutely necessary
Always use top-levelimport; do not userequireor dynamicimport()
Files:
packages/models/src/get-cheapest-from-available-providers.tspackages/db/src/provider-metrics.ts
packages/db/**/*.ts
📄 CodeRabbit inference engine (CLAUDE.md)
Use Drizzle ORM with the latest object syntax for database schema and code
Files:
packages/db/src/provider-metrics.ts
{apps/{api,gateway}/**/*.ts,packages/db/**/*.ts}
📄 CodeRabbit inference engine (CLAUDE.md)
For read operations, use
db().query.<table>.findMany()ordb().query.<table>.findFirst()
Files:
packages/db/src/provider-metrics.ts
🧬 Code graph analysis (2)
packages/models/src/get-cheapest-from-available-providers.ts (2)
packages/models/src/types.ts (2)
AvailableModelProvider(265-268)ModelWithPricing(253-262)packages/models/src/models.ts (1)
ProviderModelMapping(24-121)
packages/db/src/provider-metrics.ts (2)
packages/db/src/db.ts (1)
db(21-25)packages/db/src/schema.ts (1)
modelProviderMappingHistory(679-726)
🪛 ESLint
packages/db/src/provider-metrics.ts
[error] 1-1: Resolve error: EACCES: permission denied, open '/HjoUnatxrC'
at Object.writeFileSync (node:fs:2409:20)
at l (/home/jailuser/git/node_modules/.pnpm/get-tsconfig@4.10.1/node_modules/get-tsconfig/dist/index.cjs:7:13685)
at createFilesMatcher (/home/jailuser/git/node_modules/.pnpm/get-tsconfig@4.10.1/node_modules/get-tsconfig/dist/index.cjs:7:14437)
at Object.resolve (/home/jailuser/git/node_modules/.pnpm/eslint-import-resolver-typescript@4.4.4_eslint-plugin-import@2.32.0_eslint@9.34.0_jiti@2.6.1_/node_modules/eslint-import-resolver-typescript/lib/index.cjs:298:107)
at withResolver (/home/jailuser/git/node_modules/.pnpm/eslint-module-utils@2.12.1_@typescript-eslint+parser@8.39.1_eslint@9.34.0_jiti@2.6.1__t_eb0afb446ca3f59399f5d681f0059e64/node_modules/eslint-module-utils/resolve.js:180:23)
at fullResolve (/home/jailuser/git/node_modules/.pnpm/eslint-module-utils@2.12.1_@typescript-eslint+parser@8.39.1_eslint@9.34.0_jiti@2.6.1__t_eb0afb446ca3f59399f5d681f0059e64/node_modules/eslint-module-utils/resolve.js:201:22)
at relative (/home/jailuser/git/node_modules/.pnpm/eslint-module-utils@2.12.1_@typescript-eslint+parser@8.39.1_eslint@9.34.0_jiti@2.6.1__t_eb0afb446ca3f59399f5d681f0059e64/node_modules/eslint-module-utils/resolve.js:217:10)
at resolve (/home/jailuser/git/node_modules/.pnpm/eslint-module-utils@2.12.1_@typescript-eslint+parser@8.39.1_eslint@9.34.0_jiti@2.6.1__t_eb0afb446ca3f59399f5d681f0059e64/node_modules/eslint-module-utils/resolve.js:233:12)
at resolveImportType (/home/jailuser/git/node_modules/.pnpm/eslint-plugin-import@2.32.0_@typescript-eslint+parser@8.39.1_eslint@9.34.0_jiti@2.6.1___986ec7d736a20dae59d4d473ff8a6f0d/node_modules/eslint-plugin-import/lib/core/importType.js:126:2822)
at computeRank (/home/jailuser/git/node_modules/.pnpm/eslint-plugin-import@2.32.0_@typescript-eslint+parser@8.39.1_eslint@9.34.0_jiti@2.6.1___986ec7d736a20dae59d4d473ff8a6f0d/node_modules/eslint-plugin-import/lib/rules/order.js:529:43)
(import/order)
[error] 1-1: Resolve error: EACCES: permission denied, open '/RXwsvBEJwV'
at Object.writeFileSync (node:fs:2409:20)
at l (/home/jailuser/git/node_modules/.pnpm/get-tsconfig@4.10.1/node_modules/get-tsconfig/dist/index.cjs:7:13685)
at createFilesMatcher (/home/jailuser/git/node_modules/.pnpm/get-tsconfig@4.10.1/node_modules/get-tsconfig/dist/index.cjs:7:14437)
at Object.resolve (/home/jailuser/git/node_modules/.pnpm/eslint-import-resolver-typescript@4.4.4_eslint-plugin-import@2.32.0_eslint@9.34.0_jiti@2.6.1_/node_modules/eslint-import-resolver-typescript/lib/index.cjs:298:107)
at withResolver (/home/jailuser/git/node_modules/.pnpm/eslint-module-utils@2.12.1_@typescript-eslint+parser@8.39.1_eslint@9.34.0_jiti@2.6.1__t_eb0afb446ca3f59399f5d681f0059e64/node_modules/eslint-module-utils/resolve.js:180:23)
at fullResolve (/home/jailuser/git/node_modules/.pnpm/eslint-module-utils@2.12.1_@typescript-eslint+parser@8.39.1_eslint@9.34.0_jiti@2.6.1__t_eb0afb446ca3f59399f5d681f0059e64/node_modules/eslint-module-utils/resolve.js:201:22)
at relative (/home/jailuser/git/node_modules/.pnpm/eslint-module-utils@2.12.1_@typescript-eslint+parser@8.39.1_eslint@9.34.0_jiti@2.6.1__t_eb0afb446ca3f59399f5d681f0059e64/node_modules/eslint-module-utils/resolve.js:217:10)
at resolve (/home/jailuser/git/node_modules/.pnpm/eslint-module-utils@2.12.1_@typescript-eslint+parser@8.39.1_eslint@9.34.0_jiti@2.6.1__t_eb0afb446ca3f59399f5d681f0059e64/node_modules/eslint-module-utils/resolve.js:233:12)
at checkSourceValue (/home/jailuser/git/node_modules/.pnpm/eslint-plugin-import@2.32.0_@typescript-eslint+parser@8.39.1_eslint@9.34.0_jiti@2.6.1___986ec7d736a20dae59d4d473ff8a6f0d/node_modules/eslint-plugin-import/lib/rules/no-useless-path-segments.js:85:53)
at checkSourceValue (/home/jailuser/git/node_modules/.pnpm/eslint-module-utils@2.12.1_@typescript-eslint+parser@8.39.1_eslint@9.34.0_jiti@2.6.1__t_eb0afb446ca3f59399f5d681f0059e64/node_modules/eslint-module-utils/moduleVisitor.js:32:5)
(import/no-useless-path-segments)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (10)
- GitHub Check: generate / run
- GitHub Check: lint / run
- GitHub Check: test / run
- GitHub Check: build / run
- GitHub Check: e2e-shards (5)
- GitHub Check: e2e-shards (2)
- GitHub Check: e2e-shards (3)
- GitHub Check: e2e-shards (4)
- GitHub Check: e2e-shards (1)
- GitHub Check: autofix
🔇 Additional comments (6)
packages/db/src/provider-metrics.ts (1)
14-71: Verify the uptime calculation logic.The uptime calculation currently only subtracts
upstreamErrorsCountfromlogsCountto determine successful requests (lines 35, 56). However, the schema includes other error types:errorsCount,clientErrorsCount, andgatewayErrorsCount.Should uptime account for all error types, or is the current approach (only upstream errors) intentional? If all errors represent failures from the user's perspective, the calculation should be:
const successfulRequests = totalLogs - (totalUpstreamErrors + totalClientErrors + totalGatewayErrors);Please clarify the intended uptime definition and verify whether other error types should be included in the calculation.
packages/models/src/get-cheapest-from-available-providers.ts (5)
12-27: LGTM: Well-designed scoring system.The weighted scoring approach (40% price, 30% uptime, 30% latency) with reasonable default values provides a balanced provider selection mechanism while maintaining backward compatibility.
72-101: LGTM: Robust metrics integration with proper fallback.The implementation correctly falls back to price-only selection when metrics are unavailable (lines 72-75) and properly constructs the metrics lookup key (line 91) matching the format from the db package.
103-151: LGTM: Normalization and scoring logic is mathematically sound.The normalization correctly handles edge cases (e.g., zero range when all providers have identical values) and consistently treats lower scores as better across all metrics. The weighted combination (lines 136-139) properly applies the scoring weights.
153-181: LGTM: Clean fallback preserves original behavior.The
selectByPriceOnlyhelper maintains backward compatibility and properly isolates the price-only logic for when metrics are unavailable.
38-44: Type signature fix verified across all call sites.All 5 call sites of
getCheapestFromAvailableProviderspass variables that include theidproperty:
- models.spec.ts (3 calls): Variables derive from the imported
modelsarray, which has confirmedidproperties- chat.ts line 1025:
selectedModeluses.idat line 1019- chat.ts line 1131:
modelWithPricingis found from themodelsarray and uses.idat line 1134The intersection type
ModelWithPricing & { id: string }correctly resolves the previous TypeScript error and all callers comply with the new signature.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (4)
packages/db/src/provider-metrics.spec.ts (1)
86-322: Good coverage of aggregation semantics; consider one tiny edge-case test.The
getProviderMetricstests thoroughly exercise the main behaviors: empty data, aggregation over multiple minutes, multiple provider/model pairs, time-window filtering, zero-log exclusion, and uptime/latency extremes. The expectations line up with the implementation (usingupstreamErrorsCountfor failures and keying as${modelId}:${providerId}), so this gives high confidence in the DB layer.Minor optional improvement: the “null or zero totalDuration” case currently only tests zero; if you ever allow manual backfills that might store
NULLfortotalDurationdespite the default, a dedicated test insertingtotalDuration: nullwould directly assert that theNumber(row.totalDuration) || 0coercion behaves as intended.packages/models/src/get-cheapest-from-available-providers.ts (3)
3-21: Weights and default metrics are reasonable but might deserve configurability.Importing
ProviderMetricsfrom@llmgateway/dbremoves the prior type duplication, and theProviderScoreshape + weight constants make the scoring criteria explicit. The 0.4/0.3/0.3 split and defaults of 95% uptime / 1000 ms latency seem sensible as an initial heuristic.Design-wise, two optional thoughts:
- You might eventually want these weights and defaults to be configurable (env or config file), so tuning routing behavior doesn’t require a deploy.
- Treating missing metrics as “95% / 1000 ms” can make providers with no telemetry look quite good compared to ones with merely decent real metrics; if you’d rather be conservative, consider slightly penalizing unknown metrics instead of assuming they’re solid.
65-144: Metrics-based scoring and price-only fallback are coherent; watch behavior for missing per-provider metrics.The control flow is sound: if
metricsMapis absent or empty you delegate toselectByPriceOnly, otherwise you:
- compute a per-provider price (reusing the existing discount-aware formula),
- look up metrics under
${modelWithPricing.id}:${provider.providerId},- normalize price (cheaper → lower score), uptime (higher → lower score), and latency (lower → lower score),
- and select the provider with the lowest weighted score.
A couple of nuanced behaviors to be aware of:
- When some stable providers lack entries in
metricsMap, they getDEFAULT_UPTIME/DEFAULT_LATENCY. This can advantage “unknown” providers over ones with modest but real metrics (e.g., uptime 80% vs default 95%). If that’s intentional, great; otherwise you might want to bias defaults slightly worse than typical telemetry or even treat missing metrics as a pure price-only dimension.- When
metricsMaphas no entries for the current model but isn’t empty (e.g., metrics for other models), all providers share the same default uptime/latency, so the normalization naturally collapses to a price-only ordering, which matches the intended fallback.Overall the scoring math (ranges, normalization, and tie-breaking via the first element) is consistent and should behave predictably.
146-173: Price-only fallback preserves existing behavior; consider deduplicating the pricing formula.
selectByPriceOnlycorrectly mirrors the original logic: it iterates the stable providers, computes the average of input/output prices with discount applied, and returns the cheapest. This keeps the no-metrics path backward compatible.You now have the same pricing calculation in both the main scoring loop and this helper. If you plan to tweak pricing in the future (e.g., weight input vs output differently or add request-level pricing), extracting a small
computeEffectivePrice(modelWithPricing, providerId)helper would avoid divergence between the metrics and price-only paths.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
packages/db/src/provider-metrics.spec.ts(1 hunks)packages/models/src/get-cheapest-from-available-providers.ts(2 hunks)
🧰 Additional context used
🧬 Code graph analysis (2)
packages/db/src/provider-metrics.spec.ts (3)
packages/db/src/db.ts (1)
db(21-25)packages/db/src/schema.ts (4)
modelProviderMappingHistory(679-726)modelProviderMapping(624-677)model(593-622)provider(559-591)packages/db/src/provider-metrics.ts (2)
getProviderMetrics(25-71)getProviderMetricsForCombinations(81-145)
packages/models/src/get-cheapest-from-available-providers.ts (3)
packages/models/src/types.ts (2)
AvailableModelProvider(265-268)ModelWithPricing(253-262)packages/db/src/provider-metrics.ts (1)
ProviderMetrics(6-12)packages/models/src/models.ts (1)
ProviderModelMapping(24-121)
🪛 ESLint
packages/db/src/provider-metrics.spec.ts
[error] 1-1: Resolve error: EACCES: permission denied, open '/rVTiAJkuqz'
at Object.writeFileSync (node:fs:2409:20)
at l (/home/jailuser/git/node_modules/.pnpm/get-tsconfig@4.10.1/node_modules/get-tsconfig/dist/index.cjs:7:13685)
at createFilesMatcher (/home/jailuser/git/node_modules/.pnpm/get-tsconfig@4.10.1/node_modules/get-tsconfig/dist/index.cjs:7:14437)
at Object.resolve (/home/jailuser/git/node_modules/.pnpm/eslint-import-resolver-typescript@4.4.4_eslint-plugin-import@2.32.0_eslint@9.34.0_jiti@2.6.1_/node_modules/eslint-import-resolver-typescript/lib/index.cjs:298:107)
at withResolver (/home/jailuser/git/node_modules/.pnpm/eslint-module-utils@2.12.1_@typescript-eslint+parser@8.39.1_eslint@9.34.0_jiti@2.6.1__t_eb0afb446ca3f59399f5d681f0059e64/node_modules/eslint-module-utils/resolve.js:180:23)
at fullResolve (/home/jailuser/git/node_modules/.pnpm/eslint-module-utils@2.12.1_@typescript-eslint+parser@8.39.1_eslint@9.34.0_jiti@2.6.1__t_eb0afb446ca3f59399f5d681f0059e64/node_modules/eslint-module-utils/resolve.js:201:22)
at relative (/home/jailuser/git/node_modules/.pnpm/eslint-module-utils@2.12.1_@typescript-eslint+parser@8.39.1_eslint@9.34.0_jiti@2.6.1__t_eb0afb446ca3f59399f5d681f0059e64/node_modules/eslint-module-utils/resolve.js:217:10)
at resolve (/home/jailuser/git/node_modules/.pnpm/eslint-module-utils@2.12.1_@typescript-eslint+parser@8.39.1_eslint@9.34.0_jiti@2.6.1__t_eb0afb446ca3f59399f5d681f0059e64/node_modules/eslint-module-utils/resolve.js:233:12)
at resolveImportType (/home/jailuser/git/node_modules/.pnpm/eslint-plugin-import@2.32.0_@typescript-eslint+parser@8.39.1_eslint@9.34.0_jiti@2.6.1___986ec7d736a20dae59d4d473ff8a6f0d/node_modules/eslint-plugin-import/lib/core/importType.js:126:2822)
at computeRank (/home/jailuser/git/node_modules/.pnpm/eslint-plugin-import@2.32.0_@typescript-eslint+parser@8.39.1_eslint@9.34.0_jiti@2.6.1___986ec7d736a20dae59d4d473ff8a6f0d/node_modules/eslint-plugin-import/lib/rules/order.js:529:43)
(import/order)
[error] 1-1: Resolve error: EACCES: permission denied, open '/azxrkKKmom'
at Object.writeFileSync (node:fs:2409:20)
at l (/home/jailuser/git/node_modules/.pnpm/get-tsconfig@4.10.1/node_modules/get-tsconfig/dist/index.cjs:7:13685)
at createFilesMatcher (/home/jailuser/git/node_modules/.pnpm/get-tsconfig@4.10.1/node_modules/get-tsconfig/dist/index.cjs:7:14437)
at Object.resolve (/home/jailuser/git/node_modules/.pnpm/eslint-import-resolver-typescript@4.4.4_eslint-plugin-import@2.32.0_eslint@9.34.0_jiti@2.6.1_/node_modules/eslint-import-resolver-typescript/lib/index.cjs:298:107)
at withResolver (/home/jailuser/git/node_modules/.pnpm/eslint-module-utils@2.12.1_@typescript-eslint+parser@8.39.1_eslint@9.34.0_jiti@2.6.1__t_eb0afb446ca3f59399f5d681f0059e64/node_modules/eslint-module-utils/resolve.js:180:23)
at fullResolve (/home/jailuser/git/node_modules/.pnpm/eslint-module-utils@2.12.1_@typescript-eslint+parser@8.39.1_eslint@9.34.0_jiti@2.6.1__t_eb0afb446ca3f59399f5d681f0059e64/node_modules/eslint-module-utils/resolve.js:201:22)
at relative (/home/jailuser/git/node_modules/.pnpm/eslint-module-utils@2.12.1_@typescript-eslint+parser@8.39.1_eslint@9.34.0_jiti@2.6.1__t_eb0afb446ca3f59399f5d681f0059e64/node_modules/eslint-module-utils/resolve.js:217:10)
at resolve (/home/jailuser/git/node_modules/.pnpm/eslint-module-utils@2.12.1_@typescript-eslint+parser@8.39.1_eslint@9.34.0_jiti@2.6.1__t_eb0afb446ca3f59399f5d681f0059e64/node_modules/eslint-module-utils/resolve.js:233:12)
at checkSourceValue (/home/jailuser/git/node_modules/.pnpm/eslint-plugin-import@2.32.0_@typescript-eslint+parser@8.39.1_eslint@9.34.0_jiti@2.6.1___986ec7d736a20dae59d4d473ff8a6f0d/node_modules/eslint-plugin-import/lib/rules/no-useless-path-segments.js:85:53)
at checkSourceValue (/home/jailuser/git/node_modules/.pnpm/eslint-module-utils@2.12.1_@typescript-eslint+parser@8.39.1_eslint@9.34.0_jiti@2.6.1__t_eb0afb446ca3f59399f5d681f0059e64/node_modules/eslint-module-utils/moduleVisitor.js:32:5)
(import/no-useless-path-segments)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (10)
- GitHub Check: test / run
- GitHub Check: lint / run
- GitHub Check: build / run
- GitHub Check: generate / run
- GitHub Check: e2e-shards (5)
- GitHub Check: e2e-shards (2)
- GitHub Check: autofix
- GitHub Check: e2e-shards (3)
- GitHub Check: e2e-shards (4)
- GitHub Check: e2e-shards (1)
🔇 Additional comments (3)
packages/db/src/provider-metrics.spec.ts (2)
17-25: Test setup and time control look coherent; just confirm timer behavior.The DB cleanup + seeding in
beforeEachcorrectly respects FK constraints (history → mappings → model → provider) and gives each test an isolated, deterministic state. Usingvi.setSystemTime(mockDate)plus static timestamps makes the “last N minutes” filters reproducible, andvi.useRealTimers()inafterEachrestores the environment.One thing to verify: depending on your Vitest configuration, you may need
vi.useFakeTimers()beforevi.setSystemTimeforDate.now()-based code to be fully controlled. If your existing suite already relies on this pattern and passes consistently, you’re fine; otherwise, consider addingvi.useFakeTimers()inbeforeEachto avoid any environment-dependent flakiness.Also applies to: 82-84
324-559: Combination-specific metrics tests are solid and match the query behavior.The
getProviderMetricsForCombinationssuite cleanly validates all key behaviors: empty combinations, selective inclusion of only requested pairs, multi-combination aggregation, custom time windows, dropping combinations with no data or zero logs, and correctness of uptime/latency/totalRequests for partial failures and the “single combination” path.These tests closely mirror the query contract in
provider-metrics.ts(per-combination OR conditions, same${modelId}:${providerId}keying, same aggregation formula), so the higher-level routing code can safely rely on these results.packages/models/src/get-cheapest-from-available-providers.ts (1)
31-63: Signature and stability filtering look correct; just ensure the model id contract is clear.Requiring
modelWithPricing: ModelWithPricing & { id: string }addresses the earlier TS issue and gives you a concrete identifier for metrics lookup. The stability filter correctly excludesunstableandexperimentalproviders by preferring provider-level stability over model-level stability, while treating missing stability as acceptable.One thing to verify across the stack:
modelWithPricing.idmust match themodelIdstored inmodel_provider_mapping_history(used as the key in the DB metrics map). If any caller passes a different identifier (e.g., a display name), metrics lookups will silently fall back to the default uptime/latency instead of using real data.
…67Fqts91qog59vi3sa
- Fix uptime calculation to use all errors instead of only upstream errors - Adjust routing weights to prioritize uptime (50%) over price (20%) - Add routing metadata to logs for debugging routing decisions - Fix price display in routing metadata to show full precision 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
packages/db/src/provider-metrics.ts (1)
92-97: Prefer Drizzle'seq()helper over raw SQL templates for safety.The raw
sqltemplate mixing column references and values is harder to read and more error-prone. Drizzle'seq()helper provides better type safety and clarity.Apply this diff:
+import { and, eq, gte, sql, sum } from "drizzle-orm"; // Build OR conditions for each combination const conditions = combinations.map((combo) => and( - sql`${modelProviderMappingHistory.modelId} = ${combo.modelId}`, - sql`${modelProviderMappingHistory.providerId} = ${combo.providerId}`, + eq(modelProviderMappingHistory.modelId, combo.modelId), + eq(modelProviderMappingHistory.providerId, combo.providerId), ), );
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (6)
apps/gateway/src/chat/chat.ts(15 hunks)apps/gateway/src/chat/tools/create-log-entry.ts(3 hunks)packages/db/src/provider-metrics.ts(1 hunks)packages/db/src/schema.ts(1 hunks)packages/models/src/get-cheapest-from-available-providers.ts(3 hunks)packages/models/src/models.spec.ts(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (4)
packages/db/src/provider-metrics.ts (2)
packages/db/src/db.ts (1)
db(21-25)packages/db/src/schema.ts (1)
modelProviderMappingHistory(692-739)
apps/gateway/src/chat/tools/create-log-entry.ts (1)
packages/models/src/get-cheapest-from-available-providers.ts (1)
RoutingMetadata(23-34)
apps/gateway/src/chat/chat.ts (2)
packages/models/src/get-cheapest-from-available-providers.ts (2)
RoutingMetadata(23-34)getCheapestFromAvailableProviders(50-180)packages/db/src/provider-metrics.ts (1)
getProviderMetricsForCombinations(81-145)
packages/models/src/get-cheapest-from-available-providers.ts (3)
packages/models/src/types.ts (2)
AvailableModelProvider(265-268)ModelWithPricing(253-262)packages/db/src/provider-metrics.ts (1)
ProviderMetrics(6-12)packages/models/src/models.ts (1)
ProviderModelMapping(25-122)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (10)
- GitHub Check: e2e-shards (3)
- GitHub Check: e2e-shards (5)
- GitHub Check: e2e-shards (4)
- GitHub Check: e2e-shards (2)
- GitHub Check: e2e-shards (1)
- GitHub Check: test / run
- GitHub Check: generate / run
- GitHub Check: lint / run
- GitHub Check: build / run
- GitHub Check: autofix
🔇 Additional comments (8)
packages/models/src/models.spec.ts (1)
530-537: LGTM! Test assertions correctly updated for new return type.The test assertions now correctly access
cheapestProvider?.provider.providerIdto match the newProviderSelectionResult<T>return type fromgetCheapestFromAvailableProviders.apps/gateway/src/chat/tools/create-log-entry.ts (1)
4-4: LGTM! Routing metadata properly threaded through log entries.The
routingMetadataparameter is cleanly added and propagated through the log entry creation flow, enabling downstream tracking of provider selection decisions.Also applies to: 32-32, 65-65
packages/db/src/schema.ts (1)
411-422: LGTM! Schema properly extended with routing metadata.The
routingMetadatafield correctly mirrors theRoutingMetadatainterface structure, enabling persistence of provider selection decisions and scores.packages/db/src/provider-metrics.ts (1)
25-71: LGTM! Metrics calculation logic is sound.Both functions correctly:
- Aggregate logs, errors, and duration over the time window
- Calculate uptime as
(successful requests / total requests) * 100- Calculate average latency as
totalDuration / totalRequests- Skip entries with zero logs to avoid division by zero
The use of Map with
"modelId:providerId"keys enables efficient lookups in the routing logic.Also applies to: 81-145
apps/gateway/src/chat/chat.ts (1)
21-21: LGTM! Metrics integration is clean and consistent.The routing logic correctly:
- Fetches metrics for relevant provider combinations before selection
- Passes the
metricsMaptogetCheapestFromAvailableProviders- Extracts routing metadata from the result
- Propagates metadata through log entries
The pattern is consistently applied in both auto-routing and regular routing paths.
Also applies to: 44-44, 689-689, 1024-1043, 1131-1150
packages/models/src/get-cheapest-from-available-providers.ts (3)
57-82: Good practice: filtering unstable and experimental providers.The stability filtering at the beginning of the function ensures that only production-ready providers are considered for routing, reducing the risk of routing to unreliable endpoints.
129-160: Normalization and scoring logic is correctly implemented.The normalized scoring correctly:
- Maps each metric to 0-1 range where lower is better
- Handles edge cases where all providers have the same metric value (range = 0)
- Applies weighted combination with lower scores preferred
- Selects the provider with the minimum composite score
The logic is sound and will properly balance price, uptime, and latency (once the weights are corrected).
182-231: Clean fallback preserves backward compatibility.The
selectByPriceOnlyhelper maintains the original price-based selection behavior when metrics are unavailable, while still returning the newProviderSelectionResultstructure with routing metadata. The placeholderscore: 0is acceptable for the price-only case.
…-latency-metrics-011CV267Fqts91qog59vi3sa
…ction Added a mechanism to randomly select a provider 1% of the time to ensure all providers receive periodic traffic and build up metrics.
…blem - Add 1% random exploration rate to ensure all providers get traffic - Disable exploration during tests to maintain deterministic behavior - Fix price precision display in routing metadata 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
packages/models/src/get-cheapest-from-available-providers.ts (2)
87-107: Epsilon‑greedy exploration behavior is global; consider making rate configurableThe 1% exploration happens for all non‑test environments, regardless of whether metrics are present, and returns
"random-exploration"with emptyproviderScores. That matches the stated goal of spreading traffic but does introduce non‑determinism in prod.If you foresee needing finer control (e.g., temporarily disabling exploration or tuning the rate per environment), consider driving
EXPLORATION_RATEfrom an env var (with 0.01 as the default) and optionally populatingproviderScoreseven on exploration to keep metadata shape consistent.
206-251: Handling of providers without explicit pricing could be tightened (currently treated as free)In
selectByPriceOnly, if aProviderModelMappingis missing or has noinputPrice/outputPrice,totalPricebecomes0, making that provider always “cheapest”. This mirrors the previous behavior but is semantically odd for an unpriced provider.If this is not intentional, you could:
- Skip providers with no pricing, or
- Treat missing prices as
Infinity(or a large sentinel) rather than0.For example:
- const totalPrice = - (((providerInfo?.inputPrice || 0) + (providerInfo?.outputPrice || 0)) / 2) * - discountMultiplier; + if (!providerInfo || (providerInfo.inputPrice == null && providerInfo.outputPrice == null)) { + continue; // or set totalPrice = Number.POSITIVE_INFINITY + } + + const totalPrice = + (((providerInfo.inputPrice || 0) + (providerInfo.outputPrice || 0)) / 2) * + discountMultiplier;This would keep the routing logic aligned with real pricing data and avoid unintentionally favoring unpriced mappings.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
packages/db/migrations/1763458310_dark_silver_fox.sql(1 hunks)packages/db/migrations/meta/_journal.json(1 hunks)packages/models/src/get-cheapest-from-available-providers.ts(3 hunks)packages/models/src/models.spec.ts(2 hunks)
✅ Files skipped from review due to trivial changes (1)
- packages/db/migrations/1763458310_dark_silver_fox.sql
🚧 Files skipped from review as they are similar to previous changes (1)
- packages/models/src/models.spec.ts
🧰 Additional context used
🧬 Code graph analysis (1)
packages/models/src/get-cheapest-from-available-providers.ts (3)
packages/models/src/types.ts (2)
AvailableModelProvider(265-268)ModelWithPricing(253-262)packages/db/src/provider-metrics.ts (1)
ProviderMetrics(6-12)packages/models/src/models.ts (1)
ProviderModelMapping(25-122)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (10)
- GitHub Check: e2e-shards (5)
- GitHub Check: build / run
- GitHub Check: test / run
- GitHub Check: generate / run
- GitHub Check: lint / run
- GitHub Check: e2e-shards (3)
- GitHub Check: e2e-shards (4)
- GitHub Check: e2e-shards (2)
- GitHub Check: e2e-shards (1)
- GitHub Check: autofix
🔇 Additional comments (3)
packages/models/src/get-cheapest-from-available-providers.ts (2)
3-42: Types and routing metadata shape look consistent with the metrics/logging design
ProviderScore,RoutingMetadata, andProviderSelectionResultalign cleanly withProviderMetricsfrom@llmgateway/dband the describedrouting_metadatalog schema (providerId, score, uptime, latency, price). Shapes are coherent and should serialize cleanly downstream.
110-201: Metrics scoring and normalization logic looks sound and degrades to price‑only when metrics are missingThe scoring path:
- Computes per‑provider price (including discount) once and builds
providerScores.- Uses min/max normalization for price, uptime, and latency, with sane handling when ranges collapse to 0.
- Correctly treats lower price, higher uptime, and lower latency as better by flipping uptime and latency in the normalized scores.
- Uses the weights (0.2 price / 0.5 uptime / 0.3 latency) consistently in the final composite score, and picks the min‑score provider.
When
metricsMaphas entries but none match the current providers, all providers end up with the same default uptime/latency, so uptime/latency terms cancel out and the choice reduces effectively to price ranking, which is exactly the intended fallback behavior.Overall this path looks correct and robust for edge cases (single provider, partially missing metrics, etc.).
packages/db/migrations/meta/_journal.json (1)
454-466: New migration journal entry is well‑formed and orderedThe added entry for idx 65 follows the existing structure (version/tag/when/breakpoints) and preserves the monotonic idx ordering, so it should integrate cleanly with the migration tooling.
Updated the routing documentation to reflect improvements in the smart routing algorithm, including a weighted scoring system and epsilon-greedy exploration to address the cold start problem. Added details on routing metadata for better transparency in provider selection.
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
apps/docs/content/features/routing.mdx (1)
197-204: Update "Coming Soon" section to reflect already-implemented features.Based on code verification:
Already implemented (should be moved to "Current Implementation" section):
- Performance-based routing: Fully implemented in
get-cheapest-from-available-providers.tswith weighted scoring (50% uptime, 30% latency, 20% price) based on historical ProviderMetrics data- Tool call optimization: Auto-routing in
chat.ts(lines 976-987) automatically filters to models supporting tools when tools are specified in the requestGenuinely "Coming Soon" (can remain):
- Content-aware routing (no message content analysis currently)
- Multi-model orchestration (no evidence of combining multiple models)
Move the two implemented features to your "Current Implementation" section and revise the "Coming Soon" section to remove them.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
apps/docs/content/features/routing.mdx(2 hunks)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (9)
- GitHub Check: test / run
- GitHub Check: build / run
- GitHub Check: lint / run
- GitHub Check: e2e-shards (4)
- GitHub Check: e2e-shards (3)
- GitHub Check: e2e-shards (5)
- GitHub Check: e2e-shards (1)
- GitHub Check: e2e-shards (2)
- GitHub Check: autofix
🔇 Additional comments (4)
apps/docs/content/features/routing.mdx (4)
78-100: Provider-Specific Routing updates are clear and well-documented.The clarification about "without any fallbacks," concrete examples, and the note explaining that explicit provider routing bypasses smart routing are all helpful additions that accurately describe the expected behavior.
102-124: Auto routing documentation accurately reflects implemented capabilities—no changes needed.The verification confirms all three "Current Implementation" features are fully implemented in the codebase:
- Cost-effective model selection: Confirmed via active use of
getCheapestFromAvailableProviders()andgetCheapestModelForProvider()functions throughout the routing logic- Automatic scaling to more powerful models: Confirmed via context size estimation (
apps/gateway/src/chat/chat.ts:835–836) and capability-based filtering of suitable providers (apps/gateway/src/chat/chat.ts:951–952)- Large context handling: Confirmed via hardcoded model pool (
gpt-oss-120b,gpt-5-nano,gpt-4.1-nano) with context-aware selection and filter options for free models and reasoning capabilitiesAll "Coming Soon" features are correctly identified as not yet implemented. The auto routing system is production-ready.
218-235: Auto routing recommendation for production is appropriate—verification confirms adequate stability and edge case handling.The codebase demonstrates production readiness:
- Hardcoded allowed models (gpt-oss-120b, gpt-5-nano, gpt-4.1-nano) prevent unpredictable behavior
- Explicit error handling: HTTP 400 responses when constraints are violated (no free models available, no non-reasoning models available)
- Fallback strategy: Defaults to gpt-5-nano/openai when suitable models exhaust
- Parameterized filtering: free_models_only and no_reasoning options properly filter provider capabilities
- Cost optimization: Selects cheapest available provider from filtered pool
- E2E test coverage: api-individual.e2e.ts tests auto routing with reasoning_effort scenarios
The guidance in the Best Practices section correctly positions auto routing for production use with confidence.
31-74: All technical specifications in the documentation are verified as accurate.The weight percentages (50% uptime, 30% latency, 20% price), epsilon-greedy exploration rate (1%), and metrics time window (5 minutes) all match the code implementation exactly. The routing metadata is properly defined in the interface, returned from the routing function, and passed through the request lifecycle for logging.
| **Routing Metadata**: | ||
|
|
||
| Every request includes detailed routing metadata in the logs, showing: | ||
|
|
||
| - Available providers that were considered | ||
| - Selected provider and selection reason | ||
| - Scores for each provider (including uptime, latency, and price) | ||
|
|
||
| This transparency allows you to understand and debug routing decisions. |
There was a problem hiding this comment.
🧩 Analysis chain
Verify routing metadata is actually logged and available to users.
The documentation states that routing metadata (available providers, selected provider and reason, provider scores) is included in request logs. Confirm that this logging is implemented and accessible to users through the standard logging/observability channels.
🏁 Script executed:
#!/bin/bash
# Description: Verify routing metadata is logged
# Search for logging of routing decisions
rg -n 'routing.*metadata|routingMetadata|provider.*score|selected.*provider' --type ts --type js -C 3
# Search for log statements related to provider selection
rg -n 'log.*provider|logger.*routing|console.*routing' --type ts --type js -C 2 | head -20Length of output: 30769
🏁 Script executed:
#!/bin/bash
# Check for logging API endpoints that might expose routing metadata to users
# Search for API endpoints that might return logs with routing metadata
rg -n 'logs|getLogs|getLog|fetchLogs|routingMetadata' --type ts -C 2 \
| grep -E '(route|handler|endpoint|get.*logs|post.*logs)' \
| head -30
# Search for any public API or dashboard features related to logs
fd -e ts -e tsx -e js | xargs grep -l "log.*route\|logs.*api\|getLog" | head -10Length of output: 2213
🏁 Script executed:
#!/bin/bash
# Search for documentation or API documentation about accessing logs
find . -type f \( -name "*.md" -o -name "*.mdx" \) | xargs grep -l "log\|routing.*metadata" 2>/dev/null | head -10Length of output: 395
🏁 Script executed:
#!/bin/bash
# Search for API endpoints or features that expose logs to users
rg -n 'logs|getLog|fetchLog' --type ts -C 2 | grep -E 'export|function|route|handler' | head -40
# Look for log retrieval endpoints in gateway
fd -type f -name "*.ts" | xargs grep -l "logs" --include="*.ts" | grep -E 'gateway|route|api'Length of output: 3007
🏁 Script executed:
#!/bin/bash
# Examine the logs route implementation to see what it returns
cat apps/api/src/routes/logs.ts | head -100Length of output: 2862
🏁 Script executed:
#!/bin/bash
# Check what fields are returned in the logs response
rg -A 30 'export.*logs|function logs' apps/api/src/routes/logs.ts | head -60Length of output: 1029
🏁 Script executed:
#!/bin/bash
# Check the test to see what fields are expected in the response
cat apps/api/src/routes/logs.spec.ts | grep -A 20 "expect(json"Length of output: 3724
🏁 Script executed:
#!/bin/bash
# Get the complete logSchema definition to see all fields
rg -A 100 'const logSchema = z.object' apps/api/src/routes/logs.ts | head -120Length of output: 3315
🏁 Script executed:
#!/bin/bash
# Search explicitly for routingMetadata in logs.ts
rg -n 'routingMetadata' apps/api/src/routes/logs.tsLength of output: 0
🏁 Script executed:
#!/bin/bash
# Get full logs.ts file to see the complete schema and endpoint
wc -l apps/api/src/routes/logs.ts
tail -n 200 apps/api/src/routes/logs.tsLength of output: 4691
routing metadata is not exposed in the logs API response
The documentation claims routing metadata is included in request logs for user visibility, but verification shows the routingMetadata field is stored in the database but not included in the /logs API schema. The logSchema at apps/api/src/routes/logs.ts returns many fields (requestId, usedProvider, cost, customHeaders, source, etc.) but omits routingMetadata entirely, meaning users cannot actually access this data through the logs API to debug routing decisions as documented.
Either add routingMetadata to the logSchema response at apps/api/src/routes/logs.ts lines 28-59, or update the documentation to remove claims about routing metadata transparency.
🤖 Prompt for AI Agents
In apps/api/src/routes/logs.ts around lines 28-59 the logs API response schema
omits the stored routingMetadata field so clients cannot access routing decision
details; add routingMetadata to the logSchema response (and any TypeScript
types/serializers used there) and map it from the DB record into the response
payload—define its shape (e.g., availableProviders array, selectedProvider,
selectionReason, scores object) or reuse an existing interface, and ensure any
validation/serialization middleware includes it; alternatively, if exposing it
is not desired, update apps/docs/content/features/routing.mdx to remove the
claim that routing metadata is available via the /logs API.
2 participants
Enhance provider selection logic to consider uptime and latency from the last 5 minutes in addition to price.
PRICE_WEIGHT = 0.2 (20%)
UPTIME_WEIGHT = 0.5 (50%)
LATENCY_WEIGHT = 0.3 (30%)
Summary by CodeRabbit
New Features
Tests
Documentation