feat(routing): add provider metrics to routing metadata

[steebchen]

Include uptime, latency, and throughput metrics in routing metadata for direct provider selection and low-uptime fallback cases. This provides full visibility into provider scores and performance characteristics in the routing logs.

For direct provider selection (direct-provider-specified, single-provider-available, fallback-first-available), metrics are now fetched and included. For low-uptime fallback, the original requested provider is added to the scores array with a negative score indicating it was skipped due to low uptime.

Summary by CodeRabbit

Release Notes

• Improvements
  • Enhanced provider routing logic to consider performance metrics including uptime, latency, and throughput when selecting between providers.
  • Improved fallback routing to include original provider performance data alongside alternative provider options.
  • Direct provider selection now incorporates real-time performance metrics for more informed routing decisions.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Walkthrough

The PR enhances provider routing logic in the gateway's chat service by adding retrieval and propagation of provider metrics (price, uptime, latency, throughput) across multiple routing paths. It enriches routing metadata with originalProviderScore and providerScores containing metrics for both the original provider and fallback providers in low-uptime scenarios, and extends direct-provider routing to include fetched provider metrics.

Changes

Cohort / File(s)

Summary

Provider metrics enrichment in routing apps/gateway/src/chat/chat.ts

Added retrieval of original provider's price and performance metrics (uptime, latency, throughput); constructed originalProviderScore when original provider is skipped due to low uptime; enriched routing metadata with providerScores containing metrics for original and fallback providers in low-uptime fallback path; fetched direct provider metrics in direct-provider routing path; extended both fallback and direct-routing branches to include uptime/latency/throughput alongside price and score in routing metadata; attached provider metrics to final non-streaming routing path.

Sequence Diagram

sequenceDiagram
    participant Chat as Chat Handler
    participant Metrics as Metrics Service
    participant Router as Router
    participant Metadata as Routing Metadata

    Chat->>Metrics: Fetch original provider metrics<br/>(price, uptime, latency, throughput)
    Metrics-->>Chat: Provider metrics
    
    alt Low Uptime Detected
        Chat->>Metrics: Fetch fallback provider metrics
        Metrics-->>Chat: Fallback metrics
        Chat->>Chat: Create originalProviderScore
        Chat->>Router: Build providerScores array<br/>(original + fallback)
        Chat->>Metadata: Enrich with metrics<br/>(uptime, latency, throughput)
    else Direct Provider Selected
        Chat->>Metrics: Fetch selected provider metrics
        Metrics-->>Chat: Provider metrics
        Chat->>Metadata: Attach provider metrics<br/>to routing data
    end
    
    Chat->>Router: Route request with enriched metadata

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

• Extra attention areas:
  • Verify metric fetching logic handles missing or stale metrics gracefully
  • Confirm originalProviderScore construction is consistent across all conditional branches (fallback and direct-provider paths)
  • Validate that provider metrics are correctly propagated through both streaming and non-streaming paths
  • Review the order of operations when metrics are fetched and when scores are constructed to ensure data consistency

Possibly related PRs

• feat(metrics): add throughput and enhance metric calculations #1223: Adds throughput field to ProviderScore and propagates it through routingMetadata.providerScores alongside uptime/latency.
• feat(gateway): add uptime & latency metrics #1146: Modifies provider-metrics integration and routing in the same file, including metrics passed to provider selection logic.
• feat: route away from low-uptime providers #1230: Implements low-uptime fallback with enriched routing metadata including originalProvider and additional metrics.

Suggested reviewers

• smakosh

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The pull request title accurately and concisely summarizes the main change: adding provider metrics (uptime, latency, throughput) to routing metadata, which aligns with the primary objective and all file changes.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch add-scores-direct-routing

---

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between fbf9b5f and aaaea96.

📒 Files selected for processing (1)

• apps/gateway/src/chat/chat.ts (5 hunks)

🧰 Additional context used

📓 Path-based instructions (4)

**/*.{js,ts,jsx,tsx}

📄 CodeRabbit inference engine (AGENTS.md)

**/*.{js,ts,jsx,tsx}: Always use top-level import, never use require or dynamic imports
No unnecessary code comments

Files:

• apps/gateway/src/chat/chat.ts

**/*.{ts,tsx,js,jsx}

📄 CodeRabbit inference engine (AGENTS.md)

Use cookies for user-settings which are not saved in the database to ensure SSR works

**/*.{ts,tsx,js,jsx}: Always use tabs for indentation
No unnecessary code comments

Files:

• apps/gateway/src/chat/chat.ts

**/*.{ts,tsx}

📄 CodeRabbit inference engine (AGENTS.md)

**/*.{ts,tsx}: Use Drizzle ORM with latest object syntax for database operations
For database reads: Use db().query.<table>.findMany() or db().query.<table>.findFirst()

**/*.{ts,tsx}: Never use any or as any unless absolutely necessary in TypeScript code
Always use top-level import, never use require or dynamic imports
Use cookies for user-settings which are not saved in the database to ensure SSR works
Use Drizzle ORM with latest object syntax for database operations
For database reads: Use db().query.<table>.findMany() or db().query.<table>.findFirst()

Files:

• apps/gateway/src/chat/chat.ts

apps/{gateway,api}/**/*.{ts,tsx}

📄 CodeRabbit inference engine (AGENTS.md)

Use Zod schemas for validation in Hono applications

Files:

• apps/gateway/src/chat/chat.ts

🧬 Code graph analysis (1)

apps/gateway/src/chat/chat.ts (2)

packages/models/src/models.ts (1)

• ModelDefinition (161-221)

packages/db/src/provider-metrics.ts (1)

• getProviderMetricsForCombinations (102-185)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (9)

• GitHub Check: lint / run
• GitHub Check: test / run
• GitHub Check: build / run
• GitHub Check: e2e-shards (3)
• GitHub Check: e2e-shards (5)
• GitHub Check: e2e-shards (4)
• GitHub Check: e2e-shards (1)
• GitHub Check: e2e-shards (2)
• GitHub Check: autofix

🔇 Additional comments (2)

apps/gateway/src/chat/chat.ts (2)

1189-1235: Defensive handling for providerScores when augmenting low-uptime fallback metadata

The low-uptime fallback enrichment looks good and matches the PR intent (negative score entry for the skipped provider plus metrics). One edge case worth guarding:

• cheapestResult.metadata.providerScores is assumed to be always defined and iterable. If in any scenario getCheapestFromAvailableProviders returns metadata without a providerScores array (e.g., older callers, partial construction, or future refactors), the spread ...cheapestResult.metadata.providerScores will throw at runtime.

A small defensive refactor keeps behavior the same when providerScores is present but avoids a hard failure if it’s absent:

-						if (cheapestResult) {
+						if (cheapestResult) {
+							const baseScores =
+								cheapestResult.metadata.providerScores ?? [];
 							usedProvider = cheapestResult.provider.providerId;
 							usedModel = cheapestResult.provider.modelName;
 							routingMetadata = {
 								...cheapestResult.metadata,
 								selectionReason: "low-uptime-fallback",
 								originalProvider: requestedProvider,
 								originalProviderUptime: metrics.uptime,
-								// Add the original provider's score to the scores array
-								providerScores: [
-									originalProviderScore,
-									...cheapestResult.metadata.providerScores,
-								],
+								// Add the original provider's score to the scores array
+								providerScores: [originalProviderScore, ...baseScores],
 							};

(Optional) In the else branch you currently only emit providerScores: [originalProviderScore]. If you later have price/metrics for the chosen fallback provider even when cheapestResult is null, including a second score entry there as well would keep routing metadata structurally consistent across low-uptime paths.

[ suggest_recommended_refactor ]
[ request_verification ]

---

1347-1374: Base model ID used for direct-provider metrics may not match final routed model in rare fallback cases

Fetching metrics for direct provider selection and attaching them to providerScores is aligned with the PR goals and the implementation is generally sound (guarded for custom providers and providerMetrics is optional).

One nuance to be aware of:

• const baseModelId = (modelInfo as ModelDefinition).id; is used for the metrics lookup, but modelInfo is the earlier model definition (e.g., "auto"), not necessarily the final usedModel. In the rare default fallback path where auto-routing ends up hardcoding usedModel = "gpt-5-nano" / usedProvider = "openai" without setting routingMetadata, this block will still run and request metrics for (modelId: "auto", providerId: "openai"). If your modelProviderMappingHistory.modelId entries are keyed by the concrete base model (e.g., "gpt-5-nano"), this combination might never return metrics, so the new fields will always be undefined in that scenario.

If you rely on these metrics operationally, consider aligning the lookup key with the final selected model ID (e.g., using the same base model name you eventually log in metadata/usage), or explicitly documenting that in the “hardcoded default” path metrics may be unavailable.

[ suggest_optional_refactor ]
[ request_verification ]

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}