fix(gateway): respect RPM limits in low-uptime fallback by steebchen · Pull Request #2042 · theopenco/llmgateway

steebchen · 2026-04-20T17:27:40Z

Summary

The low-uptime fallback path in apps/gateway/src/chat/chat.ts selected an alternative provider based only on uptime and price, ignoring per-provider RPM/RPD caps. A request that was rerouted away from a low-uptime provider could land on another provider that was already at its rate limit, causing an avoidable 429 at checkProviderRateLimit right after.
Filter availableModelProviders through filterRateLimitedProviders before scoring, with fail-open if every alternative is capped. This mirrors the existing rate-limit fallback path (same file, lines ~2056–2071), so the two fallback paths now share the same RPM-aware candidate selection.

Test plan

pnpm build — passed locally
pnpm lint — passed locally
Manually exercise: requested provider below 90% uptime, best-priced alternative at RPM cap → request routes to the next-cheapest non-capped alternative, not the capped one
Manually exercise: all alternatives are capped → fails open and still attempts fallback instead of getting stuck on the low-uptime provider

🤖 Generated with Claude Code

Summary by CodeRabbit

Bug Fixes
- Improved fallback provider selection to prefer non-rate-limited candidates when available, ensuring uptime comparisons, scoring, and routing use healthier alternatives while still allowing fail-open behavior when all candidates are capped.
Tests
- Added coverage for selecting non-rate-limited candidates, deduplication of rate-limit checks, empty-input handling, and fail-open scenarios.

chatgpt-codex-connector · 2026-04-20T17:27:45Z

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

coderabbitai · 2026-04-20T17:28:00Z

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 4ba90610-45e0-445f-9a97-635735ff3925

📥 Commits

Reviewing files that changed from the base of the PR and between 8002624 and 7b09a51.

📒 Files selected for processing (3)

apps/gateway/src/chat/chat.ts
apps/gateway/src/lib/provider-rate-limit.spec.ts
apps/gateway/src/lib/provider-rate-limit.ts

Walkthrough

Adds exported helper pickNonRateLimitedCandidates; uses it in chat routing (requested-provider reroute and low-uptime fallback) to filter out rate-limited provider/model candidates (with fail-open), and adds tests covering filtering, fail-open, deduping, and empty-input behavior.

Changes

Rate-Limit Candidate Selection

Layer / File(s)	Summary
Helper implementation `apps/gateway/src/lib/provider-rate-limit.ts`	Adds exported `pickNonRateLimitedCandidates` that deduplicates provider+model peeks, calls `filterRateLimitedProviders`, filters candidates, and returns original candidates if filtering yields none (fail-open).
Chat import & requested-provider reroute `apps/gateway/src/chat/chat.ts`	Imports `pickNonRateLimitedCandidates` and replaces inline `filterRateLimitedProviders` + manual fail-open logic in the "requested provider is rate-limited" reroute with a call to the helper.
Low-uptime fallback scoring `apps/gateway/src/chat/chat.ts`	Compute `uptimeFallbackCandidates` via the helper, gate fallback scoring on its length, and use it for metrics combination lookups and `collapseProvidersToBestRegionPerProvider`.
Tests `apps/gateway/src/lib/provider-rate-limit.spec.ts`	Adds tests for `pickNonRateLimitedCandidates`: dropping rate-limited candidates, failing open when all are limited, deduping region-expanded variants (single DB/Redis peek), and early return on empty candidate lists; updates imports.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

theopenco/llmgateway#2033: Modifies the same low-uptime fallback routing logic in apps/gateway/src/chat/chat.ts to update usedRegion handling in the fallback path.
theopenco/llmgateway#2042: Also modifies low-uptime fallback candidate selection in apps/gateway/src/chat/chat.ts; similar intent to prefer non–rate-limited alternatives before uptime/price scoring.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and specifically describes the main change: applying RPM rate-limit filtering to the low-uptime fallback routing logic. It directly matches the PR's core objective of respecting rate limits during fallback selection.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix-uptime-fallback-rpm

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Copilot

Pull request overview

Aligns the gateway’s low-uptime fallback routing with the existing rate-limit fallback routing by avoiding (when possible) providers already at their configured RPM/RPD caps, reducing avoidable immediate 429s after reroute.

Changes:

Filters low-uptime fallback alternatives via filterRateLimitedProviders, with fail-open behavior when all alternatives are capped.
Routes scoring/metrics collection using the resulting uptimeFallbackCandidates instead of the full availableModelProviders.
Keeps behavior consistent with the pre-existing rate-limit fallback candidate filtering logic in the same file.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-20T17:31:41Z

+				const rateLimitedAlternatives = await filterRateLimitedProviders(
+					project.organizationId,
+					availableModelProviders.map((p) => ({
+						providerId: p.providerId,
+						model: baseModelId,
+						providerModelName: p.modelName,
+					})),


filterRateLimitedProviders is called once per entry in availableModelProviders, which likely includes multiple regions per provider (it’s later collapsed via collapseProvidersToBestRegionPerProvider). This can trigger redundant rate-limit peeks (DB + Redis) for the same provider/model, increasing latency in the low-uptime fallback path. Consider de-duplicating the candidates (e.g., by providerId + providerModelName) before calling filterRateLimitedProviders, while still filtering the full availableModelProviders list using the resulting Set.

Suggested change

const rateLimitedAlternatives = await filterRateLimitedProviders(

project.organizationId,

availableModelProviders.map((p) => ({

providerId: p.providerId,

model: baseModelId,

providerModelName: p.modelName,

})),

const uniqueRateLimitCandidates = Array.from(

new Map(

availableModelProviders.map((p) => [

`${p.providerId}:${p.modelName}`,

{

providerId: p.providerId,

model: baseModelId,

providerModelName: p.modelName,

},

]),

).values(),

);

const rateLimitedAlternatives = await filterRateLimitedProviders(

project.organizationId,

uniqueRateLimitCandidates,

Copilot · 2026-04-20T17:31:41Z

+				// Exclude alternatives that are already at their RPM/RPD cap so the
+				// low-uptime fallback doesn't route into a rate-limited provider.
+				// Fail-open: if all alternatives are rate-limited, keep them all.
+				const rateLimitedAlternatives = await filterRateLimitedProviders(
+					project.organizationId,
+					availableModelProviders.map((p) => ({
+						providerId: p.providerId,
+						model: baseModelId,
+						providerModelName: p.modelName,
+					})),
+				);
+				const nonRateLimitedAlternatives = availableModelProviders.filter(
+					(p) => !rateLimitedAlternatives.has(p.providerId),
+				);
+				const uptimeFallbackCandidates =
+					nonRateLimitedAlternatives.length > 0
+						? nonRateLimitedAlternatives
+						: availableModelProviders;


This change introduces new routing behavior (skipping rate-limited alternatives during low-uptime fallback) but there doesn’t appear to be an integration test covering the scenario where the cheapest/healthiest alternative is at its provider RPM/RPD cap and the router should pick the next-best candidate (and also the fail-open case when all alternatives are capped). Please add a regression test (likely alongside existing low-uptime fallback tests in apps/gateway/src/fallback.spec.ts) to prevent this from regressing.

Copilot · 2026-04-20T17:31:41Z

+				// Exclude alternatives that are already at their RPM/RPD cap so the
+				// low-uptime fallback doesn't route into a rate-limited provider.
+				// Fail-open: if all alternatives are rate-limited, keep them all.
+				const rateLimitedAlternatives = await filterRateLimitedProviders(
+					project.organizationId,
+					availableModelProviders.map((p) => ({
+						providerId: p.providerId,
+						model: baseModelId,
+						providerModelName: p.modelName,
+					})),
+				);
+				const nonRateLimitedAlternatives = availableModelProviders.filter(
+					(p) => !rateLimitedAlternatives.has(p.providerId),
+				);
+				const uptimeFallbackCandidates =
+					nonRateLimitedAlternatives.length > 0
+						? nonRateLimitedAlternatives
+						: availableModelProviders;


This rate-limit-aware candidate filtering duplicates the earlier rate-limit fallback block (same file around the rate-limit-fallback routing). Consider extracting a small helper for “pick fallback candidates with fail-open rate-limit filtering” to avoid the two paths drifting over time (and to centralize any future tweaks to the filtering rules).

The low-uptime fallback path selected an alternative provider purely on uptime and price, ignoring per-provider RPM/RPD caps. Filter out rate-limited alternatives (fail-open if all are capped), matching the rate-limit fallback behavior. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

availableModelProviders comes from the region-expanded list, so multiple variants of the same provider+model triggered redundant peekProviderRateLimit calls (Redis hit per region). Dedupe by providerId+modelName before calling filterRateLimitedProviders. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Both rate-limit and low-uptime fallback paths reused the same dedupe → peek → filter → fail-open dance. Pull it into a single helper next to filterRateLimitedProviders so future tweaks to the filtering rules apply to both paths at once. Add unit tests covering the next-best, fail-open, and region-dedupe cases. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings April 20, 2026 17:27

Copilot AI reviewed Apr 20, 2026

View reviewed changes

steebchen and others added 2 commits May 7, 2026 20:55

steebchen force-pushed the fix-uptime-fallback-rpm branch from b4e6f99 to 8002624 Compare May 7, 2026 13:59

steebchen merged commit 57884db into main May 7, 2026

steebchen deleted the fix-uptime-fallback-rpm branch May 7, 2026 14:27

steebchen added a commit that referenced this pull request May 7, 2026

fix(gateway): respect RPM limits in low-uptime fallback (#2042)

f0d3ca0

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>

This was referenced May 7, 2026

feat: global daily aggregation + admin stats page #2195

Merged

fix: prefer provider keys in hybrid mode #1963

Merged

This was referenced May 15, 2026

refactor: drop virtual model IDs and routing logic #2199

Merged

feat: per-project enterprise routing overrides #2363

Merged

This was referenced May 25, 2026

fix: use root model id for rate limits and UI lookups #2404

Merged

feat(gateway): sticky session routing via x-session-id #2445

Merged

coderabbitai Bot mentioned this pull request Jun 8, 2026

fix: make sticky session routing cost-aware #2596

Merged

Navigation Menu

fix(gateway): respect RPM limits in low-uptime fallback - #2042

fix(gateway): respect RPM limits in low-uptime fallback#2042
steebchen merged 3 commits into
mainfrom
fix-uptime-fallback-rpm

steebchen commented Apr 20, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

chatgpt-codex-connector Bot commented Apr 20, 2026

Uh oh!

coderabbitai Bot commented Apr 20, 2026 •

edited

Loading

Review failed

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 20, 2026

Uh oh!

Copilot AI Apr 20, 2026

Uh oh!

Copilot AI Apr 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

steebchen commented Apr 20, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Summary by CodeRabbit

Uh oh!

chatgpt-codex-connector Bot commented Apr 20, 2026

Uh oh!

coderabbitai Bot commented Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review failed

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

steebchen commented Apr 20, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Apr 20, 2026 •

edited

Loading