Skip to content

refactor(gateway): centralize service tier checks #3467

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
steebchen merged 2 commits into from
Aug 7, 2026
Merged

refactor(gateway): centralize service tier checks #3467

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a70ed90
fix(gateway): enforce service tier across retries
steebchen Aug 7, 2026
060590a
Merge branch 'main' into service-tier-fallback-enforcement
steebchen Aug 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions apps/docs/content/features/service-tiers.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,38 @@ the exact tiers exposed by each provider card.
default) are always eligible.
</Callout>

## Retries and fallback never downgrade the tier

A request can change provider or credential mid-flight: LLM Gateway falls back
to another provider when one fails, and rotates to another key for the same
provider when a credential returns a 429 or an auth error. A requested tier is
carried through all of it.

- Provider routing is narrowed to mappings that support the requested tier
**before** a provider is picked, so no fallback candidate can be one that
would serve the request as standard.
- Key selection — BYOK keys, platform-managed credentials, and env credentials
alike — skips any credential that cannot carry the tier (a proxy base URL, or
a Vertex credential pinned to a regional endpoint), on the first attempt and
on every retry.
- Every attempt re-resolves the tier against the provider, region and
credential it actually resolved to, and fails rather than sending at a lower
tier. If no eligible candidate is left, you get the upstream error instead of
a silently downgraded response.

This applies to a tier you requested yourself. The optional coding-plan default
tier is a cost preference rather than a requirement, so a request that cannot be
served at that tier runs at standard instead of failing; `used_service_tier` in
the response metadata always reports what was actually served.

<Callout type="info">
Rotating to another key for the same provider is a separate upstream account
as far as prompt caching is concerned, so a retried request re-writes its
cached prefix rather than reading the original one. Send `x-no-fallback: true`
to have the original upstream error returned to you — note that this disables
cross-provider fallback, not key rotation within a provider.
</Callout>

## Pricing uses multipliers

Service tiers do not define separate model prices in LLM Gateway. They multiply
Expand Down
185 changes: 185 additions & 0 deletions apps/gateway/src/api.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import {
resetKeyHealth,
} from "./lib/api-key-health.js";
import { createGatewayApiTestHarness } from "./test-utils/gateway-api-test-harness.js";
import { resetFailOnceCounter } from "./test-utils/mock-openai-server.js";
import {
readAll,
waitForLogByRequestId,
Expand Down Expand Up @@ -2358,6 +2359,188 @@ describe("api", () => {
expect(logs[0].usedServiceTier).toBe("priority");
});

test("/v1/chat/completions keeps the service tier when a retry rotates keys", async () => {
// The customer-visible failure this guards against: an upstream 429 on
// one key rotates the request onto another key for the same provider, and
// the second attempt is served (and billed) at the standard tier because
// the requested tier was rebuilt from the fallback context.
await db.insert(tables.apiKey).values({
id: "token-id-tier-key-rotation",
token: "real-token-tier-key-rotation",
projectId: "project-id",
description: "Test API Key",
createdBy: "user-id",
});

await db.insert(tables.providerKey).values([
{
id: "provider-key-tier-rotation-primary",
token: "sk-primary-key",
provider: "openai",
organizationId: "org-id",
baseUrl: mockServerUrl,
},
{
id: "provider-key-tier-rotation-secondary",
token: "sk-secondary-key",
provider: "openai",
organizationId: "org-id",
baseUrl: mockServerUrl,
},
]);

resetFailOnceCounter();

const res = await app.request("/v1/chat/completions", {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: "Bearer real-token-tier-key-rotation",
},
body: JSON.stringify({
model: "openai/gpt-5.6-sol",
service_tier: "flex",
messages: [{ role: "user", content: "TRIGGER_FAIL_ONCE hello" }],
}),
});

expect(res.status).toBe(200);
const json = await res.json();
// Two attempts on the same provider with two different credentials.
expect(json.metadata.routing).toHaveLength(2);
expect(json.metadata.routing[0]).toMatchObject({
provider: "openai",
succeeded: false,
});
expect(json.metadata.routing[1]).toMatchObject({
provider: "openai",
succeeded: true,
});
expect(json.metadata.routing[0].apiKeyHash).not.toBe(
json.metadata.routing[1].apiKeyHash,
);
// The tier survived the rotation: the mock echoes back the tier it was
// sent, so a dropped `service_tier` would surface as a standard-tier
// response and a null usedServiceTier on the log.
expect(json.service_tier).toBe("flex");
expect(json.metadata?.used_service_tier).toBe("flex");

const logs = await waitForLogs(2);
const successLog = logs.find((log) => !log.hasError);
expect(successLog?.requestedServiceTier).toBe("flex");
expect(successLog?.usedServiceTier).toBe("flex");
});

test("/v1/chat/completions never routes a tier request to a provider without it", async () => {
// gpt-5.6-sol is served by openai (flex/priority), azure and aws-mantle.
// A flex request must stay on openai for every attempt — falling back to a
// provider with no premium tier would serve, and bill, standard silently.
await db.insert(tables.apiKey).values({
id: "token-id-tier-no-downgrade-fallback",
token: "real-token-tier-no-downgrade-fallback",
projectId: "project-id",
description: "Test API Key",
createdBy: "user-id",
});

await db.insert(tables.providerKey).values([
{
id: "provider-key-tier-fallback-openai",
token: "sk-openai-test-key",
provider: "openai",
organizationId: "org-id",
baseUrl: mockServerUrl,
},
{
id: "provider-key-tier-fallback-azure",
token: "azure-test-key",
provider: "azure",
organizationId: "org-id",
baseUrl: mockServerUrl,
},
]);

resetFailOnceCounter();

const res = await app.request("/v1/chat/completions", {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: "Bearer real-token-tier-no-downgrade-fallback",
},
body: JSON.stringify({
// No provider prefix: routing is free to pick any mapping.
model: "gpt-5.6-sol",
service_tier: "flex",
messages: [{ role: "user", content: "TRIGGER_ERROR" }],
}),
});

// openai is the only flex-capable mapping, so the upstream failure is
// returned instead of being retried on azure at the standard tier.
expect(res.status).not.toBe(200);

const logs = await waitForLogs(1);
expect(logs.length).toBeGreaterThanOrEqual(1);
for (const log of logs) {
expect(log.usedProvider).toBe("openai");
expect(log.requestedServiceTier).toBe("flex");
expect(log.usedServiceTier).toBeNull();
}
Comment on lines +2479 to +2489

@coderabbitai coderabbitai Bot Aug 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Assert the exact status code.

expect(res.status).not.toBe(200) passes for any failure, including a setup error unrelated to service-tier routing. Assert the specific upstream status that the TRIGGER_ERROR path returns, so the test proves the request stayed on openai and returned the upstream error rather than falling back.

💚 Proposed assertion tightening 
-		expect(res.status).not.toBe(200);
+		expect(res.status).toBe(500);
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
// openai is the only flex-capable mapping, so the upstream failure is
// returned instead of being retried on azure at the standard tier.
expect(res.status).not.toBe(200);
const logs = await waitForLogs(1);
expect(logs.length).toBeGreaterThanOrEqual(1);
for (const log of logs) {
expect(log.usedProvider).toBe("openai");
expect(log.requestedServiceTier).toBe("flex");
expect(log.usedServiceTier).toBeNull();
}
// openai is the only flex-capable mapping, so the upstream failure is
// returned instead of being retried on azure at the standard tier.
expect(res.status).toBe(500);
const logs = await waitForLogs(1);
expect(logs.length).toBeGreaterThanOrEqual(1);
for (const log of logs) {
expect(log.usedProvider).toBe("openai");
expect(log.requestedServiceTier).toBe("flex");
expect(log.usedServiceTier).toBeNull();
}
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/gateway/src/api.spec.ts` around lines 2414 - 2424, Update the response
assertion in the flex-tier routing test to expect the exact status code returned
by the TRIGGER_ERROR upstream path, replacing the broad not-200 check. Keep the
existing waitForLogs and provider/service-tier assertions unchanged.

});

test("/v1/chat/completions still falls back across providers without a tier", async () => {
// The control for the test above: the same failure without service_tier
// does reach azure, so the tier — not some unrelated routing constraint —
// is what keeps the request on openai.
await db.insert(tables.apiKey).values({
id: "token-id-tier-fallback-control",
token: "real-token-tier-fallback-control",
projectId: "project-id",
description: "Test API Key",
createdBy: "user-id",
});

await db.insert(tables.providerKey).values([
{
id: "provider-key-tier-control-openai",
token: "sk-openai-test-key",
provider: "openai",
organizationId: "org-id",
baseUrl: mockServerUrl,
},
{
id: "provider-key-tier-control-azure",
token: "azure-test-key",
provider: "azure",
organizationId: "org-id",
baseUrl: mockServerUrl,
},
]);

resetFailOnceCounter();

const res = await app.request("/v1/chat/completions", {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: "Bearer real-token-tier-fallback-control",
},
body: JSON.stringify({
model: "gpt-5.6-sol",
messages: [{ role: "user", content: "TRIGGER_FAIL_ONCE hello" }],
}),
});

expect(res.status).toBe(200);
const json = await res.json();
expect(
json.metadata.routing.map(
(attempt: { provider: string }) => attempt.provider,
),
).toContain("azure");
});

test("/v1/chat/completions forwards generated request id upstream", async () => {
await db.insert(tables.apiKey).values({
id: "token-id-generated-request-id",
Expand Down Expand Up @@ -7735,6 +7918,8 @@ describe("api", () => {
},
]);

resetFailOnceCounter();

const res = await app.request("/v1/chat/completions", {
method: "POST",
headers: {
Expand Down
Loading
Loading