Conversation
Pin a logical session to a single upstream provider so provider-side prompt caches stay warm across a multi-turn conversation, falling back only when the pinned provider leaves the available list. - Resolve a sticky session key from the x-session-id header, then the OpenAI-native prompt_cache_key, then user (added to the chat schema). - For the Anthropic /messages route, derive the key from metadata.user_id (Claude Code embeds the session id there) and forward it to chat/completions as x-session-id. - Select the provider/region deterministically via rendezvous hashing when a session id is present, bypassing price/uptime scoring. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughDerive session id (header/body/Anthropic metadata), thread it through routing and provider-selection calls, deterministically pin sessions via rendezvous hashing in provider selection, add DB/log observability and UI filters, and document Sessions and Sticky Session Routing. ChangesSticky Session Routing Implementation
Sequence Diagram(s)sequenceDiagram
participant Client
participant Gateway_ChatHandler
participant ProviderSelectionService
participant ProviderRegion
Client->>Gateway_ChatHandler: POST /completions (x-session-id or prompt_cache_key/user)
Gateway_ChatHandler->>ProviderSelectionService: getCheapestFromAvailableProviders(options {sessionId})
ProviderSelectionService->>ProviderRegion: selectStickyProvider(sessionId) / rendezvous-hash
ProviderRegion-->>ProviderSelectionService: return provider mapping & metrics
ProviderSelectionService-->>Gateway_ChatHandler: selection (selectionReason: "session-sticky")
Gateway_ChatHandler->>ProviderRegion: Forward request (with x-session-id header)
ProviderRegion-->>Gateway_ChatHandler: response
Gateway_ChatHandler-->>Client: response
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
Adds sticky session routing to gateway provider selection so chat/Anthropic requests can pin a logical session to a deterministic upstream provider and improve prompt-cache locality.
Changes:
- Adds
sessionIdrouting support using rendezvous hashing in provider selection. - Resolves session IDs from
x-session-id,prompt_cache_key,user, and Anthropicmetadata.user_id. - Adds unit tests for sticky routing determinism and Anthropic session extraction.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
packages/actions/src/get-cheapest-from-available-providers.ts |
Adds sticky session provider selection and routing metadata. |
packages/actions/src/models.spec.ts |
Adds sticky routing behavior tests. |
apps/gateway/src/lib/session-id.ts |
Adds Anthropic session ID extraction helper. |
apps/gateway/src/lib/session-id.spec.ts |
Tests Anthropic session ID extraction behavior. |
apps/gateway/src/chat/schemas/completions.ts |
Adds OpenAI user field to chat request schema. |
apps/gateway/src/chat/chat.ts |
Resolves and forwards sticky session IDs into provider selection. |
apps/gateway/src/anthropic/anthropic.ts |
Extracts Anthropic session metadata and forwards it to chat completions. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| }), | ||
| metadata: z | ||
| .object({ | ||
| user_id: z.string().optional(), |
| return { | ||
| providerId: provider.providerId, | ||
| region: provider.region, | ||
| score: 0, |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
packages/actions/src/models.spec.ts (1)
577-612: ⚡ Quick winAdd the complementary re-pinning case.
This covers stability when an unrelated provider is removed, but the core documented behavior — a session moves to a new provider only when its pinned provider leaves the list — isn't asserted. Consider dropping the pinned provider and verifying the session deterministically re-pins to a remaining one (and that two different sessions don't all collapse onto the same provider).
Want me to draft this additional test case?
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/actions/src/models.spec.ts` around lines 577 - 612, Add a complementary test that verifies re-pinning when the pinned provider is removed: using modelWithMultipleProviders and getCheapestFromAvailableProviders, create a sessionId, compute the initial pinned provider, then remove that pinned provider from the availableProviders list and call getCheapestFromAvailableProviders again to assert the session now deterministically pins to a different provider; also create a second distinct sessionId and verify it does not collapse onto the same provider as the first (i.e., two different sessions should not both re-pin to the same single provider unless deterministic hashing dictates different results).packages/actions/src/get-cheapest-from-available-providers.ts (1)
415-446: ⚡ Quick winExtract the duplicated
providerScoresbuilder.This mapping is identical to the random-exploration block (Lines 463-494). Pull it into one helper so both early-exit paths stay in sync.
♻️ Proposed extraction
Add a shared helper (e.g. near the other private helpers):
function buildZeroScoreProviderScores<T extends AvailableModelProvider>( stableProviders: T[], modelWithPricing: ModelWithPricing & { id: string }, metricsMap: Map<string, ProviderMetrics> | undefined, videoPricing: VideoPricingContext | undefined, ): RoutingMetadata["providerScores"] { return stableProviders.map((provider) => { const providerInfo = findProviderMapping( modelWithPricing.providers, provider, ); const priority = getProviderDefinition(provider.providerId)?.priority ?? 1; const metrics = metricsMap?.get( metricsKey(modelWithPricing.id, provider.providerId, provider.region), ); return { providerId: provider.providerId, region: provider.region, score: 0, uptime: metrics?.uptime, latency: metrics?.averageLatency, throughput: metrics?.throughput, price: getProviderSelectionPrice(providerInfo, videoPricing).toNumber(), priority, cacheSupported: providerSupportsCaching( providerInfo as ProviderModelMapping | undefined, ), }; }); }Then both branches collapse to:
- providerScores: stableProviders.map((provider) => { - const providerInfo = findProviderMapping( - modelWithPricing.providers, - provider, - ); - const providerDef = getProviderDefinition(provider.providerId); - const priority = providerDef?.priority ?? 1; - const metrics = metricsMap?.get( - metricsKey( - modelWithPricing.id, - provider.providerId, - provider.region, - ), - ); - - return { - providerId: provider.providerId, - region: provider.region, - score: 0, - uptime: metrics?.uptime, - latency: metrics?.averageLatency, - throughput: metrics?.throughput, - price: getProviderSelectionPrice( - providerInfo, - videoPricing, - ).toNumber(), - priority, - cacheSupported: providerSupportsCaching( - providerInfo as ProviderModelMapping | undefined, - ), - }; - }), + providerScores: buildZeroScoreProviderScores( + stableProviders, + modelWithPricing, + metricsMap, + videoPricing, + ),🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/actions/src/get-cheapest-from-available-providers.ts` around lines 415 - 446, Extract the duplicated providerScores mapping into a single helper (e.g. buildZeroScoreProviderScores) and call it from both early-exit branches so they stay in sync. The helper should accept stableProviders, modelWithPricing (with id), metricsMap, and videoPricing and return RoutingMetadata["providerScores"]; inside it reuse findProviderMapping, getProviderDefinition (for priority), metricsKey, getProviderSelectionPrice (call .toNumber()), and providerSupportsCaching (cast to ProviderModelMapping | undefined) to build each entry (providerId, region, score: 0, uptime, latency, throughput, price, priority, cacheSupported). Replace the inline stableProviders.map(...) blocks in get-cheapest-from-available-providers.ts with calls to this new helper.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/gateway/src/chat/chat.ts`:
- Around line 1105-1109: The sessionId selection currently trims only the header
value but not the body-derived values, so prompt_cache_key and user can contain
leading/trailing whitespace and break sticky routing; update the sessionId
assignment to normalize (trim) prompt_cache_key and user the same way as
c.req.header does, ensuring the ternary/OR chain that builds sessionId (using
variables sessionId, prompt_cache_key, user, and c.req.header) calls .trim() or
an equivalent normalization on each non-header branch before falling back to
undefined.
---
Nitpick comments:
In `@packages/actions/src/get-cheapest-from-available-providers.ts`:
- Around line 415-446: Extract the duplicated providerScores mapping into a
single helper (e.g. buildZeroScoreProviderScores) and call it from both
early-exit branches so they stay in sync. The helper should accept
stableProviders, modelWithPricing (with id), metricsMap, and videoPricing and
return RoutingMetadata["providerScores"]; inside it reuse findProviderMapping,
getProviderDefinition (for priority), metricsKey, getProviderSelectionPrice
(call .toNumber()), and providerSupportsCaching (cast to ProviderModelMapping |
undefined) to build each entry (providerId, region, score: 0, uptime, latency,
throughput, price, priority, cacheSupported). Replace the inline
stableProviders.map(...) blocks in get-cheapest-from-available-providers.ts with
calls to this new helper.
In `@packages/actions/src/models.spec.ts`:
- Around line 577-612: Add a complementary test that verifies re-pinning when
the pinned provider is removed: using modelWithMultipleProviders and
getCheapestFromAvailableProviders, create a sessionId, compute the initial
pinned provider, then remove that pinned provider from the availableProviders
list and call getCheapestFromAvailableProviders again to assert the session now
deterministically pins to a different provider; also create a second distinct
sessionId and verify it does not collapse onto the same provider as the first
(i.e., two different sessions should not both re-pin to the same single provider
unless deterministic hashing dictates different results).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: d006bd6d-e9e6-4e2f-8bef-20bfbfe89106
📒 Files selected for processing (7)
apps/gateway/src/anthropic/anthropic.tsapps/gateway/src/chat/chat.tsapps/gateway/src/chat/schemas/completions.tsapps/gateway/src/lib/session-id.spec.tsapps/gateway/src/lib/session-id.tspackages/actions/src/get-cheapest-from-available-providers.tspackages/actions/src/models.spec.ts
Comment on lines
+1105
to
+1109
| const sessionId = | ||
| c.req.header("x-session-id")?.trim() || | ||
| prompt_cache_key || | ||
| user || | ||
| undefined; |
Contributor
There was a problem hiding this comment.
Normalize body-derived session IDs before sticky routing.
prompt_cache_key and user are not trimmed, so whitespace variants can pin to different providers unexpectedly. Normalize them like the header path.
💡 Proposed fix
const sessionId =
c.req.header("x-session-id")?.trim() ||
- prompt_cache_key ||
- user ||
+ prompt_cache_key?.trim() ||
+ user?.trim() ||
undefined;📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| const sessionId = | |
| c.req.header("x-session-id")?.trim() || | |
| prompt_cache_key || | |
| user || | |
| undefined; | |
| const sessionId = | |
| c.req.header("x-session-id")?.trim() || | |
| prompt_cache_key?.trim() || | |
| user?.trim() || | |
| undefined; |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/gateway/src/chat/chat.ts` around lines 1105 - 1109, The sessionId
selection currently trims only the header value but not the body-derived values,
so prompt_cache_key and user can contain leading/trailing whitespace and break
sticky routing; update the sessionId assignment to normalize (trim)
prompt_cache_key and user the same way as c.req.header does, ensuring the
ternary/OR chain that builds sessionId (using variables sessionId,
prompt_cache_key, user, and c.req.header) calls .trim() or an equivalent
normalization on each non-header branch before falling back to undefined.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
# Conflicts: # apps/gateway/src/chat/chat.ts # packages/actions/src/get-cheapest-from-available-providers.ts
# Conflicts: # apps/docs/content/features/routing.mdx
Persist the resolved session id (x-session-id / prompt_cache_key / user, or Anthropic metadata.user_id) on each request log. Surface it in the activity log card metadata and add a Session ID filter next to the custom metadata search. Document the sessions feature in a dedicated docs page. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The /v1/messages hybrid message schema accepted OpenAI-style tool/function roles but not system/developer, so clients (e.g. Claude Code) that place a system message in the messages array got a 400. The handler already maps roles straight through to valid OpenAI messages, so allow them in the enum. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude Code sends metadata.user_id as a JSON string
({"device_id":..,"account_uuid":..,"session_id":"<uuid>"}). The old
regex matched the literal "session_id" key instead of the UUID value, so
every Claude Code request logged the session id as "session_id". Parse the
JSON and use its session_id field, keeping the structured-string fallback.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a session-stickiness setting to the per-project custom routing config (default enabled). When disabled, the gateway ignores session ids for provider selection and scores every request independently; the session id is still recorded for observability. Exposed as a toggle in the routing settings UI, validated/persisted via the routing-config API, and gated in getCheapestFromAvailableProviders on cfg.session.enabled. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
opencode sends its session id in the x-session-affinity header (and x-source: opencode), which the session-key resolver didn't read, so opencode sessions weren't tracked or pinned. Add x-session-affinity to the resolution chain below the explicit x-session-id header. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
steebchen
enabled auto-merge
May 31, 2026 14:53
steebchen
added this pull request to the merge queue
May 31, 2026
analogpvt
pushed a commit
to analogpvt/llmgateway
that referenced
this pull request
Jun 2, 2026
## Why Two issues raised by a DevPass user (see support email): 1. Clicking **Switch to Pro** charged the card immediately with no confirmation step. 2. DevPass billing doesn't appear in the standard LLM Gateway billing dashboard, and it wasn't clear where to manage the subscription / payment method. This PR adds a confirmation flow before switching plans, makes downgrades non-punitive, and restructures the DevPass dashboard into clear Usage / Billing / Settings sections behind a sidebar. ## Changes ### Confirm before switching plans - Every **Switch to X** button now opens an `AlertDialog` first, with distinct copy: - **Upgrade** → charged a prorated amount today, then $X/mo, allowance increases right away (green ↑ Upgrade badge). - **Downgrade** → keep your current allowance until next renewal, **no refund** for the current period (↓ Downgrade badge). ### Downgrade backend (safe tweak) - `change-tier`: downgrade `proration_behavior` `create_prorations` → **`none`** (no refund/credit issued). - On downgrade we **no longer lower the credit allowance immediately** — the user keeps what they paid for and the existing renewal webhook resets it to the lower tier next cycle, so a downgrade never claws back paid credits. Upgrades unchanged. - _Deferred:_ scheduling the price swap exactly at period end via Stripe subscription schedules (needs a pending-downgrade state + webhook); flagged as follow-up. ### Dashboard sidebar + route split - New `(main)` route group with a `layout.tsx` → `DashboardShell` (header + sidebar: **Usage / Billing / Settings**, active state via `usePathname`, responsive rail/row). - Split the old monolithic `DashboardClient` into `/dashboard` (Usage), `/dashboard/billing`, `/dashboard/settings`. - Billing page adds a current-plan summary + an info note explaining DevPass billing is separate from pay-as-you-go credits and managed there. - Shared `plans.ts` + `useDevPlanStatus()` hook (React Query dedupes the status query); `ApiKeySection` extracted. - `agents/[agentId]` and `feedback/…` intentionally stay **outside** the shell (the feedback page must work for users without an active plan). ## Validation - `pnpm format` ✓, code app lint ✓, code `tsc --noEmit` ✓, API lint ✓ - Full `next build` ✓ — route map confirms `/dashboard`, `/dashboard/billing`, `/dashboard/settings` resolve under the shell while agents/feedback stay standalone. - Pre-existing `routing-config.ts` tsc errors (PR theopenco#2445) are unrelated and untouched. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **New Features** * Added dedicated billing page to manage subscription and view billing details * Added ability to upgrade or downgrade subscription tiers with clear visual indicators * Added subscription cancellation and resumption options * Added API key rotation with double-confirmation flow * Added settings page for plan configuration * **Improvements** * Downgrade behavior now preserves your current credit allowance through the rest of your billing period <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds sticky session routing so a logical session is pinned to a single upstream provider, keeping provider-side prompt caches warm across a multi-turn conversation. The session only moves to another provider when its pinned provider leaves the available list (health filtering or retry-fallback exclusion after a failure) — i.e. stick unless the provider is effectively down.
How the session key is resolved
Chat completions (
/v1/chat/completions), in priority order:x-session-idheaderprompt_cache_key(OpenAI body field)user(OpenAI body field — newly added to the schema)Anthropic Messages (
/v1/messages): the key is derived frommetadata.user_id(Claude Code embeds the session UUID as…_session_<uuid>), then forwarded to the internal chat/completions call asx-session-id. An explicitx-session-idheader on the incoming request still wins.Routing mechanism
When a session id is present,
getCheapestFromAvailableProviderspicks the provider/region deterministically via rendezvous (highest-random-weight) hashing, bypassing price/uptime scoring and epsilon-greedy exploration. Rendezvous hashing means removing one provider only reassigns the sessions pinned to that provider — every other session keeps its provider. "Unless down" is handled for free: down providers are excluded from the available list upstream / by the retry-fallback loop, so the hash naturally re-pins to the next provider.When no session id is present, behavior is unchanged.
Tests
packages/actions/src/models.spec.ts— sticky selection is deterministic and survives removal of an unrelated provider.apps/gateway/src/lib/session-id.spec.ts— Anthropicmetadata.user_idextraction.pnpm format,pnpm build, and the targeted unit tests all pass.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Tests
Documentation
Chores