Repository navigation
feat(logs): track requested vs used service tier - #2816
Merged
Merged
Conversation
Rename the log `service_tier` column to `requested_service_tier` and add a new `used_service_tier` column so logs record both the tier the client requested and the tier the provider actually served. The gateway now writes the requested tier (flex/priority) alongside the served tier on every log; the API exposes both fields and the UI activity card and log detail view show "Requested Service Tier" and "Used Service Tier" separately. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
WalkthroughThe PR splits service-tier logging into requested and used fields across the log database schema, gateway log writers, API log responses, and UI displays. Tests and media error logging update to the new field names. ChangesService tier logging split
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 04101394c8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+1447
to
+1449
| const requestedServiceTier = isRequestedServiceTier(service_tier) | ||
| ? service_tier | ||
| : null; |
There was a problem hiding this comment.
For chat requests that include service_tier but exit through existing log paths like guardrail/client validation, cache hits, streaming timeouts/upstream errors, or non-streaming fetch errors, those insertLogEntry payloads still omit both requestedServiceTier and usedServiceTier. Because these nullable insert fields default to null, the API/UI will show no requested tier for those error/cached logs even though it is computed here; thread requestedServiceTier through each chat log insert and set usedServiceTier where a provider tier is known.
Useful? React with 👍 / 👎.
Rename the existing `service_tier` column (which stored the served tier) to `used_service_tier` instead of `requested_service_tier`, so existing log rows keep their served-tier value under the correct field. RENAME COLUMN is a catalog-only operation, so this is instant even on a table with hundreds of millions of rows and needs no backfill UPDATE. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/db/migrations/1782388997_absurd_shinko_yamashiro.sql`:
- Around line 1-8: This migration appears to have been hand-edited, including a
custom rationale and a manual column rename, so regenerate it through the local
migrations skill workflow instead of keeping the handwritten SQL. Use the
migrations skill to produce the migration and confirm the corresponding snapshot
was created, then replace this file with the skill-generated output for the
`log` table change involving `service_tier`, `used_service_tier`, and
`requested_service_tier`.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: d83a21c3-3454-4a3c-8c12-b2bd0062d97b
📒 Files selected for processing (3)
packages/db/migrations/1782388997_absurd_shinko_yamashiro.sqlpackages/db/migrations/meta/1782388997_snapshot.jsonpackages/db/migrations/meta/_journal.json
steebchen
added a commit
that referenced
this pull request
Jun 25, 2026
Follow-up to #2816, which populated requestedServiceTier / usedServiceTier only on the chat success paths and the two direct _insertLog rejections. Other paths (guardrail/validation rejections, cache hits, streaming timeouts/upstream errors, fetch errors) go through the shared insertLogEntry wrapper and omitted both fields. - Default requestedServiceTier + usedServiceTier (from servedServiceTier) in the insertLogEntry wrapper so every chat log path records them; explicit values still win. Declare servedServiceTier ahead of the wrapper and drop the now-redundant explicit fields from the two success inserts. - Reset servedServiceTier at the start of each retry attempt (both the streaming and non-streaming loops) so a fallback that fails before fetch returns logs no served tier instead of inheriting the previous provider's. - Update the upstream-error test to openai/gpt-5.5 (priority, not subject to the upstream base-URL restriction added in #2821) so it still reaches the error path, and make the mock /v1/responses handler detect TRIGGER_ERROR / status triggers in array-form input. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
steebchen
added a commit
that referenced
this pull request
Jun 25, 2026
Surface the requested vs served processing tier (Flex/Priority) on both the response and every chat log path. Response: - Add requested_service_tier + used_service_tier to the response metadata for non-streaming completions and the streaming final usage chunk (via buildFinalResponseMetadata / toResponseMetadataExtras). Emitted only when a premium tier was requested; used_service_tier is null when the upstream downgraded to standard. Logs (follow-up to #2816, which only covered success + the two direct _insertLog rejections): - Default requestedServiceTier + usedServiceTier in the insertLogEntry wrapper so guardrail/validation rejections, cache hits, streaming timeouts/upstream errors and fetch errors record them too. Declare servedServiceTier ahead of the wrapper and drop the now-redundant explicit fields from the two success inserts. - Reset servedServiceTier at the start of each retry attempt (both loops) so a fallback that fails before fetch returns logs no served tier instead of inheriting the previous provider's. Tests/infra: - Update the upstream-error test to openai/gpt-5.5 (priority, not subject to the upstream base-URL restriction from #2821) so it still reaches the error path; make the mock /v1/responses handler array-safe for TRIGGER_ERROR / status triggers and echo service_tier in its streamed response object. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
pull Bot
pushed a commit
to soitun/llmgateway
that referenced
this pull request
Jun 26, 2026
## The fix: Vertex Priority/Flex PayGo needs two headers Vertex requires **both**: - `X-Vertex-AI-LLM-Shared-Request-Type: priority|flex` — selects the tier (the gateway already sent this) - `X-Vertex-AI-LLM-Request-Type: shared` — forces the shared PayGo path (**was missing**) Without the second header, a project that has **Provisioned Throughput** allocated consumes PT first and never reaches the shared Flex/Priority tier — so the request is silently served (and billed) as **standard** even though `priority` was requested. This is why production showed `requested=priority` / `used=null` on `aiplatform.googleapis.com` projects with PT, while projects without PT worked. We now send both headers for flex/priority (no-op on projects without PT, so it's safe everywhere). Per [Google's docs](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/priority-paygo) and [liteLLM](https://docs.litellm.ai/docs/tutorials/vertex_ai_pay_go): *"To bypass Provisioned Throughput entirely, also set requestType: 'shared'."* ## Also in this PR: surface & log requested vs served tier - **Response**: `requested_service_tier` + `used_service_tier` added to the response `metadata` (non-streaming) and the **streaming final usage chunk** (via `buildFinalResponseMetadata` / `toResponseMetadataExtras`). Emitted only when a premium tier was requested; `used` is `null` on a downgrade so callers can detect it. - **Logs** (follow-up to theopenco#2816): default both fields in the `insertLogEntry` wrapper so every chat log path (guardrail/validation rejections, cache hits, streaming/upstream errors, fetch errors) records them; **reset `servedServiceTier` per retry attempt** so a fallback that fails before `fetch` returns logs no served tier instead of inheriting the previous provider's. ## Tests / infra - `get-provider-headers` test asserts **both** Vertex headers on flex/priority and their absence otherwise. - Upstream-error test uses `openai/gpt-5.5` (not subject to theopenco#2821's upstream base-URL rule) so it still reaches the error path; mock `/v1/responses` handler is array-safe for `TRIGGER_ERROR` / `TRIGGER_STATUS_*` and echoes `service_tier` when streaming. - New tests: non-streaming metadata present on a priority request / absent without one; streaming final chunk carries both fields. ## Verification `pnpm test:unit` (132 files), full `apps/gateway/src/api.spec.ts` (101 passed / 2 skipped), `pnpm format`, full `pnpm build` all pass. The two-header behavior was confirmed against Google's Priority PayGo docs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Logs now record both the service tier the client requested and the tier the provider actually served.
log.service_tiercolumn toused_service_tier(it already stored the served tier).requested_service_tiercolumn.flex/priority) alongside the served tier on every chat/image/video log, including the unsupported-tier and budget rejection paths.apps/api/src/routes/logs.ts) exposes both fields.packages/shared/src/components/log-card.tsx) and the log detail view now show Requested Service Tier and Used Service Tier as separate metadata fields. The used tier keeps the cost-multiplier annotation, since billed token costs reflect the served tier.Why
Google (and others) silently downgrade an unsupported tier to standard, so the served tier can differ from what the client asked for. Tracking both makes that visible in the activity log instead of only showing the served value.
Migration
1782388997_absurd_shinko_yamashiro.sql:The old
service_tiercolumn stored the served tier, so it is renamed in place toused_service_tier— existing rows keep their value under the correct field.requested_service_tieris added empty (NULL for historical rows, since the requested tier was not recorded before).Performance at scale: both statements are catalog-only metadata operations in Postgres —
RENAME COLUMNnever rewrites the table, andADD COLUMN ... textwith no default is also instant (PG 11+). This runs in well under a second even on alogtable with hundreds of millions of rows, with no per-row backfillUPDATE. Only a briefACCESS EXCLUSIVElock is held.Testing
pnpm build— all 17 tasks passpnpm formatapps/gateway/src/api.spec.tsservice-tier tests updated and passing (3 passed)🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes