fix: bill for serving fallback model in anthropic - #5387
Conversation
|
|
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 56 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (11)
📝 WalkthroughWalkthroughAnthropic usage now identifies server-side fallback models, propagates them through chat and Responses routing metadata, preserves them during routing updates, and prioritizes them for pricing. Fallback request fields, count-token handling, and fast-mode headers are also covered. ChangesAnthropic fallback routing
Estimated code review effort: 3 (Moderate) | ~25 minutes Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant AnthropicUsage
participant AnthropicResponseConverter
participant RoutingInfo
participant resolvePricing
participant ModelCatalog
AnthropicUsage->>AnthropicResponseConverter: expose fallback-serving model
AnthropicResponseConverter->>RoutingInfo: attach ServerSideFallbackModel
RoutingInfo->>resolvePricing: provide serving and requested model metadata
resolvePricing->>ModelCatalog: try serving-model pricing first
ModelCatalog-->>resolvePricing: return rates or no match
resolvePricing->>ModelCatalog: fall back to requested-model pricing
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
This stack of pull requests is managed by Graphite. Learn more about stacking. |
Confidence Score: 4/5The cancellation billing gap should be fixed before merging.
core/providers/anthropic/anthropic.go Important Files Changed
Reviews (5): Last reviewed commit: "fix: bill for serving fallback model in ..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
core/providers/anthropic/responses.go (1)
2271-2298: 🗄️ Data Integrity & Integration | 🔴 Critical | ⚡ Quick winServer-side-fallback latch only runs for
integration_type == "anthropic"— streaming under other integrations never captures the served model.The
state.ServerSideFallbackModellatch is nested inside thectx.Value(schemas.BifrostContextKeyIntegrationType) == "anthropic"branch. For every other integration (the default OpenAI-compatible Responses API streaming path), the function returns early at Line 2348 without ever callingchunk.Usage.ServerSideFallbackModel(). Sincemessage_stop'scompletedResp(Line 2412) only reads back whatever was latched, streaming responses on the default integration will never carryServerSideFallbackModel, so a fallback-served streamed turn is billed against the requested model instead of the model that actually served it — the exact bug this PR targets, left unfixed for the primary streaming path.
chunk.Usageis the raw Anthropic-format usage regardless of output integration, so this bookkeeping should not depend on integration type.🐛 Proposed fix — hoist the latch out of the integration-type gate
if chunk.Delta.StopDetails != nil { state.StopDetails = stopDetailsToBifrost(chunk.Delta.StopDetails) } + // The handoff is only visible on this chunk's iterations, but the chunk + // that ends up being priced may be a later one, so latch it in state, + // regardless of which integration format is being emitted downstream. + if served := chunk.Usage.ServerSideFallbackModel(); served != nil { + state.ServerSideFallbackModel = served + } // Check if integration type in ctx is anthropic if ctx.Value(schemas.BifrostContextKeyIntegrationType) == "anthropic" { // Convert usage from Anthropic format to Bifrost bifrostUsage := ConvertAnthropicUsageToBifrostUsage(chunk.Usage) - // The handoff is only visible on this chunk's iterations, but the chunk - // that ends up being priced may be a later one, so latch it in state. - if served := chunk.Usage.ServerSideFallbackModel(); served != nil { - state.ServerSideFallbackModel = served - } - // Use the already-remapped stop reason so SO overrides are preserved.Also applies to: 2337-2343, 2407-2413
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@core/providers/anthropic/responses.go` around lines 2271 - 2298, Move the ServerSideFallbackModel latch using chunk.Usage.ServerSideFallbackModel() out of the integration_type == "anthropic" branch in the AnthropicStreamEventTypeMessageDelta handling, so it runs for every streaming integration before any early return. Keep ConvertAnthropicUsageToBifrostUsage and other integration-specific processing inside the existing branch, and preserve the message_stop completedResp path that reads state.ServerSideFallbackModel.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@core/providers/anthropic/responses.go`:
- Line 78: Update the AnthropicResponsesStreamState cleanup in flush() to clear
ServerSideFallbackModel before returning the struct to
anthropicResponsesStreamStatePool. Ensure every pooled-state reuse starts with a
nil value, preserving the existing resets for all other fields.
---
Outside diff comments:
In `@core/providers/anthropic/responses.go`:
- Around line 2271-2298: Move the ServerSideFallbackModel latch using
chunk.Usage.ServerSideFallbackModel() out of the integration_type == "anthropic"
branch in the AnthropicStreamEventTypeMessageDelta handling, so it runs for
every streaming integration before any early return. Keep
ConvertAnthropicUsageToBifrostUsage and other integration-specific processing
inside the existing branch, and preserve the message_stop completedResp path
that reads state.ServerSideFallbackModel.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 89bfa7f3-fd9d-4716-a084-209e68ef0e31
📒 Files selected for processing (7)
core/providers/anthropic/chat.gocore/providers/anthropic/responses.gocore/providers/anthropic/serversidefallback_test.gocore/providers/anthropic/types.gocore/schemas/bifrost.goframework/modelcatalog/datasheet/cost.goframework/modelcatalog/datasheet/cost_test.go
66c2122 to
7834f85
Compare
dd9aa5f to
5e28f23
Compare
Merge activity
|
The base branch was changed.
5e28f23 to
f8b5f2c
Compare
| if served := usageToProcess.ServerSideFallbackModel(); served != nil { | ||
| servedFallbackModel = served | ||
| usage.ServerSideFallbackModel = served | ||
| } |
There was a problem hiding this comment.
ServerSideFallbackModel() becomes available only when the terminal message_delta includes usage.iterations. If the stream is cancelled or times out after fallback content starts but before that event is processed, the deferred billing handler snapshots usage while this field is still nil. The partial turn is then priced using the requested model instead of the model that served it. Retain the serving model from metadata available before the terminal usage event so cancellation billing can use it.
3 participants
Summary
When Anthropic's server-side fallback feature hands off a request mid-call to a different model, the orchestrator never sees the swap — it only knows what the caller originally asked for. This means cost calculations were being applied against the requested model's rates rather than the model that actually produced the tokens, causing incorrect (typically inflated) billing figures. This PR surfaces the serving model through the response's
RoutingInfoand ensures the cost engine prices against it.Changes
ServerSideFallbackModeltoRoutingInfo— a provider-owned field that names the model that actually served the response when a server-side handoff occurred. Nil on all ordinary responses, so existing pricing behaviour is unchanged.AnthropicUsageIterationTypeFallbackMessageconstant andServerSideFallbackModel()method onAnthropicUsage, which inspectsusage.iterationsfor afallback_message-typed entry and returns its model name.RoutingInfo.ServerSideFallbackModelin both the chat and responses non-streaming paths before usage is flattened, since the neutral usage representation drops iterations.AnthropicResponsesStreamStatewhen first seen on a usage chunk, then stamped it onto both themessage_deltaandcompletedresponse chunks — whichever one ends up being priced will carry it.PopulateRoutingInfoto preserve any pre-existingServerSideFallbackModelvalue when core's routing snapshot overwritesRoutingInfo. Without this, streaming would lose the field entirely since the snapshot predates the final usage chunk.resolvePricingin the cost engine to rankServerSideFallbackModelfirst in the candidate lookup chain, ahead of alias model name, alias model ID, and the requested model name. Override keys also follow the serving model so negotiated rates for the model that actually ran are the ones applied.Type of change
Affected areas
How to test
go test ./core/providers/anthropic/... ./core/schemas/... ./framework/modelcatalog/datasheet/...Key scenarios covered by new tests:
ServerSideFallbackModel()returns the correct model fromfallback_messageiterations, nil for ordinary responses, nil for compaction-only iterations, and nil for a nil receiver.ExtraFields.RoutingInfo.ServerSideFallbackModel.PopulateRoutingInfopreserves the provider-set serving model while still applying core's routing snapshot for all other fields.Breaking changes
Security considerations
None. This change only affects cost attribution metadata and pricing lookups; no auth, secrets, or PII are involved.
Checklist
docs/contributing/README.mdand followed the guidelines