fix: inference geo cost on anthropic (#5072)

[TejasGhatte]

Summary

Adds end-to-end support for Anthropic's data-residency billing feature (inference_geo). When Anthropic serves a request with inference_geo: "us", a 1.1× multiplier is applied to all token and cache costs. This PR propagates the inference_geo field from the Anthropic API response through all converter paths (chat, responses, streaming, passthrough) and wires it into the cost engine so US-residency requests bill correctly. It also fixes a related streaming bug where web search request counts were lost on the terminal chunk of streamed Responses requests, causing web search to go unbilled.

Changes

• inference_geo field propagation: Added InferenceGeo *string to BifrostChatResponse, BifrostResponsesResponse, and BifrostPassthroughUsage schemas. All Anthropic converters (ToBifrostChatResponse, ToBifrostResponsesResponse, ToBifrostResponsesStream, passthrough usage) now forward inference_geo from the provider's usage block.
• Streaming terminal chunk fix: In both HandleAnthropicChatCompletionStreaming and HandleAnthropicResponsesStream, servedSpeed and servedInferenceGeo are captured across events and re-applied to the terminal chunk. The terminal response.completed chunk is built fresh with no served modifiers, so without this re-apply, fast-mode and data-residency billing silently fell back to standard rates.
• Web search count accumulation fix: accumulateAnthropicResponsesUsage now carries ServerToolUse.WebSearchRequests into both the response usage accumulator and the mirrored billed usage. Previously the per-event count was overwritten by the terminal chunk's fresh usage, causing web search to go unbilled on streamed Responses requests.
• Cost engine: serviceTier gains an inferenceGeoUS bool field. tierFromResponse accepts a third inferenceGeo *string argument and sets inferenceGeoUS when the value is "us" (case-insensitive). computeTextCost applies InferenceGeoUSMultiplier to all token/cache costs when inferenceGeoUS is true; the flat per-search fee is intentionally excluded from the multiplier.
• Database migration: Adds inference_geo_us_multiplier column to TableModelPricing via a new migration step. The column is included in pricing sync, override patching, and the entry↔table conversion helpers.
• UI: Exposes inference_geo_us_multiplier as a configurable field in the custom pricing override sheet.
• Tests: New unit tests cover the accumulator web-search fix, the chat and responses converters forwarding inference_geo and web search counts, the cost engine multiplier logic (including the safe no-op when the column is unpopulated), tierFromResponse for inferenceGeoUS, and an end-to-end streaming test (TestResponsesStream_TerminalChunkCarriesServedModifiers) that replays message_start → message_delta → message_stop and asserts the terminal billed chunk carries speed, inference_geo, web search count, and cache-creation tokens.

Type of change

• Bug fix
• Feature
• Refactor
• Documentation
• Chore/CI

Affected areas

• Core (Go)
• Transports (HTTP)
• Providers/Integrations
• Plugins
• UI (React)
• Docs

How to test

# Core/Transports
go test ./core/providers/anthropic/... ./framework/modelcatalog/datasheet/...

# UI
cd ui
pnpm i
pnpm build

To validate end-to-end: send a chat or responses request to an Anthropic model with data residency enabled. Confirm the response carries inference_geo: "us" and that the billed cost reflects the 1.1× multiplier on token/cache costs with the per-search fee unchanged. For streaming, confirm the terminal chunk carries both speed and inference_geo after message_stop.

The inference_geo_us_multiplier column must be populated in the model pricing datasheet for the multiplier to apply; if the column is null, billing is a safe no-op (standard rates).

Breaking changes

• No

tierFromResponse gains a third parameter but is an internal function. All call sites have been updated.

Related issues

Security considerations

No auth, secrets, or PII implications. The multiplier is a pricing configuration value stored in the existing model pricing table with the same access controls as other pricing columns.

Checklist

• I read docs/contributing/README.md and followed the guidelines
• I added/updated tests where appropriate
• I updated documentation where needed
• I verified builds succeed (Go and UI)
• I verified the CI pipeline passes locally if applicable

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[TejasGhatte]

• tests: openai cache coverage #5091
• fix: openai cache write pricings #5090
• fix: service tier in openai chat completion (#5073) #5089
• fix: inference geo cost on anthropic (#5072) #5088 👈 (View in Graphite)
• fix: annthropic cache rate for fast mode (#5063) #5087
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[greptile-apps]

Confidence Score: 5/5

This looks safe to merge.

• No blocking issues found in the changed code.

Important Files Changed

Filename	Overview
core/providers/anthropic/anthropic.go	Updates Anthropic streaming billing state for served speed, inference geo, and web-search usage.
framework/modelcatalog/datasheet/cost.go	Adds inference geo tier handling and applies the multiplier to token and cache costs.
framework/configstore/migrations.go	Adds a nullable model pricing column for the inference geo multiplier.
ui/app/workspace/custom-pricing/overrides/pricingOverrideSheet.tsx	Exposes the inference geo multiplier in the custom pricing override sheet.

_{Reviews (3): Last reviewed commit: "fix: inference geo cost on anthropic (#5..." | Re-trigger Greptile}

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 1c11974c-61cc-4978-9a26-56cb54b9b1df

📥 Commits

Reviewing files that changed from the base of the PR and between 19fb627 and 41d99a8.

📒 Files selected for processing (17)

• core/providers/anthropic/anthropic.go
• core/providers/anthropic/chat.go
• core/providers/anthropic/passthrough_usage.go
• core/providers/anthropic/responses.go
• core/providers/anthropic/utils_test.go
• core/schemas/chatcompletions.go
• core/schemas/passthrough.go
• core/schemas/responses.go
• framework/configstore/migrations.go
• framework/configstore/rdb.go
• framework/configstore/tables/modelpricing.go
• framework/modelcatalog/datasheet/cost.go
• framework/modelcatalog/datasheet/cost_test.go
• framework/modelcatalog/datasheet/overrides.go
• framework/modelcatalog/datasheet/types.go
• ui/app/workspace/custom-pricing/overrides/pricingOverrideSheet.tsx
• ui/lib/types/governance.ts

---

📝 Walkthrough

Summary by CodeRabbit

• New Features

  • Anthropic responses now surface inference geography, fast-mode status, and web-search usage.
  • Billing accurately accounts for web-search queries and US inference-geography pricing.
  • Custom pricing supports configuring an inference-geo US multiplier.

• Bug Fixes

  • Streaming and non-streaming usage now consistently preserve served pricing details, including terminal responses and passthrough usage.

Walkthrough

Anthropic usage handling now propagates web-search counts, served speed, and inference geography through streaming and response conversions. Model pricing persists and applies an inference-geography multiplier to token costs, with configstore and custom-pricing support.

Changes

Anthropic usage propagation

Layer / File(s)	Summary
Response usage contracts and converters core/schemas/*, core/providers/anthropic/chat.go, core/providers/anthropic/responses.go, core/providers/anthropic/passthrough_usage.go	Response and passthrough usage now carry inference geography, while Anthropic converters forward inference geography and web-search request counts.
Streaming usage and terminal billing core/providers/anthropic/anthropic.go, core/providers/anthropic/utils_test.go	Streaming handlers retain served speed and inference geography, accumulate web-search requests, and apply these values to terminal billing responses.
Inference-geography pricing and persistence framework/configstore/*, framework/modelcatalog/datasheet/*	Model pricing stores the US inference multiplier, synchronizes it through configstore and overrides, derives inference-geography tiers, and applies the multiplier to token costs while excluding search fees.
Pricing override interface ui/app/workspace/custom-pricing/overrides/pricingOverrideSheet.tsx, ui/lib/types/governance.ts	Custom pricing overrides recognize the inference-geography US multiplier field.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Anthropic
  participant BifrostStreaming
  participant UsageBilling
  participant ModelPricing
  Anthropic->>BifrostStreaming: stream usage with Speed, InferenceGeo, and web-search counts
  BifrostStreaming->>UsageBilling: accumulate usage and served-tier metadata
  UsageBilling->>ModelPricing: derive pricing tier
  ModelPricing->>UsageBilling: apply US multiplier to token costs

Loading

Possibly related PRs

• maximhq/bifrost#5072: Directly overlaps Anthropic usage propagation and inference-geography billing paths.
• maximhq/bifrost#4842: Also changes streamed usage accumulation for mid-stream billing.
• maximhq/bifrost#4917: Modifies the shared model tier and cost calculation paths.

Suggested reviewers: danpiths, Pratham-Mishra04, roroghost17, akshaydeo

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 72.73% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title is concise and accurately summarizes the Anthropic inference-geo billing change.
Description check	✅ Passed	The description follows the template closely and covers the required sections, though UI screenshots are omitted.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch 07-10-cp-inference-geo-cost

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.12.2)

level=error msg="[linters_context] typechecking error: pattern ./...: directory prefix . does not contain main module or its selected dependencies"

---

_{Comment @coderabbitai help to get the list of available commands.}

[akshaydeo]

Merge activity

• Jul 10, 10:32 AM UTC: A user started a stack merge that includes this pull request via Graphite.
• Jul 10, 10:34 AM UTC: Graphite rebased this pull request as part of a merge.
• Jul 10, 10:35 AM UTC: @akshaydeo merged this pull request with Graphite.

The base branch was changed.