Skip to content

fix: inference geo cost on anthropic - #5072

Merged
akshaydeo merged 1 commit into
devfrom
07-09-fix_inference_geo_cost_on_anthropic
Jul 9, 2026
Merged

akshaydeo merged 1 commit into
devfrom
07-09-fix_inference_geo_cost_on_anthropic

Conversation

@TejasGhatte

@TejasGhatte TejasGhatte commented Jul 9, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds support for Anthropic's data-residency billing feature (inference_geo). When Anthropic serves inference in the US (inference_geo: "us"), a 1.1x multiplier is applied to all token and cache costs. This PR propagates the inference_geo field through the full response pipeline (chat, responses, streaming, passthrough) and wires it into the cost computation layer, including a new database column and UI override field.

A secondary fix ensures Anthropic server-tool web search request counts (ServerToolUse.WebSearchRequests) are correctly forwarded and billed in both streaming and non-streaming paths, where previously the count could be lost when the terminal chunk overwrote the accumulated usage.

Changes

  • inference_geo propagation: InferenceGeo *string added to BifrostChatResponse, BifrostResponsesResponse, and BifrostPassthroughUsage. The field is forwarded from AnthropicUsage in all conversion paths: non-streaming chat, non-streaming responses, streaming chat (captured across events and set on the final chunk), streaming responses (message_delta), and passthrough.
  • Data-residency multiplier in cost computation: serviceTier gains an inferenceGeoUS bool flag. tierFromResponse now accepts and evaluates inferenceGeo. computeTextCost applies InferenceGeoUSMultiplier to token/cache costs only — the flat per-search fee is intentionally excluded.
  • Database migration: migrationAddInferenceGeoMultiplierColumn adds inference_geo_us_multiplier to TableModelPricing. The column is included in the pricing sync update list and mapped through convertEntryToTablePricing / convertTablePricingToEntry.
  • Web search billing fix: accumulateAnthropicResponsesUsage now carries ServerToolUse.WebSearchRequests into the accumulator so the count survives the terminal-chunk overwrite on streamed Responses requests. The non-streaming chat converter also forwards the count into CompletionTokensDetails.NumSearchQueries.
  • UI: inference_geo_us_multiplier added to the custom pricing override sheet and PricingOverridePatch TypeScript type.
  • Tests: New unit tests cover the accumulator web-search fix, chat/responses converter forwarding of InferenceGeo and web search counts, multiplier application (including the no-multiplier-column safe no-op), and tierFromResponse geo detection (case-insensitive).

Type of change

  • Bug fix
  • Feature
  • Refactor
  • Documentation
  • Chore/CI

Affected areas

  • Core (Go)
  • Transports (HTTP)
  • Providers/Integrations
  • Plugins
  • UI (React)
  • Docs

How to test

# Core/Transports
go test ./core/providers/anthropic/... ./framework/modelcatalog/datasheet/...

# UI
cd ui
pnpm i || npm i
pnpm build || npm run build

To validate end-to-end: send a request to an Anthropic model with data residency enabled. The response should include inference_geo: "us" and the computed cost should reflect the 1.1x multiplier on token/cache costs, with the per-search fee unchanged.

Breaking changes

  • Yes
  • No

Security considerations

No auth, secrets, or PII implications. The inference_geo value is sourced from Anthropic's API response and used only for billing computation.

Checklist

  • I read docs/contributing/README.md and followed the guidelines
  • I added/updated tests where appropriate
  • I updated documentation where needed
  • I verified builds succeed (Go and UI)
  • I verified the CI pipeline passes locally if applicable

@coderabbitai

coderabbitai Bot commented Jul 9, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 11 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 72e839ac-d4b4-438a-8fe3-dd069ab9010f

📥 Commits

Reviewing files that changed from the base of the PR and between aa19027 and c1a6a91.

📒 Files selected for processing (17)
  • core/providers/anthropic/anthropic.go
  • core/providers/anthropic/chat.go
  • core/providers/anthropic/passthrough_usage.go
  • core/providers/anthropic/responses.go
  • core/providers/anthropic/utils_test.go
  • core/schemas/chatcompletions.go
  • core/schemas/passthrough.go
  • core/schemas/responses.go
  • framework/configstore/migrations.go
  • framework/configstore/rdb.go
  • framework/configstore/tables/modelpricing.go
  • framework/modelcatalog/datasheet/cost.go
  • framework/modelcatalog/datasheet/cost_test.go
  • framework/modelcatalog/datasheet/overrides.go
  • framework/modelcatalog/datasheet/types.go
  • ui/app/workspace/custom-pricing/overrides/pricingOverrideSheet.tsx
  • ui/lib/types/governance.ts
📝 Walkthrough

Walkthrough

Adds Anthropic inference_geo and web-search billing support: provider conversions forward inference_geo and web-search counts into Bifrost schemas and streaming outputs, pricing storage gains an inference_geo multiplier column, cost calculation applies it to token costs, and UI pricing overrides expose the new field.

Changes

Anthropic Inference Geo and Web Search Billing

Layer / File(s) Summary
Schema fields for InferenceGeo and search queries
core/schemas/chatcompletions.go, core/schemas/responses.go, core/schemas/passthrough.go
Adds optional InferenceGeo fields to BifrostChatResponse, BifrostResponsesResponse, and BifrostPassthroughUsage.
Anthropic streaming usage accumulation
core/providers/anthropic/anthropic.go
Streaming usage accounting maps web search requests into NumSearchQueries and carries served Speed/InferenceGeo into final streamed chunks.
Non-streaming conversion and passthrough usage
core/providers/anthropic/chat.go, core/providers/anthropic/responses.go, core/providers/anthropic/passthrough_usage.go, core/providers/anthropic/utils_test.go
Chat/responses conversion and passthrough usage preserve InferenceGeo and web search counts, with tests covering the forwarding behavior and terminal streamed chunk reconstruction.
Model pricing column and migration
framework/configstore/tables/modelpricing.go, framework/configstore/migrations.go, framework/configstore/rdb.go
Adds InferenceGeoUSMultiplier to model pricing storage, migration, and pricing sync updates.
Cost computation and tiering
framework/modelcatalog/datasheet/cost.go, framework/modelcatalog/datasheet/types.go, framework/modelcatalog/datasheet/overrides.go, framework/modelcatalog/datasheet/cost_test.go
Adds geo-aware tiering and applies the multiplier to token costs, while propagating the new field through datasheet types, overrides, and tests.
UI pricing override support
ui/app/workspace/custom-pricing/overrides/pricingOverrideSheet.tsx, ui/lib/types/governance.ts
Adds inference_geo_us_multiplier to pricing override field definitions and patch typing.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant AnthropicStreamHandler
  participant UsageAccumulator
  participant Client
  AnthropicStreamHandler->>UsageAccumulator: usageToProcess (WebSearchRequests, Speed, InferenceGeo)
  UsageAccumulator-->>AnthropicStreamHandler: NumSearchQueries, servedSpeed, servedInferenceGeo
  AnthropicStreamHandler->>Client: final chunk with response.Speed and response.InferenceGeo
Loading

Possibly related PRs

  • maximhq/bifrost#4842: Both PRs modify Anthropic Responses streaming usage accounting in core/providers/anthropic/anthropic.go.
  • maximhq/bifrost#4917: Both PRs modify framework/modelcatalog/datasheet/cost.go and its related cost tests.
  • maximhq/bifrost#3829: Both PRs touch model pricing storage and sync behavior around TableModelPricing.

Suggested reviewers: akshaydeo, danpiths, roroghost17, Pratham-Mishra04

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 70.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title concisely and accurately summarizes the main change: Anthropic inference geo cost handling.
Description check ✅ Passed The description covers summary, changes, testing, risk, and checklist items, with only minor omissions like screenshots and related issues.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch 07-09-fix_inference_geo_cost_on_anthropic

Comment @coderabbitai help to get the list of available commands.

@TejasGhatte
TejasGhatte marked this pull request as ready for review July 9, 2026 15:09

TejasGhatte commented Jul 9, 2026 •

Copy link
Copy Markdown
Collaborator Author

@greptile-apps

greptile-apps Bot commented Jul 9, 2026 •

Copy link
Copy Markdown
Contributor

Confidence Score: 4/5

This is close, but the compaction billing path should be fixed before merging.

  • The main Anthropic chat, responses, streaming, passthrough, pricing, and UI paths are wired through.
  • Compaction responses with US geography can still skip the new multiplier.
  • The remaining issue can cause incorrect billing for token and cache costs.

framework/modelcatalog/datasheet/cost.go and the compaction response schema

Important Files Changed

Filename Overview
core/providers/anthropic/anthropic.go Captures Anthropic streaming usage modifiers and web-search counts for final billing chunks.
core/providers/anthropic/responses.go Forwards inference_geo from Anthropic Responses conversions.
framework/modelcatalog/datasheet/cost.go Adds geo multiplier billing for text costs, while compaction cost extraction still omits served geography.
framework/configstore/migrations.go Adds a nullable pricing column for the inference geo multiplier.
ui/app/workspace/custom-pricing/overrides/pricingOverrideSheet.tsx Adds the inference geo multiplier field to custom pricing overrides.

Reviews (3): Last reviewed commit: "fix: inference geo cost on anthropic" | Re-trigger Greptile

Comment thread framework/modelcatalog/datasheet/cost.go
Comment thread core/providers/anthropic/responses.go
Comment thread core/providers/anthropic/responses.go
@CLAassistant

CLAassistant commented Jul 9, 2026 •

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
core/providers/anthropic/anthropic.go (1)

886-889: 📐 Maintainability & Code Quality | 🔵 Trivial

Web search accumulation duplicated across two sites; minor pointer-idiom inconsistency.

The max-tracking-and-set logic for NumSearchQueries here (lines 966-973) is structurally identical to accumulateAnthropicResponsesUsage (lines 624-643) but operates on BifrostLLMUsage instead of ResponsesResponseUsage. Also, &n is used at line 971 instead of schemas.Ptr(n), which is used consistently in the sibling accumulator function in this same file.

Not blocking — behavior is correct in both places — but consider extracting a small shared helper to avoid future divergence, and align on schemas.Ptr() for consistency.

Also applies to: 965-980, 1142-1148

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@core/providers/anthropic/anthropic.go` around lines 886 - 889, The
`NumSearchQueries` max-tracking and assignment logic is duplicated between
`accumulateAnthropicResponsesUsage` and the `BifrostLLMUsage` accumulator in
`anthropic.go`, and the latter also uses `&n` instead of the file’s consistent
`schemas.Ptr()` pattern. Extract the shared web-search accumulation into a small
helper that both paths call, keep the max-of-values behavior identical for both
usage types, and update the pointer assignment in the `BifrostLLMUsage` path to
use `schemas.Ptr()` so the two accumulator sites stay aligned.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@core/providers/anthropic/anthropic.go`:
- Around line 886-889: The `NumSearchQueries` max-tracking and assignment logic
is duplicated between `accumulateAnthropicResponsesUsage` and the
`BifrostLLMUsage` accumulator in `anthropic.go`, and the latter also uses `&n`
instead of the file’s consistent `schemas.Ptr()` pattern. Extract the shared
web-search accumulation into a small helper that both paths call, keep the
max-of-values behavior identical for both usage types, and update the pointer
assignment in the `BifrostLLMUsage` path to use `schemas.Ptr()` so the two
accumulator sites stay aligned.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 9ff0ab71-3796-492c-85cf-41f43044f743

📥 Commits

Reviewing files that changed from the base of the PR and between 406f85a and 7b0bd1e.

📒 Files selected for processing (17)
  • core/providers/anthropic/anthropic.go
  • core/providers/anthropic/chat.go
  • core/providers/anthropic/passthrough_usage.go
  • core/providers/anthropic/responses.go
  • core/providers/anthropic/utils_test.go
  • core/schemas/chatcompletions.go
  • core/schemas/passthrough.go
  • core/schemas/responses.go
  • framework/configstore/migrations.go
  • framework/configstore/rdb.go
  • framework/configstore/tables/modelpricing.go
  • framework/modelcatalog/datasheet/cost.go
  • framework/modelcatalog/datasheet/cost_test.go
  • framework/modelcatalog/datasheet/overrides.go
  • framework/modelcatalog/datasheet/types.go
  • ui/app/workspace/custom-pricing/overrides/pricingOverrideSheet.tsx
  • ui/lib/types/governance.ts

coderabbitai[bot]
coderabbitai Bot previously approved these changes Jul 9, 2026
@TejasGhatte
TejasGhatte force-pushed the 07-09-fix_inference_geo_cost_on_anthropic branch from 7b0bd1e to aa19027 Compare July 9, 2026 15:36
Comment thread framework/modelcatalog/datasheet/cost.go

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
core/providers/anthropic/anthropic.go (1)

970-971: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Minor: use schemas.Ptr for pointer consistency.

accumulateAnthropicResponsesUsage (Line 633/640) uses schemas.Ptr(n) for the same field, but here the address operator is used. Aligning both keeps the pattern consistent within this change.

-				if n := usageToProcess.ServerToolUse.WebSearchRequests; usage.CompletionTokensDetails.NumSearchQueries == nil || n > *usage.CompletionTokensDetails.NumSearchQueries {
-					usage.CompletionTokensDetails.NumSearchQueries = &n
-				}
+				if n := usageToProcess.ServerToolUse.WebSearchRequests; usage.CompletionTokensDetails.NumSearchQueries == nil || n > *usage.CompletionTokensDetails.NumSearchQueries {
+					usage.CompletionTokensDetails.NumSearchQueries = schemas.Ptr(n)
+				}

Based on learnings, prefer bifrost.Ptr()/schemas.Ptr() over the address operator & even when & is syntactically valid, applied consistently across code paths.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@core/providers/anthropic/anthropic.go` around lines 970 - 971, The
NumSearchQueries assignment in accumulateAnthropicResponsesUsage is using the
address operator directly instead of the pointer helper used elsewhere in this
change. Update the WebSearchRequests handling in the Anthropic usage
accumulation path to use schemas.Ptr(n) when setting
CompletionTokensDetails.NumSearchQueries, matching the existing pointer style
used in the same function and keeping pointer creation consistent across code
paths.

Source: Learnings

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@core/providers/anthropic/anthropic.go`:
- Around line 970-971: The NumSearchQueries assignment in
accumulateAnthropicResponsesUsage is using the address operator directly instead
of the pointer helper used elsewhere in this change. Update the
WebSearchRequests handling in the Anthropic usage accumulation path to use
schemas.Ptr(n) when setting CompletionTokensDetails.NumSearchQueries, matching
the existing pointer style used in the same function and keeping pointer
creation consistent across code paths.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 5a9f7e52-a220-4800-9fc9-774e3cdfbf77

📥 Commits

Reviewing files that changed from the base of the PR and between 7b0bd1e and aa19027.

📒 Files selected for processing (17)
  • core/providers/anthropic/anthropic.go
  • core/providers/anthropic/chat.go
  • core/providers/anthropic/passthrough_usage.go
  • core/providers/anthropic/responses.go
  • core/providers/anthropic/utils_test.go
  • core/schemas/chatcompletions.go
  • core/schemas/passthrough.go
  • core/schemas/responses.go
  • framework/configstore/migrations.go
  • framework/configstore/rdb.go
  • framework/configstore/tables/modelpricing.go
  • framework/modelcatalog/datasheet/cost.go
  • framework/modelcatalog/datasheet/cost_test.go
  • framework/modelcatalog/datasheet/overrides.go
  • framework/modelcatalog/datasheet/types.go
  • ui/app/workspace/custom-pricing/overrides/pricingOverrideSheet.tsx
  • ui/lib/types/governance.ts
🚧 Files skipped from review as they are similar to previous changes (15)
  • ui/lib/types/governance.ts
  • core/schemas/responses.go
  • core/schemas/chatcompletions.go
  • framework/modelcatalog/datasheet/overrides.go
  • framework/configstore/tables/modelpricing.go
  • core/providers/anthropic/responses.go
  • framework/configstore/rdb.go
  • core/schemas/passthrough.go
  • framework/modelcatalog/datasheet/types.go
  • core/providers/anthropic/chat.go
  • core/providers/anthropic/passthrough_usage.go
  • ui/app/workspace/custom-pricing/overrides/pricingOverrideSheet.tsx
  • framework/configstore/migrations.go
  • framework/modelcatalog/datasheet/cost.go
  • framework/modelcatalog/datasheet/cost_test.go

akshaydeo
akshaydeo previously approved these changes Jul 9, 2026

akshaydeo commented Jul 9, 2026 •

Copy link
Copy Markdown
Contributor

Merge activity

  • Jul 9, 4:10 PM UTC: A user started a stack merge that includes this pull request via Graphite.
  • Jul 9, 4:12 PM UTC: Graphite rebased this pull request as part of a merge.
  • Jul 9, 4:13 PM UTC: Graphite couldn't merge this PR because it was not satisfying all requirements (PR does not have required approvals).
  • Jul 9, 4:27 PM UTC: A user started a stack merge that includes this pull request via Graphite.
  • Jul 9, 4:27 PM UTC: @akshaydeo merged this pull request with Graphite.

@akshaydeo
akshaydeo changed the base branch from 07-09-fix_annthropic_cache_rate_for_fast_mode to graphite-base/5072 July 9, 2026 16:10
@akshaydeo
akshaydeo changed the base branch from graphite-base/5072 to dev July 9, 2026 16:11
@akshaydeo
akshaydeo dismissed stale reviews from coderabbitai[bot] and themself July 9, 2026 16:11

The base branch was changed.

@akshaydeo
akshaydeo force-pushed the 07-09-fix_inference_geo_cost_on_anthropic branch from aa19027 to c1a6a91 Compare July 9, 2026 16:11
Comment thread framework/modelcatalog/datasheet/cost.go
@akshaydeo
akshaydeo merged commit c911b56 into dev Jul 9, 2026
15 of 16 checks passed
@akshaydeo
akshaydeo deleted the 07-09-fix_inference_geo_cost_on_anthropic branch July 9, 2026 16:27
akshaydeo pushed a commit that referenced this pull request Jul 10, 2026
## Summary

Adds support for Anthropic's data-residency billing feature (`inference_geo`). When Anthropic serves inference in the US (`inference_geo: "us"`), a 1.1x multiplier is applied to all token and cache costs. This PR propagates the `inference_geo` field through the full response pipeline (chat, responses, streaming, passthrough) and wires it into the cost computation layer, including a new database column and UI override field.

A secondary fix ensures Anthropic server-tool web search request counts (`ServerToolUse.WebSearchRequests`) are correctly forwarded and billed in both streaming and non-streaming paths, where previously the count could be lost when the terminal chunk overwrote the accumulated usage.

## Changes

- **`inference_geo` propagation**: `InferenceGeo *string` added to `BifrostChatResponse`, `BifrostResponsesResponse`, and `BifrostPassthroughUsage`. The field is forwarded from `AnthropicUsage` in all conversion paths: non-streaming chat, non-streaming responses, streaming chat (captured across events and set on the final chunk), streaming responses (`message_delta`), and passthrough.
- **Data-residency multiplier in cost computation**: `serviceTier` gains an `inferenceGeoUS bool` flag. `tierFromResponse` now accepts and evaluates `inferenceGeo`. `computeTextCost` applies `InferenceGeoUSMultiplier` to token/cache costs only — the flat per-search fee is intentionally excluded.
- **Database migration**: `migrationAddInferenceGeoMultiplierColumn` adds `inference_geo_us_multiplier` to `TableModelPricing`. The column is included in the pricing sync update list and mapped through `convertEntryToTablePricing` / `convertTablePricingToEntry`.
- **Web search billing fix**: `accumulateAnthropicResponsesUsage` now carries `ServerToolUse.WebSearchRequests` into the accumulator so the count survives the terminal-chunk overwrite on streamed Responses requests. The non-streaming chat converter also forwards the count into `CompletionTokensDetails.NumSearchQueries`.
- **UI**: `inference_geo_us_multiplier` added to the custom pricing override sheet and `PricingOverridePatch` TypeScript type.
- **Tests**: New unit tests cover the accumulator web-search fix, chat/responses converter forwarding of `InferenceGeo` and web search counts, multiplier application (including the no-multiplier-column safe no-op), and `tierFromResponse` geo detection (case-insensitive).

## Type of change

- [ ] Bug fix
- [x] Feature
- [ ] Refactor
- [ ] Documentation
- [ ] Chore/CI

## Affected areas

- [x] Core (Go)
- [ ] Transports (HTTP)
- [x] Providers/Integrations
- [ ] Plugins
- [x] UI (React)
- [ ] Docs

## How to test

```sh
# Core/Transports
go test ./core/providers/anthropic/... ./framework/modelcatalog/datasheet/...

# UI
cd ui
pnpm i || npm i
pnpm build || npm run build
```

To validate end-to-end: send a request to an Anthropic model with data residency enabled. The response should include `inference_geo: "us"` and the computed cost should reflect the 1.1x multiplier on token/cache costs, with the per-search fee unchanged.

## Breaking changes

- [ ] Yes
- [x] No

## Security considerations

No auth, secrets, or PII implications. The `inference_geo` value is sourced from Anthropic's API response and used only for billing computation.

## Checklist

- [ ] I read `docs/contributing/README.md` and followed the guidelines
- [x] I added/updated tests where appropriate
- [ ] I updated documentation where needed
- [x] I verified builds succeed (Go and UI)
- [ ] I verified the CI pipeline passes locally if applicable
akshaydeo pushed a commit that referenced this pull request Jul 10, 2026
## Summary

Adds end-to-end support for Anthropic's data-residency billing feature (`inference_geo`). When Anthropic serves a request with `inference_geo: "us"`, a 1.1× multiplier is applied to all token and cache costs. This PR propagates the `inference_geo` field from the Anthropic API response through all converter paths (chat, responses, streaming, passthrough) and wires it into the cost engine so US-residency requests bill correctly. It also fixes a related streaming bug where web search request counts were lost on the terminal chunk of streamed Responses requests, causing web search to go unbilled.

## Changes

- **`inference_geo` field propagation**: Added `InferenceGeo *string` to `BifrostChatResponse`, `BifrostResponsesResponse`, and `BifrostPassthroughUsage` schemas. All Anthropic converters (`ToBifrostChatResponse`, `ToBifrostResponsesResponse`, `ToBifrostResponsesStream`, passthrough usage) now forward `inference_geo` from the provider's usage block.
- **Streaming terminal chunk fix**: In both `HandleAnthropicChatCompletionStreaming` and `HandleAnthropicResponsesStream`, `servedSpeed` and `servedInferenceGeo` are captured across events and re-applied to the terminal chunk. The terminal `response.completed` chunk is built fresh with no served modifiers, so without this re-apply, fast-mode and data-residency billing silently fell back to standard rates.
- **Web search count accumulation fix**: `accumulateAnthropicResponsesUsage` now carries `ServerToolUse.WebSearchRequests` into both the response usage accumulator and the mirrored billed usage. Previously the per-event count was overwritten by the terminal chunk's fresh usage, causing web search to go unbilled on streamed Responses requests.
- **Cost engine**: `serviceTier` gains an `inferenceGeoUS bool` field. `tierFromResponse` accepts a third `inferenceGeo *string` argument and sets `inferenceGeoUS` when the value is `"us"` (case-insensitive). `computeTextCost` applies `InferenceGeoUSMultiplier` to all token/cache costs when `inferenceGeoUS` is true; the flat per-search fee is intentionally excluded from the multiplier.
- **Database migration**: Adds `inference_geo_us_multiplier` column to `TableModelPricing` via a new migration step. The column is included in pricing sync, override patching, and the entry↔table conversion helpers.
- **UI**: Exposes `inference_geo_us_multiplier` as a configurable field in the custom pricing override sheet.
- **Tests**: New unit tests cover the accumulator web-search fix, the chat and responses converters forwarding `inference_geo` and web search counts, the cost engine multiplier logic (including the safe no-op when the column is unpopulated), `tierFromResponse` for `inferenceGeoUS`, and an end-to-end streaming test (`TestResponsesStream_TerminalChunkCarriesServedModifiers`) that replays `message_start → message_delta → message_stop` and asserts the terminal billed chunk carries `speed`, `inference_geo`, web search count, and cache-creation tokens.

## Type of change

- [ ] Bug fix
- [x] Feature
- [ ] Refactor
- [ ] Documentation
- [ ] Chore/CI

## Affected areas

- [x] Core (Go)
- [ ] Transports (HTTP)
- [x] Providers/Integrations
- [ ] Plugins
- [x] UI (React)
- [ ] Docs

## How to test

```sh
# Core/Transports
go test ./core/providers/anthropic/... ./framework/modelcatalog/datasheet/...

# UI
cd ui
pnpm i
pnpm build
```

To validate end-to-end: send a chat or responses request to an Anthropic model with data residency enabled. Confirm the response carries `inference_geo: "us"` and that the billed cost reflects the 1.1× multiplier on token/cache costs with the per-search fee unchanged. For streaming, confirm the terminal chunk carries both `speed` and `inference_geo` after `message_stop`.

The `inference_geo_us_multiplier` column must be populated in the model pricing datasheet for the multiplier to apply; if the column is `null`, billing is a safe no-op (standard rates).

## Breaking changes

- [x] No

`tierFromResponse` gains a third parameter but is an internal function. All call sites have been updated.

## Related issues

## Security considerations

No auth, secrets, or PII implications. The multiplier is a pricing configuration value stored in the existing model pricing table with the same access controls as other pricing columns.

## Checklist

- [ ] I read `docs/contributing/README.md` and followed the guidelines
- [x] I added/updated tests where appropriate
- [ ] I updated documentation where needed
- [x] I verified builds succeed (Go and UI)
- [ ] I verified the CI pipeline passes locally if applicable
akshaydeo pushed a commit that referenced this pull request Jul 14, 2026
Adds support for Anthropic's data-residency billing feature (`inference_geo`). When Anthropic serves inference in the US (`inference_geo: "us"`), a 1.1x multiplier is applied to all token and cache costs. This PR propagates the `inference_geo` field through the full response pipeline (chat, responses, streaming, passthrough) and wires it into the cost computation layer, including a new database column and UI override field.

A secondary fix ensures Anthropic server-tool web search request counts (`ServerToolUse.WebSearchRequests`) are correctly forwarded and billed in both streaming and non-streaming paths, where previously the count could be lost when the terminal chunk overwrote the accumulated usage.

- **`inference_geo` propagation**: `InferenceGeo *string` added to `BifrostChatResponse`, `BifrostResponsesResponse`, and `BifrostPassthroughUsage`. The field is forwarded from `AnthropicUsage` in all conversion paths: non-streaming chat, non-streaming responses, streaming chat (captured across events and set on the final chunk), streaming responses (`message_delta`), and passthrough.
- **Data-residency multiplier in cost computation**: `serviceTier` gains an `inferenceGeoUS bool` flag. `tierFromResponse` now accepts and evaluates `inferenceGeo`. `computeTextCost` applies `InferenceGeoUSMultiplier` to token/cache costs only — the flat per-search fee is intentionally excluded.
- **Database migration**: `migrationAddInferenceGeoMultiplierColumn` adds `inference_geo_us_multiplier` to `TableModelPricing`. The column is included in the pricing sync update list and mapped through `convertEntryToTablePricing` / `convertTablePricingToEntry`.
- **Web search billing fix**: `accumulateAnthropicResponsesUsage` now carries `ServerToolUse.WebSearchRequests` into the accumulator so the count survives the terminal-chunk overwrite on streamed Responses requests. The non-streaming chat converter also forwards the count into `CompletionTokensDetails.NumSearchQueries`.
- **UI**: `inference_geo_us_multiplier` added to the custom pricing override sheet and `PricingOverridePatch` TypeScript type.
- **Tests**: New unit tests cover the accumulator web-search fix, chat/responses converter forwarding of `InferenceGeo` and web search counts, multiplier application (including the no-multiplier-column safe no-op), and `tierFromResponse` geo detection (case-insensitive).

- [ ] Bug fix
- [x] Feature
- [ ] Refactor
- [ ] Documentation
- [ ] Chore/CI

- [x] Core (Go)
- [ ] Transports (HTTP)
- [x] Providers/Integrations
- [ ] Plugins
- [x] UI (React)
- [ ] Docs

```sh
go test ./core/providers/anthropic/... ./framework/modelcatalog/datasheet/...

cd ui
pnpm i || npm i
pnpm build || npm run build
```

To validate end-to-end: send a request to an Anthropic model with data residency enabled. The response should include `inference_geo: "us"` and the computed cost should reflect the 1.1x multiplier on token/cache costs, with the per-search fee unchanged.

- [ ] Yes
- [x] No

No auth, secrets, or PII implications. The `inference_geo` value is sourced from Anthropic's API response and used only for billing computation.

- [ ] I read `docs/contributing/README.md` and followed the guidelines
- [x] I added/updated tests where appropriate
- [ ] I updated documentation where needed
- [x] I verified builds succeed (Go and UI)
- [ ] I verified the CI pipeline passes locally if applicable
akshaydeo pushed a commit that referenced this pull request Jul 14, 2026
Adds support for Anthropic's data-residency billing feature (`inference_geo`). When Anthropic serves inference in the US (`inference_geo: "us"`), a 1.1x multiplier is applied to all token and cache costs. This PR propagates the `inference_geo` field through the full response pipeline (chat, responses, streaming, passthrough) and wires it into the cost computation layer, including a new database column and UI override field.

A secondary fix ensures Anthropic server-tool web search request counts (`ServerToolUse.WebSearchRequests`) are correctly forwarded and billed in both streaming and non-streaming paths, where previously the count could be lost when the terminal chunk overwrote the accumulated usage.

- **`inference_geo` propagation**: `InferenceGeo *string` added to `BifrostChatResponse`, `BifrostResponsesResponse`, and `BifrostPassthroughUsage`. The field is forwarded from `AnthropicUsage` in all conversion paths: non-streaming chat, non-streaming responses, streaming chat (captured across events and set on the final chunk), streaming responses (`message_delta`), and passthrough.
- **Data-residency multiplier in cost computation**: `serviceTier` gains an `inferenceGeoUS bool` flag. `tierFromResponse` now accepts and evaluates `inferenceGeo`. `computeTextCost` applies `InferenceGeoUSMultiplier` to token/cache costs only — the flat per-search fee is intentionally excluded.
- **Database migration**: `migrationAddInferenceGeoMultiplierColumn` adds `inference_geo_us_multiplier` to `TableModelPricing`. The column is included in the pricing sync update list and mapped through `convertEntryToTablePricing` / `convertTablePricingToEntry`.
- **Web search billing fix**: `accumulateAnthropicResponsesUsage` now carries `ServerToolUse.WebSearchRequests` into the accumulator so the count survives the terminal-chunk overwrite on streamed Responses requests. The non-streaming chat converter also forwards the count into `CompletionTokensDetails.NumSearchQueries`.
- **UI**: `inference_geo_us_multiplier` added to the custom pricing override sheet and `PricingOverridePatch` TypeScript type.
- **Tests**: New unit tests cover the accumulator web-search fix, chat/responses converter forwarding of `InferenceGeo` and web search counts, multiplier application (including the no-multiplier-column safe no-op), and `tierFromResponse` geo detection (case-insensitive).

- [ ] Bug fix
- [x] Feature
- [ ] Refactor
- [ ] Documentation
- [ ] Chore/CI

- [x] Core (Go)
- [ ] Transports (HTTP)
- [x] Providers/Integrations
- [ ] Plugins
- [x] UI (React)
- [ ] Docs

```sh
go test ./core/providers/anthropic/... ./framework/modelcatalog/datasheet/...

cd ui
pnpm i || npm i
pnpm build || npm run build
```

To validate end-to-end: send a request to an Anthropic model with data residency enabled. The response should include `inference_geo: "us"` and the computed cost should reflect the 1.1x multiplier on token/cache costs, with the per-search fee unchanged.

- [ ] Yes
- [x] No

No auth, secrets, or PII implications. The `inference_geo` value is sourced from Anthropic's API response and used only for billing computation.

- [ ] I read `docs/contributing/README.md` and followed the guidelines
- [x] I added/updated tests where appropriate
- [ ] I updated documentation where needed
- [x] I verified builds succeed (Go and UI)
- [ ] I verified the CI pipeline passes locally if applicable
akshaydeo pushed a commit that referenced this pull request Jul 24, 2026
Adds support for Anthropic's data-residency billing feature (`inference_geo`). When Anthropic serves inference in the US (`inference_geo: "us"`), a 1.1x multiplier is applied to all token and cache costs. This PR propagates the `inference_geo` field through the full response pipeline (chat, responses, streaming, passthrough) and wires it into the cost computation layer, including a new database column and UI override field.

A secondary fix ensures Anthropic server-tool web search request counts (`ServerToolUse.WebSearchRequests`) are correctly forwarded and billed in both streaming and non-streaming paths, where previously the count could be lost when the terminal chunk overwrote the accumulated usage.

- **`inference_geo` propagation**: `InferenceGeo *string` added to `BifrostChatResponse`, `BifrostResponsesResponse`, and `BifrostPassthroughUsage`. The field is forwarded from `AnthropicUsage` in all conversion paths: non-streaming chat, non-streaming responses, streaming chat (captured across events and set on the final chunk), streaming responses (`message_delta`), and passthrough.
- **Data-residency multiplier in cost computation**: `serviceTier` gains an `inferenceGeoUS bool` flag. `tierFromResponse` now accepts and evaluates `inferenceGeo`. `computeTextCost` applies `InferenceGeoUSMultiplier` to token/cache costs only — the flat per-search fee is intentionally excluded.
- **Database migration**: `migrationAddInferenceGeoMultiplierColumn` adds `inference_geo_us_multiplier` to `TableModelPricing`. The column is included in the pricing sync update list and mapped through `convertEntryToTablePricing` / `convertTablePricingToEntry`.
- **Web search billing fix**: `accumulateAnthropicResponsesUsage` now carries `ServerToolUse.WebSearchRequests` into the accumulator so the count survives the terminal-chunk overwrite on streamed Responses requests. The non-streaming chat converter also forwards the count into `CompletionTokensDetails.NumSearchQueries`.
- **UI**: `inference_geo_us_multiplier` added to the custom pricing override sheet and `PricingOverridePatch` TypeScript type.
- **Tests**: New unit tests cover the accumulator web-search fix, chat/responses converter forwarding of `InferenceGeo` and web search counts, multiplier application (including the no-multiplier-column safe no-op), and `tierFromResponse` geo detection (case-insensitive).

- [ ] Bug fix
- [x] Feature
- [ ] Refactor
- [ ] Documentation
- [ ] Chore/CI

- [x] Core (Go)
- [ ] Transports (HTTP)
- [x] Providers/Integrations
- [ ] Plugins
- [x] UI (React)
- [ ] Docs

```sh
go test ./core/providers/anthropic/... ./framework/modelcatalog/datasheet/...

cd ui
pnpm i || npm i
pnpm build || npm run build
```

To validate end-to-end: send a request to an Anthropic model with data residency enabled. The response should include `inference_geo: "us"` and the computed cost should reflect the 1.1x multiplier on token/cache costs, with the per-search fee unchanged.

- [ ] Yes
- [x] No

No auth, secrets, or PII implications. The `inference_geo` value is sourced from Anthropic's API response and used only for billing computation.

- [ ] I read `docs/contributing/README.md` and followed the guidelines
- [x] I added/updated tests where appropriate
- [ ] I updated documentation where needed
- [x] I verified builds succeed (Go and UI)
- [ ] I verified the CI pipeline passes locally if applicable
akshaydeo pushed a commit that referenced this pull request Jul 29, 2026
Adds support for Anthropic's data-residency billing feature (`inference_geo`). When Anthropic serves inference in the US (`inference_geo: "us"`), a 1.1x multiplier is applied to all token and cache costs. This PR propagates the `inference_geo` field through the full response pipeline (chat, responses, streaming, passthrough) and wires it into the cost computation layer, including a new database column and UI override field.

A secondary fix ensures Anthropic server-tool web search request counts (`ServerToolUse.WebSearchRequests`) are correctly forwarded and billed in both streaming and non-streaming paths, where previously the count could be lost when the terminal chunk overwrote the accumulated usage.

- **`inference_geo` propagation**: `InferenceGeo *string` added to `BifrostChatResponse`, `BifrostResponsesResponse`, and `BifrostPassthroughUsage`. The field is forwarded from `AnthropicUsage` in all conversion paths: non-streaming chat, non-streaming responses, streaming chat (captured across events and set on the final chunk), streaming responses (`message_delta`), and passthrough.
- **Data-residency multiplier in cost computation**: `serviceTier` gains an `inferenceGeoUS bool` flag. `tierFromResponse` now accepts and evaluates `inferenceGeo`. `computeTextCost` applies `InferenceGeoUSMultiplier` to token/cache costs only — the flat per-search fee is intentionally excluded.
- **Database migration**: `migrationAddInferenceGeoMultiplierColumn` adds `inference_geo_us_multiplier` to `TableModelPricing`. The column is included in the pricing sync update list and mapped through `convertEntryToTablePricing` / `convertTablePricingToEntry`.
- **Web search billing fix**: `accumulateAnthropicResponsesUsage` now carries `ServerToolUse.WebSearchRequests` into the accumulator so the count survives the terminal-chunk overwrite on streamed Responses requests. The non-streaming chat converter also forwards the count into `CompletionTokensDetails.NumSearchQueries`.
- **UI**: `inference_geo_us_multiplier` added to the custom pricing override sheet and `PricingOverridePatch` TypeScript type.
- **Tests**: New unit tests cover the accumulator web-search fix, chat/responses converter forwarding of `InferenceGeo` and web search counts, multiplier application (including the no-multiplier-column safe no-op), and `tierFromResponse` geo detection (case-insensitive).

- [ ] Bug fix
- [x] Feature
- [ ] Refactor
- [ ] Documentation
- [ ] Chore/CI

- [x] Core (Go)
- [ ] Transports (HTTP)
- [x] Providers/Integrations
- [ ] Plugins
- [x] UI (React)
- [ ] Docs

```sh
go test ./core/providers/anthropic/... ./framework/modelcatalog/datasheet/...

cd ui
pnpm i || npm i
pnpm build || npm run build
```

To validate end-to-end: send a request to an Anthropic model with data residency enabled. The response should include `inference_geo: "us"` and the computed cost should reflect the 1.1x multiplier on token/cache costs, with the per-search fee unchanged.

- [ ] Yes
- [x] No

No auth, secrets, or PII implications. The `inference_geo` value is sourced from Anthropic's API response and used only for billing computation.

- [ ] I read `docs/contributing/README.md` and followed the guidelines
- [x] I added/updated tests where appropriate
- [ ] I updated documentation where needed
- [x] I verified builds succeed (Go and UI)
- [ ] I verified the CI pipeline passes locally if applicable
akshaydeo pushed a commit that referenced this pull request Jul 30, 2026
Adds support for Anthropic's data-residency billing feature (`inference_geo`). When Anthropic serves inference in the US (`inference_geo: "us"`), a 1.1x multiplier is applied to all token and cache costs. This PR propagates the `inference_geo` field through the full response pipeline (chat, responses, streaming, passthrough) and wires it into the cost computation layer, including a new database column and UI override field.

A secondary fix ensures Anthropic server-tool web search request counts (`ServerToolUse.WebSearchRequests`) are correctly forwarded and billed in both streaming and non-streaming paths, where previously the count could be lost when the terminal chunk overwrote the accumulated usage.

- **`inference_geo` propagation**: `InferenceGeo *string` added to `BifrostChatResponse`, `BifrostResponsesResponse`, and `BifrostPassthroughUsage`. The field is forwarded from `AnthropicUsage` in all conversion paths: non-streaming chat, non-streaming responses, streaming chat (captured across events and set on the final chunk), streaming responses (`message_delta`), and passthrough.
- **Data-residency multiplier in cost computation**: `serviceTier` gains an `inferenceGeoUS bool` flag. `tierFromResponse` now accepts and evaluates `inferenceGeo`. `computeTextCost` applies `InferenceGeoUSMultiplier` to token/cache costs only — the flat per-search fee is intentionally excluded.
- **Database migration**: `migrationAddInferenceGeoMultiplierColumn` adds `inference_geo_us_multiplier` to `TableModelPricing`. The column is included in the pricing sync update list and mapped through `convertEntryToTablePricing` / `convertTablePricingToEntry`.
- **Web search billing fix**: `accumulateAnthropicResponsesUsage` now carries `ServerToolUse.WebSearchRequests` into the accumulator so the count survives the terminal-chunk overwrite on streamed Responses requests. The non-streaming chat converter also forwards the count into `CompletionTokensDetails.NumSearchQueries`.
- **UI**: `inference_geo_us_multiplier` added to the custom pricing override sheet and `PricingOverridePatch` TypeScript type.
- **Tests**: New unit tests cover the accumulator web-search fix, chat/responses converter forwarding of `InferenceGeo` and web search counts, multiplier application (including the no-multiplier-column safe no-op), and `tierFromResponse` geo detection (case-insensitive).

- [ ] Bug fix
- [x] Feature
- [ ] Refactor
- [ ] Documentation
- [ ] Chore/CI

- [x] Core (Go)
- [ ] Transports (HTTP)
- [x] Providers/Integrations
- [ ] Plugins
- [x] UI (React)
- [ ] Docs

```sh
go test ./core/providers/anthropic/... ./framework/modelcatalog/datasheet/...

cd ui
pnpm i || npm i
pnpm build || npm run build
```

To validate end-to-end: send a request to an Anthropic model with data residency enabled. The response should include `inference_geo: "us"` and the computed cost should reflect the 1.1x multiplier on token/cache costs, with the per-search fee unchanged.

- [ ] Yes
- [x] No

No auth, secrets, or PII implications. The `inference_geo` value is sourced from Anthropic's API response and used only for billing computation.

- [ ] I read `docs/contributing/README.md` and followed the guidelines
- [x] I added/updated tests where appropriate
- [ ] I updated documentation where needed
- [x] I verified builds succeed (Go and UI)
- [ ] I verified the CI pipeline passes locally if applicable
akshaydeo pushed a commit that referenced this pull request Jul 30, 2026
Adds support for Anthropic's data-residency billing feature (`inference_geo`). When Anthropic serves inference in the US (`inference_geo: "us"`), a 1.1x multiplier is applied to all token and cache costs. This PR propagates the `inference_geo` field through the full response pipeline (chat, responses, streaming, passthrough) and wires it into the cost computation layer, including a new database column and UI override field.

A secondary fix ensures Anthropic server-tool web search request counts (`ServerToolUse.WebSearchRequests`) are correctly forwarded and billed in both streaming and non-streaming paths, where previously the count could be lost when the terminal chunk overwrote the accumulated usage.

- **`inference_geo` propagation**: `InferenceGeo *string` added to `BifrostChatResponse`, `BifrostResponsesResponse`, and `BifrostPassthroughUsage`. The field is forwarded from `AnthropicUsage` in all conversion paths: non-streaming chat, non-streaming responses, streaming chat (captured across events and set on the final chunk), streaming responses (`message_delta`), and passthrough.
- **Data-residency multiplier in cost computation**: `serviceTier` gains an `inferenceGeoUS bool` flag. `tierFromResponse` now accepts and evaluates `inferenceGeo`. `computeTextCost` applies `InferenceGeoUSMultiplier` to token/cache costs only — the flat per-search fee is intentionally excluded.
- **Database migration**: `migrationAddInferenceGeoMultiplierColumn` adds `inference_geo_us_multiplier` to `TableModelPricing`. The column is included in the pricing sync update list and mapped through `convertEntryToTablePricing` / `convertTablePricingToEntry`.
- **Web search billing fix**: `accumulateAnthropicResponsesUsage` now carries `ServerToolUse.WebSearchRequests` into the accumulator so the count survives the terminal-chunk overwrite on streamed Responses requests. The non-streaming chat converter also forwards the count into `CompletionTokensDetails.NumSearchQueries`.
- **UI**: `inference_geo_us_multiplier` added to the custom pricing override sheet and `PricingOverridePatch` TypeScript type.
- **Tests**: New unit tests cover the accumulator web-search fix, chat/responses converter forwarding of `InferenceGeo` and web search counts, multiplier application (including the no-multiplier-column safe no-op), and `tierFromResponse` geo detection (case-insensitive).

- [ ] Bug fix
- [x] Feature
- [ ] Refactor
- [ ] Documentation
- [ ] Chore/CI

- [x] Core (Go)
- [ ] Transports (HTTP)
- [x] Providers/Integrations
- [ ] Plugins
- [x] UI (React)
- [ ] Docs

```sh
go test ./core/providers/anthropic/... ./framework/modelcatalog/datasheet/...

cd ui
pnpm i || npm i
pnpm build || npm run build
```

To validate end-to-end: send a request to an Anthropic model with data residency enabled. The response should include `inference_geo: "us"` and the computed cost should reflect the 1.1x multiplier on token/cache costs, with the per-search fee unchanged.

- [ ] Yes
- [x] No

No auth, secrets, or PII implications. The `inference_geo` value is sourced from Anthropic's API response and used only for billing computation.

- [ ] I read `docs/contributing/README.md` and followed the guidelines
- [x] I added/updated tests where appropriate
- [ ] I updated documentation where needed
- [x] I verified builds succeed (Go and UI)
- [ ] I verified the CI pipeline passes locally if applicable
akshaydeo pushed a commit that referenced this pull request Jul 30, 2026
Adds support for Anthropic's data-residency billing feature (`inference_geo`). When Anthropic serves inference in the US (`inference_geo: "us"`), a 1.1x multiplier is applied to all token and cache costs. This PR propagates the `inference_geo` field through the full response pipeline (chat, responses, streaming, passthrough) and wires it into the cost computation layer, including a new database column and UI override field.

A secondary fix ensures Anthropic server-tool web search request counts (`ServerToolUse.WebSearchRequests`) are correctly forwarded and billed in both streaming and non-streaming paths, where previously the count could be lost when the terminal chunk overwrote the accumulated usage.

- **`inference_geo` propagation**: `InferenceGeo *string` added to `BifrostChatResponse`, `BifrostResponsesResponse`, and `BifrostPassthroughUsage`. The field is forwarded from `AnthropicUsage` in all conversion paths: non-streaming chat, non-streaming responses, streaming chat (captured across events and set on the final chunk), streaming responses (`message_delta`), and passthrough.
- **Data-residency multiplier in cost computation**: `serviceTier` gains an `inferenceGeoUS bool` flag. `tierFromResponse` now accepts and evaluates `inferenceGeo`. `computeTextCost` applies `InferenceGeoUSMultiplier` to token/cache costs only — the flat per-search fee is intentionally excluded.
- **Database migration**: `migrationAddInferenceGeoMultiplierColumn` adds `inference_geo_us_multiplier` to `TableModelPricing`. The column is included in the pricing sync update list and mapped through `convertEntryToTablePricing` / `convertTablePricingToEntry`.
- **Web search billing fix**: `accumulateAnthropicResponsesUsage` now carries `ServerToolUse.WebSearchRequests` into the accumulator so the count survives the terminal-chunk overwrite on streamed Responses requests. The non-streaming chat converter also forwards the count into `CompletionTokensDetails.NumSearchQueries`.
- **UI**: `inference_geo_us_multiplier` added to the custom pricing override sheet and `PricingOverridePatch` TypeScript type.
- **Tests**: New unit tests cover the accumulator web-search fix, chat/responses converter forwarding of `InferenceGeo` and web search counts, multiplier application (including the no-multiplier-column safe no-op), and `tierFromResponse` geo detection (case-insensitive).

- [ ] Bug fix
- [x] Feature
- [ ] Refactor
- [ ] Documentation
- [ ] Chore/CI

- [x] Core (Go)
- [ ] Transports (HTTP)
- [x] Providers/Integrations
- [ ] Plugins
- [x] UI (React)
- [ ] Docs

```sh
go test ./core/providers/anthropic/... ./framework/modelcatalog/datasheet/...

cd ui
pnpm i || npm i
pnpm build || npm run build
```

To validate end-to-end: send a request to an Anthropic model with data residency enabled. The response should include `inference_geo: "us"` and the computed cost should reflect the 1.1x multiplier on token/cache costs, with the per-search fee unchanged.

- [ ] Yes
- [x] No

No auth, secrets, or PII implications. The `inference_geo` value is sourced from Anthropic's API response and used only for billing computation.

- [ ] I read `docs/contributing/README.md` and followed the guidelines
- [x] I added/updated tests where appropriate
- [ ] I updated documentation where needed
- [x] I verified builds succeed (Go and UI)
- [ ] I verified the CI pipeline passes locally if applicable
akhsaul pushed a commit to akhsaul/bifrost that referenced this pull request Aug 27, 2026
## Summary

Adds end-to-end support for Anthropic's data-residency billing feature (`inference_geo`). When Anthropic serves a request with `inference_geo: "us"`, a 1.1× multiplier is applied to all token and cache costs. This PR propagates the `inference_geo` field from the Anthropic API response through all converter paths (chat, responses, streaming, passthrough) and wires it into the cost engine so US-residency requests bill correctly. It also fixes a related streaming bug where web search request counts were lost on the terminal chunk of streamed Responses requests, causing web search to go unbilled.

## Changes

- **`inference_geo` field propagation**: Added `InferenceGeo *string` to `BifrostChatResponse`, `BifrostResponsesResponse`, and `BifrostPassthroughUsage` schemas. All Anthropic converters (`ToBifrostChatResponse`, `ToBifrostResponsesResponse`, `ToBifrostResponsesStream`, passthrough usage) now forward `inference_geo` from the provider's usage block.
- **Streaming terminal chunk fix**: In both `HandleAnthropicChatCompletionStreaming` and `HandleAnthropicResponsesStream`, `servedSpeed` and `servedInferenceGeo` are captured across events and re-applied to the terminal chunk. The terminal `response.completed` chunk is built fresh with no served modifiers, so without this re-apply, fast-mode and data-residency billing silently fell back to standard rates.
- **Web search count accumulation fix**: `accumulateAnthropicResponsesUsage` now carries `ServerToolUse.WebSearchRequests` into both the response usage accumulator and the mirrored billed usage. Previously the per-event count was overwritten by the terminal chunk's fresh usage, causing web search to go unbilled on streamed Responses requests.
- **Cost engine**: `serviceTier` gains an `inferenceGeoUS bool` field. `tierFromResponse` accepts a third `inferenceGeo *string` argument and sets `inferenceGeoUS` when the value is `"us"` (case-insensitive). `computeTextCost` applies `InferenceGeoUSMultiplier` to all token/cache costs when `inferenceGeoUS` is true; the flat per-search fee is intentionally excluded from the multiplier.
- **Database migration**: Adds `inference_geo_us_multiplier` column to `TableModelPricing` via a new migration step. The column is included in pricing sync, override patching, and the entry↔table conversion helpers.
- **UI**: Exposes `inference_geo_us_multiplier` as a configurable field in the custom pricing override sheet.
- **Tests**: New unit tests cover the accumulator web-search fix, the chat and responses converters forwarding `inference_geo` and web search counts, the cost engine multiplier logic (including the safe no-op when the column is unpopulated), `tierFromResponse` for `inferenceGeoUS`, and an end-to-end streaming test (`TestResponsesStream_TerminalChunkCarriesServedModifiers`) that replays `message_start → message_delta → message_stop` and asserts the terminal billed chunk carries `speed`, `inference_geo`, web search count, and cache-creation tokens.

## Type of change

- [ ] Bug fix
- [x] Feature
- [ ] Refactor
- [ ] Documentation
- [ ] Chore/CI

## Affected areas

- [x] Core (Go)
- [ ] Transports (HTTP)
- [x] Providers/Integrations
- [ ] Plugins
- [x] UI (React)
- [ ] Docs

## How to test

```sh
# Core/Transports
go test ./core/providers/anthropic/... ./framework/modelcatalog/datasheet/...

# UI
cd ui
pnpm i
pnpm build
```

To validate end-to-end: send a chat or responses request to an Anthropic model with data residency enabled. Confirm the response carries `inference_geo: "us"` and that the billed cost reflects the 1.1× multiplier on token/cache costs with the per-search fee unchanged. For streaming, confirm the terminal chunk carries both `speed` and `inference_geo` after `message_stop`.

The `inference_geo_us_multiplier` column must be populated in the model pricing datasheet for the multiplier to apply; if the column is `null`, billing is a safe no-op (standard rates).

## Breaking changes

- [x] No

`tierFromResponse` gains a third parameter but is an internal function. All call sites have been updated.

## Related issues

## Security considerations

No auth, secrets, or PII implications. The multiplier is a pricing configuration value stored in the existing model pricing table with the same access controls as other pricing columns.

## Checklist

- [ ] I read `docs/contributing/README.md` and followed the guidelines
- [x] I added/updated tests where appropriate
- [ ] I updated documentation where needed
- [x] I verified builds succeed (Go and UI)
- [ ] I verified the CI pipeline passes locally if applicable
akhsaul pushed a commit to akhsaul/bifrost that referenced this pull request Aug 27, 2026
Adds support for Anthropic's data-residency billing feature (`inference_geo`). When Anthropic serves inference in the US (`inference_geo: "us"`), a 1.1x multiplier is applied to all token and cache costs. This PR propagates the `inference_geo` field through the full response pipeline (chat, responses, streaming, passthrough) and wires it into the cost computation layer, including a new database column and UI override field.

A secondary fix ensures Anthropic server-tool web search request counts (`ServerToolUse.WebSearchRequests`) are correctly forwarded and billed in both streaming and non-streaming paths, where previously the count could be lost when the terminal chunk overwrote the accumulated usage.

- **`inference_geo` propagation**: `InferenceGeo *string` added to `BifrostChatResponse`, `BifrostResponsesResponse`, and `BifrostPassthroughUsage`. The field is forwarded from `AnthropicUsage` in all conversion paths: non-streaming chat, non-streaming responses, streaming chat (captured across events and set on the final chunk), streaming responses (`message_delta`), and passthrough.
- **Data-residency multiplier in cost computation**: `serviceTier` gains an `inferenceGeoUS bool` flag. `tierFromResponse` now accepts and evaluates `inferenceGeo`. `computeTextCost` applies `InferenceGeoUSMultiplier` to token/cache costs only — the flat per-search fee is intentionally excluded.
- **Database migration**: `migrationAddInferenceGeoMultiplierColumn` adds `inference_geo_us_multiplier` to `TableModelPricing`. The column is included in the pricing sync update list and mapped through `convertEntryToTablePricing` / `convertTablePricingToEntry`.
- **Web search billing fix**: `accumulateAnthropicResponsesUsage` now carries `ServerToolUse.WebSearchRequests` into the accumulator so the count survives the terminal-chunk overwrite on streamed Responses requests. The non-streaming chat converter also forwards the count into `CompletionTokensDetails.NumSearchQueries`.
- **UI**: `inference_geo_us_multiplier` added to the custom pricing override sheet and `PricingOverridePatch` TypeScript type.
- **Tests**: New unit tests cover the accumulator web-search fix, chat/responses converter forwarding of `InferenceGeo` and web search counts, multiplier application (including the no-multiplier-column safe no-op), and `tierFromResponse` geo detection (case-insensitive).

- [ ] Bug fix
- [x] Feature
- [ ] Refactor
- [ ] Documentation
- [ ] Chore/CI

- [x] Core (Go)
- [ ] Transports (HTTP)
- [x] Providers/Integrations
- [ ] Plugins
- [x] UI (React)
- [ ] Docs

```sh
go test ./core/providers/anthropic/... ./framework/modelcatalog/datasheet/...

cd ui
pnpm i || npm i
pnpm build || npm run build
```

To validate end-to-end: send a request to an Anthropic model with data residency enabled. The response should include `inference_geo: "us"` and the computed cost should reflect the 1.1x multiplier on token/cache costs, with the per-search fee unchanged.

- [ ] Yes
- [x] No

No auth, secrets, or PII implications. The `inference_geo` value is sourced from Anthropic's API response and used only for billing computation.

- [ ] I read `docs/contributing/README.md` and followed the guidelines
- [x] I added/updated tests where appropriate
- [ ] I updated documentation where needed
- [x] I verified builds succeed (Go and UI)
- [ ] I verified the CI pipeline passes locally if applicable
occcat pushed a commit to occcat/bifrost that referenced this pull request Sep 2, 2026
## Summary

Adds end-to-end support for Anthropic's data-residency billing feature (`inference_geo`). When Anthropic serves a request with `inference_geo: "us"`, a 1.1× multiplier is applied to all token and cache costs. This PR propagates the `inference_geo` field from the Anthropic API response through all converter paths (chat, responses, streaming, passthrough) and wires it into the cost engine so US-residency requests bill correctly. It also fixes a related streaming bug where web search request counts were lost on the terminal chunk of streamed Responses requests, causing web search to go unbilled.

## Changes

- **`inference_geo` field propagation**: Added `InferenceGeo *string` to `BifrostChatResponse`, `BifrostResponsesResponse`, and `BifrostPassthroughUsage` schemas. All Anthropic converters (`ToBifrostChatResponse`, `ToBifrostResponsesResponse`, `ToBifrostResponsesStream`, passthrough usage) now forward `inference_geo` from the provider's usage block.
- **Streaming terminal chunk fix**: In both `HandleAnthropicChatCompletionStreaming` and `HandleAnthropicResponsesStream`, `servedSpeed` and `servedInferenceGeo` are captured across events and re-applied to the terminal chunk. The terminal `response.completed` chunk is built fresh with no served modifiers, so without this re-apply, fast-mode and data-residency billing silently fell back to standard rates.
- **Web search count accumulation fix**: `accumulateAnthropicResponsesUsage` now carries `ServerToolUse.WebSearchRequests` into both the response usage accumulator and the mirrored billed usage. Previously the per-event count was overwritten by the terminal chunk's fresh usage, causing web search to go unbilled on streamed Responses requests.
- **Cost engine**: `serviceTier` gains an `inferenceGeoUS bool` field. `tierFromResponse` accepts a third `inferenceGeo *string` argument and sets `inferenceGeoUS` when the value is `"us"` (case-insensitive). `computeTextCost` applies `InferenceGeoUSMultiplier` to all token/cache costs when `inferenceGeoUS` is true; the flat per-search fee is intentionally excluded from the multiplier.
- **Database migration**: Adds `inference_geo_us_multiplier` column to `TableModelPricing` via a new migration step. The column is included in pricing sync, override patching, and the entry↔table conversion helpers.
- **UI**: Exposes `inference_geo_us_multiplier` as a configurable field in the custom pricing override sheet.
- **Tests**: New unit tests cover the accumulator web-search fix, the chat and responses converters forwarding `inference_geo` and web search counts, the cost engine multiplier logic (including the safe no-op when the column is unpopulated), `tierFromResponse` for `inferenceGeoUS`, and an end-to-end streaming test (`TestResponsesStream_TerminalChunkCarriesServedModifiers`) that replays `message_start → message_delta → message_stop` and asserts the terminal billed chunk carries `speed`, `inference_geo`, web search count, and cache-creation tokens.

## Type of change

- [ ] Bug fix
- [x] Feature
- [ ] Refactor
- [ ] Documentation
- [ ] Chore/CI

## Affected areas

- [x] Core (Go)
- [ ] Transports (HTTP)
- [x] Providers/Integrations
- [ ] Plugins
- [x] UI (React)
- [ ] Docs

## How to test

```sh
# Core/Transports
go test ./core/providers/anthropic/... ./framework/modelcatalog/datasheet/...

# UI
cd ui
pnpm i
pnpm build
```

To validate end-to-end: send a chat or responses request to an Anthropic model with data residency enabled. Confirm the response carries `inference_geo: "us"` and that the billed cost reflects the 1.1× multiplier on token/cache costs with the per-search fee unchanged. For streaming, confirm the terminal chunk carries both `speed` and `inference_geo` after `message_stop`.

The `inference_geo_us_multiplier` column must be populated in the model pricing datasheet for the multiplier to apply; if the column is `null`, billing is a safe no-op (standard rates).

## Breaking changes

- [x] No

`tierFromResponse` gains a third parameter but is an internal function. All call sites have been updated.

## Related issues

## Security considerations

No auth, secrets, or PII implications. The multiplier is a pricing configuration value stored in the existing model pricing table with the same access controls as other pricing columns.

## Checklist

- [ ] I read `docs/contributing/README.md` and followed the guidelines
- [x] I added/updated tests where appropriate
- [ ] I updated documentation where needed
- [x] I verified builds succeed (Go and UI)
- [ ] I verified the CI pipeline passes locally if applicable
occcat pushed a commit to occcat/bifrost that referenced this pull request Sep 2, 2026
Adds support for Anthropic's data-residency billing feature (`inference_geo`). When Anthropic serves inference in the US (`inference_geo: "us"`), a 1.1x multiplier is applied to all token and cache costs. This PR propagates the `inference_geo` field through the full response pipeline (chat, responses, streaming, passthrough) and wires it into the cost computation layer, including a new database column and UI override field.

A secondary fix ensures Anthropic server-tool web search request counts (`ServerToolUse.WebSearchRequests`) are correctly forwarded and billed in both streaming and non-streaming paths, where previously the count could be lost when the terminal chunk overwrote the accumulated usage.

- **`inference_geo` propagation**: `InferenceGeo *string` added to `BifrostChatResponse`, `BifrostResponsesResponse`, and `BifrostPassthroughUsage`. The field is forwarded from `AnthropicUsage` in all conversion paths: non-streaming chat, non-streaming responses, streaming chat (captured across events and set on the final chunk), streaming responses (`message_delta`), and passthrough.
- **Data-residency multiplier in cost computation**: `serviceTier` gains an `inferenceGeoUS bool` flag. `tierFromResponse` now accepts and evaluates `inferenceGeo`. `computeTextCost` applies `InferenceGeoUSMultiplier` to token/cache costs only — the flat per-search fee is intentionally excluded.
- **Database migration**: `migrationAddInferenceGeoMultiplierColumn` adds `inference_geo_us_multiplier` to `TableModelPricing`. The column is included in the pricing sync update list and mapped through `convertEntryToTablePricing` / `convertTablePricingToEntry`.
- **Web search billing fix**: `accumulateAnthropicResponsesUsage` now carries `ServerToolUse.WebSearchRequests` into the accumulator so the count survives the terminal-chunk overwrite on streamed Responses requests. The non-streaming chat converter also forwards the count into `CompletionTokensDetails.NumSearchQueries`.
- **UI**: `inference_geo_us_multiplier` added to the custom pricing override sheet and `PricingOverridePatch` TypeScript type.
- **Tests**: New unit tests cover the accumulator web-search fix, chat/responses converter forwarding of `InferenceGeo` and web search counts, multiplier application (including the no-multiplier-column safe no-op), and `tierFromResponse` geo detection (case-insensitive).

- [ ] Bug fix
- [x] Feature
- [ ] Refactor
- [ ] Documentation
- [ ] Chore/CI

- [x] Core (Go)
- [ ] Transports (HTTP)
- [x] Providers/Integrations
- [ ] Plugins
- [x] UI (React)
- [ ] Docs

```sh
go test ./core/providers/anthropic/... ./framework/modelcatalog/datasheet/...

cd ui
pnpm i || npm i
pnpm build || npm run build
```

To validate end-to-end: send a request to an Anthropic model with data residency enabled. The response should include `inference_geo: "us"` and the computed cost should reflect the 1.1x multiplier on token/cache costs, with the per-search fee unchanged.

- [ ] Yes
- [x] No

No auth, secrets, or PII implications. The `inference_geo` value is sourced from Anthropic's API response and used only for billing computation.

- [ ] I read `docs/contributing/README.md` and followed the guidelines
- [x] I added/updated tests where appropriate
- [ ] I updated documentation where needed
- [x] I verified builds succeed (Go and UI)
- [ ] I verified the CI pipeline passes locally if applicable
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants