Skip to content

fix(cost): update Bing Grounding pricing - #43016

Closed
nvishnu22 wants to merge 2 commits into
BerriAI:mainfrom
nvishnu22:fix/bing-grounding-pricing
Closed

nvishnu22 wants to merge 2 commits into
BerriAI:mainfrom
nvishnu22:fix/bing-grounding-pricing

Conversation

@nvishnu22

@nvishnu22 nvishnu22 commented Sep 24, 2026 •

Copy link
Copy Markdown

TLDR

Problem this solves:

  • Bing Grounding spend uses outdated $35/1K pricing.
  • Microsoft currently lists $14 per 1,000 transactions.
  • This causes LiteLLM to overestimate Bing search spend.

How it solves it:

  • Updates bing_grounding/search pricing to $14 per 1,000 ($0.014 per query).
  • Keeps primary and backup cost maps synchronized (enforced by CI's diff check).
  • Updates the existing behavioral test test_connection_mode_tracks_the_g1_cost, which runs litellm.search and asserts the charged response_cost is 0.014, with a dated pricing source.

Official Microsoft pricing (accessed 2026-09-24):
https://www.microsoft.com/en-us/bing/apis/grounding-pricing

User Flow

Before: a user running Bing Grounding searches sees inflated spend.

  1. They run Grounding with Bing Search through LiteLLM.
  2. LiteLLM prices each transaction at $0.035.
  3. 607 transactions calculate to $21.245.

After: the same usage uses Microsoft's current pricing.

  1. They run Grounding with Bing Search through LiteLLM.
  2. LiteLLM prices each transaction at $0.014.
  3. 607 transactions calculate to $8.498.

Web-search mode (no connection ID) still charges $0, covered by test_web_search_mode_is_not_billed_the_g1_price.

Pre-Submission checklist

  • I have added meaningful tests
  • The handful of test files covering my change pass locally
  • My PR passes all required CI/CD checks
  • My PR's scope is as isolated as possible
  • I have received a Greptile Confidence Score of at least 4/5

Screenshots / Proof of Fix

Before (57eb3ff)

  1. Read the bing_grounding/search price from the cost map.
  2. Observed:
input_cost_per_query: 0.035
607 transactions: 21.245

After (35dcc33)

  1. Read the same bing_grounding/search price from the updated cost map.
  2. Observed:
input_cost_per_query: 0.014
607 transactions: 8.498

Tests:

pytest tests/search_tests/test_bing_grounding_search.py tests/unit/llms/azure/search/test_bing_grounding_search_transformation.py
53 passed, 3 skipped (live-API tests, local only)

Type

🐛 Bug Fix

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

@CLAassistant

CLAassistant commented Sep 24, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@greptile-apps

greptile-apps Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 4/5

The PR is not ready to merge until the existing connection-mode pricing test is updated and the new test meets the repository’s test requirements

Findings

  1. P1 Existing pricing test fails ▶
  2. P2 Price test misses charged result ▶

Summary

The PR changes Bing Grounding’s listed price from $0.035 to $0.014 per query in the primary and backup cost maps and adds price-map assertions

  • The existing runtime pricing test still expects the old charge
  • The new assertions do not test the charged result

Reviews (1) · Last reviewed commit: "fix(cost): update Bing Grounding pricing"

},
"bing_grounding/search": {
"input_cost_per_query": 0.035,
"input_cost_per_query": 0.014,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Existing pricing test fails

The connection-mode test loads the updated local price map, but still expects $0.035. It now receives $0.014, so its assertion fails and blocks the test suite. Update the existing runtime test alongside this price change

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated test_connection_mode_tracks_the_g1_cost to assert 0.014.

def test_bing_grounding_pricing():
model_info = _load(MAIN_PATH)[MODEL]

assert model_info["input_cost_per_query"] == BING_GROUNDING_COST_PER_QUERY

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Price test misses charged result

These tests read JSON but never check what a search charges. They also pin a vendor price without a source and date. The repository requires behavioral regression tests and dated provenance for external facts; satisfy that requirement before merging

Context Used: AGENTS.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the JSON-only test. Coverage is now the behavioral test_connection_mode_tracks_the_g1_cost, which checks the charged response_cost and cites the dated pricing source. CI's diff check already keeps the main and backup JSON in sync.

@codspeed

codspeed Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing nvishnu22:fix/bing-grounding-pricing (70addc0) with main (342f9c3)1

Open in CodSpeed

Footnotes

  1. No successful run was found on main (759c216) during the generation of this report, so 342f9c3 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@codecov

codecov Bot commented Sep 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@devin-ai-integration

Copy link
Copy Markdown
Contributor

Superseded by the rolling registry PR #42951, which carries these values re-verified against the official sources with a co-author credit. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants