Skip to content

test(pricing): cover gpt-5.6 cache-cost plumbing and bedrock_mantle responses billing - #35324

Merged
mateo-berri merged 1 commit into
litellm_internal_stagingfrom
litellm_gpt56_pricing_test_gaps
Jul 31, 2026
Merged

test(pricing): cover gpt-5.6 cache-cost plumbing and bedrock_mantle responses billing#35324
mateo-berri merged 1 commit into
litellm_internal_stagingfrom
litellm_gpt56_pricing_test_gaps

Conversation

@mateo-berri

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • Follow-up to fix(pricing): correct gpt-5.6 prices for openai, bedrock, and flex long context #35270, closing the two QA-runbook gaps that kept its last checkboxes unchecked
  • The cache-cost fields wired through get_model_info there (cache_creation_input_token_cost_flex/_priority, and the *_above_272k_tokens_flex cache variants) had no billing test; re-dropping them from the explicit kwargs list passed the whole suite
  • No test exercised a /v1/responses call against a bedrock_mantle gpt-5.6 deployment end to end through the cost calculator

How it solves it:

  • test_generic_cost_per_token_gpt56_terra_cache_costs_by_tier_and_context bills a request with cache-read and cache-write tokens across five (service tier, context size) combinations and asserts the exact prompt cost. Mutation-verified: reverting the cache wiring in litellm/utils.py fails the flex, priority, and flex-long-context rows
  • test_gpt_5_6_responses_call_cost runs a ResponsesAPIResponse for all three bedrock_mantle gpt-5.6 models through litellm.completion_cost and asserts the exact spend at the corrected rates

The standard above-272k cache-write row passes even without the explicit wiring because _get_model_info_helper has a regex passthrough for keys ending in _above_<n>k_tokens; tier-suffixed keys do not match it, which is exactly why the explicit wiring needs its own coverage.

The remaining wired-but-untested fields (cache_*_above_272k_tokens_priority) have no data in the cost map because OpenAI publishes no long-context column for the Fast tier, so there is no billing behavior to lock down.

Relevant issues

Follow-up to #35270

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • My PR passes all CI/CD checks (e.g., lint, format, unit tests)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Type

🧪 Tests

Changes

Test-only; no runtime code changes.

@greptile-apps

greptile-apps Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Adds focused regression coverage for GPT-5.6 billing:

  • Verifies OpenAI GPT-5.6 Terra cache-read and cache-write pricing across standard, flex, priority, and long-context tiers
  • Verifies Responses API cost calculation for three Bedrock Mantle GPT-5.6 models

Confidence Score: 5/5

The PR appears safe to merge; the added tests exercise the intended pricing paths without changing production behavior

The cache test reaches tier-specific and long-context cache pricing selection, while the Bedrock Mantle test uses the same provider-prefixed model and Responses API usage conversion used by production cost calculation

Important Files Changed

Filename Overview
tests/test_litellm/litellm_core_utils/llm_cost_calc/test_llm_cost_calc_utils.py Adds exact cache-cost assertions across service-tier and context-size combinations using the calculator's expected OpenAI token accounting shape
tests/test_litellm/llms/bedrock_mantle/test_bedrock_mantle_responses_transformation.py Adds Responses API completion-cost coverage for all registered Bedrock Mantle GPT-5.6 variants

Reviews (1): Last reviewed commit: "test(pricing): cover gpt-5.6 cache-cost ..." | Re-trigger Greptile

@codecov

codecov Bot commented Jul 31, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@mateo-berri
mateo-berri merged commit d0fe305 into litellm_internal_staging Jul 31, 2026
75 checks passed
@mateo-berri
mateo-berri deleted the litellm_gpt56_pricing_test_gaps branch July 31, 2026 05:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants