Skip to content

test(pricing): allow a source link on Bedrock GovCloud rows the AWS offer file prices - #41167

Open
devin-ai-integration[bot] wants to merge 1 commit into
mainfrom
litellm_bedrock_usgov_source_field
Open

devin-ai-integration[bot] wants to merge 1 commit into
mainfrom
litellm_bedrock_usgov_source_field

Conversation

@devin-ai-integration

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

How it solves it:

  • Drop the "source" not in gov assertion. Capabilities, limits, and the 1.2x premium checks stay

User Flow

Before: the catalog sync PR cannot merge because a test rejects a sourced GovCloud row

  1. The sync opens chore(prices): sync Azure, Azure AI, Gemini, OpenAI, Bedrock, Together AI, Fireworks and Vertex prices: 278 models, 59 new, 30 deprecated #41154 with a source link on us-gov.anthropic.claude-fable-5-1
  2. The misc test job fails with assert 'source' not in {...} for that row
  3. The catalog update stalls

After: the same sync PR passes the GovCloud test

  1. The sync opens the same PR with the same source link
  2. test_usgov_rows_keep_commercial_limits_and_capabilities passes for all 19 gov rows
  3. The catalog update can merge

Relevant issues

Unblocks #41154

Affected release

Linear ticket

Pre-Submission checklist

  • I have added meaningful tests
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Raw provider evidence. The assertion was written on 2026-09-04 (bef3585) when no public AWS source listed Claude GovCloud rows. The AWS Price List offer file https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonBedrockFoundationModels/current/index.json (publicationDate 2026-09-11T12:44:10Z, snapshot in litellm-providers src/server/pricing/__tests__/fixtures/bedrock-foundation-models.json) now carries us-gov-east-1 products for "Claude Fable 5.1 (Amazon Bedrock Edition)". Per-1M-token USD rates and the catalog value the sync wrote:

usagetype offer USD / 1M catalog field catalog value
UGE1-MP:UGE1_input_tokens_standard-Units 12.0 input_cost_per_token 1.2e-05
UGE1-MP:UGE1_output_tokens_standard-Units 60.0 output_cost_per_token 6e-05
UGE1-MP:UGE1_cache_write_tokens_standard-Units 15.0 cache_creation_input_token_cost 1.5e-05
UGE1-MP:UGE1_cache_write_tokens_1h_standard-Units 24.0 cache_creation_input_token_cost_above_1hr 2.4e-05
UGE1-MP:UGE1_cache_read_tokens_standard-Units 0.3 cache_read_input_token_cost 3e-07

Every rate is exactly 1.2x the commercial anthropic.claude-fable-5-1 row, so the existing premium test keeps holding. The gov row in #41154 differs from main only by the added source link

Before (d2859e1)

  1. uv run pytest tests/test_litellm/test_bedrock_usgov_pricing.py against the chore(prices): sync Azure, Azure AI, Gemini, OpenAI, Bedrock, Together AI, Fireworks and Vertex prices: 278 models, 59 new, 30 deprecated #41154 catalog
  2. FAILED ...test_usgov_rows_keep_commercial_limits_and_capabilities[us-gov.anthropic.claude-fable-5-1] - AssertionError: assert 'source' not in {...}

After (e9770e9)

  1. uv run pytest tests/test_litellm/test_bedrock_usgov_pricing.py
  2. 22 passed, and the non-pricing fields of the chore(prices): sync Azure, Azure AI, Gemini, OpenAI, Bedrock, Together AI, Fireworks and Vertex prices: 278 models, 59 new, 30 deprecated #41154 gov row still equal the commercial row

Type

✅ Test

Caveats (if any)

Low

  • Gov rows may now carry any source link, the test no longer guards against a stale one

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/a6532463fd19440d8490b0393c54d776
Open in Devin Desktop: https://app.devin.ai/desktop/session/a6532463fd19440d8490b0393c54d776?variant=devin

…ffer file prices

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration
devin-ai-integration Bot requested a review from a team September 15, 2026 01:50
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@greptile-apps

greptile-apps Bot commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR permits GovCloud pricing rows to include source metadata by removing the assertion that previously prohibited it

  • Commercial capability and limit comparisons remain unchanged
  • Optional source values are no longer validated for AWS provenance

Confidence Score: 4/5

The behavior change is reasonable, but the explicit test-integrity requirement must be satisfied by validating source metadata when present

The remaining assertions exclude source metadata, so malformed or unrelated provenance can pass without detection; replacing the prohibition with conditional validation preserves coverage

Files Needing Attention: tests/test_litellm/test_bedrock_usgov_pricing.py

Important Files Changed

Filename Overview
tests/test_litellm/test_bedrock_usgov_pricing.py Removes the blanket source prohibition but leaves optional GovCloud source metadata unconstrained

Reviews (1): Last reviewed commit: "test(pricing): allow a source link on Be..." | Re-trigger Greptile

@@ -125,4 +125,3 @@ def test_usgov_rows_keep_commercial_limits_and_capabilities(model_data, gov_key)
gov = model_data[gov_key]
assert _non_pricing_fields(gov) == _non_pricing_fields(model_data[GOV_ROW_SOURCES[gov_key]])
assert "search_context_cost_per_query" not in gov

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Source validation removed

This deletion leaves GovCloud source URLs unchecked, violating the requirement that existing test changes preserve regression coverage before merging

Rule Used: What: Flag any modifications to existing tests and verify they don't weaken test coverage or mask regressions. Why: Developers may alter tests to make failing code pass rather than fix the actual bug, hiding regressions. Good: ``` // Test updated t... (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

@codecov

codecov Bot commented Sep 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant