test(e2e): pin regional cost-map uplift and rate-limit attribution

[mubashir1osmani]

TLDR

Problem this solves:

• Regional bedrock pricing regression had no live coverage
• A 429's source was never asserted in any test
• Both are numbers customers reconcile externally

How it solves it:

• Pins each regional variant's resolved rate from /model/info
• Asserts the exact uplift, not just regional above global
• Drives a key past its own RPM and checks the failure labels
• Requires the gateway class and a named provider

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Both captured against a local proxy on port 4000 at this branch's commit c2b54892fa, on deployments and keys the run created through the management API exactly the way the tests do. This is a test-only PR, so the proof is that the behavior each test asserts is real and observable over HTTP, not that a pytest run went green

1. Regional variants resolve their own uplifted cost-map rate (LIT-3912). Three deployments of the same model differing only in region prefix, each read back through /model/info with no pricing override configured. The regional rates are the 10% uplift over global, which is the mapping that regressed

$ for V in global us eu; do
    curl -s -X POST http://localhost:4000/model/new -H "Authorization: Bearer sk-1234" \
      -d "{\"model_name\":\"proof-rp-$V\",\"litellm_params\":{\"model\":\"bedrock/$V.anthropic.claude-opus-4-7\",\"aws_region_name\":\"us-east-1\"}}"
    curl -s "http://localhost:4000/model/info?litellm_model_id=$ID" -H "Authorization: Bearer sk-1234"
  done

  proof-rp-global-14486        input=5e-06    output=2.5e-05
  proof-rp-us-628              input=5.5e-06  output=2.75e-05
  proof-rp-eu-14231            input=5.5e-06  output=2.75e-05

No provider call is made, so this needs no AWS credentials and spends nothing. That is deliberate: the regression is in rate resolution, and resolution is fully observable before any token is billed

2. A gateway 429 is attributed to the gateway and names a provider (PR #27687). A key limited to one request a minute, driven three times

$ KEY=$(curl -s -X POST http://localhost:4000/key/generate -H "Authorization: Bearer sk-1234" \
    -d '{"key_alias":"proof-rl-19519","models":["gemini-2.5-flash"],"rpm_limit":1}' | jq -r .key)

$ for i in 1 2 3; do curl -s -o /dev/null -w "call $i -> %{http_code}\n" -X POST \
    http://localhost:4000/v1/chat/completions -H "Authorization: Bearer $KEY" \
    -d '{"model":"gemini-2.5-flash","messages":[{"role":"user","content":"say ok"}],"max_tokens":8}'; done
  call 1 -> 200
  call 2 -> 429
  call 3 -> 429

$ curl -sL http://localhost:4000/metrics -H "Authorization: Bearer sk-1234" \
    | grep litellm_proxy_failed_requests_metric_total | grep proof-rl-19519
  exception_class=HTTPException exception_status=429 api_provider=vertex_ai route=/v1/chat/completions count=2.0

exception_class=HTTPException is the gateway's own limiter, as against a vendor rejection which surfaces its provider error class. For contrast, a real vendor 429 observed on the same proxy during this work looked like exception_class="Openai.RateLimitError" exception_status="429" api_provider="openai", which is the distinction the test pins

Four mutations were run against the live proxy and each failed the relevant test: claiming no regional uplift, claiming the gateway 429 carries a vendor error class, scraping for a key alias that made no calls, and requiring an empty api_provider. Both tests pass again once reverted, which is the evidence they would have failed before the fixes rather than passing vacuously

Type

✅ Test

Changes

Two independent regression tests plus their registry cells, mgmt.model.info.reports_regional_uplift and logging.prometheus.failure.attributes_rate_limit_source. No product code and no harness plumbing changes

The pricing test asserts the exact expected uplift rather than an inequality. regional > global would pass on a mapping that resolved the regional variant to some unrelated model's higher rate, which is the same class of bug wearing a different number. Rates are compared with a relative tolerance because the uplift arithmetic is not exact in binary, and pytest.approx is untyped under the suite's basedpyright gate

The rate-limit test asserts on exception_class because that is the only label that separates a gateway rejection from a vendor one; exception_status is 429 either way. It also requires api_provider to be non-empty, since a multi-provider deployment cannot act on a 429 that does not say which upstream was involved

QA runbook

• tests/e2e/llm_translation/test_regional_pricing_e2e.py::TestRegionalUpliftPricing::test_regional_variants_price_above_global - a regional bedrock deployment bills at its own uplifted rate, not the global one

  • Register three deployments of anthropic.claude-opus-4-7, one each for the global., us. and eu. prefixes, with no input_cost_per_token override: curl -X POST http://localhost:4000/model/new -H "Authorization: Bearer sk-1234" -d '{"model_name":"rp-us","litellm_params":{"model":"bedrock/us.anthropic.claude-opus-4-7","aws_region_name":"us-east-1"}}' (needs store_model_in_db: true; no AWS credentials required)
  • GET /model/info and read each row's model_info block, which is the rate the proxy resolved rather than anything configured
  • Expect global at input=5e-06 / output=2.5e-05, and both us. and eu. at exactly 1.1x those, input=5.5e-06 / output=2.75e-05
  • Confirm the assertion is not merely directional: a regional rate that is higher than global but not the 1.1x uplift must fail
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/logging/test_prometheus_ratelimit_failure_e2e.py::TestPrometheusRateLimitAttribution::test_gateway_rate_limit_is_labelled_with_source_and_provider - a 429 raised by the gateway's own limiter is labelled as the gateway's, and names the provider

  • Generate a key limited to one request a minute: curl -X POST http://localhost:4000/key/generate -H "Authorization: Bearer sk-1234" -d '{"key_alias":"rl-manual","models":["gemini-2.5-flash"],"rpm_limit":1}' (needs prometheus in litellm_settings.callbacks)
  • Send three /v1/chat/completions requests with that key inside one minute and expect the first to return 200 and the rest 429
  • Scrape /metrics (follow the 307 with curl -L) and find litellm_proxy_failed_requests_metric_total for api_key_alias=rl-manual
  • Expect exception_status="429", exception_class="HTTPException" (the gateway's limiter, not a vendor class) and a non-empty api_provider
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Greptile Summary

Adds live end-to-end coverage for two customer-visible attribution paths.

• Registers global, US, and EU Bedrock variants and verifies their resolved token rates through /model/info.
• Exercises a key-level RPM limit and validates the resulting Prometheus failure labels.
• Registers both scenarios in the management and logging coverage registries.

Confidence Score: 4/5

The PR appears safe to merge, with a non-blocking test-integrity gap in the rate-limit scenario.

The registry and regional-pricing additions are consistent with the harness, while the rate-limit test should verify that its first provider request succeeds so it cannot validate gateway attribution after an unrelated initial failure.

Files Needing Attention: tests/e2e/logging/test_prometheus_ratelimit_failure_e2e.py

Important Files Changed

Filename	Overview
tests/e2e/llm_translation/test_regional_pricing_e2e.py	Adds a well-scoped model-management test that checks exact global and regional cost-map rates and defers cleanup using the created model IDs.
tests/e2e/logging/test_prometheus_ratelimit_failure_e2e.py	Adds Prometheus attribution coverage, but does not establish that a provider request succeeds before the gateway rate limit is exercised.
tests/e2e/coverage_registry/logging.yaml	Adds a valid logging coverage cell matching the new Prometheus test marker.
tests/e2e/coverage_registry/mgmt.yaml	Adds a valid management coverage cell matching the new regional-pricing test marker.

_{Reviews (1): Last reviewed commit: "test(e2e): pin regional cost-map uplift ..." | Re-trigger Greptile}

[greptile-apps]

Successful request is never established

The status assertion only requires one 429, so the test can validate the gateway labels even when the initial provider request fails. Assert that the first response succeeds before checking that later requests exceed the key's RPM limit; otherwise the advertised end-to-end provider scenario remains unverified.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!