test(integration): regression tests for August cost tracking and budgeting bugs

[devin-ai-integration]

TLDR

Problem this solves:

• 24 cost tracking, budgeting and spend reporting bugs customers reported in August had merged fixes but no integration or e2e test
• Any of those fixes could be silently undone by a later refactor

How it solves it:

• Adds 24 tests under tests/integration/spend, pricing, management and observability, one per bug
• Each test drives the real proxy against the scripted upstream and asserts the response cost header, the persisted spend rows or the budget decision
• Each test was proven red with the fix commit reverted and green with it restored (evidence per test below)
• One test that wrote provider env vars to the shared proxy config was moved onto owned_proxy so shard order cannot leak state

User Flow

Before: an engineer refactors spend tracking and reintroduces one of the August regressions, for example OCR annotation pages billed at the plain page rate

1. They open a PR and CI runs the unit suite and the integration shards
2. Every check is green because nothing exercises that request through the proxy against the spend table
3. A customer sends POST https://litellm-domain/v1/ocr with a document annotation and the x-litellm-response-cost header and https://litellm-domain/ui/?page=logs both under-report the spend

After: the same refactor turns the accounting integration shard red before merge

1. They open the same PR
2. The CircleCI integration-accounting shard fails on test_ocr_annotation_pages_are_billed_at_annotation_cost_per_page, showing the expected and actual cost
3. The customer keeps seeing the correct spend because the PR cannot merge

Relevant issues

Regression coverage for August customer bug reports, one Pylon ticket per test, listed in the table below. No open issue is fixed by this PR

Affected release

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

This PR ships tests only, so the proof is the mutation check tests/AGENTS.md asks for: each test was run with its fix commit reverted (red) and with the fix restored (green), through the same real proxy, Postgres, Redis and scripted upstream the CircleCI shard uses. The harness is deterministic by design and does not call real providers. The accounting shard passed locally on the tip with two order seeds (30 passed each), plus management (21 passed) and extensions (21 passed)

Before (fix commit reverted on top of df7db2c)

Per test, the fix sha in the table was reverted (git revert --no-commit <sha>, or a worktree at the fix parent where the revert conflicted), the proxy restarted, and the new node run. Every node failed on the behavioral assertion. Output is in the fold-outs below

After (df7db2c)

Fix restored, same node rerun, passed. The shards: python tests/integration/run.py accounting reports 30 passed with INTEGRATION_ORDER_SEED=1 and again with INTEGRATION_ORDER_SEED=2, management 21 passed, extensions 21 passed

Tests and their red/green evidence

Pylon	Fix PR	Fix sha	Test node
6870	#36222	a969319fb5	tests/integration/pricing/test_configured_prices.py::test_saving_echoed_model_info_does_not_freeze_cost_map_price_into_deployment
6913	#30555	96b9437bdd	tests/integration/management/test_budget_updates.py::test_shortening_budget_duration_moves_reset_at_onto_the_new_schedule
6966	#36718	599daea985	tests/integration/spend/test_cache_and_quota.py::test_repeated_count_tokens_on_budgeted_key_does_not_reserve_budget_or_block_later_completion
7014	#35880	9bfe593241	tests/integration/pricing/test_configured_prices.py::test_cost_estimate_reports_configured_prices_for_model_absent_from_cost_map
7180	#37695	3ea1c16b0d	tests/integration/management/test_team_member_budget_cache.py::test_team_member_default_budget_lands_in_redis_after_first_member_call
7224	#36562	9ec0145986	tests/integration/spend/test_team_daily_activity_aggregated.py::test_aggregated_team_activity_reports_the_whole_range_team_spend_in_one_page
7268	#41878	f6d4766ebe	tests/integration/spend/test_daily_rollup_retry.py::test_failed_daily_user_rollup_commit_is_retried_so_spend_report_and_daily_activity_agree
7291	#34577	579f41d57f	tests/integration/spend/test_org_budget_cli_session_token.py::test_cli_session_token_without_org_id_charges_and_caps_the_team_organization
7295	#36592	258fe3e4ba	tests/integration/spend/test_passthrough_budget_reservation.py::test_repeated_gemini_passthrough_calls_stay_served_while_key_spend_is_below_max_budget
7307	#36718	599daea985	tests/integration/spend/test_cache_and_quota.py::test_in_flight_count_tokens_does_not_reserve_key_budget_away_from_a_completion
7341	#37208	20cfccaf5f	tests/integration/spend/test_batch_observability.py::test_batch_retrieval_row_sums_reasoning_tokens_and_counts_output_and_error_file_failures
7342	#36714	2b63919f67	tests/integration/spend/test_batch_poll_starvation.py::test_batches_gone_at_provider_do_not_starve_a_newer_batch_out_of_cost_polling
7363	#41349	006080ea6d	tests/integration/spend/test_team_member_spend.py::test_member_added_without_any_budget_is_charged_on_its_membership_row
7519	#37365	d6afe728aa	tests/integration/spend/test_failed_dispatch_tokens.py::test_provider_500_after_dispatch_records_estimated_prompt_tokens_on_failure_row
7524	#39281	853fed824e	tests/integration/observability/test_guardrail_effects.py::test_bedrock_passthrough_converse_guardrail_ignores_denied_term_in_tool_definition
7536	#36699	262ed530f8	tests/integration/management/test_team_budget_duration_defaults.py::test_team_new_explicit_null_budget_duration_is_not_replaced_by_default
7577	#39670	a502c728ac	tests/integration/management/test_organization_budget_clear.py::test_patch_organization_update_with_null_tpm_limit_clears_it_and_keeps_sibling_limits
7587	#37355	6b7adf011e	tests/integration/pricing/test_service_tier_pricing.py::test_ultrafast_service_tier_bills_ultrafast_rates_and_keeps_pricing_off_the_wire
7636	#37770	3db6c5ab18	tests/integration/spend/test_model_router_selected_model.py::test_model_router_alias_without_router_in_name_keeps_selected_model_in_response_and_spend_log
7685	#36008	99703a30f0	tests/integration/spend/test_disconnected_bedrock_messages_stream_billing.py::test_client_disconnect_mid_bedrock_messages_stream_still_bills_terminal_usage
7738	#37975	3122600e21	tests/integration/pricing/test_databricks_cache_pricing.py::test_databricks_cached_prompt_tokens_bill_at_cache_rates_not_input_rate
7775	#38131	c19d49d919	tests/integration/observability/test_callback_delivery.py::test_streamed_responses_success_callback_carries_provider_apim_request_id
7928	#37208	20cfccaf5f	tests/integration/spend/test_batch_completion_accounting.py::test_completed_batch_spend_row_records_reasoning_tokens_and_error_file_failures
7958	#38985	b86791b4b9	tests/integration/pricing/test_ocr_page_pricing.py::test_ocr_annotation_pages_are_billed_at_annotation_cost_per_page

Pylon 6870: PATCH /model/{id}/update (and POST /model/new) must not persist server-derived pricing keys carried in the request's mod [...]

Red method: revert_fix_at_head

Red (fix reverted):

git checkout -b scratch-broken-6870 && git revert --no-commit -m 1 a969319fb5e4557de9713b3232e376a153641f18 (conflicts in all three src files; model_management_endpoints.py resolved to the pre-fix call sites, i.e. merged_model_info.update(updated_patch.model_info.model_dump(exclude_none=True)) and no without_server_derived_pricing wrap on /model/new; litellm/utils.py reverted cleanly; litellm/types/utils.py kept at HEAD because later commits import other symbols from it and reverting it would ImportError at proxy boot instead of exercising the behavior) && git checkout HEAD -- tests/integration (test file + contracts.json re-applied) && ~/integ_local.sh restart-proxy && ~/integ_local.sh run- [...]

Green (fix present):

~/integ_local.sh up && ~/integ_local.sh run-node tests/integration/pricing/test_configured_prices.py::test_saving_echoed_model_info_does_not_freeze_cost_map_price_into_deployment  (leg A, fixed code) -> PASSED [100%], 1 passed in 0.28s

After returning to pylon-regress-6870 and ~/integ_local.sh rest [...]

Pylon 6913: POST /budget/update with a JSON body containing an existing budget_id and a changed budget_duration, without budget_rese [...]

Red method: revert_fix_at_head

Red (fix reverted):

On scratch branch off pylon-regress-6913: `git revert --no-commit 96b9437bdd85c99ccd72b59bace716fc49c7423a` conflicted in all three files (main has since rewritten the conditional to also clear reset_at when duration is set to null). Resolved by `git checkout HEAD -- tests/ litellm/proxy/common_utils/timezone_utils.py` and hand-editing budget_management_endpoints.py to drop only the recompute branch (`{"budget_reset_at": get_budget_reset_time(...)} if budget_obj.budget_duration is not None and "budget_reset_at" not in model_fields_set`), leaving the rest of HEAD intact; git status showed only that one production file modified, no test files. Then `integ_local.sh restart-proxy` and `integ_loc [...]

Green (fix present):

On pylon-regress-6913 tip (leg A before commit and leg C after restoring the branch and `integ_local.sh restart-proxy`): `integ_local.sh run-node tests/integration/management/test_budget_updates.py::test_shortening_budget_duration_moves_reset_at_onto_the_new_schedule`
collecting ... collected 1 item [...]

Pylon 6966: The customer-facing regression is on authenticated POST /v1/messages/count_tokens, with an Anthropic-native body such as [...]

Red method: revert_fix_at_head

Red (fix reverted):

git checkout -b scratch-red-6966 && git revert --no-commit -m 1 599daea985782841813eeffc92f02bc18c863851 && git checkout HEAD -- tests/integration (git status: only M litellm/proxy/spend_tracking/budget_reservation.py) && bash integ_local.sh restart-proxy && bash integ_local.sh run-node tests/integration/spend/test_cache_and_quota.py::test_repeated_count_tokens_on_budgeted_key_does_not_reserve_budget_or_block_later_completion

        for count in counts:
>           assert count.status_code == 200, count.text
E           AssertionError: {"error":{"message":"Budget has been exceeded! Key=key (sk-...zLnA) Current cost: 0.1, Max budget: 0.1","type":"budget_exceeded","param":null,"code":"422"}} [...]

Green (fix present):

bash /home/ubuntu/attachments/.../integ_local.sh run-node tests/integration/spend/test_cache_and_quota.py::test_repeated_count_tokens_on_budgeted_key_does_not_reserve_budget_or_block_later_completion (on branch tip, after restart-proxy from the fixed tree)

collecting ... collected 1 item

tests/int [...]

Pylon 7014: POST /cost/estimate with a model name that is a router model_group for an on-prem/self-hosted deployment (e.g. nvidia) w [...]

Red method: revert_fix_at_head

Red (fix reverted):

On scratch branch off pylon-regress-7014: git revert --no-commit -m 1 9bfe593241d3acdb4e39e172ae6311319f8f2d38 (conflicted in cost_tracking_settings.py and its unit test; production file resolved with --theirs = exact pre-fix file, unit test kept at HEAD, tests/integration untouched); ~/integ_local.sh restart-proxy; ~/integ_local.sh run-node tests/integration/pricing/test_configured_prices.py::test_cost_estimate_reports_configured_prices_for_model_absent_from_cost_map

>           assert body["input_cost_per_token"] == pytest.approx(0.003), response.text
E           AssertionError: {"model":"integration-53307ec533f446c5bf957a99776cca5d",...,"cost_per_request":0.0,...,"input_cost_per_token":0 [...]

Green (fix present):

~/integ_local.sh up; ~/integ_local.sh run-node tests/integration/pricing/test_configured_prices.py::test_cost_estimate_reports_configured_prices_for_model_absent_from_cost_map (leg A, and again as leg C after restart-proxy on branch tip)

tests/integration/pricing/test_configured_prices.py::test_cos [...]

Pylon 7180: Proxy runs with enable_redis_auth_cache: true and a redis cache backend (the integration proxy_config.yaml already enabl [...]

Red method: revert_fix_at_head

Red (fix reverted):

git checkout -b scratch-red-7180 && git revert --no-commit 3ea1c16b0d07621198600603a7184d3e55070254 && git checkout HEAD -- tests/integration (git status: only litellm/proxy/auth/auth_checks.py and tests/test_litellm/proxy/auth/test_auth_checks.py staged, no tests/integration files) && ./integ_local.sh restart-proxy && ./integ_local.sh run-node tests/integration/management/test_team_member_budget_cache.py::test_team_member_default_budget_lands_in_redis_after_first_member_call

>           assert time.monotonic() < deadline, f"State did not converge: {observed!r}"
E           AssertionError: State did not converge: None

tests/integration/_support/client.py:50: AssertionError
================ [...]

Green (fix present):

on pylon-regress-7180 tip 7119ec0988 after ./integ_local.sh restart-proxy: ./integ_local.sh run-node tests/integration/management/test_team_member_budget_cache.py::test_team_member_default_budget_lands_in_redis_after_first_member_call

collecting ... collected 1 item

tests/integration/management/te [...]

Pylon 7224: The customer-facing team usage flow queried GET /team/daily/activity with start_date, end_date, page_size and page repea [...]

Red method: worktree_at_fix_parent_then_cherry_pick

Red (fix reverted):

git revert --no-commit 9ec0145986 conflicted in litellm/proxy/_types.py, common_daily_activity.py, team_endpoints.py (plus tests/ui), so fell back: git worktree add /tmp/broken 9ec0145986^; copied tests/integration and .circleci/scripts/wait_integration_services.py from the branch; ln -s .venv; dropped/recreated circle_test (older schema cannot drop columns the newer view depends on); INTEGREPO=/tmp/broken integ_local.sh restart-proxy; INTEGREPO=/tmp/broken integ_local.sh run-node tests/integration/spend/test_team_daily_activity_aggregated.py::test_aggregated_team_activity_reports_the_whole_range_team_spend_in_one_page

tests/integration/spend/test_team_daily_activity_aggregated.py::test_agg [...]

Green (fix present):

cd /home/ubuntu/repos/litellm && /home/ubuntu/integ_local.sh restart-proxy && /home/ubuntu/integ_local.sh run-node tests/integration/spend/test_team_daily_activity_aggregated.py::test_aggregated_team_activity_reports_the_whole_range_team_spend_in_one_page   (branch tip 818e80e54d)

rootdir: /home/ub [...]

Pylon 7268: A successful POST /v1/chat/completions through a virtual key can persist its raw spend log while a daily user rollup com [...]

Red method: revert_fix_at_head

Red (fix reverted):

On scratch branch off pylon-regress-7268: git revert --no-commit -m 1 f6d4766ebeb089f6daa33966dfa69cb999300423 (conflicts in db_spend_update_writer.py and exception_handler.py resolved toward the pre-merge side, keeping postgres_sqlstate and _spend_tables_left_to_send helpers that later code imports); git checkout HEAD -- tests/; git status showed only litellm/proxy/db/db_spend_update_writer.py and litellm/proxy/db/exception_handler.py modified; bash ~/integ_local.sh restart-proxy; bash ~/integ_local.sh run-node "tests/integration/spend/test_daily_rollup_retry.py::test_failed_daily_user_rollup_commit_is_retried_so_spend_report_and_daily_activity_agree"

Result: FAILED. Failed: Timeout (>90.0 [...]

Green (fix present):

bash ~/integ_local.sh run-node "tests/integration/spend/test_daily_rollup_retry.py::test_failed_daily_user_rollup_commit_is_retried_so_spend_report_and_daily_activity_agree"

tests/integration/spend/test_daily_rollup_retry.py::test_failed_daily_user_rollup_commit_is_retried_so_spend_report_and_daily [...]

Pylon 7291: Drive a real proxy request with POST /v1/chat/completions using Authorization: Bearer produced by th [...]

Red method: revert_fix_at_head

Red (fix reverted):

git checkout -b scratch off pylon-regress-7291; git revert --no-commit 579f41d57fbff105c9430beb8cd08179ef414edb (conflicted because the fix's two lines were later moved into _inherit_org_identity; resolved by deleting the org_id backfill from _inherit_org_identity, git checkout HEAD -- tests/integration tests/test_litellm; git status showed only litellm/proxy/auth/user_api_key_auth.py changed); integ_local.sh restart-proxy; bash /home/ubuntu/attachments/1c796615-d004-456f-98b2-7ed5f3932c12/integ_local.sh run-node tests/integration/spend/test_org_budget_cli_session_token.py::test_cli_session_token_without_org_id_charges_and_caps_the_team_organization

tests/integration/spend/test_org_budget_c [...]

Green (fix present):

On pylon-regress-7291 tip 0dde259547 after restart-proxy: bash /home/ubuntu/attachments/1c796615-d004-456f-98b2-7ed5f3932c12/integ_local.sh run-node tests/integration/spend/test_org_budget_cli_session_token.py::test_cli_session_token_without_org_id_charges_and_caps_the_team_organization
configfile:  [...]

Pylon 7295: A caller sends a successful native Gemini passthrough request as POST /gemini/v1beta/models/{model}:generateContent with [...]

Red method: worktree_at_fix_parent_then_cherry_pick

Red (fix reverted):

revert_fix_at_head was attempted first: `git revert --no-commit 258fe3e4bac73f3dbf31653ff19ab3ce0e1a6909` conflicted (code moved); resolved by deleting only the one line `_metadata["user_api_key_budget_reservation"] = user_api_key_dict.budget_reservation` from litellm/proxy/pass_through_endpoints/pass_through_endpoints.py (git status: that single file, no tests), restart-proxy, run-node -> PASSED (1 passed in 12.35s). Reason: e7cd97c6b6 (#42304, BudgetReservationReleaseMiddleware) now releases any reservation the callbacks did not claim at request end, so at head the fix line is redundant and the leak cannot be reproduced by reverting it alone. Fallback used: `git worktree add /tmp/broken 25 [...]

Green (fix present):

bash /home/ubuntu/attachments/2f7fa92b-ee53-412d-8dd3-b1b46eca6631/integ_local.sh run-node "tests/integration/spend/test_passthrough_budget_reservation.py::test_repeated_gemini_passthrough_calls_stay_served_while_key_spend_is_below_max_budget" on pylon-regress-7295 at 8f332692c7 (final run after res [...]

Pylon 7307: For a budgeted proxy key with a small max_budget, POST /v1/messages/count_tokens with an Anthropic Messages request such [...]

Red method: revert_fix_at_head

Red (fix reverted):

bash /home/ubuntu/attachments/aad63fb2-60b6-459e-8266-ab19547b5678/integ_local.sh run-node "tests/integration/spend/test_cache_and_quota.py::test_in_flight_count_tokens_does_not_reserve_key_budget_away_from_a_completion" (scratch branch, git revert --no-commit -m 1 599daea9, only litellm/proxy/spend_tracking/budget_reservation.py reverted, restart-proxy)

            completion_answered.set()
            count: Final = counting.result(timeout=30)
>           assert completion.status_code == 200 and completion.json()["usage"]["total_tokens"] == 40, completion.text
E           AssertionError: {"error":{"message":"Budget has been exceeded! Key=012f34c1a0f617a050b461b3993af30695e611b6e23f603beff [...]

Green (fix present):

bash /home/ubuntu/attachments/aad63fb2-60b6-459e-8266-ab19547b5678/integ_local.sh run-node "tests/integration/spend/test_cache_and_quota.py::test_in_flight_count_tokens_does_not_reserve_key_budget_away_from_a_completion" (on pylon-regress-7307 tip, after restart-proxy)

rootdir: /home/ubuntu/repos/l [...]

Pylon 7341: Drive the proxy batch flow with POST /v1/files using a multipart JSONL batch input, POST /v1/batches with the returned i [...]

Red method: worktree_at_fix_parent_then_cherry_pick

Red (fix reverted):

git worktree add /tmp/broken 20cfccaf5ff704fc75f32de01f456d208226d5d4^ (c62c2afa09); copied tests/integration and .circleci/scripts/wait_integration_services.py from branch; ln -s main .venv; DROP SCHEMA public CASCADE (old prisma schema cannot db push over the newer one); INTEGREPO=/tmp/broken integ_local.sh restart-proxy; INTEGREPO=/tmp/broken integ_local.sh run-node "tests/integration/spend/test_batch_observability.py::test_batch_retrieval_row_sums_reasoning_tokens_and_counts_output_and_error_file_failures"

                seconds=30,
            )
            assert retrieval.status_code == 200, retrieval.text
            rows: Final = eventually(lambda: _retrieval_rows(key), lambda val [...]

Green (fix present):

integ_local.sh restart-proxy && integ_local.sh run-node "tests/integration/spend/test_batch_observability.py::test_batch_retrieval_row_sums_reasoning_tokens_and_counts_output_and_error_file_failures" (branch tip 92b5bc106e, after returning from the worktree; the pre-red run on the same tip also pass [...]

Pylon 7342: A customer creates a managed batch with POST /v1/batches, supplying input_file_id, endpoint /v1/chat/completions, comple [...]

Red method: revert_fix_at_head

Red (fix reverted):

On scratch branch off pylon-regress-7342: `git revert --no-commit -m 1 2b63919f67c012e71c129e621327d524298ad45a` conflicted in check_batch_cost.py and its unit test (the file was refactored since the fix), so the conflict was resolved by keeping HEAD and removing the two retirement call sites the fix added (`_retire_job` on unroutable unified id, and `_retire_job` on provider 404), which restores the pre-fix bare `continue`; `git checkout HEAD -- tests/integration` left only the production file modified. Then `bash /home/ubuntu/integ_local.sh restart-proxy` and `bash /home/ubuntu/integ_local.sh run-node "tests/integration/spend/test_batch_poll_starvation.py::test_batches_gone_at_provider_do_ [...]

Green (fix present):

On pylon-regress-7342 tip dfc50d682d after `bash /home/ubuntu/integ_local.sh restart-proxy`: `bash /home/ubuntu/integ_local.sh run-node "tests/integration/spend/test_batch_poll_starvation.py::test_batches_gone_at_provider_do_not_starve_a_newer_batch_out_of_cost_polling"`
collecting ... collected 1 i [...]

Pylon 7363: Regression flow: an admin POSTs /team/member_add with {"team_id":"","member":{"user_id":"","role":"use [...]

Red method: worktree_at_fix_parent_then_cherry_pick

Red (fix reverted):

git worktree add /tmp/broken 006080ea6ddcc1866dbd0b4d2531e9b104e81b32^ (HEAD c553bc92bd); cp -r tests/integration into it; ln -s .venv; cd /tmp/broken && INTEGREPO=/tmp/broken /home/ubuntu/integ_local.sh restart-proxy && INTEGREPO=/tmp/broken /home/ubuntu/integ_local.sh run-node tests/integration/spend/test_team_member_spend.py::test_member_added_without_any_budget_is_charged_on_its_membership_row

>           assert [
                {"user_id": row["user_id"], "team_id": row["team_id"], "budget_id": row["budget_id"], "spend": row["spend"]}
                for row in memberships
            ] == [{"user_id": user, "team_id": team, "budget_id": None, "spend": 0}], added.text
E           Asse [...]

Green (fix present):

cd /home/ubuntu/repos/litellm (branch pylon-regress-7363, tip 89220b0630) && /home/ubuntu/integ_local.sh restart-proxy && /home/ubuntu/integ_local.sh run-node tests/integration/spend/test_team_member_spend.py::test_member_added_without_any_budget_is_charged_on_its_membership_row

collecting ... coll [...]

Pylon 7519: For a dispatched provider failure, exercise POST /v1/chat/completions with a configured model whose scripted upstream re [...]

Red method: revert_fix_at_head

Red (fix reverted):

On scratch-broken-7519 (off pylon-regress-7519): git revert --no-commit -m 1 d6afe728aad9e9d97e43cac17fe552930a821331 conflicted in litellm/proxy/utils.py and three tests/test_litellm files; ran git checkout HEAD -- tests/integration and git checkout HEAD -- tests/test_litellm (no test files reverted), resolved utils.py by keeping the fixed helper definitions but making _failure_usage_to_lift return None after the recovered-usage branch (the reverted behaviour), re-adding the Usage import the revert had dropped so the proxy imports. Then: /home/ubuntu/integ_local.sh restart-proxy && /home/ubuntu/integ_local.sh run-node "tests/integration/spend/test_failed_dispatch_tokens.py::test_provider_50 [...]

Green (fix present):

On pylon-regress-7519 tip 74009c6c5b (after git checkout -- . on scratch and deleting it): /home/ubuntu/integ_local.sh restart-proxy && /home/ubuntu/integ_local.sh run-node "tests/integration/spend/test_failed_dispatch_tokens.py::test_provider_500_after_dispatch_records_estimated_prompt_tokens_on_fa [...]

Pylon 7524: POST /bedrock/model/{modelId}/converse (and converse-stream) through the proxy with a pre_call guardrail enabled (defaul [...]

Red method: revert_fix_at_head

Red (fix reverted):

On scratch branch off pylon-regress-7524: git revert --no-commit 853fed824e841ce3e3952f137b6a3d577ced07d4 && git checkout HEAD -- tests/integration (git status: only litellm/llms/bedrock/passthrough/guardrail_translation/handler.py and tests/unit/.../test_handler.py reverted, no tests/integration files); integ_local.sh restart-proxy; integ_local.sh run-node tests/integration/observability/test_guardrail_effects.py::test_bedrock_passthrough_converse_guardrail_ignores_denied_term_in_tool_definition

tests/integration/observability/test_guardrail_effects.py::test_bedrock_passthrough_converse_guardrail_ignores_denied_term_in_tool_definition FAILED [100%]
=================================== FAILU [...]

Green (fix present):

On pylon-regress-7524 at bf98163ecd (after git revert --abort and restart-proxy): integ_local.sh run-node tests/integration/observability/test_guardrail_effects.py::test_bedrock_passthrough_converse_guardrail_ignores_denied_term_in_tool_definition

collected 1 item
tests/integration/observability/te [...]

Pylon 7536: The regression is on authenticated POST /team/new. With litellm.default_team_params.budget_duration set to 30d, sending [...]

Red method: revert_fix_at_head

Red (fix reverted):

On a scratch worktree at the merge tip with `262ed530f8` reverted (litellm/ reverted cleanly, ui/ and tests/test_litellm conflicts resolved to HEAD): `integ_local.sh run-node tests/integration/management/test_team_budget_duration_defaults.py::test_team_new_explicit_null_budget_duration_is_not_replaced_by_default`

FAILED tests/integration/management/test_team_budget_duration_defaults.py::test_team_new_explicit_null_budget_duration_is_not_replaced_by_default - AssertionError: {...,"max_budget":500.0,...,"budget_duration":"30d",...,"budget_reset_at":"2026-10-01T00:00:00Z",...}
assert '30d' is None
============================== 1 failed in 16.45s ==============================

Green (fix present):

On 240b274 (test runs on an owned proxy with `default_team_params.budget_duration: 30d` set in yaml, STORE_MODEL_IN_DB=False): `integ_local.sh run-node tests/integration/management/test_team_budget_duration_defaults.py::test_team_new_explicit_null_budget_duration_is_not_replaced_by_default`

============================== 1 passed in 16.57s ==============================

Pylon 7577: The customer sent PATCH /organization/update with Authorization: Bearer and JSON {"organization_id":""," [...]

Red method: revert_fix_at_head

Red (fix reverted):

On scratch branch scratch-red-7577 off pylon-regress-7577: git revert --no-commit -m 1 a502c728ac8f8cf2f74ab534d61bd984f9dc4f87 && git checkout HEAD -- tests/integration (git status showed only litellm/proxy/management_endpoints/organization_endpoints.py and the pre-existing unit test file tests/test_litellm/proxy/management_endpoints/test_organization_endpoints.py modified, no tests/integration files), then /home/ubuntu/integ_local.sh restart-proxy && /home/ubuntu/integ_local.sh run-node "tests/integration/management/test_organization_budget_clear.py::test_patch_organization_update_with_null_tpm_limit_clears_it_and_keeps_sibling_limits"

tests/integration/management/test_organization_budget [...]

Green (fix present):

On pylon-regress-7577 tip 75f14fe6 after restart-proxy from the fixed tree: /home/ubuntu/integ_local.sh run-node "tests/integration/management/test_organization_budget_clear.py::test_patch_organization_update_with_null_tpm_limit_clears_it_and_keeps_sibling_limits"

platform linux -- Python 3.12.13,  [...]

Pylon 7587: A proxy caller sends POST /v1/chat/completions with a registered model, messages, and service_tier=ultrafast. When the d [...]

Red method: revert_fix_at_head

Red (fix reverted):

On scratch branch off pylon-regress-7587: git revert --no-commit -m 1 6b7adf011eae3a1963427f13e85f1a1bf4eeb376 (one trivial conflict in litellm/utils.py resolved by dropping the cache_read_input_token_cost_ultrafast line), git checkout HEAD -- tests/, git status showed only litellm/litellm_core_utils/llm_cost_calc/utils.py, litellm/types/utils.py, litellm/utils.py, ui/litellm-dashboard/src/lib/http/schema.d.ts modified. ~/integ_local.sh restart-proxy; ~/integ_local.sh run-node tests/integration/pricing/test_service_tier_pricing.py::test_ultrafast_service_tier_bills_ultrafast_rates_and_keeps_pricing_off_the_wire

tests/integration/pricing/test_service_tier_pricing.py::test_ultrafast_service_t [...]

Green (fix present):

With test committed (4415177605) on pylon-regress-7587: ~/integ_local.sh up; ~/integ_local.sh run-node tests/integration/pricing/test_service_tier_pricing.py::test_ultrafast_service_tier_bills_ultrafast_rates_and_keeps_pricing_off_the_wire

collecting ... collected 1 item
tests/integration/pricing/t [...]

Pylon 7636: For POST /v1/chat/completions using an Azure AI Model Router deployment exposed through an arbitrary model-group alias s [...]

Red method: revert_fix_at_head

Red (fix reverted):

git checkout -b scratch-broken-7636 && git revert --no-commit -m 1 3db6c5ab18ec205cb388e44bbc6d1453ce855c4a (conflicts in common_utils.py and spend_tracking_utils.py resolved to the pre-fix code by hand, budget json and tests/ restored to HEAD) && git checkout HEAD -- tests/integration && /home/ubuntu/integ_local.sh restart-proxy && /home/ubuntu/integ_local.sh run-node tests/integration/spend/test_model_router_selected_model.py::test_model_router_alias_without_router_in_name_keeps_selected_model_in_response_and_spend_log

E             + integration-d08dd3fb7ac248baaf54446ae4670964

tests/integration/spend/test_model_router_selected_model.py:58: AssertionError
============================= s [...]

Green (fix present):

on pylon-regress-7636 tip 6809e8087d after restart-proxy: /home/ubuntu/integ_local.sh run-node tests/integration/spend/test_model_router_selected_model.py::test_model_router_alias_without_router_in_name_keeps_selected_model_in_response_and_spend_log


tests/integration/spend/test_model_router_select [...]

Pylon 7685: For a proxy POST /v1/messages request with stream=true routed to a Bedrock Anthropic Claude model, a client that stops r [...]

Red method: revert_fix_at_head

Red (fix reverted):

Scratch branch off pylon-regress-7685: git revert --no-commit -m 1 99703a30f06dde3e3fbd6283ff388fcaa29eb6b7 (conflicts in streaming_iterator.py resolved by taking the pre-fix side; budget json and tests restored with git checkout HEAD -- tests/integration), restart-proxy, then
bash integ_local.sh run-node "tests/integration/spend/test_disconnected_bedrock_messages_stream_billing.py::test_client_disconnect_mid_bedrock_messages_stream_still_bills_terminal_usage"

 "WHERE api_key=%s",
                    (sha256(key.encode()).hexdigest(),),
                ),
                lambda values: len(values) == 1,
                seconds=70,
            )
            assert rows[0]["request_id"] == me [...]

Green (fix present):

Leg A (committed tip, before red):
bash integ_local.sh run-node "tests/integration/spend/test_disconnected_bedrock_messages_stream_billing.py::test_client_disconnect_mid_bedrock_messages_stream_still_bills_terminal_usage"
configfile: pyproject.toml
plugins: cov-5.0.0, mock-3.15.1, respx-0.22.0, post [...]

Pylon 7738: A caller sends POST /v1/chat/completions with an OpenAI-compatible body such as {"model":"databricks/databricks-claude-s [...]

Red method: revert_fix_at_head

Red (fix reverted):

git checkout -b scratch-broken-7738 && git revert --no-commit -m 1 3122600e21d402dd958023d6eda7a95903f7b131 (conflicts in model_prices_and_context_window.json, its backup, and unit tests were resolved by taking HEAD for those files; only litellm/llms/databricks/cost_calculator.py stayed reverted, git status showed no tests/integration changes) && ./integ_local.sh restart-proxy && ./integ_local.sh run-node tests/integration/pricing/test_databricks_cache_pricing.py::test_databricks_cached_prompt_tokens_bill_at_cache_rates_not_input_rate

tests/integration/pricing/test_databricks_cache_pricing.py::test_databricks_cached_prompt_tokens_bill_at_cache_rates_not_input_rate FAILED [100%]
>            [...]

Green (fix present):

./integ_local.sh run-node tests/integration/pricing/test_databricks_cache_pricing.py::test_databricks_cached_prompt_tokens_bill_at_cache_rates_not_input_rate (run on branch tip 122d5dc32b after restart-proxy, following the red leg; the first run before the red leg also passed)

collecting ... collec [...]

Pylon 7775: For a proxy POST to /v1/responses with stream=true, the provider receives the Responses request and returns a 200 SSE st [...]

Red method: revert_fix_at_head

Red (fix reverted):

git checkout -b scratch-revert-7775 && git revert --no-commit c19d49d9192f26a7e15c9571d1c3a4e7ee45092a (conflicts in litellm/responses/streaming_iterator.py resolved by dropping the _restore_provider_response_headers call and method, keeping HEAD's _logging_copy) && git checkout HEAD -- tests/integration tests/test_litellm && integ_local.sh restart-proxy && /home/ubuntu/integ_local.sh run-node "tests/integration/observability/test_callback_delivery.py::test_streamed_responses_success_callback_carries_provider_apim_request_id"
=========================== short test summary info ============================
FAILED tests/integration/observability/test_callback_delivery.py::test_streamed_respons [...]

Green (fix present):

on pylon-regress-7775 tip afcdf5e659 after restart-proxy: /home/ubuntu/integ_local.sh run-node "tests/integration/observability/test_callback_delivery.py::test_streamed_responses_success_callback_carries_provider_apim_request_id"
configfile: pyproject.toml
plugins: cov-5.0.0, mock-3.15.1, respx-0.22 [...]

Pylon 7928: PR 37208 fixes completed batch accounting on the proxy batch lifecycle: after POST /v1/files and POST /v1/batches, GET / [...]

Red method: worktree_at_fix_parent_then_cherry_pick

Red (fix reverted):

git revert -m 1 --no-commit 20cfccaf5ff704fc75f32de01f456d208226d5d4 hit content conflicts in check_batch_cost.py, batch_utils.py and litellm_logging.py (later refactors on main), so revert was aborted and the fallback used: git worktree add /tmp/broken 20cfccaf5ff704fc75f32de01f456d208226d5d4^ ; rm -rf /tmp/broken/tests/integration && cp -r tests/integration /tmp/broken/tests/integration ; ln -s ~/repos/litellm/.venv /tmp/broken/.venv ; cp .circleci/scripts/wait_integration_services.py into the worktree (absent at that commit) ; docker exec postgres-db psql -U postgres -d circle_test -c 'DROP SCHEMA public CASCADE; CREATE SCHEMA public;' (old prisma schema could not db push over the newer s [...]

Green (fix present):

/home/ubuntu/integ_local.sh run-node tests/integration/spend/test_batch_completion_accounting.py::test_completed_batch_spend_row_records_reasoning_tokens_and_error_file_failures (on branch tip b08d9f7233, proxy restarted from ~/repos/litellm after the red leg)

collecting ... collected 1 item

tests [...]

Pylon 7958: POST /v1/ocr with a response whose `usage_info` includes `pages_processed_annotation` should bill those pages at `annota [...]

Red method: revert_fix_at_head

Red (fix reverted):

git checkout -b scratch-red-7958 && git revert --no-commit -m 1 b86791b4b968df6960b9a7a1f66bde794e7ee80e (conflict in litellm/cost_calculator.py resolved to pre-fix `ocr_cost_per_page: Final = pricing.get("ocr_cost_per_page")` with no annotation rate; deleted unit test left deleted) && git checkout HEAD -- tests/integration && /home/ubuntu/integ_local.sh restart-proxy && /home/ubuntu/integ_local.sh run-node tests/integration/pricing/test_ocr_page_pricing.py::test_ocr_annotation_pages_are_billed_at_annotation_cost_per_page

>           assert float(response.headers["x-litellm-response-cost"]) == pytest.approx(expected), response.text
E           AssertionError: {"pages":[...],"model":"integra [...]

Green (fix present):

git checkout pylon-regress-7958 && /home/ubuntu/integ_local.sh restart-proxy && /home/ubuntu/integ_local.sh run-node tests/integration/pricing/test_ocr_page_pricing.py::test_ocr_annotation_pages_are_billed_at_annotation_cost_per_page

collected 1 item
tests/integration/pricing/test_ocr_page_pricing. [...]

Commit df7db2c answers the review: the budget reset test asserts the new reset_at falls inside the next daily window from the update time instead of matching a second control budget, and the callback delivery poll keeps drained batches in a Final list with a mutable-ok reason since drain() consumes the queue; both were rerun red (fix reverted) and green. In CircleCI, pipeline 90108 (f54bc74) ran all eight integration shards green, including accounting (30 passed), management and extensions. Pipeline 90117 ran 767cb7d green on every integration shard

Commit 2d2bdf8 merges main (contracts.json union plus main's appended test in test_callback_delivery.py), pipeline 90122 ran it green on the integration workflow. Commit 240b274 answers the follow-up Bugbot review: the budget reset test starts from a 10d duration, since 30d is a monthly reset that can share a midnight with the 1d reset at month end, and the team default test moves to an owned proxy with default_team_params.budget_duration set in yaml and STORE_MODEL_IN_DB=False, so it no longer writes default team settings into the shared config; both tests were rerun red (fix reverted) and green

Type

✅ Test

Caveats (if any)

Low

• Pylon 7583 (Anthropic workspace header) was dropped: its fix only exists on the staging line, so the test is red on main by design
• Pylon 7872 (Responses websocket spend) was dropped: the child branch also edited production code, it needs its own PR
• Pylon 7691 (fail closed budget enforcement) was dropped: it needs a global config flag that breaks sibling tests, so it needs an owned_proxy rewrite
• 2247 added lines across 25 files, review per test using the table

Link to Devin session: https://app.devin.ai/sessions/d330510cfc8f463c823bd79c40b7e5e2
Open in Devin Desktop: https://app.devin.ai/desktop/session/d330510cfc8f463c823bd79c40b7e5e2?variant=devin

---

Note

Low Risk
Test-only changes plus harness/config tweaks for integration CI; no production proxy or billing logic is modified in this diff.

Overview
Adds 24 end-to-end integration tests (one per previously fixed customer bug) that drive the real proxy against scripted upstreams and assert x-litellm-response-cost, Postgres spend/budget rows, Redis cache, callbacks, or 422 budget decisions. New suites sit under tests/integration/spend, pricing, management, and observability, each wired into contracts.json via @pytest.mark.covers for shard contract checks.

The shared wire test server gains Reply.headers, pause_between_chunks, and chunked flush behavior so tests can simulate slow Bedrock event streams (client disconnect billing) and SSE /v1/responses with provider headers in generic-API callbacks. proxy_config.yaml sets proxy_batch_polling_interval: 1 to keep batch cost-polling tests deterministic.

A few existing integration tests are tightened (callback batch draining, cache partition helper) and owned_proxy is used where tests must not mutate shared default team params.

^{Reviewed by Cursor Bugbot for commit 865c3c2. Bugbot is set up for automated code reviews on this repo. Configure here.}

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge, with no outstanding correctness, security, or repository-rule findings.

Summary

This test-only PR adds integration regression coverage for 24 previously fixed cost tracking, budgeting, spend reporting, pricing, management, and observability defects.

• Extends the local wire harness with response headers and controlled streaming delays
• Adds real-proxy assertions covering persisted spend, response costs, budget decisions, callbacks, and guardrail behavior
• Registers each regression test in the integration contract map
• Moves configuration-sensitive team-default coverage to an owned proxy
• Both previous review findings are resolved

_{Reviews (8) · Last reviewed commit: "Merge remote-tracking branch 'origin/mai..."}

[cursor]

Stale Bugbot comment from a previous run.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[cursor]

Stale Bugbot comment from a previous run.

[devin-ai-integration]

Addressed the outside-diff finding in 96213de and 767cb7d: a completion_id helper and three Final ids replace the rebound dict

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Team defaults leak across shared proxy

Medium Severity

_default_team_budget_duration patches /update/default_team_settings on the shared proxy and persists it to the config DB. Cleanup sends {}, which stores {models: []} rather than restoring the original unset default_team_params, so later team-create tests on this shard (or after a reload) inherit that leftover default.

 

^{Reviewed by Cursor Bugbot for commit 2d2bdf8. Configure here.}

[devin-ai-integration]

Fixed in 240b274: the test now runs on an owned proxy with the default set in yaml, so nothing is persisted

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[greptile-apps]

These tests are not vacuous. Most exercise the real proxy and verify independent outcomes such as response cost plus persisted spend, budget decisions plus upstream dispatch, or agreement between daily activity and global spend reports. Several also assert the exact request received by the scripted upstream, so a canned success response cannot make them pass. The documented red/green mutation runs further support that they detect regressions.

There is moderate intentional coupling: some tests assert internal details such as Redis key names, call_type, metadata.cost_breakdown, callback hidden_params, and batch_processed; exact synthetic token totals can also make fixtures somewhat brittle. The contract mappings add metadata but no behavioral protection, and the mutation evidence is described rather than encoded as a runnable mutation suite.

Overall: high-signal regression coverage with moderate intentional coupling, not overfit or vacuous. If brittleness appears, relax implementation-name/storage-shape assertions while preserving the independent accounting invariants and upstream-dispatch checks.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 865c3c2. Configure here.}

[cursor]

Batch status asserted without polling

Low Severity

GET /v1/batches/{id} is asserted completed on the first response. The same lifecycle in test_batch_observability.py polls for that status. A retrieve that still reflects the create-time validating row fails immediately even though spend is polled for 70s afterward.

Additional Locations (1)

• tests/integration/spend/test_batch_observability.py#L183-L188

 

^{Reviewed by Cursor Bugbot for commit 865c3c2. Configure here.}