fix(caching): key response cache by router model group in litellm_metadata

[devin-ai-integration]

TLDR

Problem this solves:

• Scheduled CircleCI on main is red in unit, integration-management(-replica), integration-accounting, integration-extensions, plus parts of integration-providers and integration-cost
• Each failure pins behavior a merged PR changed on purpose, or was wrong from the day it landed

How it solves it:

• Updates each test to assert the new behavior just as strictly, no skips, retries or longer waits
• One real bug fix in Cache._get_model_param_value: it now reads model_group from litellm_metadata too, so /v1/responses cache entries stop leaking across model groups

Failures fixed

Red scheduled run on main (pipeline 90845, sha 1d52985d03), same failures in 90840, 90836 and 90824

Job on main	Test	Culprit	Mechanism	Fix
[unit]([REDACTED SECRET]/gh/BerriAI/litellm/2240217)	tests/unit/proxy/proxy_server/test_routes_model_cost_map.py::test_reload_model_cost_map_surfaces_the_blob_id_of_the_bytes_served_on_every_status_surface	347798b (#35491) replays runtime registrations after a reload	models_count was compared to len(litellm.model_cost), which includes models other tests registered in the same xdist worker (4472 == 4477)	Compare against the count of the served map itself
[integration-management]([REDACTED SECRET]/gh/BerriAI/litellm/2240196), [replica]([REDACTED SECRET]/gh/BerriAI/litellm/2240197)	tests/integration/management/test_vector_store_file_list_managed_ids.py::test_provider_errors_reach_the_caller_and_other_models_keep_mapping[401,404,500]	6d8434f (#43787)	Provider error statuses are now raised through LiteLLM's mapped exceptions, so the caller gets the standard error envelope instead of the raw provider body	Pin status code, provider message, code and the mapped type per status
same	tests/integration/authorization/test_team_scoped_models.py::test_v2_model_info_reports_a_team_model_to_team_and_non_team_keys	f850b2c (#44451) added two shared YAML deployments	The helper sent model_name=, which /v2/model/info does not accept, so the filter was ignored and the new deployments showed up	Use the real model filter
[integration-accounting]([REDACTED SECRET]/gh/BerriAI/litellm/2240195)	tests/integration/spend/test_roi_branch_spend.py::test_documented_header_and_body_tags_reach_recorded_branch_and_pr_cost	1a70233 (#44426), red since it landed	Three identical requests, so two were response-cache hits with zero spend (0.011 == 0.033)	Distinct message per request, plus assert three upstream calls
same	tests/integration/pricing/test_per_second_pricing.py::test_streaming_chat_per_second_pricing_covers_the_full_stream[new_field]	6d8434f (#43787) records GET requests on the shared upstream	A background GET /v1/models landed in the observations next to the single chat call	Count only POST observations, still exactly one
[integration-extensions]([REDACTED SECRET]/gh/BerriAI/litellm/2240194)	tests/integration/observability/test_otel_v1_request_trace.py and 8 tests in tests/integration/observability/test_otel_excluded_services.py	797353f (#44240)	The batch_write_to_db cost-callback span was removed on purpose; the spend flush is now its own postgres.update LiteLLM_VerificationToken trace	Assert the new span, its separate trace, and its exclusion at the tenant
[integration-providers]([REDACTED SECRET]/gh/BerriAI/litellm/2240198)	tests/integration/routing/test_stale_cost_map_boot.py::test_config_deployment_dropped_by_stale_boot_cost_map_is_restored_after_reload	6d8434f (#43787) / shared upstream now serializes method and api_key	Exact observation dict lacked the two new keys	Add them to the expected dict
[integration-cost]([REDACTED SECRET]/gh/BerriAI/litellm/2240200)	test_case_bills_expected_cost[perplexity/pplx-decider-v1-27b-decisions], [claude-sonnet-5-messages_upstream_500_zero_spend]	6d8434f (#43787)	Same observation shape; /v1/messages without max_tokens is now a 400 before the upstream	Add the keys; send max_tokens so the case still reaches the upstream 500
integration-cost	test_case_bills_expected_cost[gpt-5.6-responses_web_search_medium], [gpt-5.6-responses_web_search_reported_count_mixed_actions], [gpt-5.6-responses_web_search_reported_count_open_page_only]	a849093 (#44414) added two cases whose request is identical to web_search_medium, exposing a latent cache bug	The router puts model_group in litellm_metadata for /v1/responses, and the cache key only read metadata, so it fell back to the shared openai/gpt-5.6 and one case was served another's cached response (0.021488 != 0.008988, or spend 0.0)	_get_model_param_value reads model_group and caching_groups from metadata and litellm_metadata in both kwargs and litellm_params

Deferred

Still red on main and not claimed here, left for the next run:

• integration-providers: test_bedrock_mantle_claude_chat_wire.py::test_health_check_on_a_mantle_claude_deployment_probes_the_native_messages_route, test_bedrock_stream_timeout_wire.py::test_c8_the_bedrock_passthrough_stream_already_retries_at_the_deployment_timeout
• integration-cost: claude-sonnet-5-stream_no_usage and claude-sonnet-5-stream_no_usage_image_input (recounted prompt tokens 48 vs 47 and 303 vs 301, red since the 2026-10-03 12:09 run, range a76ba8c..e768ad5, most likely 7b432d7 (fix(proxy): stamp the client alias on a copy of each streamed chunk so pricing sees the deployment model #44341))
• integration-providers / build_and_test: xAI provider jobs fail with provider errors on main too, not code
• integration-database-replica: test_managed_file_flat_ids_index.py, test_partition_transactions.py
• integration-mcp, integration-database: no stable test-level failure signature yet
• integration-security: the S2 route sweep intermittently gets RemoteProtocolError on GET /organization/daily/activity* as internal_user, the same signature that failed main in pipelines 90758 and 90795, so it is a flake owned by the deflake automation. It failed here in 2240853, and in the rerun, then passed all 131 tests in rerun 2241029 at the tip. The failing tests also passed 2 of 2 locally at both base and head
• Flakes owned by the deflake automation: test_azure_storage_chaos, test_passthrough_upstream_error_chaos, test_get_favicon_*, plus one-off unit failures on this branch in test_arize_dynamic_params and test_premium_check_receives_the_field_name, and an integration-extensions proxy that exited before readiness in test_guardrail_effects.py
• GitHub Actions proxy-behavior and roi-database failed only in their Codecov upload step at 30a8b32b19, a signature check that was red on main at the same time

CircleCI on this branch

• Pipeline 90847 at ce7ed5b1f5
• Pipeline 90848 at 864d9844fd, rerun from failed build_and_test and integration. Every claimed test passes there
• Pipeline 90854 at 30a8b32b19, after the cache key fix. The three gpt-5.6-responses_web_search_* cases pass in integration-cost and again in its rerun
• Pipeline 90858 at 5d5a47fbfe, after merging current main. The three gpt-5.6-responses_web_search_* cases pass in integration-cost, unit and every claimed job are green, and the remaining reds match main pipeline 90859 at 02f61c9c42

Pre-Submission checklist

• I have added meaningful tests
• The handful of test files covering my change pass locally
• My PR passes all required CI/CD checks
• My PR's scope is as isolated as possible; it only solves 1 specific problem

Screenshots / Proof of Fix

The test changes are proven by the previously red CircleCI jobs passing on this branch, linked above

For the cache fix, two model groups group-a and group-b both pointing at openai/gpt-5.4-nano with cache: true (local), real OpenAI calls, the same /v1/responses body sent to group-a then group-b, base origin/main on :4001 and this branch on :4002

for g in group-a group-b; do curl -s -D h.txt -o b.json localhost:$p/v1/responses -H "Authorization: Bearer sk-qa-1234" -H "Content-Type: application/json" -d "{\"model\":\"$g\",\"input\":\"Reply with one word: ping $p\"}"; grep -i "x-litellm-cache-key\|x-litellm-model-group" h.txt; done

== port 4001 (base origin/main)
group-a -> model-group: group-a
group-b -> cache-key: 5acc60dd6bf92028bc299e9fdd4c8cb987b45756508dbcbb6b6633dcf086c242 model-group: group-b
== port 4002 (head)
group-a -> model-group: group-a
group-b -> model-group: group-b

On base, group-b was answered from group-a's cache entry. On this branch it reaches its own deployment, and repeating the same request to group-a is still a cache hit on both sides (x-litellm-cache-key: 92b0c7f37e98... on :4002)

Type

Bug Fix
Test

Caveats (if any)

Low

• The legacy OTel v1 logger no longer exports a spend-write span at all
  • Its export hook drops spans without a parent, and the flush has none since fix(otel): name postgres service spans by operation and table #44240
  • The test pins that; not changed here
• With OTel v2, tenant destinations no longer receive the spend-flush span either
  • The flush runs outside any request, so only the operator destination gets it
  • Left as is, out of scope for this PR

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/eee2353031f946a0a9f5f4787568e664
Open in Devin Desktop: https://app.devin.ai/desktop/session/eee2353031f946a0a9f5f4787568e664?variant=devin
Requested by: @mateo-berri

---

Note

Medium Risk
The cache key change affects response caching for router model groups (especially /v1/responses); incorrect metadata precedence could still mis-key caches, but the fix narrows cross-group leakage.

Overview
Aligns red CircleCI integration and unit tests with intentional behavior on main, and fixes a cache key bug where /v1/responses could share entries across router model groups.

Cache fix: Cache._get_model_param_value now resolves model_group and caching_groups from metadata and litellm_metadata on both top-level kwargs and litellm_params, so cache keys respect the router’s model group instead of collapsing to the underlying provider model.

Test updates (same strictness, new expectations):

• OTel: spend flush is postgres.update LiteLLM_VerificationToken on its own trace, not batch_write_to_db on the request trace; tenant postgres exclusion assertions follow suit.
• Provider/management: vector-store list errors expect LiteLLM’s mapped error type; /v2/model/info filters with model= not model_name=.
• Cost/spend/pricing: upstream observation dicts include method/api_key; ROI tag test uses distinct prompts and expects three upstream calls; messages cost case sends max_tokens; per-second pricing ignores non-POST observations.
• Unit: model cost map reload models_count matches the served finalized map, not polluted global litellm.model_cost.

^{Reviewed by Cursor Bugbot for commit 5d5a47f. Bugbot is set up for automated code reviews on this repo. Configure here.}

• 5d5a47f passes /live-pr-risk

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[greptile-apps]

Confidence Score: 5/5

[Low risk] Test updates to match changed behavior on main.

The PR appears safe to merge based on the changes since the previous review.

Summary

The PR updates tests and fixture data to match current proxy behavior without changing product code. Since the previous review, it adds a spend-persistence poll before the OTel v1 request-trace assertion.

_{Reviews (2) · Last reviewed commit: "test(integration): wait for the spend fl..."}

[mateo-berri]

bugbot run

[mateo-berri]

@greptileai

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_circleci_heal_20261005 (5d5a47f) with main (02f61c9)}

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 5d5a47f. Configure here.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mateo-berri]

LGTM