fix(proxy): keep request metadata out of the cost tracking failure alert

[devin-ai-integration]

TLDR

Problem this solves:

• The cost tracking failure alert f-strings the whole request metadata
• One 250-byte request became a 23 KB Slack alert
• The string is built on every failing request, whatever the log level
• A read-only Redis replica or a Redis OOM hits that path on every request

How it solves it:

• The alert keeps the exception, the traceback, the model, and the call type
• The metadata keys are logged once, at debug level, through lazy %s args
• Metadata values are never stringified on the failure path
• Regression test: a metadata value whose repr raises, run at WARNING and DEBUG

Decisions:

• Keys instead of a bounded repr: a repr still stringifies every value, which is what blew up, and can carry secrets
• One sorted tuple of keys per source (chosen, litellm_metadata, old_metadata); a source that is not a mapping logs ()
• Debug level only, so at WARNING nothing is computed

User Flow

Before: a proxy admin whose Redis cache turned into a read-only replica after a failover gets a 23 KB failed_tracking_spend Slack alert carrying a developer's whole request metadata, headers, and key details, rebuilt on every request whatever the log level

1. The admin runs the proxy with LITELLM_LOG=WARNING, Slack alerting on, and alert_types: ["failed_tracking_spend"]; a failover leaves the Redis cache pointing at a read-only replica
2. A developer sends POST https://litellm-domain/v1/chat/completions with "model": "healthy-chat", a one-line prompt, and "metadata": {"tags": ["lit7506-repro"], "user_context": "<a 90-character note>"}, about 250 bytes in all
3. They get 200 with the completion, as usual
4. The admin's Slack channel gets a failed_tracking_spend alert of 23 to 24 KB: after Error in tracking cost callback - ... You can't write against a read only replica and the traceback it carries chosen_metadata: {...} and old_metadata: {...}, which hold the developer's metadata blob four times, the request headers four times, two key-auth dumps with the key hash and budgets, and the deployment's api_base
5. The proxy log shows one ERROR ... Error in tracking cost callback line per request, and the 23 KB string is rebuilt for every request on that model even though the alert itself is sent once a day per model

After: the same failover gets the admin a short alert with the error, the traceback, the model, and the call type, and the metadata keys only show up at debug level

1. The admin runs the proxy with LITELLM_LOG=WARNING, Slack alerting on, and alert_types: ["failed_tracking_spend"]; a failover leaves the Redis cache pointing at a read-only replica
2. A developer sends POST https://litellm-domain/v1/chat/completions with "model": "healthy-chat", a one-line prompt, and "metadata": {"tags": ["lit7506-repro"], "user_context": "<a 90-character note>"}, about 250 bytes in all
3. They get 200 with the completion, as usual
4. The admin's Slack channel gets a failed_tracking_spend alert of 5,295 bytes: the You can't write against a read only replica error, the traceback, model: healthy-chat, and call_type: acompletion, with nothing from the developer's metadata, headers, or key
5. The proxy log shows the same one ERROR line per request and nothing else at WARNING; only with LITELLM_LOG=DEBUG does it add one line naming the metadata keys, such as chosen_metadata keys=('headers', 'user_api_key', ...), never their values

Relevant issues

Affected release

Linear ticket

Resolves LIT-7506

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Same rig for both legs, only the proxy commit differs. Each run boots a two-worker proxy (--num_workers 2) on a random port with Postgres behind it and litellm_settings.cache pointing at a Redis read-only replica on 127.0.0.1:59361, which is what a failover leaves behind, so every request's spend counter INCRBYFLOAT fails and the cost tracking callback hits its except block. SLACK_WEBHOOK_URL points at a local receiver that records each Slack POST as one JSON line. Config: qa-chat, qa-responses, and qa-messages all on openai/gpt-5.4-mini (real OpenAI calls), alerting: ["slack"], alert_types: ["failed_tracking_spend"]. Each leg ran twice, LITELLM_LOG=WARNING and LITELLM_LOG=DEBUG. Every request carries a per-run sentinel string in an x-lit7506-note header and in metadata.user_context, so grepping the alerts and the proxy log for $SENTINEL shows whether the request metadata got out. The WARNING run is shown in full and the DEBUG run shows the alert summary and the log greps

Before (5fc510a)

LITELLM_LOG=WARNING, SENTINEL=lit7506-sentinel-1789813802-576956

1. The three requests, one per unified endpoint, all 200 with a real completion

$ curl -s http://127.0.0.1:$PORT/v1/chat/completions -H "Authorization: Bearer sk-lit7506" -H "Content-Type: application/json" -H "x-lit7506-note: $SENTINEL" -d '{"model":"qa-chat","messages":[{"role":"user","content":"Say hi in three words"}],"metadata":{"tags":["lit7506-qa"],"user_context":"$SENTINEL lorem ipsum dolor sit amet, a realistic customer metadata blob"}}' | jq -c "{id, model, text: .choices[0].message.content}"
{"id":"chatcmpl-EPmcy6GHQaTfDZHnLVmDon8viUFxa","model":"qa-chat","text":"Hi there, friend."}

$ curl -s http://127.0.0.1:$PORT/v1/responses -H "Authorization: Bearer sk-lit7506" -H "Content-Type: application/json" -H "x-lit7506-note: $SENTINEL" -d '{"model":"qa-responses","input":"Say hi in three words","metadata":{"user_context":"$SENTINEL lorem ipsum dolor sit amet, a realistic customer metadata blob"}}' | jq -c "{id, model, text: .output[-1].content[0].text}"
{"id":"resp_esy1iFiY_F7vCDtFEwWiYz7XMF-ctoTeIGzt5rutP4aHg-362V4G6pKlPJhwvnSQC1RW9Wo7J-ExelNpqRF3b_LgIM9OEWqeaNWSXnSngtH3iN1xn7LFO_zwzjFT8ljeXuaTuyl5JerA0QsPBPLxhzth1d0kPwUbfLzAxdankKbWuNkUHukmWZfZ6kNv4Cobc3ZzIBs6N8sZ314xMcRHkExM5Ivg_QRSmDVobuiWqWqLOZ729M0ekOBGB5qgFnsp7K1COYso_2NgRiGXMJAGd3ZfpYuesnYX0iYnngbsTw1ePWwsb4NMxLWsggm0iS0V-tkxuHrydUm2BOnxJHHovelCM2-GuaTOYPtFD9wWoJxJaGy2kC4k2dQ4EMGxUyW5W0fV7lMLTBhiOQWAzNdVTPH1aDaykkdxrDMsuXKuP3n0ERJ3_AC71PZ8zBiS2NHH6Ao9rxoUr48SjimNawfSGPL7","model":"qa-responses","text":"Hi there, friend."}

$ curl -s http://127.0.0.1:$PORT/v1/messages -H "Authorization: Bearer sk-lit7506" -H "Content-Type: application/json" -H "anthropic-version: 2023-06-01" -H "x-lit7506-note: $SENTINEL" -d '{"model":"qa-messages","max_tokens":32,"messages":[{"role":"user","content":"Say hi in three words"}],"metadata":{"user_id":"lit7506-qa-user"}}' | jq -c "{id, model, text: .content[0].text}"
{"id":"resp_bGl0ZWxsbTpjdXN0b21fbGxtX3Byb3ZpZGVyOm9wZW5haTttb2RlbF9pZDo1M2E5YTE4ZTQ4Mjk4MmM0MzFmNjUxNDcxZDM3ZDVjZTkxZjA5OTFhZTYxNjNhOWU2M2JhYzUyN2RiMmM5NWIwO3Jlc3BvbnNlX2lkOnJlc3BfMGRkODM4Mzg2MzRkMjQ0ODAwNmFhZTY0NGE0MjFjODdkMGFiNGM5M2FmYjhmZjg1NjI=","model":"qa-messages","text":"Hi there, friend."}

2. Alerts at the Slack webhook: bytes, and whether the text carries the request metadata, the headers, the key auth dump, or the sentinel

bytes=24130 model=gpt-5.4-mini chosen_metadata=True headers=True UserAPIKeyAuth=True sentinel=True sentinel_hits=7 call_type=True
bytes=23642 model=gpt-5.4-mini chosen_metadata=True headers=True UserAPIKeyAuth=True sentinel=True sentinel_hits=8 call_type=True

3. Tail of the first alert, each line cut at 220 characters; the three metadata lines are 5,008, 5,009, and 8,676 characters long and each repeats the request headers and the developer's metadata blob

 Args to _PROXY_track_cost_callback
 model: gpt-5.4-mini
 chosen_metadata: {'headers': {'host': '127.0.0.1:29658', 'user-agent': 'curl/8.7.1', 'accept': '*/*', 'content-type': 'application/json', 'x-lit7506-note': '$SENTINEL', 'content-length': '184'}, ...
 litellm_metadata: {'headers': {'host': '127.0.0.1:29658', 'user-agent': 'curl/8.7.1', 'accept': '*/*', 'content-type': 'application/json', 'x-lit7506-note': '$SENTINEL', 'content-length': '184'} ...
 old_metadata: {'headers': {'host': '127.0.0.1:29658', 'user-agent': 'curl/8.7.1', 'accept': '*/*', 'content-type': 'application/json', 'x-lit7506-note': '$SENTINEL', 'content-length': '184'}, 'r ...
 call_type: aresponses

Proxy URL: `http://localhost:4000`

4. Proxy log: one ERROR per request, no keys line, and the sentinel stays out of the WARNING log (it only ever reached Slack)

$ grep -c "Error in tracking cost callback" proxy.log
3
$ grep "Cost tracking callback failed" proxy.log
$ grep -c "$SENTINEL" proxy.log
0

LITELLM_LOG=DEBUG, SENTINEL=lit7506-sentinel-1789813853-523069, same three requests, all 200

bytes=23638 model=gpt-5.4-mini chosen_metadata=True headers=True UserAPIKeyAuth=True sentinel=True sentinel_hits=8 call_type=True

$ grep -c "Error in tracking cost callback" proxy.log
3
$ grep "Cost tracking callback failed" proxy.log
$ grep -c "$SENTINEL" proxy.log
23

After (9e8a847)

LITELLM_LOG=WARNING, SENTINEL=lit7506-sentinel-1789813801-284411

1. The three requests, one per unified endpoint, all 200 with a real completion

$ curl -s http://127.0.0.1:$PORT/v1/chat/completions -H "Authorization: Bearer sk-lit7506" -H "Content-Type: application/json" -H "x-lit7506-note: $SENTINEL" -d '{"model":"qa-chat","messages":[{"role":"user","content":"Say hi in three words"}],"metadata":{"tags":["lit7506-qa"],"user_context":"$SENTINEL lorem ipsum dolor sit amet, a realistic customer metadata blob"}}' | jq -c "{id, model, text: .choices[0].message.content}"
{"id":"chatcmpl-EPmcxZOhk57C5UAtDGET7HMj4UJ4w","model":"qa-chat","text":"Hi there, friend."}

$ curl -s http://127.0.0.1:$PORT/v1/responses -H "Authorization: Bearer sk-lit7506" -H "Content-Type: application/json" -H "x-lit7506-note: $SENTINEL" -d '{"model":"qa-responses","input":"Say hi in three words","metadata":{"user_context":"$SENTINEL lorem ipsum dolor sit amet, a realistic customer metadata blob"}}' | jq -c "{id, model, text: .output[-1].content[0].text}"
{"id":"resp_UajfOcGXCD5YS8GxLieLCrwINE-6LSKGTvK0D3Vvj1lH4rG2WtyWzVnoLRVYWqIU0sX01ZqhTtibxpEG6ZJu5ld6JiVDSxJdyBaJN48UmMmSOO7uaOXfwztboic0H6njAVWBvL08mj_9wmiRAo0zwb-dXlz6955Th61xgvoiqlU0_hoBo2Duw7RypFlJqQNDMPO7KZuR6DcIQDgZ6EWZeP2xRK7bn0N0aCBTwg_VA4rvUDx1oGZgQelhsiR7-_c4TItrOIV1lVW9HXBc-wqeFeoUMaWGhFz2SeSIdW1igLcE716tKQzk6yop587H3VXhEybmuRp7WUck1fXWrWRXIeC999IVCaE1c49kCDSQzjCSw2cBiMD4ZbylnVH34K0cyYgBNKQKOYWyF9U6TiuSWqT-DatkeN5RZepAP_vefvQmXq2qN60ydhKWK2-8Z11wKiDxWyzTwrjU8qMCMH7uYcFA","model":"qa-responses","text":"Hi there, friend."}

$ curl -s http://127.0.0.1:$PORT/v1/messages -H "Authorization: Bearer sk-lit7506" -H "Content-Type: application/json" -H "anthropic-version: 2023-06-01" -H "x-lit7506-note: $SENTINEL" -d '{"model":"qa-messages","max_tokens":32,"messages":[{"role":"user","content":"Say hi in three words"}],"metadata":{"user_id":"lit7506-qa-user"}}' | jq -c "{id, model, text: .content[0].text}"
{"id":"resp_bGl0ZWxsbTpjdXN0b21fbGxtX3Byb3ZpZGVyOm9wZW5haTttb2RlbF9pZDo1M2E5YTE4ZTQ4Mjk4MmM0MzFmNjUxNDcxZDM3ZDVjZTkxZjA5OTFhZTYxNjNhOWU2M2JhYzUyN2RiMmM5NWIwO3Jlc3BvbnNlX2lkOnJlc3BfMDI4ZjgwOGFiOGUyZDg1OTAwNmFhZTY0NDk3NGI0ODdkMDhlZGNkYzAzZTZjODBiNGI=","model":"qa-messages","text":"Hi there, friend."}

2. Alerts at the Slack webhook: 5,295 bytes, no metadata, no headers, no key auth dump, no sentinel

bytes=5295 model=gpt-5.4-mini chosen_metadata=False headers=False UserAPIKeyAuth=False sentinel=False sentinel_hits=0 call_type=True

3. Tail of the first alert: the exception and traceback above it are unchanged, then only the model and the call type

 Args to _PROXY_track_cost_callback
 model: gpt-5.4-mini
 call_type: acompletion

Proxy URL: `http://localhost:4000`

4. Proxy log: the same one ERROR per request, nothing else at WARNING

$ grep -c "Error in tracking cost callback" proxy.log
3
$ grep "Cost tracking callback failed" proxy.log
$ grep -c "$SENTINEL" proxy.log
0

LITELLM_LOG=DEBUG, SENTINEL=lit7506-sentinel-1789813881-287290, same three requests, all 200

bytes=5295 model=gpt-5.4-mini chosen_metadata=False headers=False UserAPIKeyAuth=False sentinel=False sentinel_hits=0 call_type=True
bytes=5294 model=gpt-5.4-mini chosen_metadata=False headers=False UserAPIKeyAuth=False sentinel=False sentinel_hits=0 call_type=True

One keys line per request (3,201, 4,554, and 4,562 characters; shown cut after the first six keys), naming the metadata keys and never a value

$ grep -c "Error in tracking cost callback" proxy.log
3
$ grep "Cost tracking callback failed" proxy.log
LiteLLM Proxy:DEBUG: proxy_track_cost_callback.py:456 - Cost tracking callback failed for model=gpt-5.4-mini call_type=acompletion; chosen_metadata keys=('_litellm_router_usage_counted_tokens', '_routing_request_tags', 'agent_id', 'api_base', 'attempted_fallbacks', 'attempted_retries', ... 62 key names, no values) litellm_metadata keys=(...) old_metadata keys=(...)
LiteLLM Proxy:DEBUG: proxy_track_cost_callback.py:456 - Cost tracking callback failed for model=gpt-5.4-mini call_type=aresponses; chosen_metadata keys=('_litellm_router_usage_counted_tokens', '_routing_request_tags', 'agent_id', 'api_base', 'attempted_fallbacks', 'attempted_retries', ... 59 key names, no values) litellm_metadata keys=(...) old_metadata keys=(...)
LiteLLM Proxy:DEBUG: proxy_track_cost_callback.py:456 - Cost tracking callback failed for model=gpt-5.4-mini call_type=anthropic_messages; chosen_metadata keys=('_litellm_router_usage_counted_tokens', '_routing_request_tags', 'agent_id', 'api_base', 'attempted_fallbacks', 'attempted_retries', ... 59 key names, no values) litellm_metadata keys=(...) old_metadata keys=(...)
$ grep -c "$SENTINEL" proxy.log
23

Observations from the runs, none caused by this PR:

• Alert dedupe is per deployment model, hence 1 or 2 alerts
• model=gpt-5.4-mini is the deployment model, not the alias, both legs
• 23 DEBUG sentinel hits come from request logging, both legs
• Chat completions leaves litellm_metadata empty, pre-existing

Type

🐛 Bug Fix

Caveats (if any)

Three CircleCI jobs are red at 9e8a847 and none of them reaches this change (Low). litellm_utils_testing fails test_models_by_provider on every main pipeline since #41515 added the Amazon Transcribe cost map entry (main pipelines 89818 and 89834 fail it at tips without this PR, and the merge base 5fc510a already carries that entry); tracked separately. proxy_e2e_anthropic_messages_tests fails the two test_bedrock_invoke_messages_with_all_beta_headers cases with Bedrock's invalid beta flag on the same two main pipelines; tracked in LIT-8149. google_generate_content_endpoint_testing hit a Vertex 429 RESOURCE_EXHAUSTED on gemini-2.5-flash-lite that main's scheduled runs pass; the from-failed rerun (workflow d2ca33c2) got the same 429 on two of the streaming tests, and every PR pipeline started after 10:25Z that day (89859, 89860, 89861) fails the same test_proxy_genai_sdk_* tests with the same 429 while every pipeline before 10:25Z passes them, so it is the shared Vertex quota, not this change

Live PR risk at 9e8a847

No breaking change and no regression found. The one contract change is the intended one: the failed_tracking_spend Slack alert text no longer carries the chosen_metadata, litellm_metadata and old_metadata blocks (24,130 bytes to 5,295 bytes for the same failure), so an operator who read those blocks out of the Slack message loses them. The model: and call_type: lines stay, the exception and traceback stay, and LIT-7506 asks for exactly this removal, so it ships as asked

Regression risk: none left unreached. Every dependent of the changed block was driven live on base 5fc510a and head 9e8a847 (2 uvicorn workers, /v1/chat/completions, /v1/responses and /v1/messages against a real gpt-5.4-mini deployment, the failure forced by a read-only Redis replica, LITELLM_LOG at WARNING and at DEBUG, the alert captured by a forwarding recorder at the webhook boundary)

Dependency graph

• ProxyLogging.failed_tracking_alert to SlackAlerting.failed_tracking_alert (litellm/proxy/utils.py, litellm/integrations/SlackAlerting/slack_alerting.py): the only reader of error_msg; the dedupe key uses failing_model, which is unchanged. Verified live (same alert count and same per-deployment dedupe on both sides) and tested (tests/test_litellm/proxy/utils/proxy_logging/test_alerting.py, plus the new regression test)
• spend_log_error(...) ERROR line: outside the diff, arguments unchanged. Verified live (3 lines per run on both sides)
• new verbose_proxy_logger.debug(...) line: readers are the log handlers, JSON formatter included, which format through getMessage(). Tested by the new parametrized test with a logging.Handler at WARNING and at DEBUG; verified live (absent at WARNING, one keys-only line per failing request at DEBUG)
• _metadata_keys: private, one caller. Tested; Codecov reports every modified line covered
• string readers of the removed lines (chosen_metadata, Args to _PROXY_track_cost_callback) across litellm/, enterprise/, tests/, ui/ and the docs checkout: none; docs/proxy/alerting.md names only the alert type
• merge ref: main moved 13 commits past 5fc510a, two of them touching slack_alerting.py budget wording and none touching failed_tracking_alert or this hook. The merged tree (3aa49e8fd0) merges clean, the mapped test file passes on it, and the WARNING scenario re-run on it gives a 5,428-byte alert with no metadata, headers, key auth or sentinel in it, matching the head leg

Not verified

• rendering in a real Slack channel: the recorder captured the exact webhook POST body on both sides, but no real Slack workspace received it
• JSON_LOGS=True formatting of the new debug line on a live proxy: covered by the unit test's handler only

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

• 9e8a847 passes /live-pr-risk

---

Note

Low Risk
Observability-only change to failure alerting and debug logging; spend tracking behavior is unchanged aside from shorter alert text.

Overview
When cost tracking fails, failed_tracking_spend alerts no longer embed full request metadata (chosen_metadata, litellm_metadata, old_metadata). The Slack/error payload keeps the exception, traceback, model, and call_type only, avoiding huge alerts and accidental leakage of headers or customer metadata on paths like read-only Redis.

Metadata is surfaced only at DEBUG via a lazy verbose_proxy_logger.debug line that logs sorted key names per metadata bucket (never values), using a new _metadata_keys helper. A parametrized test asserts metadata values are never stringified on the failure path at WARNING or DEBUG.

^{Reviewed by Cursor Bugbot for commit 9e8a847. Bugbot is set up for automated code reviews on this repo. Configure here.}

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_cost_callback_bounded_error_msg (9e8a847) with main (6b8db68)}

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge, with no outstanding correctness or repository-rule violations.

Findings

1.  Caller-Controlled Metadata Keys ▶

Summary

This PR prevents request metadata values from being included or stringified when cost tracking fails.

• Keeps the exception, traceback, model, and call type in failure alerts.
• Moves metadata diagnostics to lazy debug logging and emits only sorted key names.
• Adds regression coverage at WARNING and DEBUG levels using metadata values that fail if stringified.

_{Reviews (2) · Last reviewed commit: "fix(proxy): keep request metadata out of..."}

[greptile-apps]

Caller-Controlled Metadata Keys

Debug logging includes every caller-controlled metadata key, so sensitive identifiers embedded in key names can persist in logs. Consider logging fixed categories or counts instead

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[mateo-berri]

At DEBUG the proxy already logs whole request bodies, so key names add no new exposure; values are never stringified, per the ticket's ask

[greptile-apps]

That context addresses the concern. Since DEBUG already emits the full request body, logging the metadata key names does not add meaningful exposure at that level; importantly, _metadata_keys() never stringifies metadata values, and the failure alert remains free of them. No change is needed for this comment.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 9e8a847. Configure here.}

[mateo-berri]

LGTM