fix(sentry): scrub PII and secrets inside object reprs and nested locals, add SENTRY_SEND_DEFAULT_PII opt-in

[devin-ai-integration]

TLDR

Problem this solves:

• Sentry frame locals carried the caller's email and hashed key inside object reprs
• The master key, the database URL, and the raw virtual key rode along in nested dicts
• Only a top-level user_id local read [Filtered]; send_default_pii was hardwired off with no opt-in

How it solves it:

• Scrubs every string in the event for secret and PII field names, sk- keys, emails, and hashes; the bare sk- match starts at the shorter of the custom key minimum and a generated key, so both key shapes match under any pair of settings
• Keeps Sentry's own denylists and adds the proxy's auth header names to them
• SENTRY_SEND_DEFAULT_PII=true opts into identity fields; secrets stay filtered either way
• Source context lines stay readable so the stack trace still makes sense

User Flow

Before: the operator opens the Sentry issue for a rejected request and reads the caller's email, hashed key, raw virtual key, the master key, and the database URL in the frame locals, while the one user_id local reads [Filtered]

1. The admin creates an internal user with POST https://litellm-domain/user/new and {"user_id": "alice@example.com", "user_email": "alice@example.com", "user_role": "internal_user"}, and gets back a key sk-...
2. Alice sends POST https://litellm-domain/v1/chat/completions (or /v1/responses, or /v1/messages with the key in x-api-key) with "model": "no-such-model" and gets 400 Invalid model name, then POST https://litellm-domain/team/new and gets 401 Only proxy admin ... Your user_id=ali***om
3. The operator opens the issues at https://<org>.sentry.io/issues/<id>/: user_id reads [Filtered], but user_api_key_dict reads UserAPIKeyAuth(token='5e77...1ba2', ..., user_id='alice@example.com', user_email='alice@example.com', ...), data.metadata.user_api_key_hash reads 5e77...1ba2, and raw_headers.authorization reads Bearer sk-... in clear
4. general_settings reads {'master_key': 'sk-...', 'database_url': 'postgresql://user:password@host/db'} in clear, and setting SENTRY_SEND_DEFAULT_PII=true changes nothing
5. Anyone who can read the Sentry project (support, a contractor, a Slack alert integration) can take the master key, the database credentials, and Alice's own key, and can tell which user each failing request belonged to

After: the same issues read [Filtered] for the email, the hashed key, the raw key, the master key, and the database URL, and SENTRY_SEND_DEFAULT_PII=true lets the operator see the identity fields on purpose

1. The admin creates an internal user with POST https://litellm-domain/user/new and {"user_id": "alice@example.com", "user_email": "alice@example.com", "user_role": "internal_user"}, and gets back a key sk-...
2. Alice sends POST https://litellm-domain/v1/chat/completions (or /v1/responses, or /v1/messages with the key in x-api-key) with "model": "no-such-model" and gets 400 Invalid model name, then POST https://litellm-domain/team/new and gets 401 Only proxy admin ... Your user_id=ali***om
3. The operator opens the issues at https://<org>.sentry.io/issues/<id>/: user_api_key_dict reads UserAPIKeyAuth(token='[Filtered]', key_name='sk-...FJLQ', ..., user_id='[Filtered]', user_email='[Filtered]', ...), data.metadata.user_api_key_hash reads [Filtered], and raw_headers.authorization (or x-api-key) reads [Filtered]
4. general_settings reads {'master_key': '[Filtered]', 'database_url': '[Filtered]'}; after SENTRY_SEND_DEFAULT_PII=true and a restart the next issue shows user_id='alice@example.com' and the hashed key again while token, the raw key, master_key, and database_url stay [Filtered]
5. Anyone who can read the Sentry project no longer sees the master key, the database credentials, or Alice's key, and cannot link a failing request to a user unless the operator opted in

Linear ticket

Resolves LIT-8600

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Shared setup for both legs: the proxy boots from the named commit with --num_workers 2 on 127.0.0.1:51173 against the shared dev Postgres, SENTRY_DSN points at a local envelope sink on 127.0.0.1:42275 that stores every envelope Sentry's ingest would have received (the before leg would otherwise ship the dev database URL and master key to sentry.io), gpt-5-mini is a real OpenAI deployment. An internal user qa.user.lit8600@example.com with its own key sk-...FJLQ (hash 5e77...1ba2) was created once through POST /user/new. Every leg first sends 10 POST /v1/chat/completions with an unknown model so both workers have initialised Sentry (it initialises lazily on the first LLM route per worker), then runs the cases below. Leak counts come from walking every JSON path of every envelope the sink received for that case; excerpts are the actual frame locals with the master key, database URL, and the QA key replaced by <... redacted> and long values elided with ...

Cases 1 to 4 run with SENTRY_SEND_DEFAULT_PII unset, case 5 is the 200 transaction, case 6 sends request metadata whose field names collide with Sentry's own source-context fields, case 7 re-runs case 1 and case 4 on a proxy restarted with SENTRY_SEND_DEFAULT_PII=true, and case 8 restarts the proxy with litellm_settings.input_callback: ["sentry"] plus a deployment whose provider key OpenAI rejects, first with SENTRY_SEND_DEFAULT_PII unset and then set

Before (cbba146)

Chat Completions, unknown model

1. `curl -s -w '\nHTTP %{http_code}\n' http://127.0.0.1:51173/v1/chat/completions -H "Authorization: Bearer $USER_KEY" -H 'Content-Type: application/json' -d '{"model":"no-such-model-lit8600","messages":[{"role":"user","content":"hi"}]}'

2. {"error":{"message":"/chat/completions: Invalid model name passed in model=no-such-model-lit8600. Call /v1/models to view available models for your key.","type":"invalid_request_error",...}} then HTTP 400

3. The error event the sink received, frame chat_completion():

   user_api_key_dict = UserAPIKeyAuth(token='5e77...1ba2', key_name='sk-...FJLQ', key_alias=None, ..., user_id='qa.user.lit8600@example.com', ...)
   data.metadata = {..., "user_api_key_hash": "'5e77...1ba2'", ...}
   data.secret_fields.raw_headers = {"host": "'127.0.0.1:51173'", "user-agent": "'curl/8.7.1'", "accept": "'*/*'", "authorization": "'Bearer <virtual key redacted>'", "content-type": "'application/json'", "content-length": "'77'"}
   general_settings = {"master_key": "'<master key redacted>'", "database_url": "'<database url redacted>?connection_limit=10&pool_timeout=60'"}
   

4. Leak count over the event: email at 5 paths, key hash at 11, raw virtual key at 3, master key at 2, database URL at 2, [Filtered] at 0

Responses, unknown model

1. `curl -s -w '\nHTTP %{http_code}\n' http://127.0.0.1:51173/v1/responses -H "Authorization: Bearer $USER_KEY" -H 'Content-Type: application/json' -d '{"model":"no-such-model-lit8600","input":"hi"}'

2. {"error":{"message":"/responses: Invalid model name passed in model=no-such-model-lit8600. ...","type":"invalid_request_error",...}} then HTTP 400

3. Frame responses_api() in the event:

   user_api_key_dict = UserAPIKeyAuth(token='5e77...1ba2', key_name='sk-...FJLQ', ..., user_id='qa.user.lit8600@example.com', ...)
   general_settings = {"master_key": "'<master key redacted>'", "database_url": "'<database url redacted>?connection_limit=10&pool_timeout=60'"}
   data.secret_fields.raw_headers = {..., "authorization": "'Bearer <virtual key redacted>'", ...}
   

4. Leak count: email at 5 paths, key hash at 9 (data.litellm_metadata.user_api_key_hash and user_api_key_dict), raw virtual key at 2, master key at 3, database URL at 3

Messages, unknown model, key sent as x-api-key

1. `curl -s -w '\nHTTP %{http_code}\n' http://127.0.0.1:51173/v1/messages -H "x-api-key: $USER_KEY" -H 'anthropic-version: 2023-06-01' -H 'Content-Type: application/json' -d '{"model":"no-such-model-lit8600","max_tokens":16,"messages":[{"role":"user","content":"hi"}]}'

2. {"type":"error","error":{"type":"invalid_request_error","message":"anthropic_messages: Invalid model name passed in model=no-such-model-lit8600. ..."}} then HTTP 400

3. Frame route_request() in the event:

   data.secret_fields.raw_headers = {"host": "'127.0.0.1:51173'", "user-agent": "'curl/8.7.1'", "accept": "'*/*'", "x-api-key": "'<virtual key redacted>'", "anthropic-version": "'2023-06-01'", "content-type": "'application/json'", "content-length": "'93'"}
   

4. Leak count: email at 5 paths, key hash at 9, raw virtual key at 2 (under raw_headers.x-api-key), master key at 3, database URL at 3

/team/new with the internal user's key

1. `curl -s -w '\nHTTP %{http_code}\n' http://127.0.0.1:51173/team/new -H "Authorization: Bearer $USER_KEY" -H 'Content-Type: application/json' -d '{"team_alias":"lit8600"}'

2. {"error":{"message":"Authentication Error, Only proxy admin can be used to generate, delete, update info for new keys/users/teams. Route=/team/new. Your role=internal_user. Your user_id=qa.use********...",...}} then HTTP 401

3. Frames _authorize_authenticated_request() and common_checks() in the event (the only two locals the SDK filtered by name are api_key and master_key):

   user_api_key_auth_obj = UserAPIKeyAuth(token='5e77...1ba2', key_name='sk-...FJLQ', ..., user_id='qa.user.lit8600@example.com', ...)
   api_key = [Filtered]
   master_key = [Filtered]
   valid_token = UserAPIKeyAuth(token='5e77...1ba2', key_name='sk-...FJLQ', ..., user_id='qa.user.lit8600@example.com', ...)
   general_settings = {"master_key": "'<master key redacted>'", "database_url": "'<database url redacted>?connection_limit=10&pool_timeout=60'"}
   exception.values[0].value ends with: Your role=internal_user. Your user_id=qa.use*******************om
   

4. Leak count: email at 9 paths (user_api_key_auth_obj, user_obj, user_object, valid_token), key hash at 5, master key at 2, database URL at 2

Chat Completions, real gpt-5-mini call

1. `curl -s -w '\nHTTP %{http_code}\n' http://127.0.0.1:51173/v1/chat/completions -H "Authorization: Bearer $USER_KEY" -H 'Content-Type: application/json' -d '{"model":"gpt-5-mini","messages":[{"role":"user","content":"Say hi in three words"}]}'
2. {"id":"chatcmpl-ERqugVeT4lADhmmePrwEGfRxuCre6","created":1790306962,"model":"gpt-5-mini","object":"chat.completion","choices":[{"finish_reason":"stop","index":0,"message":{"content":"Hi there, friend.","role":"assistant",...}}],...} then HTTP 200
3. The sink received one transaction envelope and no error event; leak count over the transaction: email 0, key hash 0, raw virtual key 0, master key 0, database URL 0 (transactions never carried identifiers on either leg)

Chat Completions, unknown model, metadata fields named like Sentry source context

1. curl -s -w '\nHTTP %{http_code}\n' http://127.0.0.1:51173/v1/chat/completions -H "Authorization: Bearer $USER_KEY" -H 'Content-Type: application/json' -d '{"model":"no-such-model-lit8600","messages":[{"role":"user","content":"hi"}],"metadata":{"context_line":"sk-canaryCTXLINE0000000000000000000000000000000000","pre_context":["canary.ctx@example.com"],"plain_key":"sk-canaryPLAIN00000000000000000000000000000000000"}}'

2. {"error":{"message":"/chat/completions: Invalid model name passed in model=no-such-model-lit8600. Call /v1/models to view available models for your key.","type":"invalid_request_error",...}} then HTTP 400

3. The error event the sink received, frame chat_completion():

   data.metadata = {..., "context_line": "'sk-canaryCTXLINE0000000000000000000000000000000000'", "pre_context": ["'canary.ctx@example.com'"], "plain_key": "'sk-canaryPLAIN00000000000000000000000000000000000'", "requester_metadata": {"context_line": "'sk-canaryCTXLINE...'", "pre_context": ["'canary.ctx@example.com'"], "plain_key": "'sk-canaryPLAIN...'"}, ...}
   data.proxy_server_request.body.metadata = {"context_line": "'sk-canaryCTXLINE...'", "pre_context": "['canary.ctx@example.com']", "plain_key": "'sk-canaryPLAIN...'", ...}
   

4. Leak count over the event: each of the three canary values appears at 12 paths (data.metadata, its requester_metadata copy, and the proxy_server_request.body.metadata copy, in three frames each), alongside the same email, key hash, master key, database URL, and raw virtual key leaks as case 1

PII opt-in (SENTRY_SEND_DEFAULT_PII=true), same Chat Completions and /team/new requests

1. Restart the proxy with SENTRY_SEND_DEFAULT_PII=true and repeat cases 1 and 4
2. The events are byte-for-byte the same shape as above: token='5e77...1ba2' user_id='qa.user.lit8600@example.com', raw_headers.authorization = "'Bearer <virtual key redacted>'", general_settings = {"master_key": "'<master key redacted>'", "database_url": "'<database url redacted>...'"}
3. Leak count on Chat Completions: email 5, key hash 11, raw virtual key 3, master key 2, database URL 2; the variable does nothing before this PR

Chat Completions through a deployment whose provider key OpenAI rejects, litellm_settings.input_callback: ["sentry"]

1. Restart the proxy from a config that adds input_callback: ["sentry"] next to failure_callback: ["sentry"] and a second deployment gpt-5-mini-badkey (real openai/gpt-5-mini, api_key: os.environ/LIT8600_BAD_OPENAI_KEY where that variable holds sk-canaryBADKEY0000000000000000000000000000000000), SENTRY_SEND_DEFAULT_PII unset

2. 4 x curl -s -w '\nHTTP %{http_code}\n' http://127.0.0.1:51173/v1/chat/completions -H "Authorization: Bearer $USER_KEY" -H 'Content-Type: application/json' -d '{"model":"gpt-5-mini-badkey","messages":[{"role":"user","content":"hi from canary.msg@example.com"}],"metadata":{"context_line":"sk-canaryCTXLINE0000000000000000000000000000000000","pre_context":["canary.ctx@example.com"],"plain_key":"sk-canaryPLAIN00000000000000000000000000000000000"}}'

3. {"error":{"message":"litellm.AuthenticationError: AuthenticationError: OpenAIException - Incorrect API key provided: sk-canar****...0000. ... then HTTP 401, and {"error":{"message":"No deployments available for selected model, Try again in 5 seconds. ... then HTTP 429 once the router has cooled the deployment down: 401, 429, 401, 429 over the four requests

4. Each 401 error event the sink received carries one litellm.llm_call breadcrumb; breadcrumbs.values[0].message (1024 characters, Sentry's string cap) reads:

   Model Call Details pre-call: {'litellm_trace_id': '113b8925-...', 'litellm_call_id': 'dde03434-...', 'input': [{'role': 'user', 'content': 'hi from canary.msg@example.com'}], 'litellm_params': {'acompletion': True, ..., 'api_key': 'sk-canaryBADKEY0000000000000000000000000000000000', 'force_timeout': 600, ...
   

5. Leak count over the three 401 events' breadcrumb messages: provider key 1 each, prompt email 1 each, [Filtered] 0; the metadata canaries reach the frame locals as in case 6 (sk-canaryPLAIN... at 21 paths, sk-canaryCTXLINE... at 3, canary.ctx@example.com at 1, canary.msg@example.com at 4 across the run's events)

After (28ce1ad)

The tip 5dbc9fb is the merge of origin/main (6b76888) into this branch with the PR diff unchanged: every shared file's hunks are disjoint from main's, the sentry-sdk pin is the same, and the 419 targeted tests pass on the merged tree, so the run below and the /live-pr-risk pass stand at 28ce1ad and CI covers the merged tree

Chat Completions, unknown model

1. `curl -s -w '\nHTTP %{http_code}\n' http://127.0.0.1:51173/v1/chat/completions -H "Authorization: Bearer $USER_KEY" -H 'Content-Type: application/json' -d '{"model":"no-such-model-lit8600","messages":[{"role":"user","content":"hi"}]}'

2. {"error":{"message":"/chat/completions: Invalid model name passed in model=no-such-model-lit8600. Call /v1/models to view available models for your key.","type":"invalid_request_error",...}} then HTTP 400, the same response as before

3. The error event the sink received, frame chat_completion():

   user_api_key_dict = UserAPIKeyAuth(token='[Filtered]', key_name='sk-...FJLQ', key_alias=None, ..., user_id='[Filtered]', ...)
   data.metadata = {..., "user_api_key_hash": "[Filtered]", ...}
   data.secret_fields.raw_headers = {"host": "'127.0.0.1:51173'", "user-agent": "'curl/8.7.1'", "accept": "'*/*'", "authorization": "[Filtered]", "content-type": "'application/json'", "content-length": "'77'"}
   general_settings = {"master_key": "[Filtered]", "database_url": "[Filtered]"}
   

4. Leak count over the event: email at 0 paths, key hash at 0, raw virtual key at 0, master key at 0, database URL at 0, [Filtered] at 19 (every user_api_key_dict, user_api_key_hash, raw_headers.authorization, master_key, and database_url local); the frames' own pre_context, context_line, and post_context source lines carry no [Filtered]

Responses, unknown model

1. `curl -s -w '\nHTTP %{http_code}\n' http://127.0.0.1:51173/v1/responses -H "Authorization: Bearer $USER_KEY" -H 'Content-Type: application/json' -d '{"model":"no-such-model-lit8600","input":"hi"}'

2. {"error":{"message":"/responses: Invalid model name passed in model=no-such-model-lit8600. ...","type":"invalid_request_error",...}} then HTTP 400

3. Frame responses_api() in the event:

   user_api_key_dict = UserAPIKeyAuth(token='[Filtered]', key_name='sk-...FJLQ', ..., user_id='[Filtered]', ...)
   general_settings = {"master_key": "[Filtered]", "database_url": "[Filtered]"}
   data.secret_fields.raw_headers = {..., "authorization": "[Filtered]", ...}
   

4. Leak count: email 0, key hash 0, raw virtual key 0, master key 0, database URL 0

Messages, unknown model, key sent as x-api-key

1. `curl -s -w '\nHTTP %{http_code}\n' http://127.0.0.1:51173/v1/messages -H "x-api-key: $USER_KEY" -H 'anthropic-version: 2023-06-01' -H 'Content-Type: application/json' -d '{"model":"no-such-model-lit8600","max_tokens":16,"messages":[{"role":"user","content":"hi"}]}'

2. {"type":"error","error":{"type":"invalid_request_error","message":"anthropic_messages: Invalid model name passed in model=no-such-model-lit8600. ..."}} then HTTP 400

3. Frame route_request() in the event:

   data.secret_fields.raw_headers = {"host": "'127.0.0.1:51173'", "user-agent": "'curl/8.7.1'", "accept": "'*/*'", "x-api-key": "[Filtered]", "anthropic-version": "'2023-06-01'", "content-type": "'application/json'", "content-length": "'93'"}
   

4. Leak count: email 0, key hash 0, raw virtual key 0 (raw_headers.x-api-key now reads [Filtered]), master key 0, database URL 0

/team/new with the internal user's key

1. `curl -s -w '\nHTTP %{http_code}\n' http://127.0.0.1:51173/team/new -H "Authorization: Bearer $USER_KEY" -H 'Content-Type: application/json' -d '{"team_alias":"lit8600"}'

2. {"error":{"message":"Authentication Error, Only proxy admin can be used to generate, delete, update info for new keys/users/teams. Route=/team/new. Your role=internal_user. Your user_id=qa.use********...",...}} then HTTP 401, the caller still gets the masked id

3. Frames _authorize_authenticated_request() and common_checks() in the event:

   user_api_key_auth_obj = UserAPIKeyAuth(token='[Filtered]', key_name='sk-...FJLQ', ..., user_id='[Filtered]', ...)
   api_key = [Filtered]
   master_key = [Filtered]
   valid_token = UserAPIKeyAuth(token='[Filtered]', key_name='sk-...FJLQ', ..., user_id='[Filtered]', ...)
   general_settings = {"master_key": "[Filtered]", "database_url": "[Filtered]"}
   exception.values[0].value ends with: Your role=internal_user. Your user_id='[Filtered]'
   

4. Leak count: email 0 (user_api_key_auth_obj, user_obj, user_object, valid_token all read user_id='[Filtered]'), key hash 0, master key 0, database URL 0

Chat Completions, real gpt-5-mini call

1. `curl -s -w '\nHTTP %{http_code}\n' http://127.0.0.1:51173/v1/chat/completions -H "Authorization: Bearer $USER_KEY" -H 'Content-Type: application/json' -d '{"model":"gpt-5-mini","messages":[{"role":"user","content":"Say hi in three words"}]}'
2. {"id":"chatcmpl-ERrZJtv89yPGy2GrZ6OroJjCnGt2N","created":1790309481,"model":"gpt-5-mini","object":"chat.completion","choices":[{"finish_reason":"stop","index":0,"message":{"content":"Hi there, friend.","role":"assistant",...}}],...} then HTTP 200
3. The sink received one transaction envelope and no error event; leak count over the transaction: email 0, key hash 0, raw virtual key 0, master key 0, database URL 0; its request.headers.authorization reads [Filtered] (an empty string on the before leg)

Chat Completions, unknown model, metadata fields named like Sentry source context

1. Same request as the Before leg, three canary values in metadata under the field names context_line, pre_context, and plain_key

2. {"error":{"message":"/chat/completions: Invalid model name passed in model=no-such-model-lit8600. Call /v1/models to view available models for your key.","type":"invalid_request_error",...}} then HTTP 400, unchanged

3. The error event the sink received, frame chat_completion():

   data.metadata = {..., "context_line": "'[Filtered]'", "pre_context": ["'[Filtered]'"], "plain_key": "'[Filtered]'", "requester_metadata": {"context_line": "'[Filtered]'", "pre_context": ["'[Filtered]'"], "plain_key": "'[Filtered]'"}, ...}
   data.proxy_server_request.body.metadata = {"context_line": "'[Filtered]'", "pre_context": "['[Filtered]']", "plain_key": "'[Filtered]'", ...}
   

4. Leak count over the event: each canary value at 0 paths (12 each on the Before leg), email 0, key hash 0, raw virtual key 0, master key 0, database URL 0, [Filtered] at 48; the frames' own context_line still reads raise ProxyModelNotFoundError( and their pre_context lines are untouched, because the source-context exemption only applies at a real stack-frame path

PII opt-in (SENTRY_SEND_DEFAULT_PII=true), same Chat Completions and /team/new requests

1. Restart the proxy with SENTRY_SEND_DEFAULT_PII=true and repeat cases 1, 4, and 6; the responses are unchanged (HTTP 400, HTTP 401, and HTTP 400 with the same bodies)
2. Chat Completions event, frame chat_completion(): user_api_key_dict = UserAPIKeyAuth(token='[Filtered]', key_name='sk-...FJLQ', ..., user_id='qa.user.lit8600@example.com', user_email='qa.user.lit8600@example.com', ...), data.metadata.user_api_key_hash = '5e77...1ba2', raw_headers.authorization = "[Filtered]", general_settings = {"master_key": "[Filtered]", "database_url": "[Filtered]"}
3. /team/new event: user_api_key_auth_obj = UserAPIKeyAuth(token='[Filtered]', ..., user_id='qa.user.lit8600@example.com', ...), api_key = [Filtered], master_key = [Filtered], and the message ends with Your user_id=qa.use***om exactly as the caller saw it
4. Leak count on Chat Completions: email 5 (the user_api_key_dict reprs), key hash 6 (data.metadata.user_api_key_hash and its proxy_server_request.body copy in three frames), raw virtual key 0, master key 0, database URL 0; on /team/new: email 10, key hash 0, raw virtual key 0, master key 0, database URL 0; on case 6 the pre_context email canary is visible at its 12 paths (emails are what the opt-in sends) while the two sk- canaries stay [Filtered] at all 12

Chat Completions through a deployment whose provider key OpenAI rejects, litellm_settings.input_callback: ["sentry"]

1. Same restart and the same four requests as the Before leg, SENTRY_SEND_DEFAULT_PII unset

2. HTTP 401 with the same Incorrect API key provided: sk-canar****...0000 body, then HTTP 429 from the cooldown: 401, 429, 429, 429 over the four requests (which request hits the cooldown depends on which worker took it, see the observations)

3. breadcrumbs.values[0].message on each 401 event (882 characters):

   Model Call Details pre-call: {'litellm_trace_id': 'a63ace4c-...', 'litellm_call_id': '674013a5-...', 'input': [{'role': 'user', 'content': 'hi from [Filtered]'}], 'litellm_params': {'acompletion': True, ..., 'api_key': '[Filtered]', 'force_timeout': 600, ...
   

4. Leak count over the breadcrumb messages: provider key 0, prompt email 0, [Filtered] 5 each; every metadata canary at 0 paths over the run's 12 events (21, 3, 1, and 4 on the Before leg)

5. Restarted with SENTRY_SEND_DEFAULT_PII=true, same four requests, same responses: the breadcrumb reads 'content': 'hi from canary.msg@example.com' and still 'api_key': '[Filtered]' (906 characters, [Filtered] 3); canary.msg@example.com at 10 paths and canary.ctx@example.com at 20 over the run's events, the two sk- canaries and the provider key at 0
   Observations from the run:

• Sentry initialises lazily on a worker's first LLM route; pre-existing, unchanged here
• With PII off the /team/new message's user_id=qa.use***om becomes user_id='[Filtered]'
• With PII on /team/new shows no hash: its only carrier is the always-filtered token=
• Transactions never carried identifiers on either leg
• Request metadata named context_line or pre_context is scrubbed like any other field; only a real stack frame's source lines are exempt
• The same Before request through litellm.completion(..., api_key=..., metadata=...) with failure_callback=["sentry"] leaks the key and metadata on the Before tree and nothing on the After tree; with the opt-in only the email is visible
• The transaction's Authorization header reads [Filtered], an empty string before
• input_callback: ["sentry"] records a Model Call Details pre-call: {...} breadcrumb that rides the next error event; on the Before tree its 1024-character message carried the deployment's provider api_key and the prompt, on the After tree both read [Filtered] and the prompt's email comes back with the opt-in. Which of the four case 8 requests hit the router cooldown (401, 429, 401, 429 on Before, 401, 429, 429, 429 on both After runs) depends on which worker took it, the cooldown being per-worker in-memory state

/live-pr-risk

Breaking: none observed. Across cases 1 to 8, the risk scenarios A to F (streaming chat, /v1/messages, key generate, info and delete, /health, the spend row, latency), the SDK leg and the merged tree, every client-visible result (status, body keys, stream chunk count, spend row) matched the base tree; only the Sentry envelope contents changed

Backward incompatible (each visible to an operator in Sentry only, each recorded under Caveats):

• SENTRY_API_TRACE_RATE="" now means 1.0 where the base raised ValueError on the worker's first LLM call
• Sentry's own default denylists are merged with litellm's instead of replaced and the scrubber walks nested objects, so more values read [Filtered]: the transaction's Authorization header and the masked user_id= tail of the exception message with PII off
• Request metadata fields named pre_context, context_line or post_context are scrubbed like any other field; only a real stack frame's source lines keep those names exempt
• SENTRY_SEND_DEFAULT_PII is new; only true and false count and anything else keeps PII off

Regression risk: every Sentry event pays the recursive key scrub plus the regex pass, measured back to back on the same two-worker rig (numbers under Caveats: about 4 ms per captured error event, under 1 ms per transaction). Router retries produce three AuthenticationError events per bad-key request on both trees, so that cost scales with num_retries, not with this PR. Sentry still initialises lazily on the first LLM route, and an exception inside the scrubber makes the SDK drop the event without touching the request, which the fail-closed depth test pins

Dependency graph: 19 dependents from a walk over litellm/, enterprise/, tests/, ui/, the docs checkout and the installed SDK 2.21.0; nothing in ui/ or enterprise/ reads Sentry. Driven live: the Starlette transaction per request, capture_exception from failure_callback on chat, responses and messages, the pre-call breadcrumb from input_callback, the transaction's request headers under both PII modes, the YAML config path, the SDK litellm.completion path, both env parsers, and the tip merged into main. The SDK floor stays 2.21.0 (pyproject.toml proxy-runtime >=2.21.0,<3.0, proxy-dev pins ==2.21.0, every Dockerfile installs 2.21.0) and everything the module imports exists there

Not verified: the logging integration path (an ERROR log whose args carry a secret dict bypasses litellm's handler-level filter on both trees; before_send now scrubs it by field name and value pattern, covered by the unit tests only), general_settings.alerting: ["sentry"] and the DB failure_handler (name-only dependents, unchanged), background jobs and rq workers, the unpinned pip install sentry_sdk fallback in a venv without the extra (latest 2.x drops the 1024-character truncation, so breadcrumbs get longer but pass through the same scrubber), and a host app that calls sentry_sdk.init itself (its options win, as before)

Type

🐛 Bug Fix

Caveats (if any)

Low

• The exception message loses the masked user_id=qa.use***om tail with PII off; the response the caller gets is unchanged
• A secret-named field holding a value nested more than four bracket levels deep falls back to the old word match; secret-named keys and sk- values inside it are still filtered
• Every Sentry event now pays for the recursive key scrub plus the regex pass on the request path. Back to back on the same two-worker rig (base, head, base, head, 50 samples each, load 15 to 21) a 400 error event went from a 10.2 and 10.1 ms median to 14.2 and 13.8 ms and a /v1/models transaction from 3.4 and 3.7 ms to 3.8 and 4.5 ms; the scrubber alone measures 0.6 ms median and 8.4 ms worst over 143 captured events, and it only runs on requests that already fail or on sampled transactions
• SENTRY_API_TRACE_RATE="" now means 1.0, the same as unset, where the old code raised ValueError on the worker's first LLM call (that request failed and Sentry stayed off); a deployment carrying the empty value starts tracing every request, which is what the docs example already showed. SENTRY_API_SAMPLE_RATE treated empty as 1.0 on both sides
• The required lint check is red on main itself since feat(mcp): configure protocol versions and capability discovery #43169 (6b76888): ruff check --config ruff-tests.toml tests reports F601 for a repeated "UNIT_FLAG" key in tests/unit/test_unit_shard_missing_paths.py, a file this PR does not touch (same failure on fix(dashscope): pass image generation provider params at the top level of parameters #43208's lint run); the one-line fix fix(tests): drop the repeated UNIT_FLAG key in test_unit_shard_missing_paths #43212 landed on main as 25de1ab, and since the lint workflow checks out the PR head rather than the merge commit, main was merged in at a2bfb20 to pick it up
• rust-test is red on main as well (run 36090485216 at 3fa02ef fails the same model-catalog spec parity case on fal_ai/fal-ai/nano-banana-2); nothing here touches the catalog
• The proof reads each Sentry envelope at a local ingest sink instead of on a sentry.io issue page, because the Before leg would otherwise ship the dev master key and database URL to sentry.io; the envelope is the exact payload sentry.io stores, so a page screenshot adds no evidence
• The After leg ran at 28ce1ad, before main was merged in at 5dbc9fb and again at a2bfb20: the merged ranges cbba146..6b76888 and 6b76888..8327cd6 touch nothing Sentry-related, the second touches none of this PR's files at all, the first's hunks in the shared files are disjoint from this PR's, and the targeted tests pass on each merged tree, so re-running both legs for a merge commit would spend real provider calls for no new evidence
• With PII off, any sk- value carrying 13 or more key characters and any 64-hex token read [Filtered] wherever they appear in an event, so a model alias or content digest of that shape is masked too; an allowlist for those shapes would let a real key of the same shape through, and a masked alias costs a reader nothing

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

• 28ce1ad passes /live-pr-risk

---

Note

Medium Risk
Changes what operators see in Sentry (more [Filtered] fields, optional PII opt-in) and adds per-event scrubbing on the error/transaction path; client API behavior is unchanged but observability and empty-env sample-rate parsing differ slightly.

Overview
Centralizes Sentry setup in litellm_core_utils/sentry_scrubbing.py and wires it from set_callbacks(["sentry"]) via build_sentry_init_options instead of inline sentry_sdk.init.

Expands what gets redacted: SENTRY_DENYLIST / SENTRY_PII_DENYLIST gain proxy auth headers and LiteLLM identity fields; the SDK’s default denylists are merged (not replaced). A recursive before_send / before_send_transaction pass scrubs every string in the event—object reprs, nested locals, breadcrumbs, transactions—for secret field names, sk- keys, emails, and 64-char hashes, while real stack-frame source lines stay readable.

Adds SENTRY_SEND_DEFAULT_PII=true to opt into identity fields; secrets stay [Filtered] either way. proxy-dev pins sentry-sdk==2.21.0; tests move to the dedicated scrubbing module with integration-style coverage.

^{Reviewed by Cursor Bugbot for commit a2bfb20. Bugbot is set up for automated code reviews on this repo. Configure here.}

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[greptile-apps]

Confidence Score: 5/5

[Medium risk] Adds Sentry error scrubbing for secrets and personal data.

The PR appears safe to merge based on the reviewed changes.

Summary

This PR adds Sentry event scrubbing and an opt-in for sending identity fields. Changes since the previous review also move streamed Responses container-ownership recording ahead of the terminal chunk, adjust provider-budget accounting, and add coverage for those paths.

_{Reviews (8) · Last reviewed commit: "Merge remote-tracking branch 'origin/mai..."}

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_sentry_scrub_pii_reprs (a2bfb20) with main (8327cd6)}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit a2bfb20. Configure here.}

[mateo-berri]

LGTM