Repository navigation
fix(proxy): keep the raw client model out of spend logs for rejections outside the router - #41943
Conversation
…s outside the router
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
… before a deployment is picked
…when a spend row is placeholdered With store_prompts_in_spend_logs on, the persisted request body kept the client's model string even when the row's model, model_group, and error text had been replaced by the unknown-model placeholder. The body's model now takes the same placeholder on those rows. Also annotates the new test locals with Final and wraps the four test lines that ran past 120 characters.
…s stored request body
…_spend_logs_outside_router # Conflicts: # litellm/proxy/openai_files_endpoints/common_utils.py
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit aadadc6. Configure here.
1 participant
TLDR
Problem this solves:
modelstringmodelandmodel_groupHow it solves it:
model_groupunknown-modeland the scrubbed metadataUser Flow
Before: a developer's client puts prompt text into the
modelfield by mistake, the gateway rejects the call, and the prompt text is still saved in the spend logs for anyone who can read them"input": "hi"and"model": "opus-4.6 Please summarize my medical records\nPatient has diabetes", and get a 400 carrying OpenAI'sInvalid value for 'model'errormodeland get a 400 sayingModel '<that string>' not found in model_list. Please check your config.yaml.purpose=batch, the samemodel, and a JSONL file, and get a 400 with the samenot found in model_listtextmodeland get a 400 sayingcompletion: Invalid model name passed in model=<that string>model, inmodel_group, and inmetadata.error_information.error_message. The batches and adapter rows show"model": "unknown-model", but theirerror_messagestill holds the whole string. The files call returns no call id, and its row (found through GET https://litellm-domain/spend/logs?start_date=...&end_date=...) holds the string inerror_messagetoostore_prompts_in_spend_logs: truein the config, even next toturn_off_message_logging: true, each of those rows also stores the whole string as themodelof the saved request body, and the moderations row stores it twice more inside that body'smetadata(model_groupanderror_information.error_message)/spend/logs, or the spend logs table can read that user's prompt textAfter: the same calls are rejected with 400s, and none of the saved rows holds the prompt text
Invalid value for 'model'errorbatch creation: Invalid model name passed in model=<that string>. Call /v1/models to view available models for your key.file upload: Invalid model name passed in model=<that string>. Call /v1/models to view available models for your key."type": "invalid_request_error"and"code": "400", with the/v1/modelshint added to the message"model": "unknown-model", an emptymodel_group, and anerror_messagewith the string cut out:Invalid value for 'model' = unknown-modelfor moderations,batch creation: Invalid model name passed in. Call /v1/models ...for batches,file upload: ...for files, andcompletion: ...for the adapter, with no client string anywhere in the row"model": "unknown-model", and themetadatacopied into that body shows the same emptymodel_groupand the same cut-downerror_information.error_messageas the row; a healthy call's saved body keeps its real model nameDesign decisions
The placeholder gate moved from "no
model_group" to "the call failed, no deployment id was resolved, the model does not look like a model name, and the group is not a configured one".model_groupis stamped before a deployment is picked, and for calls that never reach the router it is just the client's own string, so it says nothing about whether the model is real. A deployment id only exists once a deployment was chosen. The alternative, rejecting unknown moderation models at the proxy, was left out on purpose because moderations is allowed to pass a model the config does not list straight to the provider"Looks like a model name" means at most 256 characters with no whitespace. That test alone would also placeholder a configured group whose name has spaces (
Broken GPT Mini) when every deployment is cooling down and the router fails before picking one, so the gate also asks the router whether the group is configured, by exact membership in the model list or the team-public names. Wildcard patterns are excluded on purpose: a regex match against a prompt-shaped string is exactly the guess this PR removes. When no router is in hand (the GCS pub/sub v1 path builds the same payload without one) the gate falls back to placeholderingThe persisted
model_groupis blanked only when the row is placeholdered, the group equals the raw client string, and that string does not look like a model name. Blanking it on every placeholdered row was rejected because router-rejected rows keep a sane unknown model inmodel_grouptoday and tests pin that.router_metadata.requested_modelfollows the same valueThe error text scrub runs where the spend log payload is built, since that is the only place that knows the row was placeholdered. It replaces three spellings of the raw string with
unknown-model: the literal one, the Python repr-escaped one (\nwritten as two characters, which is how a provider error dict or an exception detail gets stringified), and the JSON-escaped one. Scrubbing only the literal spelling was rejected because the moderations repro stores the repr-escaped form. Onlyerror_messageandtracebackare scrubbed; the other error fields never hold free text. The scrub builds a new mapping instead of editing the one the logging object handed over, so Prometheus, Datadog, and OTel keep reading the unscrubbed error text they already receiveError text is truncated to 2048 characters (35% head, a marker, 65% tail) before the payload is built, so the raw string can be cut in half by the marker. The scrub also removes the longest suffix of the head and the longest prefix of the tail that overlap a spelling. The alternatives were moving the scrub ahead of truncation, which would put it before the placeholder verdict exists, or dropping truncated error text entirely, which loses debugging value
With
store_prompts_in_spend_logson, the saved request body of a placeholdered row gets itsmodelreplaced byunknown-model. The proxy also injects itsmetadatadict into that body, and the logging pipeline later writesmodel_groupanderror_informationinto the same dict, so the saved body carried the client string a second and third time undermetadata.model_groupandmetadata.error_information.error_message(found by the live QA on the moderations path, which is the one whose provider 400 echoes the model). Those two keys now get the same values the row itself stores, the persisted group and the scrubbed error text, and every other key in that metadata stays as it was. Dropping the whole saved body for those rows was rejected because the rest of the body (headers minus authorization, metadata, the other fields) is what an operator debugs a rejected call with.turn_off_message_loggingwas not extended to covermodelbecause it never has, and the placeholder is what the rest of the row already says. The request body sanitizer now descends into anyMapping, not onlydict, which its own signature already promised; without that the read-only metadata copy was stored as itsstr()The adapter route raises the shared unknown-model error with route
completion, which keeps the client message prefix and theinvalid_request_error/null/"400"contract it has today. Using the request path as the route was rejected because it changes the client message shape for no gain.retryable_with_model_read_throughisFalsefor both the adapter and the files and batches helper, because only the router's request path acts on that flag and neither of these goes through it. The a2a route already does the sameThe files and batches credential lookup raises the same error with its
operation_contextas the route, so the client still gets a 400 that names the model, and the saved message is the fixed<operation>: Invalid model name passed in.text. One caller built itsoperation_contextwith the file's model inside it, which would have put a client string back into the saved message, so it now readsfile operation (file created with model)like its sibling call sites. This changes the client-facing wording on those routes fromModel '<x>' not found in model_list. Please check your config.yaml.to the shared wording, and a sane unknown model on/v1/batchesnow savesunknown-modellike the chat routes do instead of the name. Keeping the old wording with a separate exception class was rejected because it would need a second special case in the spend log sanitizer. The helper imports the error inside the function because its module is imported by SDK-only installs wherefastapiis absent, and a module-level import broke that install test in CIRelevant issues
Affected release
Linear ticket
Resolves LIT-7620
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Shared setup for both sides: a proxy booted from the commit named in each heading with
--num_workers 2and--use_v2_migration_resolver, its own Postgres database, real OpenAI and Anthropic keys, master keysk-lit7620-qa, andcustom_adapter.pynext to the config. Every row below is read back throughGET /spend/logs?request_id=<x-litellm-call-id>(thejq '<row fields>'filter picksrequest_id,call_type,status,model,model_group,model_id,metadata.error_information.error_message, and two booleans that grep the traceback and the whole JSON row for the prompt textmedical records|Patient has diabetes). The sweep cases readGET /spend/logs?start_date=...&summarize=falseat the merge base andGET /spend/logs/v2?...at the tip. The fivePrompt storage oncases ran on a second boot of the same commit withqa_prompts.yaml; their filter addsstored_request_body_model(themodelof the saved request body, first 80 characters),stored_request_body_holds_prompt_text, andstored_messages. The files case uploads a one-linebatch.jsonl({"custom_id":"r1","method":"POST","url":"/v1/chat/completions","body":{"model":"gpt-5.4-mini","messages":[{"role":"user","content":"hi"}]}}). The hostile-input case sends a 3948-character, 32-line model string (long_model.txt) and then tests every 40-character and every 12-character slice of it against the JSON-encoded row, its string values, and its string values with literal\nturned into newlines. The After side is the PR's tip, which mergesmainat 1f6e5b6: #41934 landed a per-key grant check ahead of the files and batches credential lookup after this branch's merge base, so the threeMerged tip onlycases generate a virtual key restricted to one model and send the same prompt-shapedmodelthrough it, where that grant check rejects the call with a 403 that names the model before the unknown-model error ever runs; that 403 carries nox-litellm-call-idheader, so those rows are read back throughGET /spend/logs/v2?status=failure&start_date=...&end_date=...filtered on the key alias. The merge base has no such check, so those cases have no Before sideThe output under each step is abbreviated to fit GitHub's description limit: string values inside a row are cut at 100 characters and other output lines at 160, each cut ending in
... (+N chars), the row'srequest_idis left out because the command that read it carries the same id, and the cooldown case shows one of its twelve calls. The complete, unabridged transcripts of both sides, including the cases left out here, are in this gist. Left out: the healthy-call controls on a configured model (moderations, chat completions, responses, messages, the adapter route, the group with spaces in its name, and the prompt-storage boot's sane unknown name), the control where the provider rejects a call on a configured model, and the four router-path cases (/v1/chat/completions,/v1/responses, and/v1/messageswith prompt text inmodel, and/v1/chat/completionswith a sane unknown model name), which this PR does not touch and whose rows read the same on both sides. Every one of those rows still counts in the sweep case of its sideqa.yaml:qa_prompts.yaml:custom_adapter.py:Before (5fc510a)
POST /v1/moderations with prompt text in
modelcurl -sS -X POST http://127.0.0.1:23858/v1/moderations -H 'Authorization: Bearer sk-lit7620-qa' -H 'Content-Type: application/json' -d '{"input":"hi","model":"opus-4.6 Please summarize my medical records\nPatient has diabetes"}'curl -sS 'http://127.0.0.1:23858/spend/logs?request_id=8ae826e9-0806-4791-be26-1a75d88c2bab' -H 'Authorization: Bearer sk-lit7620-qa' | jq '<row fields>'POST /v1/batches with prompt text in
modelcurl -sS -X POST http://127.0.0.1:23858/v1/batches -H 'Authorization: Bearer sk-lit7620-qa' -H 'Content-Type: application/json' -d '{"input_file_id":"file-lit7620-does-not-exist","endpoint":"/v1/chat/completions","completion_window":"24h","model":"opus-4.6 Please summarize my medical records\nPatient has diabetes"}'curl -sS 'http://127.0.0.1:23858/spend/logs?request_id=ac051450-b83d-4cdf-84b3-7a98eef54da5' -H 'Authorization: Bearer sk-lit7620-qa' | jq '<row fields>'POST /anthropic-adapter/v1/messages with prompt text in
modelcurl -sS -X POST http://127.0.0.1:23858/anthropic-adapter/v1/messages -H 'Authorization: Bearer sk-lit7620-qa' -H 'Content-Type: application/json' -d '{"model":"opus-4.6 Please summarize my medical records\nPatient has diabetes","max_tokens":16,"messages":[{"role":"user","content":"hi"}]}'curl -sS 'http://127.0.0.1:23858/spend/logs?request_id=0f2c5b14-20f8-41f7-b91a-b1a0e8fd975c' -H 'Authorization: Bearer sk-lit7620-qa' | jq '<row fields>'POST /v1/files with prompt text in
modelcurl -sS -X POST http://127.0.0.1:23858/v1/files -H 'Authorization: Bearer sk-lit7620-qa' -F purpose=batch -F 'model=<the same prompt-shaped string, with a real newline>' -F file=@batch.jsonlGET /v1/batches?model=
curl -sS 'http://127.0.0.1:23858/v1/batches?model=opus-4.6%20Please%20summarize%20my%20medical%20records%0APatient%20has%20diabetes' -H 'Authorization: Bearer sk-lit7620-qa'Sane unknown model name on POST /v1/moderations
curl -sS -X POST http://127.0.0.1:23858/v1/moderations -H 'Authorization: Bearer sk-lit7620-qa' -H 'Content-Type: application/json' -d '{"input":"hi","model":"text-moderation-nonexistent-9"}'curl -sS 'http://127.0.0.1:23858/spend/logs?request_id=5cb75e80-10e0-4961-b81e-eb6a1d00a202' -H 'Authorization: Bearer sk-lit7620-qa' | jq '<row fields>'Sane unknown model name on POST /v1/batches
curl -sS -X POST http://127.0.0.1:23858/v1/batches -H 'Authorization: Bearer sk-lit7620-qa' -H 'Content-Type: application/json' -d '{"input_file_id":"file-lit7620-does-not-exist","endpoint":"/v1/chat/completions","completion_window":"24h","model":"gpt-nonexistent-9"}'curl -sS 'http://127.0.0.1:23858/spend/logs?request_id=21c6436a-66e1-4575-8033-8f4449bd0aa1' -H 'Authorization: Bearer sk-lit7620-qa' | jq '<row fields>'Configured model group with spaces in its name, the provider rejects the call
curl -sS -X POST http://127.0.0.1:23858/v1/chat/completions -H 'Authorization: Bearer sk-lit7620-qa' -H 'Content-Type: application/json' -d '{"model":"Team GPT Mini","messages":[{"role":"narrator","content":"hi"}]}'curl -sS 'http://127.0.0.1:23858/spend/logs?request_id=e6876c31-343f-4739-9eac-40ea58aa33dd' -H 'Authorization: Bearer sk-lit7620-qa' | jq '<row fields>'Configured model group with spaces in its name, every deployment has a bad key
Twelve identical calls; statuses in order: 3 x 401, 9 x 429. Call 4 (every deployment already cooling down, so the router fails before picking one) shown; the full transcript holds all twelve
curl -sS -X POST http://127.0.0.1:23858/v1/chat/completions -H 'Authorization: Bearer sk-lit7620-qa' -H 'Content-Type: application/json' -d '{"model":"Broken GPT Mini","messages":[{"role":"user","content":"hi"}]}'curl -sS 'http://127.0.0.1:23858/spend/logs?request_id=5d88b136-4e3a-4368-909c-0cc11dd6caac' -H 'Authorization: Bearer sk-lit7620-qa' | jq '<row fields>'Every spend log row of the run, grouped by call_type, status, model, and model_group, and how many rows hold the prompt text
curl -sS 'http://127.0.0.1:23858/spend/logs?start_date=2026-09-18&end_date=2026-09-20&summarize=false' -H 'Authorization: Bearer sk-lit7620-qa' | jq '<group and count>'The rows written by POST /v1/files (call_type IGNORE_THIS), read back one by one
curl -sS 'http://127.0.0.1:23858/spend/logs?start_date=2026-09-18&end_date=2026-09-20&summarize=false' -H 'Authorization: Bearer sk-lit7620-qa' | jq '<rows with call_type IGNORE_THIS>'Hostile input: a 3948-character prompt-shaped
modelon POST /v1/moderationscurl -sS -X POST http://127.0.0.1:33372/v1/moderations -H 'Authorization: Bearer sk-lit7620-qa' -H 'Content-Type: application/json' -d @long_body.json # {"input":"hi","model":<the string above>}curl -sS 'http://127.0.0.1:33372/spend/logs?request_id=2026ae06-57a2-4d71-8a7e-0c30d0b60903' -H 'Authorization: Bearer sk-lit7620-qa' | jq '<row fields>'... | jq --rawfile m long_model.txt -f slices.jq # every 40-character slice of the model string (offsets 0 to length-40) tested against the JSON-encoded row, its string values, and its string values with literal backslash-n turned into newlines... | jq -r '.[0].metadata.error_information.traceback' | grep -n -i -E 'truncat' | cut -c1-300 # truncation markers in the traceback... | jq --rawfile m long_model.txt '<same test with 12-character slices against the JSON-encoded row>'Prompt storage on: POST /v1/moderations with prompt text in
modelcurl -sS -D - -X POST http://127.0.0.1:29160/v1/moderations -H 'Authorization: Bearer sk-lit7620-qa' -H 'Content-Type: application/json' -d '{"input":"hi","model":"opus-4.6 Please summarize my medical records\nPatient has diabetes"}'curl -sS 'http://127.0.0.1:29160/spend/logs?request_id=e9e34900-9a74-49fb-aa85-2399adbf5682' -H 'Authorization: Bearer sk-lit7620-qa' | jq '<row fields>'Prompt storage on: POST /v1/batches with prompt text in
modelcurl -sS -D - -X POST http://127.0.0.1:29160/v1/batches -H 'Authorization: Bearer sk-lit7620-qa' -H 'Content-Type: application/json' -d '{"input_file_id":"file-lit7620-does-not-exist","endpoint":"/v1/chat/completions","completion_window":"24h","model":"opus-4.6 Please summarize my medical records\nPatient has diabetes"}'curl -sS 'http://127.0.0.1:29160/spend/logs?request_id=ebbb1ef1-19df-445d-8396-954c4a14cb4f' -H 'Authorization: Bearer sk-lit7620-qa' | jq '<row fields>'Prompt storage on: POST /anthropic-adapter/v1/messages with prompt text in
modelcurl -sS -D - -X POST http://127.0.0.1:29160/anthropic-adapter/v1/messages -H 'Authorization: Bearer sk-lit7620-qa' -H 'Content-Type: application/json' -d '{"model":"opus-4.6 Please summarize my medical records\nPatient has diabetes","max_tokens":16,"messages":[{"role":"user","content":"hi"}]}'curl -sS 'http://127.0.0.1:29160/spend/logs?request_id=4f848255-6968-47a6-8f1b-7b003699a0ef' -H 'Authorization: Bearer sk-lit7620-qa' | jq '<row fields>'Prompt storage on, control: healthy POST /v1/chat/completions on a configured model
curl -sS -D - -X POST http://127.0.0.1:29160/v1/chat/completions -H 'Authorization: Bearer sk-lit7620-qa' -H 'Content-Type: application/json' -d '{"model":"gpt-5.4-mini","messages":[{"role":"user","content":"Reply with the single word ok"}],"max_tokens":5}'curl -sS 'http://127.0.0.1:29160/spend/logs?request_id=16225467-c3a0-46a7-b5a5-32a9bac1cbf0' -H 'Authorization: Bearer sk-lit7620-qa' | jq '<row fields>'After (aadadc6)
POST /v1/moderations with prompt text in
modelcurl -sS -X POST http://127.0.0.1:22193/v1/moderations -H 'Authorization: Bearer sk-lit7620-qa' -H 'Content-Type: application/json' -d '{"input":"hi","model":"opus-4.6 Please summarize my medical records\nPatient has diabetes"}'curl -sS 'http://127.0.0.1:22193/spend/logs?request_id=9831eb04-1e5b-47a2-8df2-d3d879d0ba3d' -H 'Authorization: Bearer sk-lit7620-qa' | jq '<row fields>'POST /v1/batches with prompt text in
modelcurl -sS -X POST http://127.0.0.1:22193/v1/batches -H 'Authorization: Bearer sk-lit7620-qa' -H 'Content-Type: application/json' -d '{"input_file_id":"file-lit7620-does-not-exist","endpoint":"/v1/chat/completions","completion_window":"24h","model":"opus-4.6 Please summarize my medical records\nPatient has diabetes"}'curl -sS 'http://127.0.0.1:22193/spend/logs?request_id=894237ec-bcbb-41fa-a9c9-d42909ddc2e7' -H 'Authorization: Bearer sk-lit7620-qa' | jq '<row fields>'POST /anthropic-adapter/v1/messages with prompt text in
modelcurl -sS -X POST http://127.0.0.1:22193/anthropic-adapter/v1/messages -H 'Authorization: Bearer sk-lit7620-qa' -H 'Content-Type: application/json' -d '{"model":"opus-4.6 Please summarize my medical records\nPatient has diabetes","max_tokens":16,"messages":[{"role":"user","content":"hi"}]}'curl -sS 'http://127.0.0.1:22193/spend/logs?request_id=9921634c-cdde-4dd4-b859-3cb19b16aef6' -H 'Authorization: Bearer sk-lit7620-qa' | jq '<row fields>'POST /v1/files with prompt text in
modelcurl -sS -X POST http://127.0.0.1:22193/v1/files -H 'Authorization: Bearer sk-lit7620-qa' -F purpose=batch -F 'model=<the same prompt-shaped string, with a real newline>' -F file=@batch.jsonlGET /v1/batches?model=
curl -sS 'http://127.0.0.1:22193/v1/batches?model=opus-4.6%20Please%20summarize%20my%20medical%20records%0APatient%20has%20diabetes' -H 'Authorization: Bearer sk-lit7620-qa'Sane unknown model name on POST /v1/moderations
curl -sS -X POST http://127.0.0.1:22193/v1/moderations -H 'Authorization: Bearer sk-lit7620-qa' -H 'Content-Type: application/json' -d '{"input":"hi","model":"text-moderation-nonexistent-9"}'curl -sS 'http://127.0.0.1:22193/spend/logs?request_id=8a8461c5-55dd-4334-bccb-68e0e79172ad' -H 'Authorization: Bearer sk-lit7620-qa' | jq '<row fields>'Sane unknown model name on POST /v1/batches
curl -sS -X POST http://127.0.0.1:22193/v1/batches -H 'Authorization: Bearer sk-lit7620-qa' -H 'Content-Type: application/json' -d '{"input_file_id":"file-lit7620-does-not-exist","endpoint":"/v1/chat/completions","completion_window":"24h","model":"gpt-nonexistent-9"}'curl -sS 'http://127.0.0.1:22193/spend/logs?request_id=cf87088c-955f-454d-a811-d3ebabfc7131' -H 'Authorization: Bearer sk-lit7620-qa' | jq '<row fields>'Configured model group with spaces in its name, the provider rejects the call
curl -sS -X POST http://127.0.0.1:22193/v1/chat/completions -H 'Authorization: Bearer sk-lit7620-qa' -H 'Content-Type: application/json' -d '{"model":"Team GPT Mini","messages":[{"role":"narrator","content":"hi"}]}'curl -sS 'http://127.0.0.1:22193/spend/logs?request_id=726dd001-e4f1-4ef1-bcfe-27d8b0729af5' -H 'Authorization: Bearer sk-lit7620-qa' | jq '<row fields>'Configured model group with spaces in its name, every deployment has a bad key
Twelve identical calls; statuses in order: 3 x 401, 9 x 429. Call 4 (every deployment already cooling down, so the router fails before picking one) shown; the full transcript holds all twelve
curl -sS -X POST http://127.0.0.1:22193/v1/chat/completions -H 'Authorization: Bearer sk-lit7620-qa' -H 'Content-Type: application/json' -d '{"model":"Broken GPT Mini","messages":[{"role":"user","content":"hi"}]}'curl -sS 'http://127.0.0.1:22193/spend/logs?request_id=97e54ae0-3d24-4e1e-89a9-e47f4790e08a' -H 'Authorization: Bearer sk-lit7620-qa' | jq '<row fields>'Every spend log row of the run, grouped by call_type, status, model, and model_group, and how many rows hold the prompt text
curl -sS 'http://127.0.0.1:22193/spend/logs/v2?start_date=2026-09-19%2000:00:00&end_date=2026-09-20%2023:59:59&page=1&page_size=100' -H 'Authorization: Bearer sk-lit7620-qa' | jq '{total, rows: (.data|length)}'... | jq -r '.data[] | [.call_type, .status, .model, .model_group] | @tsv' | sort | uniq -c... | jq '[.data[] | select(tostring | test("medical records|Patient has diabetes|summarize my"))] | length' # rows holding the prompt text anywhere... | jq '[.data[] | select((.call_type|test("file|list_batches|alist")) )] | map({request_id, call_type, model, model_group})' # rows from the files or list-batches routesThe rows written by POST /v1/files (call_type IGNORE_THIS), read back one by one
... | jq '.data[] | select(.call_type=="IGNORE_THIS") | {request_id, startTime, call_type, status, model, model_group, model_id, api_base}'curl -sS 'http://127.0.0.1:22193/spend/logs?request_id=ff819b50-1df6-4e2e-b6d6-eabba8e80b5c' -H 'Authorization: Bearer sk-lit7620-qa' | jq '<row fields, plus route and traceback head>'curl -sS 'http://127.0.0.1:22193/spend/logs?request_id=1afc278d-a79a-440b-a540-f2c687fa44e9' -H 'Authorization: Bearer sk-lit7620-qa' | jq '<row fields, plus route and traceback head>'Hostile input: a 3948-character prompt-shaped
modelon POST /v1/moderationscurl -sS -X POST http://127.0.0.1:22193/v1/moderations -H 'Authorization: Bearer sk-lit7620-qa' -H 'Content-Type: application/json' -d @long_body.json # {"input":"hi","model":<the string above>}curl -sS 'http://127.0.0.1:22193/spend/logs?request_id=ae3557c3-64c8-476e-9171-4df1a124f1fc' -H 'Authorization: Bearer sk-lit7620-qa' | jq '<row fields>'... | jq --rawfile m long_model.txt -f slices.jq # every 40-character slice of the model string (offsets 0 to length-40) tested against the JSON-encoded row, its string values, and its string values with literal backslash-n turned into newlines... | jq -r '.[0].metadata.error_information.traceback' | grep -n -i -E 'truncat' | cut -c1-300 # truncation markers in the traceback... | jq --rawfile m long_model.txt '<same test with 12-character slices against the JSON-encoded row>'Merged tip only: a virtual key restricted to gpt-5.4-mini
curl -sS -X POST http://127.0.0.1:27326/key/generate -H 'Authorization: Bearer sk-lit7620-qa' -H 'Content-Type: application/json' -d '{"models": ["gpt-5.4-mini"], "key_alias": "lit7620-restricted"}' | jq '{key: (.key | .[0:8] + "..."), models}'Merged tip only: the restricted key sends POST /v1/batches with prompt text in
modelcurl -sS -X POST http://127.0.0.1:27326/v1/batches -H 'Authorization: Bearer <restricted key>' -H 'Content-Type: application/json' -d '{"input_file_id": "file-abc123", "endpoint": "/v1/chat/completions", "completion_window": "24h", "model": "opus-4.6 Please summarize my medical records\nPatient has diabetes"}'curl -sS 'http://127.0.0.1:27326/spend/logs/v2?status=failure&start_date=2026-09-19%2000:00:00&end_date=2026-09-20%2023:59:59&page=1&page_size=100' -H 'Authorization: Bearer sk-lit7620-qa' | jq '[.data[] | select(.metadata.user_api_key_alias == "lit7620-restricted")] | map(<row fields>)'Merged tip only: the restricted key sends POST /v1/files with prompt text in
modelcurl -sS -X POST http://127.0.0.1:27326/v1/files -H 'Authorization: Bearer <restricted key>' -F purpose=batch -F 'model=<the same prompt-shaped string, with a real newline>' -F file=@batch.jsonlcurl -sS 'http://127.0.0.1:27326/spend/logs/v2?status=failure&start_date=2026-09-19%2000:00:00&end_date=2026-09-20%2023:59:59&page=1&page_size=100' -H 'Authorization: Bearer sk-lit7620-qa' | jq '[.data[] | select(.metadata.user_api_key_alias == "lit7620-restricted")] | map(<row fields>)'Prompt storage on: POST /v1/moderations with prompt text in
modelcurl -sS -D - -X POST http://127.0.0.1:28229/v1/moderations -H 'Authorization: Bearer sk-lit7620-qa' -H 'Content-Type: application/json' -d '{"input":"hi","model":"opus-4.6 Please summarize my medical records\nPatient has diabetes"}'curl -sS 'http://127.0.0.1:28229/spend/logs?request_id=620127d9-b1f8-486d-9d81-cd8971f50ab2' -H 'Authorization: Bearer sk-lit7620-qa' | jq '<row fields>'Prompt storage on: POST /v1/batches with prompt text in
modelcurl -sS -D - -X POST http://127.0.0.1:28229/v1/batches -H 'Authorization: Bearer sk-lit7620-qa' -H 'Content-Type: application/json' -d '{"input_file_id":"file-lit7620-does-not-exist","endpoint":"/v1/chat/completions","completion_window":"24h","model":"opus-4.6 Please summarize my medical records\nPatient has diabetes"}'curl -sS 'http://127.0.0.1:28229/spend/logs?request_id=8c01bccd-1d31-45e9-a216-e7188734007f' -H 'Authorization: Bearer sk-lit7620-qa' | jq '<row fields>'Prompt storage on: POST /anthropic-adapter/v1/messages with prompt text in
modelcurl -sS -D - -X POST http://127.0.0.1:28229/anthropic-adapter/v1/messages -H 'Authorization: Bearer sk-lit7620-qa' -H 'Content-Type: application/json' -d '{"model":"opus-4.6 Please summarize my medical records\nPatient has diabetes","max_tokens":16,"messages":[{"role":"user","content":"hi"}]}'curl -sS 'http://127.0.0.1:28229/spend/logs?request_id=bd08bf0e-3772-44ab-91a8-662441026100' -H 'Authorization: Bearer sk-lit7620-qa' | jq '<row fields>'Prompt storage on, control: healthy POST /v1/chat/completions on a configured model
curl -sS -D - -X POST http://127.0.0.1:28229/v1/chat/completions -H 'Authorization: Bearer sk-lit7620-qa' -H 'Content-Type: application/json' -d '{"model":"gpt-5.4-mini","messages":[{"role":"user","content":"Reply with the single word ok"}],"max_tokens":5}'curl -sS 'http://127.0.0.1:28229/spend/logs?request_id=feadec69-04b6-4253-8a59-1696e8b8eba4' -H 'Authorization: Bearer sk-lit7620-qa' | jq '<row fields>'Type
Bug Fix
Caveats (if any)
Medium
Low
proxy_e2e_anthropic_messages_testsis red ontest_bedrock_invoke_messages_with_all_beta_headers(Bedrock 400invalid beta flag, opus-4.5 and sonnet-4.5 converse) here, at the previous tip (89863), and on the last twomainpipelines (89818, 89834), both older than themainthis tip merges; no fix onmainyet.google_generate_content_endpoint_testinghit Vertex429 Resource exhaustedon the threetest_proxy_genai_sdk_*tests, as did every pipeline across eight branches from 10:27Z to 11:54Z (89859 to 89870); green at 10:17Z and onmain89834mainmerge:litellm_utils_testingtest_models_by_provider(fix(proxy): register transcribe as a known provider for model grants #41926) and thelocal_testing_part2MockValSerflake are green at 89869mainwas merged in; the merge moved it to 1f6e5b6, and none of the 53maincommits in between touches the spend log payload builder, the moderations route, or the adapter route. The files and batches helper gained feat(batches): support Mistral files/batches and per-page OCR batch cost tracking (internal copy of #40484) #41934's grant check, which passes the master key through to the samenot found in model_listerror the Before side shows/spend/logs/v2model_groupanderror_messagefilters no longer find these rows by the client string; filter onmodel=unknown-modelinstead/v1/batchesnow savesunknown-modelinstead of the name, matching the chat routesInvalid model name passed intextGET /v1/batches?model=<x>writes no spend row at all, before and after; left alonePOST /v1/filesreturns nox-litellm-call-idheader and saves acall_type: IGNORE_THISrow, before and after; left alone/v1/moderationsstill saves its name, since the provider rejects it and the name is not prompt-shaped; left aloneNo matching adapter given, before and after (LIT-7515); left aloneunknown-modelkeep their first-insertedmodel_groupunknown-modelwith an empty group instead of the group nameFinal Attestation
I have read the entire PR description and it is up to date with the latest commits, and I have gone through the entire PR diff line-by-line and it has no bugs, no issues, no regressions, no vulnerabilities, no drops in quality of code (e.g., DRY, code smells, magic strings, etc.), and no low quality tests. The PR meets the bar of a senior engineer at a top-tier tech company
aadadc6 passes /live-pr-risk