test(store_model_in_db): assert the 400 contract in the unknown-model spend log test

[mateo-berri]

TLDR

Problem this solves:

• proxy_store_model_in_db_tests red on staging since 2026-09-04 03:09 UTC
• the spend log test pinned model_group and litellm.BadRequestError wording
• only a stale wildcard route ever produced that wording; fix(router): evict stale global pattern_router entries on upsert/delete #39664 evicts it now

How it solves it:

• assert the HTTP 400 and the failure spend log row for an unknown model
• accept both 400 shapes: the proxy's own rejection, or the router's behind a wildcard route
• the test passes with or without a wildcard route registered

User Flow

Before: a contributor merging into litellm_internal_staging finds the required job proxy_store_model_in_db_tests red on every run since 2026-09-04 03:09 UTC, on a test their change never touched

1. They open the staging pipeline on https://app.circleci.com/pipelines/github/BerriAI/litellm and see proxy_store_model_in_db_tests failed on test_chat_completion_bad_model_with_spend_logs
2. Inside that job the test sent POST http://0.0.0.0:4000/v1/chat/completions with "model": "non-existent-model" on a key that lists that model, and got HTTP 400 with Invalid model name passed in model=non-existent-model. Call /v1/models to view available models for your key. plus an x-litellm-call-id header
3. Fifteen seconds later it read GET http://0.0.0.0:4000/spend/logs?request_id= and got one failure row: model "non-existent-model", spend 0, error_code "400", the same message, and model_group ""
4. The test insists on model_group == "non-existent-model" and on litellm.BadRequestError wording, so it fails with assert '' == 'non-existent-model' and the job stays red on every scheduled staging run (89097, 89104, 89115) and on unrelated PRs merged with staging

After: the same job is green because the test checks what a caller of an unknown model actually gets

1. They open the staging pipeline on https://app.circleci.com/pipelines/github/BerriAI/litellm and see proxy_store_model_in_db_tests green
2. Inside that job the test sent POST http://0.0.0.0:4000/v1/chat/completions with "model": "non-existent-model" on a key that lists that model, and got HTTP 400 with Invalid model name passed in model=non-existent-model. Call /v1/models to view available models for your key. plus an x-litellm-call-id header
3. Fifteen seconds later it read GET http://0.0.0.0:4000/spend/logs?request_id= and got one failure row: model "non-existent-model", spend 0, error_code "400", the same message, and model_group ""
4. The test checks the HTTP 400 and that the failure row for that call id carries model "non-existent-model", zero spend and tokens, error_code "400", model_group "", error_class "ProxyModelNotFoundError", and the model name in the error message, so it passes; it also passes when a sibling suite leaves a wildcard route behind, where the row carries model_group "non-existent-model", error_class "BadRequestError", and a litellm.BadRequestError message instead, and fails on any other model group or error class

Relevant issues

Linear ticket

Resolves LIT-6949

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Why the job flipped: the last green staging job (2155012) answered the same request from the router (litellm.BadRequestError: You passed in model=non-existent-model. There are no healthy deployments for this model. Received Model Group=non-existent-model), which only happens while a wildcard route is registered. The only wildcard in the job is the assemblyai/* model test_adding_passthrough_model.py adds through POST /model/new and deletes through POST /model/delete. Before #39664 that delete left the wildcard behind, so every later unknown-model request went through the router; #39664 evicts it, and the proxy answers the plain 400 it always gave without wildcards. Case 2 below shows the original test passing only in that stale state

Shared setup, the CI job mirrored locally: proxy at the commit named by each heading, booted with --num_workers 2 (one process, two uvicorn workers behind 127.0.0.1:41237, one fresh Postgres database), from litellm/proxy/example_config_yaml/store_model_db_config.yaml with team_metadata_validator_e2e.py beside it, STORE_MODEL_IN_DB=True, LITELLM_MASTER_KEY=sk-1234, FAKE_OPENAI_API_BASE on tests/_fake_openai_endpoint_server.py, TEAM_METADATA_VALIDATION_SERVICE_URL on tests/store_model_in_db_tests/cost_center_service.py, LITELLM_LOG=ERROR. CI itself runs one container with the default single worker; a single-worker run gave the same results line for line. Both test files run in CI glob order from copies with 0.0.0.0:4000 replaced by 127.0.0.1:41237, since the files hardcode the port. test_e2e_assemblyai_passthrough fails here on a 401 because this machine has no ASSEMBLYAI_API_KEY; its fixture still adds and deletes the wildcard, which is the part that matters

P=http://127.0.0.1:41237
KEY=$(curl -s $P/key/generate -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"models":["gpt-4","non-existent-model"]}' | jq -r .key)

Before (59d42d3)

Case 1: unknown model with no wildcard route registered (CI's state today)

1. uv run pytest <copies>/test_adding_passthrough_model.py <copies>/test_openai_error_handling.py -vv

   test_e2e_assemblyai_passthrough FAILED        (401, no ASSEMBLYAI_API_KEY on this machine)
   test_assemblyai_routes_with_bad_api_key PASSED
   test_chat_completion_bad_model PASSED
   test_completion_bad_model PASSED
   test_embeddings_bad_model PASSED
   test_images_bad_model PASSED
   test_async_chat_completion_bad_model PASSED
   test_missing_model_parameter_curl[chat] PASSED
   test_missing_model_parameter_curl[completions] PASSED
   test_missing_model_parameter_curl[embeddings] PASSED
   test_missing_model_parameter_curl[images] PASSED
   test_chat_completion_bad_model_with_spend_logs FAILED
   >           assert log_entry["model_group"] == "non-existent-model"
   E           AssertionError: assert '' == 'non-existent-model'
   ======================== 2 failed, 10 passed in 18.05s =========================
   

2. curl -s -D - $P/v1/chat/completions -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" -d '{"model":"non-existent-model","messages":[{"role":"user","content":"Hello!"}]}'

   HTTP/1.1 400 Bad Request
   x-litellm-call-id: e4aac64b-6336-49ca-8b71-797c76be0b69
   {"error":{"message":"/chat/completions: Invalid model name passed in model=non-existent-model. Call `/v1/models` to view available models for your key.","type":"invalid_request_error","param":null,"code":"400",...}}
   

3. sleep 15; curl -s "$P/spend/logs?request_id=e4aac64b-6336-49ca-8b71-797c76be0b69" -H "Authorization: Bearer sk-1234" | jq '.[0] | {request_id, model, model_group, spend, total_tokens, status: .metadata.status, error_code: .metadata.error_information.error_code, error_class: .metadata.error_information.error_class, error_message: .metadata.error_information.error_message}'

   {
     "request_id": "e4aac64b-6336-49ca-8b71-797c76be0b69",
     "model": "non-existent-model",
     "model_group": "",
     "spend": 0.0,
     "total_tokens": 0,
     "status": "failure",
     "error_code": "400",
     "error_class": "ProxyModelNotFoundError",
     "error_message": "400: {'error': '/chat/completions: Invalid model name passed in model=non-existent-model. Call `/v1/models` to view available models for your key.'}"
   }
   

Case 2: unknown model with a wildcard route registered (the state a stale assemblyai/* left before #39664)

1. curl -s $P/model/new -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"model_name":"assemblyai/*","litellm_params":{"model":"assemblyai/*","custom_llm_provider":"assemblyai","api_key":"dummy","api_base":"https://api.assemblyai.com","use_in_pass_through":true},"model_info":{}}'

   {"model_name":"assemblyai/*","id":"48f592dd-5816-4554-bfde-7f6d2eaa5abe"}
   

2. sleep 12; for i in 1 2 3 4; do curl -s $P/v1/chat/completions -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" -d '{"model":"non-existent-model","messages":[{"role":"user","content":"Hello!"}]}' | jq -r .error.message; done (both workers now hold the wildcard)

   litellm.BadRequestError: You passed in model=non-existent-model. There are no healthy deployments for this model. Received Model Group=non-existent-model
   Available Model Group Fallbacks=None
   (same answer four times)
   

3. uv run pytest "<copies>/test_openai_error_handling.py::test_chat_completion_bad_model_with_spend_logs" -vv (the original test)

   test_chat_completion_bad_model_with_spend_logs PASSED
   ============================== 1 passed in 15.55s ==============================
   

4. Same chat request as case 1

   HTTP/1.1 400 Bad Request
   x-litellm-call-id: c35fc787-48a1-4ebe-87cd-fb4fa6ce40bb
   {"error":{"message":"litellm.BadRequestError: You passed in model=non-existent-model. There are no healthy deployments for this model. Received Model Group=non-existent-model\nAvailable Model Group Fallbacks=None","type":"invalid_request_error","param":null,"code":"400"}}
   

5. Same spend log read as case 1 for that call id

   {
     "request_id": "c35fc787-48a1-4ebe-87cd-fb4fa6ce40bb",
     "model": "non-existent-model",
     "model_group": "non-existent-model",
     "spend": 0.0,
     "total_tokens": 0,
     "status": "failure",
     "error_code": "400",
     "error_class": "BadRequestError",
     "error_message": "litellm.BadRequestError: You passed in model=non-existent-model. There are no healthy deployments for this model. Received Model Group=non-existent-model\nAvailable Model Group Fallbacks=None"
   }
   

6. curl -s $P/model/delete -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"id":"48f592dd-5816-4554-bfde-7f6d2eaa5abe"}'; sleep 12 then the four-request loop from step 2 again: the proxy is back in case 1's state on both workers

   {"message":"Model: 48f592dd-5816-4554-bfde-7f6d2eaa5abe deleted successfully"}
   /chat/completions: Invalid model name passed in model=non-existent-model. Call `/v1/models` to view available models for your key.
   (same answer four times)
   

After (a61bead)

Case 1: unknown model with no wildcard route registered (CI's state today)

1. uv run pytest <copies>/test_adding_passthrough_model.py <copies>/test_openai_error_handling.py -vv

   test_e2e_assemblyai_passthrough FAILED        (401, no ASSEMBLYAI_API_KEY on this machine)
   test_assemblyai_routes_with_bad_api_key PASSED
   test_chat_completion_bad_model PASSED
   test_completion_bad_model PASSED
   test_embeddings_bad_model PASSED
   test_images_bad_model PASSED
   test_async_chat_completion_bad_model PASSED
   test_missing_model_parameter_curl[chat] PASSED
   test_missing_model_parameter_curl[completions] PASSED
   test_missing_model_parameter_curl[embeddings] PASSED
   test_missing_model_parameter_curl[images] PASSED
   test_chat_completion_bad_model_with_spend_logs PASSED
   ======================== 1 failed, 11 passed in 16.87s =========================
   

2. Same chat request as before

   HTTP/1.1 400 Bad Request
   x-litellm-call-id: 139c0c64-3c68-4f84-a3bc-d71113cdc679
   {"error":{"message":"/chat/completions: Invalid model name passed in model=non-existent-model. Call `/v1/models` to view available models for your key.","type":"invalid_request_error","param":null,"code":"400",...}}
   

3. Same spend log read as before

   {
     "request_id": "139c0c64-3c68-4f84-a3bc-d71113cdc679",
     "model": "non-existent-model",
     "model_group": "",
     "spend": 0.0,
     "total_tokens": 0,
     "status": "failure",
     "error_code": "400",
     "error_class": "ProxyModelNotFoundError",
     "error_message": "400: {'error': '/chat/completions: Invalid model name passed in model=non-existent-model. Call `/v1/models` to view available models for your key.'}"
   }
   

   This row is exactly what the test now asserts on: the 400, model, model_group one of "" or "non-existent-model", zero spend and tokens, error_code "400", error_class one of ProxyModelNotFoundError or BadRequestError, and the model name in the message

Case 2: unknown model with a wildcard route registered (the state a stale assemblyai/* left before #39664)

1. Same POST /model/new for assemblyai/* as before (model_id 0873709c-1a14-420b-8aa1-677de2919e45), same 12 second wait and four-request check: all four answers come from the router

   litellm.BadRequestError: You passed in model=non-existent-model. There are no healthy deployments for this model. Received Model Group=non-existent-model
   

2. uv run pytest "<copies>/test_openai_error_handling.py::test_chat_completion_bad_model_with_spend_logs" -vv (the fixed test)

   test_chat_completion_bad_model_with_spend_logs PASSED
   ============================== 1 passed in 15.57s ==============================
   

3. Same chat request and spend log read as before

   HTTP/1.1 400 Bad Request
   x-litellm-call-id: 20b1e071-c50e-4cca-a78c-11a92d8e06fc
   {
     "request_id": "20b1e071-c50e-4cca-a78c-11a92d8e06fc",
     "model": "non-existent-model",
     "model_group": "non-existent-model",
     "spend": 0.0,
     "total_tokens": 0,
     "status": "failure",
     "error_code": "400",
     "error_class": "BadRequestError",
     "error_message": "litellm.BadRequestError: You passed in model=non-existent-model. There are no healthy deployments for this model. Received Model Group=non-existent-model\nAvailable Model Group Fallbacks=None"
   }
   

   The fixed test accepts this row too: it is the other shape the membership checks allow
4. Same POST /model/delete as before (Model: 0873709c-1a14-420b-8aa1-677de2919e45 deleted successfully); both workers were back to case 1's answer within a minute, and GET /v2/model/info listed no assemblyai/* row

Observations from the run, none caused or changed by this PR:

• two workers: a delete takes up to a minute to leave both workers
• that lag once left the sibling suite's assemblyai/* row behind; CI runs one worker
• a spend row for a model nothing matches carries model_group "" and no traceback
• POST /model/new without litellm_params answers a 500, not a 422

Type

✅ Test

Caveats (if any)

Low

• under two workers /v2/model/info can lag a /model/new or /model/delete by a sync tick, so the sibling suite's cleanup can miss a row; CI runs one worker and this PR leaves it alone
• a spend log row for a model nothing matches carries model_group ""; that is how the proxy already behaves, unchanged here
• test_e2e_assemblyai_passthrough needs ASSEMBLYAI_API_KEY, which CI has and this machine does not

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Low Risk
Test-only assertion updates; no production code or runtime behavior changes.

Overview
Fixes flaky proxy_store_model_in_db_tests by aligning test_chat_completion_bad_model_with_spend_logs with how the proxy records unknown-model failures after wildcard routes are properly evicted (#39664).

The test now asserts HTTP 400 on the chat completion call before checking spend logs. Spend-log expectations no longer assume the router-only shape: model_group may be "" (direct proxy rejection) or "non-existent-model" (wildcard/router path), and error_class may be ProxyModelNotFoundError or BadRequestError, with the model name still required in the error message. The strict litellm.BadRequestError message check was dropped in favor of that broader contract.

^{Reviewed by Cursor Bugbot for commit a61bead. Bugbot is set up for automated code reviews on this repo. Configure here.}

• a61bead passes /live-pr-risk

[greptile-apps]

Greptile Summary

Test-only updates align the unknown-model spend-log test with both supported HTTP 400 failure paths.

• Verifies that the completion request returns HTTP 400.
• Accepts the direct proxy and wildcard-router telemetry representations.
• Continues validating the request ID, model, zero spend and tokens, failure status, error code, error class, and model-bearing error message.

Confidence Score: 5/5

The PR appears safe to merge because it changes only test expectations and preserves meaningful validation for both supported failure representations.

No actionable new issue remains. The previous telemetry-coverage finding was manually resolved after model-group and error-class checks were restored.

Important Files Changed

Filename	Overview
tests/store_model_in_db_tests/test_openai_error_handling.py	Updates unknown-model error assertions to cover both legitimate proxy and router failure shapes without changing production behavior.

_{Reviews (4): Last reviewed commit: "test(store_model_in_db): accept both 400..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit a61bead. Configure here.}