fix(proxy): stop queued registry read-throughs spending the resync budget

[yuneng-berri]

TLDR

Problem this solves:

• Bursts right after adding a model get 400 Invalid model name
• Queued requests on another worker burn the shared resync budget

How it solves it:

• After waiting on the lock, check if the model is already loaded
• If it is, return right away without spending budget or hitting the DB
• Guardrails and agents get the same check through their existing lookups

User Flow

Before: right after an admin adds a model, part of a client burst for it fails with 400

1. The admin runs the proxy with 2 workers, models stored in the DB, and no Redis
2. They send POST http://localhost:4000/model/new for a new model name backed by openai/gpt-4o-mini and get 200
3. Their clients immediately send 60 concurrent POST http://localhost:4000/v1/chat/completions for that model name
4. 3 to 12 of them come back 400 Invalid model name passed in model=<name>, the rest 200

After: the same burst succeeds in full

1. The admin runs the proxy with 2 workers, models stored in the DB, and no Redis
2. They send POST http://localhost:4000/model/new for a new model name backed by openai/gpt-4o-mini and get 200
3. Their clients immediately send 60 concurrent POST http://localhost:4000/v1/chat/completions for that model name
4. All 60 come back 200

Relevant issues

Introduced by #36263, and it started showing up in CI (tests/integration/observability/test_azure_storage_chaos.py::test_slow_sink_lands_every_id_once_without_deadlock) after #43082

Affected release

Regression since v1.99.0-rc.1

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Shared setup: a fresh local postgres:16 container on port 55440, no Redis, and this config.yaml, so the second worker only learns about the new model through the read-through. Each checkout sits outside the main repo so no parent .env gets loaded, and every boot log only mentions Redis in the login_throttle.py "Redis is not configured" warning

model_list: []
general_settings:
  master_key: <a generated test key, also exported as LITELLM_MASTER_KEY>
  store_model_in_db: true

Each run starts a fresh proxy (steps 1 and 2), since a warm proxy finishes the first resync too fast for enough requests to queue behind it. Both sides ran the exact same steps: three runs with 60 concurrent requests and three with 150, each with a new model name

Before (b94eaa6)

1. Start the proxy with 2 workers from a checkout of that commit

   env -i HOME=$HOME PATH=$PATH PYTHONPATH=$PWD OPENAI_API_KEY=$OPENAI_API_KEY STORE_MODEL_IN_DB=True LITELLM_LOG=WARNING \
     DATABASE_URL=postgresql://postgres:pw@127.0.0.1:55440/litellm \
     python -m litellm.proxy.proxy_cli --config config.yaml --port 4111 --num_workers 2 > litellm.log 2>&1 &
   until [ "$(curl -s -o /dev/null -w '%{http_code}' localhost:4111/health/liveliness)" = 200 ]; do sleep 2; done; sleep 5

2. Add the model

   MODEL=nrb2
   curl -s -X POST localhost:4111/model/new -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' \
     -d "{\"model_name\":\"$MODEL\",\"litellm_params\":{\"model\":\"openai/gpt-4o-mini\",\"api_key\":\"os.environ/OPENAI_API_KEY\"}}" \
     -o /dev/null -w 'model/new %{http_code}\n'

   model/new 200
   

3. Immediately fire 60 concurrent chat completions for it and tally the status codes

   seq 60 | xargs -P 60 -I{} curl -s -o /dev/null -w '%{http_code}\n' localhost:4111/v1/chat/completions \
     -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' \
     -d "{\"model\":\"$MODEL\",\"max_tokens\":1,\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}" | sort | uniq -c

     48 200
     12 400
   

4. Check the proxy log

   grep -c 'resync budget' litellm.log; grep -m1 'resync budget' litellm.log; grep -m1 -o "Invalid model name passed in model=[^ .]*" litellm.log

   12
   17:36:33 - LiteLLM Proxy:WARNING: registry_read_through.py:82 - registry read-through for 'nrb2' skipped: resync budget of 20 per 5.0s exhausted
   Invalid model name passed in model=nrb2
   

5. Repeating steps 1 to 4 on a fresh proxy each time gives

   run 1, 60 requests:   55 x 200,  5 x 400,  5 budget warnings
   run 2, 60 requests:   48 x 200, 12 x 400, 12 budget warnings
   run 3, 60 requests:   57 x 200,  3 x 400,  3 budget warnings
   run 4, 150 requests: 136 x 200, 14 x 400, 14 budget warnings
   run 5, 150 requests: 133 x 200, 17 x 400, 17 budget warnings
   run 6, 150 requests: 134 x 200, 16 x 400, 16 budget warnings
   

After (1a012fe)

1. Start the proxy with 2 workers from a checkout of that commit, same command on port 4222

2. Add the model, same as Before with MODEL=nra2

   model/new 200
   

3. Immediately fire 60 concurrent chat completions for it, same as Before

     60 200
   

4. Check the proxy log

   grep -c 'resync budget' litellm.log; grep -c 'Invalid model name' litellm.log

   0
   0
   

5. Repeating steps 1 to 4 on a fresh proxy each time gives

   run 1, 60 requests:   60 x 200, 0 x 400, 0 budget warnings
   run 2, 60 requests:   60 x 200, 0 x 400, 0 budget warnings
   run 3, 60 requests:   60 x 200, 0 x 400, 0 budget warnings
   run 4, 150 requests: 150 x 200, 0 x 400, 0 budget warnings
   run 5, 150 requests: 150 x 200, 0 x 400, 0 budget warnings
   run 6, 150 requests: 150 x 200, 0 x 400, 0 budget warnings
   

Type

🐛 Bug Fix

Caveats (if any)

Low

• Only reproduces on a freshly started proxy in local runs
  • A warm Before proxy served 60 and 150 request bursts without 400s
• Different new models still share one 20 per 5s budget per worker, unchanged here

[greptile-apps]

Confidence Score: 5/5

[Medium risk] Changes how the registry read-through cache decides when to resync.

The PR appears safe to merge, with both previous findings addressed and no new blocking issues

Summary

The PR checks whether a registry entry is already loaded after taking the lock. Queued requests can then return without another database read or spending resync budget

• The latest changes move the burst test’s setup into the test and type its registry parameter
• Both numbered findings are fixed. The earlier resolved findings about timed sleeps and mutable collections remain fixed in the burst test
• No new actionable issues were found

_{Reviews (3) · Last reviewed commit: "test(proxy): keep the queued read-throug..."}

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_fix_2f0746 (1a012fe) with main (a206882)¹}

Footnotes

1. No successful run was found on main (22bd49e) during the generation of this report, so a206882 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[yuneng-berri]

@greptileai

[yuneng-berri]

@greptileai