fix(router): keep custom model_info across a price data reload

[yassin-berriai]

TLDR

Problem this solves:

• Price data reload wiped custom model_info from every model group
• Model Hub showed - / - for token limits after the reload
• Operator model_info overrides silently reverted to catalog values
• With pre-call checks on, alias requests died with "LLM Provider NOT provided"
• Once registrations survived reloads, dead ones were replayed forever

How it solves it:

• Runtime model metadata is re-applied onto the freshly fetched catalog
• Both reload paths (scheduled job and manual endpoint) share one helper
• Pre-call checks resolve the deployment model before the lookup that can raise
• An unresolvable provider skips the filter instead of failing the request
• A reload rebuilds deployment metadata from the routers that are alive then

Relevant issues

Linear ticket

Resolves LIT-4675

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live proxy on a real Postgres, hitting the real OpenAI API and costing real money. Two deployments, both carrying model_info token limits:

model_list:
  - model_name: polyu-chat
    litellm_params:
      model: hosted_vllm/gpt-5.6            # OpenAI-compatible, absent from the built-in cost map
      api_base: https://api.openai.com/v1
      api_key: os.environ/OPENAI_API_KEY
    model_info:
      max_input_tokens: 128000
      max_output_tokens: 16384
  - model_name: dashscope-qwen
    litellm_params:
      model: dashscope/qwen-max             # present in the cost map, so the override is what is at stake
      api_key: os.environ/OPENAI_API_KEY
    model_info:
      max_input_tokens: 12345
      max_output_tokens: 678

router_settings:
  enable_pre_call_checks: true
general_settings:
  store_model_in_db: true

A reload interval was armed the way the Admin UI arms it:

$ curl -sX POST "http://127.0.0.1:4676/schedule/model_cost_map_reload?hours=24" -H "Authorization: Bearer $KEY"
{"message":"Model cost map reload scheduled for every 24 hours","status":"success","interval_hours":24,...}

Before, at 2a9843e

That is the current litellm_internal_staging tip, so this is the bug as it stands today. The Admin UI's Price Data Reload button, and nothing else:

$ curl -s -H "Authorization: Bearer $KEY" http://127.0.0.1:4676/model_group/info
  polyu-chat       max_input_tokens=128000.0   max_output_tokens=16384.0
  dashscope-qwen   max_input_tokens=12345.0    max_output_tokens=678.0

$ curl -sX POST http://127.0.0.1:4676/reload/model_cost_map -H "Authorization: Bearer $KEY"
{"message":"Price data reloaded successfully! 2987 models updated.",...}

$ curl -s -H "Authorization: Bearer $KEY" http://127.0.0.1:4676/model_group/info
  polyu-chat       max_input_tokens=None       max_output_tokens=None
  dashscope-qwen   max_input_tokens=30720.0    max_output_tokens=8192.0

$ curl -sX POST http://127.0.0.1:4676/v1/chat/completions -H "Authorization: Bearer $KEY" \
    -d '{"model":"polyu-chat","messages":[{"role":"user","content":"Reply with the single word OK"}]}'
  ERROR: litellm.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider you are
  trying to call. You passed model=polyu-chat

polyu-chat lost its limits outright; dashscope-qwen is the quieter half of the same bug, its configured 12345/678 replaced by the catalog's 30720/8192.

It does not take a click on the pod in question either. One admin's click publishes a revision that every other pod picks up on its own poll, so a pod that was never touched wipes its own metadata a few seconds after boot:

12:38:27 - LiteLLM Proxy:INFO: periodic_reload_schedule.py:154 - Model cost map reload triggered by manual reload request
12:38:27 - LiteLLM Proxy:INFO: proxy_server.py:6588 - Model cost map reloaded successfully. Models count: 2987

$ curl -s -H "Authorization: Bearer $KEY" http://127.0.0.1:4676/model_group/info
  polyu-chat       max_input_tokens=None       max_output_tokens=None
  dashscope-qwen   max_input_tokens=30720.0    max_output_tokens=8192.0

After, at 43d7ba0

The head moved to 856d2f7 after this was captured; the only difference is one debug log rewritten from an f-string to %-style arguments for the lazy-logging gate, which touches nothing shown here.

Same config, same database, same endpoints, and a second reload to show it is idempotent:

$ curl -s -H "Authorization: Bearer $KEY" http://127.0.0.1:4676/model_group/info
  polyu-chat       max_input_tokens=128000.0   max_output_tokens=16384.0
  dashscope-qwen   max_input_tokens=12345.0    max_output_tokens=678.0

$ curl -sX POST http://127.0.0.1:4676/reload/model_cost_map -H "Authorization: Bearer $KEY"
{"message":"Price data reloaded successfully! 2987 models updated.",...}
$ curl -s -H "Authorization: Bearer $KEY" http://127.0.0.1:4676/model_group/info
  polyu-chat       max_input_tokens=128000.0   max_output_tokens=16384.0
  dashscope-qwen   max_input_tokens=12345.0    max_output_tokens=678.0

$ curl -sX POST http://127.0.0.1:4676/reload/model_cost_map -H "Authorization: Bearer $KEY"   # again
$ curl -s -H "Authorization: Bearer $KEY" http://127.0.0.1:4676/model_group/info
  polyu-chat       max_input_tokens=128000.0   max_output_tokens=16384.0
  dashscope-qwen   max_input_tokens=12345.0    max_output_tokens=678.0

$ curl -sX POST http://127.0.0.1:4676/v1/chat/completions -H "Authorization: Bearer $KEY" \
    -d '{"model":"polyu-chat","messages":[{"role":"user","content":"Reply with the single word OK"}]}'
  content=OK  total_tokens=16

The same pod-poll path, which is what a pod that nobody clicked on runs, keeps the metadata and still serves the request:

12:39:28 - LiteLLM Proxy:INFO: periodic_reload_schedule.py:154 - Model cost map reload triggered by manual reload request
12:39:29 - LiteLLM Proxy:INFO: proxy_server.py:6595 - Model cost map reloaded successfully. Models count: 2987

$ curl -s -H "Authorization: Bearer $KEY" http://127.0.0.1:4676/model_group/info
  polyu-chat       max_input_tokens=128000.0   max_output_tokens=16384.0
  dashscope-qwen   max_input_tokens=12345.0    max_output_tokens=678.0

$ curl -sX POST http://127.0.0.1:4676/v1/chat/completions -H "Authorization: Bearer $KEY" \
    -d '{"model":"polyu-chat","messages":[{"role":"user","content":"Reply with the single word OK"}]}'
  content=OK  total_tokens=16

The same thing on the Model Hub

This is the surface the report was filed against. Admin UI, AI Hub, Model Hub tab. These two shots were taken on an earlier head of this branch, before the rebase onto current staging; the transcripts above are the same claim re-run against the head as it stands.

Before. polyu-chat reads - / -, and dashscope-qwen reads the catalog's 30.7K / 8.2K rather than the configured 12.3K / 678:

After. Both rows carry what the config asked for:

The Mode column moving from chat to - for polyu-chat is the fix working, not a regression. Booting the base branch with the reload schedule cleared, so no reload ever runs, gives mode=None, max_input_tokens=128000, max_output_tokens=16384 for polyu-chat and mode='chat', 12345, 678 for dashscope-qwen, which is exactly the fixed post-reload state. The chat in the before shot came from the wipe falling back to the catalog, not from the deployment.

Deleting a model, on the same live proxy

Restoring registrations onto every catalog is only correct while whatever registered them still exists. GET /model/cost_map/source reports len(litellm.model_cost), which makes the accumulation directly observable: 25 create-then-delete cycles through the same endpoints the Admin UI's Models page uses, then a price data reload.

for i in $(seq 1 25); do
  ID=$(curl -s -X POST $B/model/new -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
    -d "{\"model_name\":\"leak-probe-$i\",\"litellm_params\":{\"model\":\"hosted_vllm/leak-probe-$i\",\"api_base\":\"http://127.0.0.1:9/v1\",\"api_key\":\"sk-x\"},\"model_info\":{\"max_input_tokens\":1234}}" | jq -r '.model_info.id')
  curl -s -X POST $B/model/delete -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d "{\"id\":\"$ID\"}" >/dev/null
done
curl -s -X POST $B/reload/model_cost_map -H "Authorization: Bearer $KEY"
curl -s -H "Authorization: Bearer $KEY" $B/model/cost_map/source | jq -r .model_count

An earlier head of this branch recorded every registration and replayed it, so the reload put the deleted models straight back and the count never came down. At 43d7ba0 the reload is the garbage collector again, exactly as it was before this PR; the difference is that it no longer takes the live deployments with it:

baseline model_count:              2990
after 25 create+delete cycles:     3040
Price data reloaded successfully! 2987 models updated.
after a price data reload:         2990

The catalog itself is 2987 models, so 2990 is the baseline plus the config-declared deployments, and 3040 is 50 entries that no longer describe anything the proxy can route to.

At the unit level the same thing holds for the paths a curl cannot reach. Twenty per-request routers built from a caller-supplied user_config and then discarded, and twenty clientside-credential deployments, each left 40 and 20 entries behind on the previously reviewed commit; both leave nothing now, while an operator's own register_model override is still recorded and still survives a reload.

Type

🐛 Bug Fix

Changes

litellm.model_cost carries two kinds of entry: the catalog fetched from the model cost map, and whatever is registered at runtime through litellm.register_model. The Router registers every deployment's model_info there under its model id and its backend model name, and per-request custom pricing lands there too. A price data reload replaced the whole dict, so everything in the second category vanished. /model_group/info reads litellm.model_cost through get_deployment_model_info, which is why a model absent from the catalog came back with null token limits, and why a model present in it silently reverted to the catalog's numbers.

The reload now re-applies both on top of the freshly fetched catalog. That ordering matters: the catalog supplies fresh pricing and the re-applied entries re-assert the operator's intent over it, which is the same precedence a fresh boot produces. How each is restored is the subject of the next section, and it is the part the security review changed. Both reload paths already adopt a fetched catalog through the one _swap_in_model_cost_map helper, so the re-apply hangs off its tail and neither path can forget it; the count it returns is taken before anything is re-applied, so the endpoint keeps reporting on the price data alone.

The second defect is independent and outlives the first. Router._pre_call_checks resolved the per-deployment model name on the line after get_router_model_info, so any deployment that model-info lookup cannot map left the name unset, and the supported-params check then ran get_llm_provider on the bare model group name. That call sits outside the surrounding try, so a filter turned into a 400 for the whole request. The name is now resolved first, and provider resolution failing there skips the check for that deployment rather than escaping deployment selection, matching what the block already does when a provider reports no supported params.

Not everything registered at runtime is durable, and the difference matters here. Config-declared deployment metadata and explicit register_model overrides are lasting operator intent, so they are restored. Per-request custom pricing describes one call, so _register_custom_pricing_for_request opts out via persist_across_reloads=False; carrying it forward would let a one-off price beat fresh upstream pricing for the rest of the process.

Restoring a registration is only right while whatever registered it still exists, and that is what the security review caught. Before this PR a reload replaced litellm.model_cost wholesale, so a deleted deployment's entries were swept away as a side effect of the wipe. Preserving registrations removes that sweep, and a recording that only ever grows brings the deletion question with it.

So deployment metadata is not recorded at all. A reload asks the routers that are alive at that moment to re-assert the deployments they are serving right then, which is what _replay_model_cost_registrations does, over a weakref.WeakSet of live routers. Everything the accumulation question was about answers itself: a deleted deployment is not in model_list, a repointed one is there under its new backend, a deployment dropped for an inactive environment or an invalid tag_regex was never added, and a router built per request from a caller-supplied user_config is unreferenced the moment the request ends. None of them need withdrawing, so none of them can be missed, and there is no unregister path to keep in step with future removal paths.

The one thing with no owner to ask is an explicit litellm.register_model override, so those are still recorded and replayed. They come from an operator rather than from request traffic, and a registration describing a single request opts out via persist_across_reloads=False.

Registration itself moved into _register_deployment_in_model_cost, shared by _create_deployment, add_deployment and the rebuild, so a reload writes what a boot writes rather than a second implementation of it. That also settles a divergence between the two registration sites: add_deployment previously skipped the shared-backend mode preservation that _create_deployment applies, and now both get it.

Tests cover both halves and each was checked against a mutant: reverting the assignment order, dropping the provider guard, dropping the replay, dropping the recording, recording request-scoped registrations anyway, taking the reported model count after the re-apply rather than before, and reintroducing the stale variable reference in the scheduled path each turn the relevant test red. The rebuild was mutation-checked the same way: dropping the rebuild entirely, letting a discarded router keep contributing, leaving a deleted deployment in model_list so the rebuild still sees it, and recording router registrations globally again all turn the relevant test red. One mutation survives and is reported rather than counted as a kill: dropping the litellm_params custom-pricing fold from the payload changes nothing today, because both registration sites already carry that pricing on the stored model_info. It is kept so the payload stays derivable from a deployment alone, which is the property the rebuild rests on

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

The PR preserves operator model metadata across model-cost catalog reloads while preventing request-scoped and deleted deployment registrations from being replayed.

• Centralizes scheduled and manual catalog refreshes in refresh_model_cost_map.
• Rebuilds deployment metadata from currently live routers and preserves explicit operator registrations.
• Excludes per-request pricing registrations from durable replay.
• Resolves deployment models before supported-parameter pre-call filtering and tolerates unresolved providers.

Confidence Score: 5/5

The PR appears safe to merge.

The previously reported request-pricing persistence issue is fixed because all current request-scoped pricing registrations opt out of durable replay, while router-owned deployment entries are rebuilt only from live router state.

Important Files Changed

Filename	Overview
litellm/utils.py	Adds the centralized catalog-refresh and explicit-registration replay mechanism while allowing request-scoped registrations to opt out.
litellm/router.py	Rebuilds cost-map metadata from live router deployments and adjusts provider resolution during pre-call checks.
litellm/main.py	Marks completion and embedding request-scoped custom pricing as nonpersistent across catalog reloads.
litellm/proxy/proxy_server.py	Routes both scheduled and manual model-cost reloads through the shared refresh helper.
tests/test_litellm/test_router_model_cost_isolation.py	Covers live-router replay, discarded routers, deployment deletion and repointing, and request-pricing isolation.
tests/test_litellm/test_utils.py	Covers durable operator registration replay and removal of nonpersistent request registrations on refresh.

_{Reviews (4): Last reviewed commit: "fix(router): keep custom model_info acro..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 94.87179% with 4 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/router.py	96.55%	2 Missing ⚠️
litellm/utils.py	86.66%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[yassin-berriai]

@greptileai please review the current head 478a554

Good catch on the registration lifetime; the finding was right and is fixed rather than rebutted. register_model now takes persist_across_reloads (default True, so an explicit override still survives a refresh) and _register_custom_pricing_for_request in litellm/main.py passes False, so per-request pricing is applied to the current catalog and never replayed onto later ones. A regression test pins both directions in one refresh: the durable override wins over fresh catalog pricing, the request-scoped one does not.

The push also fixes a real bug your review prompted me to look harder for. The scheduled reload path still referenced the local I had removed, inside a block wrapped by a broad except, so the reload worked while its success log silently raised. There is now a test that drives _check_and_reload_model_cost_map and asserts nothing was logged as an exception, mutation-checked against exactly that regression.

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_lit4675_model_cost_reload (856d2f7) with litellm_internal_staging (2792887)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (09dd167) during the generation of this report, so 2792887 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[yassin-berriai]

@greptileai please review the current head 56d4aad

The security review's finding was right and is fixed rather than rebutted. Before this PR a reload replaced litellm.model_cost wholesale, so a deleted deployment's entries were swept away as a side effect of the wipe; once the replay preserves registrations, nothing sweeps them, and repeated create/delete grows the registry for the life of the process. Router.delete_deployment now withdraws what the deployment registered, and upsert_deployment does the same for the outgoing registration so repointing a deployment at a different backend model does not strand the old key.

The two key kinds are handled differently on purpose. A deployment id was only ever written by that one deployment, so it is dropped from litellm.model_cost outright. A backend key is shared with the catalog and with other deployments on the same backend model, so it is withdrawn from the replay registry only, and only once no remaining deployment registers under it; dropping it from litellm.model_cost would take real upstream pricing down with it. A test pins that by deleting a deployment sitting on gemini/gemini-2.5-pro and asserting the catalog price survives.

Proof of fix in the description is re-captured on a live proxy against this head: 25 create-then-delete cycles plus a price data reload leave len(litellm.model_cost) at 3036 on the previously reviewed commit and back at its 2986 baseline here. Five mutants were checked, one per branch of the withdrawal logic. The branch is also rebased onto current staging, which resolves the conflict.

[yassin-berriai]

osv-scan is red for a reason that is not in this diff. It flags `cryptography 48.0.1` in `uv.lock` (GHSA-g6cj-pr64-35w5, GHSA-jwv3-5hgf-82ww, GHSA-m2h6-j472-rp4c; fixed in 49.0.0 / 50.0.0), and `git diff litellm_internal_staging...HEAD -- uv.lock pyproject.toml` is empty here. Every open PR is failing the same scan right now, so the pin is on the base branch and the bump belongs in its own PR rather than this one.

[yassin-berriai]

@greptileai please review the current head ca31fac

This replaces the withdrawal hooks from the previous round with a change of mechanism, because the hooks were the wrong shape. The security review's finding was one instance of a class: a reload was re-asserting everything ever registered, so anything whose owner had gone kept coming back. Each hook I added closed one path and left the next one open, and one of them could not be closed at all, since _route_user_config_request builds a Router per request from caller-supplied config and discards it, and no unregister call can reach a router nothing references. Measured on the previously reviewed commit, 20 such requests left 40 entries behind and 20 clientside-credential deployments left 20.

Deployment metadata is now not recorded at all. A reload asks the routers alive at that moment to re-assert what they are serving right then, over a weakref.WeakSet. A deleted deployment is not in model_list, a repointed one is there under its new backend, one dropped for an inactive environment or an invalid tag_regex was never added, and a discarded router is gone. All of it falls out of the design rather than out of a hook, and unregister_model plus both call sites are deleted. Explicit register_model overrides have no owner to ask, so those are still recorded; they come from an operator rather than from request traffic.

Registration moved into one _register_deployment_in_model_cost used by _create_deployment, add_deployment and the rebuild, so a reload writes what a boot writes. Worth flagging that this also gives add_deployment the shared-backend mode preservation _create_deployment already had, which was an existing divergence between two near-copies.

Proof of fix is re-captured live against this head, and a test asserts the rebuilt entry contains every field the boot registration wrote, including the cache pricing that is derived rather than stored. Four mutants are killed; one survives and is reported in the description as equivalent rather than counted as a kill.