fix(proxy): treat malformed cost-map token limits as absent on /v1/models

[yuneng-berri]

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Config used for both runs; the first deployment carries a non-numeric limit, the second is a healthy control that shows whether the whole listing survives

model_list:
  - model_name: openai/regression-probe-33891
    litellm_params:
      model: openai/regression-probe-33891
      api_key: os.environ/OPENAI_API_KEY
    model_info:
      max_input_tokens: "128,000"

  - model_name: openai/gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY

general_settings:
  master_key: sk-1234

Before, at 595e724 (this branch's parent, tip of litellm_internal_staging at the time)

$ curl -s -w "\nHTTP_STATUS: %{http_code}\n" http://localhost:4006/v1/models -H "Authorization: Bearer sk-1234"
{"error":{"message":"Internal server error","type":"internal_server_error"}}
HTTP_STATUS: 500

The proxy log shows the cast that failed, inside the per-model listing loop

  File "litellm/proxy/utils.py", line 6133, in create_model_info_response
ValueError: invalid literal for int() with base 10: '128,000'
INFO:     127.0.0.1:60986 - "GET /v1/models HTTP/1.1" 500 Internal Server Error

After, at ab02127

$ curl -s -w "\nHTTP_STATUS: %{http_code}\n" http://localhost:4006/v1/models -H "Authorization: Bearer sk-1234"
HTTP_STATUS: 200

{"id": "openai/regression-probe-33891", "object": "model", "created": 1677610602, "owned_by": "openai"}
{"id": "openai/gpt-4o-mini", "object": "model", "created": 1677610602, "owned_by": "openai", "max_input_tokens": 128000, "max_output_tokens": 16384}
total listed: 37

The deployment with the malformed value is listed with its limits omitted, the healthy control keeps both of its limits, and the other 35 deployments come back instead of being lost to the 500

Same proxy still serves real traffic, hitting the live OpenAI API

$ curl -s http://localhost:4006/v1/chat/completions -H "Authorization: Bearer sk-1234" \
    -H "Content-Type: application/json" \
    -d '{"model":"openai/gpt-4o-mini","messages":[{"role":"user","content":"Reply with exactly: models listing is healthy"}],"max_tokens":20}'
{"id":"chatcmpl-E3B44CYMc7OrjsrqMrgoE9bgMnkVm","created":1784426224,"model":"openai/gpt-4o-mini","object":"chat.completion","choices":[{"finish_reason":"stop","index":0,"message":{"content":"models listing is healthy","role":"assistant"}}],"usage":{"completion_tokens":4,"prompt_tokens":15,"total_tokens":19}}
HTTP_STATUS: 200

Type

🐛 Bug Fix

Changes

create_model_info_response read max_input_tokens and max_output_tokens out of the cost map and cast both with a bare int(). The surrounding try/except covers only the get_model_info lookup, not the casts, so a deployment whose model_info carries a non-numeric limit raised inside the per-model loop and failed the entire GET /v1/models and /models response with a 500. Every healthy deployment went down with it. Before the cost-map switch the same config returned 200 and simply omitted that deployment's limits

The value gets there because a deployment's model_info is registered into litellm.model_cost verbatim, so it reaches the cost map and not only the router index. Router.get_configured_token_limits already coerced this safely for the deployment path, so guarding one path and not the other still left the pair regressing

Both call sites now share coerce_token_limit in litellm_core_utils/core_helpers.py. It returns None for anything that is not a usable number, so the listing omits that one limit and keeps serving, and it rejects bools since True/False is never a meaningful token limit. Coercion of well-formed values is unchanged, including numeric strings like "32000"

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Greptile Summary

This PR fixes a 500 Internal Server Error on GET /v1/models triggered when a deployment's model_info contains a non-numeric token limit (e.g., "128,000"). The root cause was bare int() casts in create_model_info_response that sat outside the surrounding try/except, so a single malformed value would crash the entire per-model loop and drop all other healthy deployments from the response.

• Introduces coerce_token_limit() in litellm_core_utils/core_helpers.py — a shared helper that returns None for bools, empty strings, locale-formatted numbers, and other non-coercible types — and replaces both the bare int() casts in proxy/utils.py and the local _as_int helper in router.py with it.
• Adds three new unit tests: a parametrized bad-value sweep covering the cost-map path, a mixed valid/malformed limit case, and a Router integration test that exercises the real litellm.model_cost registration path to prevent future regressions on the same code route.

Confidence Score: 5/5

Safe to merge — the change is narrowly scoped to token-limit coercion, the shared helper is correctly implemented, and the router refactor is behaviorally equivalent to the code it replaces.

The fix is minimal and targeted: coerce_token_limit correctly orders the bool-before-int check (bool subclasses int), adds OverflowError handling for inf/nan floats that the old router helper missed, and falls through cleanly for None and unsupported types. Both changed call sites now produce identical semantics to what they did for well-formed values. The new Router integration test covers the exact registration path described in the bug report, preventing a future regression through the real litellm.model_cost lookup.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/litellm_core_utils/core_helpers.py	Adds coerce_token_limit() — a safe coercion helper that returns None for bools, non-numeric strings, and unsupported types. Bool check correctly precedes int check (since bool is a subclass of int) and OverflowError is caught for inf/nan floats.
litellm/proxy/utils.py	Replaces bare int() casts on cost-map token limits with coerce_token_limit(). The old casts were outside the surrounding try/except, so a malformed value like '128,000' raised ValueError mid-loop and failed the entire /v1/models response with a 500.
litellm/router.py	Removes the local _as_int helper from get_configured_token_limits and replaces it with the shared coerce_token_limit. Behavior is equivalent: None/bool → None, OverflowError now also caught (improvement for inf/nan floats).
tests/test_litellm/proxy/test_proxy_utils.py	Adds three new tests: parametrized bad-value coverage for the cost-map path, a mixed valid/malformed limit test, and a Router integration test that exercises the real litellm.model_cost registration path described in the bug report. litellm.model_cost is saved/restored in a try/finally for isolation.

_{Reviews (1): Last reviewed commit: "fix(proxy): treat malformed cost-map tok..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_/quirky-heyrovsky-faafca (ab02127) with litellm_internal_staging (595e724)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (7891388) during the generation of this report, so 595e724 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩