diff --git a/e2e/cases/12_custom_pricing_must_honor_cache_tokens.md b/e2e/cases/12_custom_pricing_must_honor_cache_tokens.md index 368ad034c33e..0c529de28391 100644 --- a/e2e/cases/12_custom_pricing_must_honor_cache_tokens.md +++ b/e2e/cases/12_custom_pricing_must_honor_cache_tokens.md @@ -1,115 +1,60 @@ -# Case 12 — Deployment UUID entry in `litellm.model_cost` must not silently strip cache pricing +# Case 12 — Router must backfill cost fields from canonical entry for known models ## Goal -Regression guard for the **actual** production bug that triggered the -"cache pricing only correct after clicking Reload Price Data" report. - -### Root cause (verified end-to-end against the running proxy) - -Three pieces interact: - -1. **Router register-on-startup** - (`litellm/router.py:7230-7237`) - - ```python - _model_id = deployment.model_info.id - if _model_id is not None: - _model_info_dict = deployment.model_info.model_dump(exclude_none=True) - for field in CustomPricingLiteLLMParams.model_fields.keys(): - field_value = deployment.litellm_params.get(field) - if field_value is not None: - _model_info_dict[field] = field_value - litellm.register_model(model_cost={_model_id: _model_info_dict}) - ``` - - For each deployment loaded from DB / config, the router writes an - entry into `litellm.model_cost` keyed by the deployment's **UUID**. - The dict is `model_info.model_dump(exclude_none=True)` merged with - any custom-pricing keys in `litellm_params`. When the dashboard - `/model/new` form was used to add the model — that form exposes - only `input_cost_per_token` and `output_cost_per_token` — and if - `deployment.model_info` did not have static-map cache rates merged - in by the time `_create_deployment` ran, **the UUID entry written - into `litellm.model_cost` permanently lacks - `cache_read_input_token_cost` / `cache_creation_input_token_cost`**. - -2. **Cost calc prefers the UUID over the bare model name** - (`litellm/cost_calculator.py:661-672`) - - ```python - if custom_pricing is True: - if router_model_id is not None and router_model_id in litellm.model_cost: - entry = litellm.model_cost[router_model_id] - if entry.get("input_cost_per_token") is not None or ...: - return_model = router_model_id # ← UUID wins - else: - return_model = model - ``` - - Because the deployment has `input_cost_per_token` set, - `custom_pricing` is `True`. Because the router wrote a UUID entry - in step 1, it is found in `model_cost`. The model name handed to - `cost_per_token` is therefore the UUID — and `cost_per_token` - reads the partial UUID entry, finds `cache_*_input_token_cost = None`, - and drops cache tokens from the bill. - -3. **"Reload Price Data" is a coincidental band-aid** - (`litellm/proxy/proxy_server.py:13319`) - - ```python - litellm.model_cost = new_model_cost_map # ← whole-dict replacement - ``` - - The reload endpoint replaces `litellm.model_cost` wholesale with - the freshly-fetched static JSON. This **incidentally evicts every - deployment-UUID entry** the router previously wrote. On the next - request, `_select_model_name_for_cost_calc` finds the UUID no - longer present, falls through to the bare model name, hits the - complete static-map row, and bills correctly. - - Reload is **not** repopulating cache fields. It is clearing the - stale partial entry so the lookup falls through. - -### Why "Reload fixes some requests but not all" - -- **Single-process effect, multi-worker fleet** — the reload endpoint - only updates `litellm.model_cost` in the worker that handled the - HTTP POST. Other workers (and other machines) read a - `force_reload=True` flag in `LiteLLM_Config` and try to reload on - their next 10-second poll. Whichever worker gets there first clears - the flag back to `False` (`proxy_server.py:5192-5213`), so **any - worker that polls later than that loses the broadcast and never - reloads**. -- **Re-registration overwrites the fix** — the periodic DB-sync / - config-reload task calls `_create_deployment` again, re-running - step 1 above. That writes the partial UUID entry back into - `litellm.model_cost` and the bug returns on the affected worker. -- **Load balancing splits the symptom** — chat requests are spread - across all workers, so some calls hit a freshly-reloaded worker - (correct bill) and some hit a stale worker (under-billed). End - result: dashboard shows partial recovery, never full. - -### Numerical impact (verified) - -For the user-reported Usage (`claude-haiku-4-5-20251001`, -`prompt=100191`, `cache_read=99774`, `cache_creation=416`): - -| Path | Total | Notes | +Guard the behavior of `Router._backfill_cost_fields_from_canonical`: +when a deployment is registered via the dashboard `/model/new` form +(or DB sync) with only `input_cost_per_token` and +`output_cost_per_token` set, the **deployment-UUID entry** in +`litellm.model_cost` must be augmented with the missing +`CustomPricingLiteLLMParams` fields (`cache_read_input_token_cost`, +`cache_creation_input_token_cost`, etc.) pulled from the canonical +static entry for the bare model name. + +Without backfill, the cost calculator's custom-pricing path +(`_select_model_name_for_cost_calc` at `cost_calculator.py:661-672`) +prefers the deployment-UUID entry and silently drops cache pricing, +under-billing cache-heavy requests by ~93%. + +### Why this is needed (the original symptom) + +`/model/new` exposes only two pricing fields. The user-reported +`cleanedLitellmParams` dump confirmed it: input/output rates set, +cache rates absent. Operators experience the gap as: + +> "Cache pricing breakdown looks correct after I click Reload Price +> Data, but only on some requests. Reload doesn't stick." + +What Reload was actually doing: replacing +`litellm.model_cost` wholesale, which incidentally **evicted the +deployment-UUID entries** the router had registered. The next +request fell through to the bare model name, hit the canonical +entry, and billed correctly — for that worker, until the next DB +sync re-registered the deployment with the same partial dict, or +until the request load-balanced to a worker that never received +the reload broadcast. + +Backfill closes the gap deterministically: every worker, at +registration time, fills the missing fields from the canonical +entry. No more reload-as-fix and no more per-worker drift. + +### Numerical impact (user's prod Usage shape) + +For `claude-haiku-4-5-20251001`, `prompt=100191`, `cache_read=99774`, +`cache_creation=416`: + +| State | Total | Notes | |---|---|---| -| UUID-path with partial entry (worker before reload, or after re-sync) | **$0.000756** | cache portion silently dropped | -| Bare-model-name path (worker right after reload) | **$0.011253** | correct | +| Pre-fix (UUID entry missing cache rates) | **$0.000756** | cache portion silently dropped | +| Post-fix (canonical fields backfilled) | **$0.011253** | math correct | -Delta: **$0.010497 missing** per request, about **−93%** of the bill. -Across many cached requests this is significant revenue lost. +Restored revenue: **+$0.010497 per request** (93% of the bill). ## Preconditions - `e2e/tools/proxy status` reports `ready` -- `LITELLM_LOCAL_MODEL_COST_MAP=True` set in docker-compose -- No provider key needed — the case calls `response_cost_calculator` - directly with a fabricated `Usage` and a sentinel deployment UUID - registered via `litellm.register_model` +- `LITELLM_LOCAL_MODEL_COST_MAP=True` (already set in docker-compose) +- No DB, no provider, no real API key — pure in-process Router test. ## Steps @@ -121,110 +66,90 @@ echo "exit=$?" ``` The fixture: -1. Reads the static `claude-haiku-4-5-20251001` entry as the correct baseline -2. `litellm.register_model({: {input/output rates only, no cache fields}})` - to mimic what `router.py:7237` does for a `/model/new`-added deployment -3. Builds a `Usage` with `cache_read=99774`, `cache_creation=416` -4. Calls `response_cost_calculator(custom_pricing=True, router_model_id=)` - — exactly the proxy's logging-path shape -5. Asserts the result equals the static-map baseline within $0.0001 -## Expected (after the fix lands) +1. Reads the canonical static entry for `claude-haiku-4-5-20251001` + as the expected baseline. +2. Builds a `Router(model_list=[...])` with a single deployment whose + `litellm_params` carries only `input_cost_per_token` / + `output_cost_per_token` (exactly what `/model/new` produces). +3. Asserts `litellm.model_cost[]` has + `cache_read_input_token_cost` and + `cache_creation_input_token_cost` populated after Router + registration. +4. Runs `response_cost_calculator(custom_pricing=True, + router_model_id=)` for the user-reported Usage shape and + asserts the total equals the static-map baseline within $0.0001. + +## Expected — GREEN ``` -expected total (correct cache billing) = $0.011253 -actual total via UUID path = $0.011253 -PASS: UUID-path total agrees with static-map total within $0.0001 -exit=0 -``` - -## Current status — RED +After Router(model_list=...) registration: + UUID entry exists: True + input_cost_per_token = 1e-06 + output_cost_per_token = 5e-06 + cache_read_input_token_cost = 1e-07 + cache_creation_input_token_cost = 1.25e-06 -On `fix/prometheus-prompt-cache-tokens` and v1.83.10: - -``` expected total (correct cache billing) = $0.011253 -actual total via UUID path = $0.000756 -FAIL: cost calc via UUID path disagrees with static-map total by $-0.010497 (-93.3%) -exit=1 -``` +actual total via UUID path = $0.011253399999999998 -## Suggested fix - -`litellm/router.py:7230-7237` — when writing the UUID entry, merge -cache rate fields from the static map for the bare model name when -the deployment's litellm_params doesn't supply them: - -```python -_model_id = deployment.model_info.id -if _model_id is not None: - _model_info_dict = deployment.model_info.model_dump(exclude_none=True) - - # NEW: backfill cache rate fields from the static model_cost map - # for the bare model name. The dashboard /model/new form does not - # surface these, so without this step a UUID-keyed entry will - # silently strip cache pricing. - bare_model_name = deployment.litellm_params.get("model") - if bare_model_name and bare_model_name in litellm.model_cost: - static_entry = litellm.model_cost[bare_model_name] - for cache_field in ( - "cache_read_input_token_cost", - "cache_read_input_token_cost_above_200k_tokens", - "cache_creation_input_token_cost", - "cache_creation_input_token_cost_above_1hr", - "cache_creation_input_token_cost_above_200k_tokens", - ): - if _model_info_dict.get(cache_field) is None: - value = static_entry.get(cache_field) - if value is not None: - _model_info_dict[cache_field] = value - - # existing override loop unchanged — litellm_params still wins - for field in CustomPricingLiteLLMParams.model_fields.keys(): - field_value = deployment.litellm_params.get(field) - if field_value is not None: - _model_info_dict[field] = field_value - - litellm.register_model(model_cost={_model_id: _model_info_dict}) +PASS: UUID-path total agrees with static-map total within $0.0001 ``` -Properties of this fix: - -- **Deterministic** — every worker writes the same UUID entry, regardless - of whether reload has been triggered or how the deployment was added -- **Reload-free** — no operator action required; correctness comes from - the registration step itself -- **Preserves user overrides** — `litellm_params` cache rates still - win if explicitly supplied (e.g. by an enterprise customer with - negotiated discount cache pricing) -- **Multi-worker safe** — every worker independently does the merge - using its own local `litellm.model_cost`; no cross-worker - coordination needed - -A second, **separate** fix is also indicated for the reload -broadcast — the `force_reload` boolean flag at -`proxy_server.py:5192-5213` should be replaced by a monotonic -`last_reload_at` timestamp so all workers definitely observe the -reload. Track this as a separate task; it's out of scope for the -case 12 assertion. +## Where the fix lives + +- `litellm/router.py` — `_backfill_cost_fields_from_canonical` + staticmethod, invoked from both `_create_deployment` (init-time + path used by `set_model_list`) and `add_deployment` (runtime path + used by `/model/new` + DB sync). +- Unit tests pinning the three scopes (known model backfill, user + override wins, unknown model leaves fields absent): + `tests/test_litellm/test_router_backfill_cost_fields.py`. ## Failure modes | Symptom | Cause | |---|---| -| Current: `FAIL: ...disagrees by -93.3%` | Expected — the fix has not landed yet | -| `FAIL: register_model didn't write the UUID entry` | `register_model` API changed signatures; update the test setup | -| `FAIL: UUID entry has cache rates already` | Some upstream code is now auto-merging cache rates at register time — the bug may already be fixed; verify with a fresh `/model/new` round-trip and update this case to GREEN, or extend the simulated litellm_params with cache_field=None overrides to keep the test meaningful | -| `FAIL: static {MODEL} entry incomplete` | Bundled JSON regressed for the baseline model — see case 10 | +| `FAIL: cache_read_input_token_cost is None` after Router init | The backfill helper isn't being called from one of the register sites — check `_create_deployment` and `add_deployment` | +| `FAIL: cost calc disagrees by ~-93%` | The UUID entry was registered with partial data; backfill ran but didn't reach this field — verify `CustomPricingLiteLLMParams.model_fields.keys()` covers cache_* | +| `FAIL: static entry incomplete in the bundled JSON` | The canonical baseline for `claude-haiku-4-5-20251001` regressed — see case 10 | +| Cost off by tiny amounts (< $1e-6) | Floating-point rounding, not a regression — tolerance is $1e-4 | ## Cross-reference -- Case 10 — guards the bare model-name path (static map cache fields - present) -- Case 11 — guards observability (`error_information.error_message` - not silenced) -- This case (12) — guards the deployment-UUID path (the one the user - actually triggered) - -All three must be GREEN for cache billing on dashboard-added -deployments to be trustworthy without operator intervention. +- **Case 10** — guards the static `model_cost` entry has full pricing + (the canonical source the backfill copies *from*). +- **Case 11** — guards observability for failure logging + (`error_information.error_message`). +- **This case (12)** — guards the deployment-UUID path produced by + Router registration. + +All three GREEN means the cost-breakdown pipeline is trustworthy +end-to-end, with no operator intervention required. + +## Design rationale (why backfill, not "fix the short-circuit") + +The cost calculator has a `custom_pricing` short-circuit at +`cost_calculator.py:326-335` and a UUID-preference branch at +`661-672`. Both are intentional: when a deployment supplies custom +pricing, that pricing wins. + +The actual gap is upstream of cost calc: the **registered UUID +entry is incomplete** for known models, because the dashboard form +doesn't expose every CustomPricingLiteLLMParams field. Two valid +fixes were considered: + +1. **Cost calc fallback** — if UUID entry lacks a cache rate, fall + back to the bare model name's static entry. Rejected because it + adds a runtime lookup on every request and conflates "user + omitted" with "user wants zero". + +2. **Router-side backfill at registration** *(chosen)* — fill missing + fields once at register-time, so every worker's UUID entry is + complete and the cost calc reads a single source of truth. Aligns + with what `model_info` merge already does for the dashboard's + display path; eliminates the inconsistency. + +Backfill **only fills slots the user left blank**; any value +explicitly set in `litellm_params` still wins. Unknown / custom +models with no static entry pass through unchanged. diff --git a/e2e/cases/data/12_custom_pricing_must_honor_cache_tokens.py b/e2e/cases/data/12_custom_pricing_must_honor_cache_tokens.py index b49e4e708fe3..e33309c1e53b 100644 --- a/e2e/cases/data/12_custom_pricing_must_honor_cache_tokens.py +++ b/e2e/cases/data/12_custom_pricing_must_honor_cache_tokens.py @@ -1,39 +1,33 @@ """ -Regression fixture for Case 12 — `router.py:register_model` must not -write a deployment-UUID entry that strips cache rates. - -REAL PROD PATH (verified against a running e2e proxy): - - 1. Proxy startup / DB sync: `router.py:7230-7237` registers each - deployment into `litellm.model_cost` under its UUID. The dict it - writes is `deployment.model_info.model_dump(exclude_none=True)` - plus any `CustomPricingLiteLLMParams` keys from - `deployment.litellm_params`. When the dashboard `/model/new` form - was used to add the model, `litellm_params` carries only - `input_cost_per_token` and `output_cost_per_token` — and if - `deployment.model_info` lacks the static-map cache rates at - register time, the UUID entry written into `litellm.model_cost` - is permanently missing cache fields. - - 2. Cost calc time: `cost_calculator._select_model_name_for_cost_calc` - (cost_calculator.py:661-672) sees `custom_pricing=True` and - prefers the UUID entry over the bare model name. It returns the - UUID, and `cost_per_token` then computes against the partial - entry — cache tokens go unbilled. - - 3. Clicking "Reload Price Data" replaces `litellm.model_cost` - wholesale (proxy_server.py:13319), which incidentally evicts the - UUID entry. Next call resolves the bare model name and gets the - full static-map row, so it bills correctly — until the DB-sync - task re-registers the deployment a few minutes later. - -This case asserts: a deployment registered with partial pricing must -NOT under-bill cache tokens. The fix is in `router.py:_create_deployment` -(see suggested patch in the case markdown). - -Run via Case 12 runbook (docker exec). Exit non-zero when the cost -calc under-bills relative to the correct static-map total. +Regression fixture for Case 12 — Router must backfill cost fields from +the canonical static `litellm.model_cost` entry when registering a +deployment-UUID model_cost row, so that cost calculation against the +UUID resolves to the correct total even when the user didn't supply +cache rates (which the dashboard /model/new form doesn't expose). + +End-to-end flow exercised: + 1. Build a `Deployment` mimicking what the dashboard `/model/new` + produces — `litellm_params.input_cost_per_token` / + `output_cost_per_token` set, but no cache rate fields, and + `litellm_params.model="claude-haiku-4-5-20251001"` (known upstream). + 2. Call `Router.add_deployment(deployment)` — the same code path + hit by /model/new and DB-sync. + 3. Assert `litellm.model_cost[]` has both + `cache_read_input_token_cost` and + `cache_creation_input_token_cost` populated (backfilled from + the static entry for `claude-haiku-4-5-20251001`). + 4. Run `response_cost_calculator(custom_pricing=True, + router_model_id=)` for the user-reported Usage shape and + assert the total equals the static-map baseline. + +Before the router backfill landed, step 3 found `None` and step 4 +under-billed by ~93%. With backfill, the cost calc through the +custom_pricing/UUID path returns the same total as the bare model +name path — no more "Reload Price Data" workaround required. + +Run via Case 12 runbook (docker exec); exit non-zero on regression. """ + import os import sys @@ -42,6 +36,7 @@ import litellm from litellm import ModelResponse from litellm.cost_calculator import response_cost_calculator +from litellm.router import Router from litellm.types.utils import PromptTokensDetailsWrapper, Usage DEPLOYMENT_UUID = "case12-539b1c62-ac07-47ae-8987-29426984bb55" @@ -59,7 +54,7 @@ def fail(msg: str) -> None: sys.exit(1) -# --- correct baseline: pure model_cost lookup (no UUID interference) --- +# Correct baseline from the static map static_entry = litellm.model_cost.get(MODEL, {}) input_rate = static_entry.get("input_cost_per_token") output_rate = static_entry.get("output_cost_per_token") @@ -67,8 +62,8 @@ def fail(msg: str) -> None: cache_create_rate = static_entry.get("cache_creation_input_token_cost") if None in (input_rate, output_rate, cache_read_rate, cache_create_rate): fail( - f"static {MODEL} entry incomplete — this case relies on the bundled " - "JSON having full pricing for the baseline model. See case 10." + f"static {MODEL} entry incomplete in the bundled JSON — case 12 " + f"relies on it for the baseline. See case 10." ) non_cache_prompt = PROMPT_TOKENS - CACHE_READ - CACHE_CREATE @@ -79,29 +74,62 @@ def fail(msg: str) -> None: + CACHE_CREATE * cache_create_rate ) -# --- simulate the broken state router.py:7237 produces ----------------- -litellm.register_model({ - DEPLOYMENT_UUID: { - "input_cost_per_token": input_rate, - "output_cost_per_token": output_rate, - "litellm_provider": PROVIDER, - "mode": "chat", - # cache_*_input_token_cost intentionally absent — exactly what the - # dashboard /model/new form produces, and what gets register_model'd - # if deployment.model_info doesn't have the static-map cache fields - # merged in by the time _create_deployment runs. - } -}) - -if DEPLOYMENT_UUID not in litellm.model_cost: - fail("register_model didn't write the UUID entry — broken assumption") -if litellm.model_cost[DEPLOYMENT_UUID].get("cache_read_input_token_cost") is not None: +# Drop any stale UUID entry left by a previous run so the test is +# reproducible. (e2e harness Postgres is ephemeral, but litellm.model_cost +# is per-process and survives across pytest runs in the same container.) +litellm.model_cost.pop(DEPLOYMENT_UUID, None) + +# Build a Router with a single deployment whose litellm_params mimics the +# dashboard /model/new output — only input/output rates, no cache fields. +router = Router( + model_list=[ + { + "model_name": "case12-claude-haiku", + "litellm_params": { + "model": MODEL, + "custom_llm_provider": PROVIDER, + "input_cost_per_token": input_rate, + "output_cost_per_token": output_rate, + # cache_*_input_token_cost intentionally absent + }, + "model_info": { + "id": DEPLOYMENT_UUID, + }, + } + ] +) +del router # the registration side-effects are what we care about + +uuid_entry = litellm.model_cost.get(DEPLOYMENT_UUID, {}) +print("After Router(model_list=...) registration:") +print(f" UUID entry exists: {DEPLOYMENT_UUID in litellm.model_cost}") +print( + f" input_cost_per_token = {uuid_entry.get('input_cost_per_token')}" +) +print( + f" output_cost_per_token = {uuid_entry.get('output_cost_per_token')}" +) +print( + f" cache_read_input_token_cost = {uuid_entry.get('cache_read_input_token_cost')}" +) +print( + f" cache_creation_input_token_cost = {uuid_entry.get('cache_creation_input_token_cost')}" +) +print() + +if uuid_entry.get("cache_read_input_token_cost") is None: + fail( + "Router registered a deployment-UUID model_cost entry without " + "cache_read_input_token_cost. The backfill from canonical static " + "entry is missing — see router.py _backfill_cost_fields_from_canonical." + ) +if uuid_entry.get("cache_creation_input_token_cost") is None: fail( - "UUID entry has cache rates already — something is auto-merging that " - "this test was meant to detect; revisit the case design" + "Router registered a deployment-UUID model_cost entry without " + "cache_creation_input_token_cost. Same fix as above." ) -# --- build the request shape the proxy passes to cost calc ------------- +# Build the cost-calc request shape — same as the prod logging path. usage = Usage( prompt_tokens=PROMPT_TOKENS, completion_tokens=COMPLETION_TOKENS, @@ -119,11 +147,13 @@ def fail(msg: str) -> None: object="chat.completion", created=0, model=MODEL, - choices=[{ - "index": 0, - "message": {"role": "assistant", "content": "ok"}, - "finish_reason": "stop", - }], + choices=[ + { + "index": 0, + "message": {"role": "assistant", "content": "ok"}, + "finish_reason": "stop", + } + ], usage=usage, ) resp._hidden_params = { @@ -131,7 +161,6 @@ def fail(msg: str) -> None: "model_id": DEPLOYMENT_UUID, } -# --- the actual prod-shaped call --------------------------------------- actual_total = response_cost_calculator( response_object=resp, model=MODEL, @@ -141,14 +170,10 @@ def fail(msg: str) -> None: cache_hit=None, base_model=None, prompt="", - custom_pricing=True, # litellm_params has input_cost_per_token set + custom_pricing=True, router_model_id=DEPLOYMENT_UUID, ) -print(f"deployment UUID = {DEPLOYMENT_UUID}") -print(f"static {MODEL} entry has cache rates: yes") -print(f"UUID entry has cache rates: no (router writes partial dict)") -print() print(f"expected total (correct cache billing) = ${expected_total:.6f}") print(f"actual total via UUID path = ${actual_total!r}") @@ -157,22 +182,16 @@ def fail(msg: str) -> None: EPS = 1e-4 if abs(actual_total - expected_total) > EPS: - print() - print(f"FAIL: cost calc via UUID path disagrees with static-map total by " - f"${actual_total - expected_total:+.6f} " - f"({100*(actual_total - expected_total)/expected_total:+.1f}%)") - print() - print("Cause: router.py:7237 registers the deployment under its UUID " - "with partial pricing. cost_calculator._select_model_name_for_cost_calc " - "prefers the UUID over the bare model name, and cost_per_token then " - "reads the partial entry — cache_*_input_token_cost are None, so " - "the cache portion is dropped.") - print() - print("Fix: in router._create_deployment, when writing the UUID entry " - "into litellm.model_cost, merge the static map's cache rate fields " - "for the bare model name when not provided in litellm_params. See " - "the case markdown 'Suggested fix' section.") - sys.exit(1) + diff_pct = 100 * (actual_total - expected_total) / expected_total + fail( + f"cost calc via UUID path disagrees with static-map total by " + f"${actual_total - expected_total:+.6f} ({diff_pct:+.1f}%). " + f"This means the Router registered a partial deployment-UUID " + f"entry and the cost calc fell through to a path that ignored " + f"cache pricing. Check router.py _backfill_cost_fields_from_canonical " + f"and confirm it is invoked from both register sites in " + f"_create_deployment and add_deployment." + ) print() print(f"PASS: UUID-path total agrees with static-map total within ${EPS}") diff --git a/litellm/router.py b/litellm/router.py index 6572d96f7b97..c306fa185850 100644 --- a/litellm/router.py +++ b/litellm/router.py @@ -6707,6 +6707,44 @@ def _generate_model_id(self, model_group: str, litellm_params: dict): return hash_object.hexdigest() + @staticmethod + def _backfill_cost_fields_from_canonical( + model_info_dict: dict, + litellm_params: dict, + ) -> None: + """ + Backfill missing CustomPricingLiteLLMParams fields in a + deployment-UUID model_cost entry from the canonical static + litellm.model_cost entry for the bare model name. + + The dashboard /model/new form only exposes input_cost_per_token + and output_cost_per_token. Without this backfill, deployments + added via the dashboard end up registered under their UUID with + cache_*_input_token_cost = None, and the cost calculator's + custom-pricing path silently drops cache token charges. + + Priority: + 1. Values already present in ``model_info_dict`` (set explicitly + by the user via dashboard or config) — kept as-is. + 2. Values in ``litellm.model_cost[bare_model_name]`` for known + upstream models — copied into missing slots. + + Only known models are eligible (those that have a static + model_cost entry). Custom in-house models with no static entry + pass through unchanged. Mutates ``model_info_dict`` in place. + """ + bare_model_name = litellm_params.get("model") + if not bare_model_name: + return + canonical = litellm.model_cost.get(bare_model_name) + if not canonical: + return + for field in CustomPricingLiteLLMParams.model_fields.keys(): + if model_info_dict.get(field) is None: + value = canonical.get(field) + if value is not None: + model_info_dict[field] = value + def _create_deployment( self, deployment_info: dict, @@ -6735,6 +6773,14 @@ def _create_deployment( if deployment.litellm_params.get(field) is not None: _model_info[field] = deployment.litellm_params[field] + # Backfill any missing cost fields from the canonical static + # entry for the bare model name. User-supplied values above + # always win; this only fills slots the user didn't touch. + self._backfill_cost_fields_from_canonical( + model_info_dict=_model_info, + litellm_params=_litellm_params, + ) + ## REGISTER MODEL INFO IN LITELLM MODEL COST MAP model_id = deployment.model_info.id if model_id is not None: @@ -7234,6 +7280,13 @@ def add_deployment(self, deployment: Deployment) -> Optional[Deployment]: field_value = deployment.litellm_params.get(field) if field_value is not None: _model_info_dict[field] = field_value + # Backfill any missing cost fields from the canonical static + # entry for the bare model name. User-supplied values above + # always win; this only fills slots the user didn't touch. + self._backfill_cost_fields_from_canonical( + model_info_dict=_model_info_dict, + litellm_params=deployment.litellm_params.model_dump(), + ) litellm.register_model(model_cost={_model_id: _model_info_dict}) # add to model names diff --git a/tests/test_litellm/test_router_backfill_cost_fields.py b/tests/test_litellm/test_router_backfill_cost_fields.py new file mode 100644 index 000000000000..f62f6f0ee88c --- /dev/null +++ b/tests/test_litellm/test_router_backfill_cost_fields.py @@ -0,0 +1,167 @@ +""" +Tests for Router._backfill_cost_fields_from_canonical. + +Background: the dashboard /model/new form exposes only input/output cost +fields. Without backfill, deployments added via the dashboard end up +registered under their UUID with cache_*_input_token_cost = None, and +the cost calculator's custom-pricing path silently drops cache token +charges. The backfill copies missing fields from +litellm.model_cost[bare_model_name] for known upstream models, while +preserving any value the user supplied explicitly. + +This file pins three behaviors: + 1. Known model + no user cache rates → backfilled from canonical + 2. Known model + explicit user cache rates → user values win + 3. Unknown model (no static entry) → no backfill, fields stay None +""" + +import os +import sys + +import pytest + +sys.path.insert( + 0, os.path.abspath("../../..") +) # Adds the parent directory to the system path + +import litellm +from litellm import Router + +KNOWN_MODEL = "claude-haiku-4-5-20251001" +UNKNOWN_MODEL = "company-private/in-house-llm-not-in-static-map" + + +@pytest.fixture(autouse=True) +def _ensure_local_model_cost_map(monkeypatch): + """Use bundled JSON deterministically — never depend on a live fetch.""" + monkeypatch.setenv("LITELLM_LOCAL_MODEL_COST_MAP", "True") + + +def _build_router(deployment_uuid: str, model: str, extra_params: dict | None = None): + """Build a single-deployment Router whose litellm_params mimics what + dashboard /model/new produces — only input/output rates, plus any + overrides callers want to add via ``extra_params``.""" + litellm_params: dict = { + "model": model, + "custom_llm_provider": "anthropic", + "input_cost_per_token": 1e-6, + "output_cost_per_token": 5e-6, + } + if extra_params: + litellm_params.update(extra_params) + return Router( + model_list=[ + { + "model_name": "alias-for-test", + "litellm_params": litellm_params, + "model_info": {"id": deployment_uuid}, + } + ] + ) + + +def test_known_model_backfills_missing_cache_fields(): + """Dashboard added a known-upstream model with only input/output rates. + The UUID entry should be backfilled from the static map so cache + pricing applies on the cost-calc UUID path.""" + deployment_uuid = "backfill-known-model-uuid" + litellm.model_cost.pop(deployment_uuid, None) + + _build_router(deployment_uuid=deployment_uuid, model=KNOWN_MODEL) + + entry = litellm.model_cost.get(deployment_uuid, {}) + assert ( + entry.get("input_cost_per_token") == 1e-6 + ), "user-supplied input_cost_per_token should be preserved" + assert ( + entry.get("output_cost_per_token") == 5e-6 + ), "user-supplied output_cost_per_token should be preserved" + + canonical = litellm.model_cost[KNOWN_MODEL] + assert ( + entry.get("cache_read_input_token_cost") + == canonical["cache_read_input_token_cost"] + ), "cache_read_input_token_cost should be backfilled from canonical entry" + assert ( + entry.get("cache_creation_input_token_cost") + == canonical["cache_creation_input_token_cost"] + ), "cache_creation_input_token_cost should be backfilled from canonical entry" + + +def test_user_supplied_cache_rates_override_backfill(): + """If the user explicitly sets cache rates in litellm_params (e.g. + they negotiated a discounted gateway rate), those values must win + over the canonical static values.""" + deployment_uuid = "backfill-user-override-uuid" + litellm.model_cost.pop(deployment_uuid, None) + + user_cache_read_rate = 2e-7 # 2x the canonical Anthropic rate + user_cache_create_rate = 9.99e-7 + + _build_router( + deployment_uuid=deployment_uuid, + model=KNOWN_MODEL, + extra_params={ + "cache_read_input_token_cost": user_cache_read_rate, + "cache_creation_input_token_cost": user_cache_create_rate, + }, + ) + + entry = litellm.model_cost.get(deployment_uuid, {}) + assert ( + entry.get("cache_read_input_token_cost") == user_cache_read_rate + ), "user-supplied cache_read_input_token_cost must win over canonical" + assert ( + entry.get("cache_creation_input_token_cost") == user_cache_create_rate + ), "user-supplied cache_creation_input_token_cost must win over canonical" + + canonical = litellm.model_cost[KNOWN_MODEL] + assert ( + entry.get("cache_read_input_token_cost") + != canonical["cache_read_input_token_cost"] + ), "test geometry weak — user value happens to equal canonical" + + +def test_unknown_model_no_backfill(): + """Custom in-house model with no static map entry: backfill must + not invent values — cache fields stay None / absent.""" + deployment_uuid = "backfill-unknown-model-uuid" + litellm.model_cost.pop(deployment_uuid, None) + litellm.model_cost.pop(UNKNOWN_MODEL, None) + assert ( + UNKNOWN_MODEL not in litellm.model_cost + ), "test setup expects UNKNOWN_MODEL to be absent from static map" + + _build_router(deployment_uuid=deployment_uuid, model=UNKNOWN_MODEL) + + entry = litellm.model_cost.get(deployment_uuid, {}) + assert ( + entry.get("input_cost_per_token") == 1e-6 + ), "user-supplied fields still register normally for unknown models" + assert ( + entry.get("cache_read_input_token_cost") is None + ), "cache_read must not be invented when no canonical entry exists" + assert ( + entry.get("cache_creation_input_token_cost") is None + ), "cache_creation must not be invented when no canonical entry exists" + + +def test_backfill_skips_when_canonical_lacks_field(): + """If the canonical static entry itself lacks a given field, the + backfill leaves the UUID entry unchanged for that field (i.e. no + None-for-None copy that would mask absence).""" + deployment_uuid = "backfill-canonical-partial-uuid" + litellm.model_cost.pop(deployment_uuid, None) + + _build_router(deployment_uuid=deployment_uuid, model=KNOWN_MODEL) + + entry = litellm.model_cost.get(deployment_uuid, {}) + # Pick a field the canonical entry definitely doesn't have. The + # bundled JSON's Anthropic claude-haiku-4-5-20251001 entry doesn't + # carry input_cost_per_audio_token, for instance. + canonical = litellm.model_cost[KNOWN_MODEL] + if canonical.get("input_cost_per_audio_token") is None: + assert ( + "input_cost_per_audio_token" not in entry + or entry["input_cost_per_audio_token"] is None + ), "backfill must not copy None-valued fields from canonical"