feat(router): backfill missing cost fields from canonical entry for known models - #4
Merged
Conversation
…nown models
When a deployment is added via the dashboard /model/new form (or
loaded from DB), the form exposes only `input_cost_per_token` and
`output_cost_per_token` — none of the cache rate fields are surfaced.
Router previously registered the resulting partial dict directly into
`litellm.model_cost` under the deployment UUID, leaving
`cache_read_input_token_cost` and `cache_creation_input_token_cost`
absent.
Downstream effect: `_select_model_name_for_cost_calc` prefers the
deployment-UUID entry when `custom_pricing=True`. With cache rate
fields missing, the cost calculator silently dropped cache token
charges — under-billing cache-heavy requests by ~93%.
Operators were treating "Reload Price Data" as a fix. It worked by
accident: replacing `litellm.model_cost` wholesale evicted the
deployment-UUID entries, causing fall-through to the canonical bare
model name entry. Multi-worker proxies experienced partial recovery
because the reload broadcast (force_reload flag in LiteLLM_Config) is
consumed by whichever worker polls first.
Fix: `Router._backfill_cost_fields_from_canonical` — at registration,
copy missing CustomPricingLiteLLMParams fields from
`litellm.model_cost[bare_model_name]` into the dict that will be
keyed by the deployment UUID. User-supplied values always win;
unknown / custom models with no static entry pass through unchanged.
Invoked from both register sites:
- `_create_deployment` (init-time path via set_model_list)
- `add_deployment` (runtime path via /model/new and DB sync)
Test plan:
- 4 unit tests in tests/test_litellm/test_router_backfill_cost_fields.py
* known model + missing cache rates → backfilled from canonical
* known model + explicit user cache rates → user values win
* unknown model (no static entry) → no backfill, fields stay absent
* canonical lacks a field → no None-for-None copy
- 4 existing tests in test_router_model_cost_isolation.py still pass
- e2e case 12 (`12_custom_pricing_must_honor_cache_tokens.md`) flips
RED → GREEN; cost calc through the deployment-UUID path now
produces $0.011253 instead of $0.000756 for the user's prod
Usage shape (Anthropic claude-haiku-4-5-20251001, prompt=100191,
cache_read=99774, cache_creation=416).
6 tasks
songkuan-zheng
added a commit
that referenced
this pull request
Jun 10, 2026
…axonomy (#78) * fix(streaming): reset Anthropic message_start cursor (output_tokens=1) when no message_delta arrives The Anthropic streaming protocol emits `message_start.usage.output_tokens=1` as a placeholder cursor; the real cumulative output count only arrives in the final `message_delta` event. When a stream is cancelled before `message_delta` lands (common for thinking models on long-tail prompts), ChunkProcessor._calculate_usage_per_chunk's last-wins accumulator left completion_tokens stuck at 1. Because 1 is truthy, the `completion_tokens or token_counter(text=...)` fallback in calculate_usage() never fired, and requests were billed for 1 output token even when several thousand tokens of text had actually streamed. Fix: track whether any chunk's completion_tokens exceeded 1 (saw_non_cursor_completion). If the only update we saw was the cursor, reset completion_tokens to 0 so the text-based fallback estimates from the real completion content. Legitimate 1-token completions (model returns "Yes." etc.) are unaffected in practice — token_counter on a 1-token completion_output also yields ~1, so billing stays approximately correct. Tier: C (universal bug fix — affects every LiteLLM user calling Anthropic with streaming + cancel/timeout). Upstream PR candidate once landed here. Tests added: - TestAnthropicCursorBug (6 cases) — pins the post-fix behavior - TestNonAnthropicStreamingIntact (2 cases) — guards against regression on providers without the cursor pattern All 9 existing streaming_chunk_builder_utils tests still pass. * feat(spend_logs): add success_partial status + cancellation metadata fields Adds the schema scaffolding for tracking client-cancelled requests as a billable "success_partial" status (instead of dropping them into the failure bucket, which both pollutes the proxy failure rate metric and zeroes out billing for compute the upstream provider already charged us for). New StandardLoggingPayloadStatus value: - "success_partial": client disconnected mid-flight (or upstream cut early) but upstream consumed billable compute. Bills prompt + the output that was generated up to cancellation. Does NOT count toward the proxy failure rate. New Literal types in litellm/types/utils.py (used by PR #3 cancel-billing path to populate metadata fields): - CancelPhase: before_upstream | during_upstream | streaming_partial | during_parsing - CancelUsageSource: upstream_truth | tokenizer_estimate | upstream_completed_after_cancel | shield_timeout | no_completion New StandardLoggingMetadata + SpendLogsMetadata fields (all Optional, populated only when the cancel-billing path fires): - cancellation_indicator: client_disconnect | upstream_disconnect - cancel_phase: lifecycle phase when cancel was detected - bytes_delivered_to_client: total bytes ACK'd to client before disconnect - upstream_completed: whether upstream call ran to completion - usage_source: provenance of the recorded usage numbers DB compatibility: - LiteLLM_SpendLogs.status is `String?` (not enum) — accepts new value without migration. - LiteLLM_SpendLogs.metadata is `Json?` — new fields land inside the JSON blob, no migration required. - Existing `metadata.error_information.error_code` (used by nginx/ Prometheus dashboards) is unchanged; cancellation fields are additive. Tier: B (internal change to a TypedDict + helper init; downstream behavior change ships in PR #3). The success_partial status name and field semantics are coordinated with the upcoming cancel-billing module. Tests: - 10 new cases covering literal acceptance, None-init, round-trip, error_code coexistence, and enum value coverage. - All 73 existing spend_tracking tests pass. * feat(cancel): catch CancelledError + re-route streaming/non-stream cancels to success_partial path Closes the 499 black hole. asyncio.CancelledError is a BaseException subclass since Python 3.8 — every `except Exception` in LiteLLM was letting it slip through silently: * SpendLogs got no row for the cancelled request * No callback fired (Langfuse trace stuck "Running", Prometheus failed_requests counter not incremented) * The upstream provider continued generating after the client TCP close and billed us for compute that never reached our ledger This PR introduces a small finalize layer that catches the cancel signal, marks the Logging object with cancel_phase / cancellation metadata (matching PR #4's schema), dispatches the partial response through the normal async_success_handler path with the new status="success_partial" classification, and re-raises CancelledError so asyncio's cancellation contract is preserved. Files ----- * litellm/litellm_core_utils/cancel_finalize.py (new) Public helpers: - mark_logging_obj_cancelled(logging_obj, phase, indicator, bytes_delivered): idempotent dict mutation; pathological inputs (None logging_obj, missing model_call_details) are no-ops. - finalize_streaming_cancel(stream_wrapper, logging_obj, ...): runs stream_chunk_builder on accumulated chunks → dispatches through async_success_handler with success_partial intent. Shielded internally so the SpendLogs write completes even if the runtime injects more cancel signals during teardown. Falls back to post_call_failure_hook if there are zero chunks. - finalize_non_stream_cancel(upstream_task, logging_obj, ..., shield_timeout_s=60.0): shields the upstream call past the cancel and waits for real usage (strategy A). On timeout, cancels upstream and records usage_source="shield_timeout". * litellm/proxy/proxy_server.py — async_data_generator Added `except asyncio.CancelledError:` before `except Exception:` that calls finalize_streaming_cancel and re-raises. * litellm/proxy/common_request_processing.py — async_streaming_data_generator Same catch for the /v1/messages and /v1beta/...generateContent paths (Anthropic + Google). * litellm/litellm_core_utils/streaming_handler.py — __anext__ Defense-in-depth: if cancel reaches the stream wrapper directly (not via the proxy generator), mark the Logging object and re-raise so whoever does finalize the request gets the marker. * CLAUDE.md — Test discipline Added "No theater tests" rule: mock the boundary (httpx transport, DB cursor), not the unit under test. Use a small spy fake to capture handler args; assert on the captured data, not on `.called`. Example: instead of mocking `stream_chunk_builder`, pass real ModelResponseStream chunks in. The rule is enforced in this PR's test file. Tests (18 cases, all real — no mocking of the unit under test): * mark_logging_obj_cancelled: dict-mutation contract incl. idempotency, None handling, no-pollution-with-Nones * is_logging_obj_cancelled: marker detection * _get_accumulated_chunks: stream wrapper variants * finalize_streaming_cancel: - Real Anthropic-shaped chunks (message_start cursor=1 + content deltas, NO message_delta) run through real stream_chunk_builder → captured response.usage.completion_tokens > 1, exercising the cross-PR contract with PR #1's cursor reset - No-chunks edge: real post_call_failure_hook spy verifies fallback fires with a CancelledError, not a swallowed signal - Downstream success_handler raising must NOT propagate * finalize_non_stream_cancel: - Shield-wait happy path: real asyncio task, real response flows through to captured success kwargs - Shield timeout: upstream task actually cancelled - Upstream exception during shield window - No upstream task (before_upstream cancel) Cost computation for the partial response — the actual implementation of "modified strategy 5'" billing — lands in PR #3 (cancel_billing.py) which reads the markers this PR sets. Until PR #3 lands, the reassembled partial response runs through the existing cost calculator unchanged, which is already a significant improvement over the prior "drop everything" behavior because PR #1's cursor=1 fix produces sensible partial usage for Anthropic. Tier: C+D (Tier C for the BaseException catch fix — purely a bug; Tier D for the success_partial taxonomy — opinionated mechanism with billing implications we want carried in the fork until upstream agrees on semantics). * feat(cancel): compute partial cost + bridge cancel metadata to SpendLogs Completes the cancel-billing path started in PR #2. cancel_finalize was catching cancels and dispatching through async_success_handler with a reassembled partial response, but two gaps remained: 1. The cancellation markers written to logging_obj.model_call_details never reached SpendLogs — _get_spend_logs_metadata pulls fields from request_data.litellm_params.metadata, not from model_call_details. Result: SpendLogs rows existed (good, no more black hole) but had no cancel_phase / usage_source / etc., so dashboards couldn't tell normal vs partial. 2. The zero-chunk fallback path (finalize → failure hook) still wrote response_cost=0.0 — the upstream provider received the prompt and started processing, but our ledger said "free". For thinking models on long prompts this is a real $ leak ($0.003-$0.01 per 499 in production). Files ----- * litellm/litellm_core_utils/cancel_billing.py (new): - compute_prompt_only_cost(messages, model, custom_llm_provider): best-effort prompt-only $ via the real LiteLLM cost map. Returns 0.0 on any pricing failure — caller is in the failure hook and cannot afford an exception. Wraps token_counter + cost_per_token so the cost-map lookup path is exercised by real provider IDs. - enrich_request_metadata_with_cancel_markers(request_data, logging_obj): copies the five cancel markers from model_call_details into litellm_params.metadata. Idempotent; no-op when there's no marker (normal requests pay zero overhead). Overrides metadata.status to "success_partial". * litellm/litellm_core_utils/cancel_finalize.py: - finalize_streaming_cancel now calls enrich_request_metadata_with_cancel_markers right after mark_logging_obj_cancelled, so the partial response dispatched through async_success_handler carries the markers into the cost-tracking pipeline. - finalize_non_stream_cancel does the same bridge, AND a second bridge after the shield-wait outcome is known (upstream_completed / usage_source can change after the asyncio.wait_for resolves). * litellm/proxy/hooks/proxy_track_cost_callback.py: - async_post_call_failure_hook: when the failure is actually a CancelledError (i.e. the cancel-finalize fallback path because no chunks were received), compute the prompt-only cost via cancel_billing.compute_prompt_only_cost instead of the hardcoded 0.0. Other failures (provider 5xx, real timeouts) still write 0.0 — there's no compute on our side to bill in those cases. - Same hook also calls enrich_request_metadata_with_cancel_markers so failure-path SpendLogs rows tagged success_partial get the right metadata even when called by external code paths that didn't go through cancel_finalize first. Tests (12 new cases, no mocking of cost_per_token / token_counter): * TestComputePromptOnlyCost: - Real Anthropic & OpenAI pricing paths exercised → cost > 0 with sanity upper bound (catches both "always 0" and "wrong rate" regressions) - Unknown model degrades to 0.0 without raising - Long-prompt cost scales with token count (sanity that we're actually counting, not constants) * TestEnrichRequestMetadataWithCancelMarkers: - No-op when no marker (must not pollute normal requests) - All five markers + status override propagate - Creates missing litellm_params.metadata dict - Partial markers don't insert keys for absent ones - Overwrites stale status (success / failure → success_partial) All 43 Phase 1 tests pass together (cursor + schema + cancel_finalize + cancel_billing). 82 streaming + spend_tracking regression tests pass. Tier: D (mechanism with billing policy — bills > 0 for cancelled requests, which is a behavior change downstream consumers can observe). * test(e2e): add case 26 cancel-billing + plumb success_partial through SpendLogs Adds an e2e case that verifies the full Phase 1 cancel-billing chain end-to-end against the live proxy + Postgres, and fixes three real plumbing bugs the case exposed: 1. FallbackStreamWrapper.__anext__ did not accumulate chunks FallbackStreamWrapper inherits from CustomStreamWrapper but overrides __anext__ to delegate to its own async generator — bypassing the parent's `self.chunks.append(chunk)`. Result: any streaming request that went through the Router fallback wrapper had empty wrapper.chunks, so finalize_streaming_cancel's reassembly path saw "no chunks" and fell through to the failure-hook fallback. Fix: append in __anext__ to mirror the parent. 2. enrich_request_metadata_with_cancel_markers only touched one of two metadata dicts The proxy failure path reads metadata from request_data["litellm_params"]["metadata"]; the success path reads from logging_obj.litellm_params["metadata"]. These are usually the same object but can diverge after construction-time copies. Fix: update both. Detected via case 26 showing status="success" completion_tokens=200 with all cancel markers blank — the success path was getting the partial response but not the markers. 3. _get_status_for_spend_log only knew "success" / "failure" metadata.status was set to "success_partial" by the cancel-finalize bridge, but _get_status_for_spend_log collapsed any non-"failure" value back to "success" before writing the LiteLLM_SpendLogs.status column. Dashboards filtering `WHERE status='success_partial'` would have matched zero rows. Fix: widen the Literal to three values and pass success_partial through unchanged. 4. _ProxyDBLogger.async_post_call_failure_hook clobbered cancel markers and hardcoded status="failure" The failure hook rebuilt litellm_params.metadata from scratch, preserving only "tags" — cancel markers set by cancel_finalize were lost on the zero-chunk fallback path. Also hardcoded metadata.status = "failure" even for CancelledError. Fix: detect CancelledError and classify as success_partial; preserve the five cancel marker fields alongside the existing tags preservation. 5. time.time() vs datetime.now() in finalize success_handler calls async_success_handler expects datetime instances for start_time / end_time (it does arithmetic on them); passing time.time() floats raised TypeError in the cost calculator and silently dropped the row. Fix: datetime.now(). Test plan (live proxy + Postgres): * Case 26 C1: streaming cancel mid-flight → status=success_partial, completion_tokens=214, cancel_phase=streaming_partial * Case 26 C2: long stream cancelled near end → status=success_partial, completion_tokens=760 * Case 26 C3: zero-chunk cancel → status=success_partial, completion_tokens=14 * Full mock-only e2e suite: 17/17 pass, 0 fail, 8 skip (expected real-provider skips). No regression in cases 04, 06, 07, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 23, 25. Unit tests: * 43 Phase 1 tests still pass (cursor + schema + cancel_finalize + cancel_billing). * 82 streaming + spend_tracking regression tests still pass. Tier: C (FallbackStreamWrapper accumulation, _get_status_for_spend_log widening — universal bug fixes) + D (success_partial classification, metadata bridging policy). * test(e2e): cases 27-32 cancel-billing expansion + Phase 2 gaps documented Extends the cancel-billing e2e coverage with five new mock-only cases. Two PASS outright, three honestly SKIP with detailed deferral reasons that point at the remaining Phase 2 work. PASSING (new): 29 failure-not-polluted — Forces upstream 503; asserts SpendLogs row classifies as status=failure with error metadata populated. Critical regression guard: without this, the CancelledError check in proxy_track_cost_callback could silently grow to catch other exceptions and start polluting success_partial. 32 concurrent-cancels — Fires 10 concurrent stream cancels, asserts all 10 SpendLogs rows write with success_partial + the marker, and a control health probe returns in < 3s (no asyncio task leak under contention). DEFERRED (skip 77, with detailed runbook comments): 27 non-stream-cancel — finalize_non_stream_cancel exists and is unit-tested, but no caller wires it up. LiteLLM removed the check_request_disconnection polling task (it's dead code in proxy_server.py), so non-stream cancel detection requires a Phase 2 polling revive. Until then non-stream cancels look like normal successes from the proxy's view (which is at least billing-correct, just unclassified). 28 v1-messages-cancel — The CancelledError catch in common_request_processing.py IS firing (verified: SpendLogs row gets correct partial completion_tokens reflecting bytes received before cancel), but the success_partial markers don't propagate because response.logging_obj is None on the anthropic_messages path. Phase 2: pull logging_obj from request_data["litellm_logging_obj"] instead of from response. 30 shield-timeout — Depends on case 27 wiring. Implemented the LITELLM_CANCEL_SHIELD_TIMEOUT_S env var (default 60s, e2e sets 2s) so the case can run deterministically once polling is wired. Other changes: * e2e/_config/mock_provider.py: added time.sleep(ttft_ms) to the Anthropic non-stream path so case 27/30 can deterministically drive client-cancel timing once the proxy plumbing lands. * e2e/_config/docker-compose.yml: sets LITELLM_CANCEL_SHIELD_TIMEOUT_S=2 on the proxy container. * litellm/litellm_core_utils/cancel_finalize.py: reads LITELLM_CANCEL_SHIELD_TIMEOUT_S env var at module load for the DEFAULT_CANCEL_SHIELD_TIMEOUT_S knob. Suite outcome: 30 cases · 19 PASS · 0 FAIL · 11 SKIP (11 skips = 6 pre-existing Tier=real + 1 pre-existing thinking-signature + 1 pre-existing anthropic-beta-overrides + 3 new Phase-2-deferred). Tier: B (e2e infrastructure + documentation of Phase 2 gaps). * feat(cancel): Phase 2 — non-stream cancel detection + /v1/messages markers Closes the remaining Phase 2 gaps documented in commit 987df5f: 1. /v1/messages cancel markers didn't reach SpendLogs (case 28) The CancelledError catch in async_streaming_data_generator was firing correctly but `getattr(response, "logging_obj", None)` returned None on the Anthropic native path (response is an async iterator without that attribute), so mark_logging_obj_cancelled became a no-op. Additionally, `_get_litellm_metadata_from_kwargs` prefers the newer `litellm_metadata` dict over `metadata` — and the cancel markers were only written to the latter, so they were filtered out by the SpendLogs pipeline even when present on the Logging instance. 2. Non-stream cancel was a silent black hole (case 27, 30) No proxy code path detected client disconnect during non-stream LLM calls. The dead-code `check_request_disconnection` helper had been in proxy_server.py for ages but with zero callers. Fixes ----- litellm/proxy/proxy_server.py + litellm/proxy/common_request_processing.py Both streaming catches: when response.logging_obj is None, fall back to request_data["litellm_logging_obj"]. Required for /v1/messages, /v1beta/...streamGenerateContent and any other endpoint where the response object isn't a CustomStreamWrapper. litellm/litellm_core_utils/cancel_billing.py enrich_request_metadata_with_cancel_markers now writes to BOTH the `metadata` and `litellm_metadata` keys on each litellm_params dict (request_data + logging_obj.litellm_params). litellm_metadata is the newer-endpoint variant; without writing to it, /v1/messages cancellations couldn't surface success_partial markers because the cost callback's metadata extractor returns litellm_metadata when both are present. litellm/proxy/common_request_processing.py — base_process_llm_request Added a _disconnect_watcher task that polls request.is_disconnected() every second while the LLM call is in flight. On detection: * Records the disconnect time and KEEPS the LLM call running. We don't want to cancel it: the upstream provider has almost certainly started processing and will charge us regardless, so the right thing is to record real upstream usage. * If the LLM call completes within LITELLM_CANCEL_SHIELD_TIMEOUT_S (default 60s, env-tunable) of the disconnect, the row gets tagged with usage_source=upstream_completed_after_cancel and upstream_completed=True. * If the budget elapses with the LLM still running, give up waiting, synthesize an empty response so the handler can return, and tag usage_source=shield_timeout / upstream_completed=False. The real response (when it eventually comes back from upstream) will fire a SECOND SpendLogs row via the normal success path — that's acceptable because it preserves the upstream-billing alignment. Implemented as a `wait_for(shield(...), poll_interval)` loop rather than a single wait_for + shield, so the watcher task gets a chance to set disconnect_flag between polls without being interrupted by the timeout itself. Cleaned up via a finally that cancels the watcher task regardless of how the LLM call resolved. e2e/cases/27, 28, 30 Un-skipped; all three now PASS against the live proxy. Test plan (mock-only e2e suite, full run): * 30 cases · 22 PASS · 0 FAIL · 8 SKIP * 8 SKIP = 6 Tier=real + 1 thinking-signature (pre-existing) + 1 anthropic-beta-overrides (Tier=real, requires Bedrock). All cancel- related cases now PASS — no remaining Phase 2 deferrals. * 125 unit tests still pass (cancel_billing 12 + cancel_finalize 18 + cursor 8 + spend_logs metadata 5 + streaming regression 9 + spend tracking regression 73). Tier: C (universal bug fix — anyone relying on non-stream cancel detection or the /v1/messages cancel taxonomy was getting wrong rows) + D (the disconnect-watcher's shield-and-wait policy is opinionated billing behavior we want to carry until upstream agrees on semantics). * feat(cancel): Phase 3 — orthogonal delivery_status/billing_status taxonomy Replace the single `status='success_partial'` marker with two orthogonal dimensions derived at SpendLog write time from the five cancel markers plus the raw status: delivery_status ∈ {full, partial, none} # what reached the client billing_status ∈ {full, partial, none} # what we billed The top-level SpendLogs status filter stays binary (success | failure); cancellations carry `status='success'` with metadata markers and the UI surfaces them through the new derived columns. Callers that need to target the cancel slice in SQL can filter on `metadata::jsonb->>'delivery_status'` / `billing_status` directly. Scope: - core derivation in `spend_tracking_utils._derive_delivery_billing_status` is the single source of truth — no other writer sets these fields. - e2e cases 26–30 and 32 updated to assert both dimensions; case 33 (real-anthropic-cancel) added end-to-end. - UI: view_logs columns and filter_options surface the new fields. - Tests added for the new derivation and for the binary status filter (test_status_filter_condition). * docs(cancel): post-Phase-3 cleanup — rename case 26, refresh case 33 doc, harden case 30 preflight, log upstream PR candidates - e2e/cases/26 renamed: success_partial → partial (refactor dropped the three-valued status sentinel). Updated dispatcher reference in run-all-cases. - e2e/cases/26 + 33 .md docs rewritten to describe the new binary status + orthogonal delivery_status / billing_status taxonomy, including the failure-mode lookup table for case 33 (real Anthropic). - e2e/cases/data/30_shield_timeout.sh: add the mock-container healthcheck preflight that the other mock-only cases already have, so real-mode runs SKIP (rc=77) instead of FAIL. - UPSTREAM_PR_QUEUE.md: log two Tier-C candidates (cursor=1 reset + CancelledError catch) and the Phase 3 taxonomy as a Tier-D ISSUE-FIRST entry. Refresh Last reviewed. Tier: B (internal infra / docs). Tried upstream first: N/A — purely internal cleanup.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a deployment is added via the dashboard
/model/newform (orloaded from DB), the form exposes only
input_cost_per_tokenandoutput_cost_per_token— none of the cache rate fields are surfaced.Router previously registered the resulting partial dict directly into
litellm.model_costunder the deployment UUID, leavingcache_read_input_token_costandcache_creation_input_token_costabsent.
Downstream effect:
_select_model_name_for_cost_calc(cost_calculator.py:661-672)prefers the deployment-UUID entry when
custom_pricing=True. Withcache rate fields missing, the cost calculator silently dropped
cache token charges — under-billing cache-heavy requests by ~93%.
Why "Reload Price Data" seemed to fix it (sometimes)
Reload replaces
litellm.model_costwholesale (proxy_server.py:13319),which incidentally evicts all deployment-UUID entries the router had
registered. The next request falls through to the canonical bare
model name entry and bills correctly — until the periodic DB-sync
task re-registers the deployment with the same partial dict, or the
request load-balances to a worker that didn't receive the reload
broadcast (the
force_reloadflag is consumed by whichever workerpolls first;
proxy_server.py:5192-5213).The fix in this PR makes the registered UUID entry complete on its
own, deterministically, on every worker. Reload becomes irrelevant
for this class of bug.
What changes
litellm/router.py_backfill_cost_fields_from_canonicalstaticmethod. Invoked from both register sites —_create_deployment(init-time path viaset_model_list) andadd_deployment(runtime path via/model/new+ DB sync)tests/test_litellm/test_router_backfill_cost_fields.pye2e/cases/12_custom_pricing_must_honor_cache_tokens.mde2e/cases/data/12_custom_pricing_must_honor_cache_tokens.pylitellm.register_modelBackfill semantics (priority)
deployment.litellm_paramsis preserved unchanged (e.g.negotiated discounted cache rates from a gateway).
Noneare filled fromlitellm.model_cost[bare_model_name]—but only if the canonical entry has a non-None value.
static map entry are unaffected (cache fields stay absent).
This mirrors what the existing
model_infomerge already does forthe dashboard's display layer, eliminating the
"display says cache pricing is set, but cost calc ignores it"
inconsistency.
Numerical impact (verified end-to-end)
For the user-reported Usage on
claude-haiku-4-5-20251001(
prompt=100191,cache_read=99774,cache_creation=416):Restored revenue: +$0.010497 per request (93% of the bill).
Test plan
test_known_model_backfills_missing_cache_fieldstest_user_supplied_cache_rates_override_backfilltest_unknown_model_no_backfilltest_backfill_skips_when_canonical_lacks_fieldtest_router_model_cost_isolation.py(4 tests) still passesOut of scope (separate work)
force_reloadbroadcast race (proxy_server.py:5192-5213).The reload broadcast mechanism has its own bug: the first worker
to see
force_reload=Trueclears it back toFalse, so otherworkers can miss the broadcast. After this PR, reload is no longer
the user-visible recovery path, so the race becomes lower priority
— track as separate work.
/model/newform gap: surfacing cache fields in theUI is the cleanest long-term fix but lives outside this codebase
(litellm dashboard / UI repo). Backfill makes the proxy correct
regardless of UI completeness.