Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 14 additions & 7 deletions config/litellm/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -5,19 +5,26 @@
# stock image's Anthropic->OpenAI translation drops prompt-cache hits on
# Qwen/DeepSeek, mis-streams reasoning models, silently drops params it does
# not recognise, manufactures a reasoning ceiling from the caller's thinking
# budget, and discards the provider's own bill during stream reassembly while
# its rate card cannot price the route either.
# patch_litellm_cache.py closes the eleven gaps at build time
# budget, and loses the BYOK half of the provider's bill during stream
# reassembly while its rate card cannot price the route either.
# patch_litellm_cache.py closes the eight gaps at build time
# (see that script for the diagnosis);
# cost_callback.py surfaces the resulting cost/cache stats into the pod log
# stream. Both mirror the host-side dotfiles setup that took Qwen cache hit
# rate from 0% to ~99.99%.
#
# Pinned to v1.86.2 — the version the patches and cost_callback were
# validated against, and the same release the dotfiles host install pins.
# Pinned to v1.94.0 — the newest stable published release, and the version
# the patches and cost_callback are validated against (#3697). The bump from
# v1.86.2 retired four patches upstream absorbed and narrowed a fifth.
#
# The patch script fails the build loudly if a needle stops matching, so a
# future bump can't silently ship an unpatched (full-input-rate) image.
FROM ghcr.io/berriai/litellm:v1.86.2
# future bump can't silently ship an unpatched (full-input-rate) image. Read
# that failure as "look at this", not as "upstream fixed it": at this bump,
# two needles stopped matching purely because upstream reflowed the code
# around them, and deleting those patches would have shipped an image paying
# full input rate on every Qwen turn. Confirm each miss against the new
# source before retiring anything.
FROM ghcr.io/berriai/litellm:v1.94.0

# Patches edit the installed litellm package (root-owned site-packages); the
# build runs as root. The runtime pod still drops to UID 1000 via the
Expand Down
6 changes: 3 additions & 3 deletions config/litellm/anthropic_thinking_policy.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,15 +28,15 @@
Historically this never mattered: the model-cost map did not carry these slugs,
``OpenrouterConfig`` advertised no reasoning knobs, and ``drop_params`` silently
discarded the synthesized param — which is precisely why these models have been
running at full depth. Patch 7 makes the OpenRouter param gate accurate, which
running at full depth. Patch 4 makes the OpenRouter param gate accurate, which
is right for an *operator-configured* ``reasoning_effort`` and wrong for this
adapter-manufactured one: it would turn a knob nobody set into the effective
setting, with no config file mentioning it and nothing in the logs (Patch 8
only fires on drops, and this param would no longer be dropped).

So Patch 9 gates the synthesis, defaulting it off. ``thinking`` stays out of
So Patch 6 gates the synthesis, defaulting it off. ``thinking`` stays out of
the OpenAI body for non-Claude models and only an explicitly configured
``reasoning_effort`` reaches the wire — exactly the property Patch 7 exists to
``reasoning_effort`` reaches the wire — exactly the property Patch 4 exists to
restore, without the adapter's bucket riding along.

The gate covers the *derived* value only. On an adaptive request
Expand Down
29 changes: 16 additions & 13 deletions config/litellm/cost_callback.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,15 +28,17 @@
calculate_usage``, a rebuild that enumerates the token/cache counts and
originally DROPPED the provider's ``cost`` / ``cost_details`` outright.
That is why this module recorded ``cost: null`` on 1252 of 1252 sampled
calls in run 6. The egg-litellm image's **patch 10** now carries those two
fields across the rebuild (``config/litellm/stream_cost_preservation.py``),
so the billed figure reaches ``_extract_cost`` on the streaming path too
calls in run 6. litellm 1.94.0 carries ``cost`` across that rebuild, and the
egg-litellm image's **patch 7** carries the ``cost_details`` it still drops
(``config/litellm/stream_cost_preservation.py``) — that is where the BYOK
number lives. Between them the billed figure reaches ``_extract_cost`` on
the streaming path too
and this module needs no change to read it — the value simply stops being
absent (#3691).

``cost: null`` therefore no longer means "streaming"; it means the cost was
genuinely unavailable — a stock (unpatched) LiteLLM under this callback, or
a provider that does not report one. It is still emitted as null rather
genuinely unavailable — a LiteLLM older than 1.94.0 under this callback, a
BYOK route on an unpatched image, or a provider that does not report one. It is still emitted as null rather
than ``0.0``: a zero would read in the logs as "this route is free", the
exact opposite of the cost-visibility signal this module exists to provide
(#2799).
Expand All @@ -46,7 +48,7 @@
(issue #3175). It is kept strictly separate from ``cost`` — an estimate
from a possibly-stale rate card must never be mistaken for a bill — and
follows the same null-not-zero rule when LiteLLM cannot price the model.
That was the case for every route egg uses until the image's **patch 11**
That was the case for every route egg uses until the image's **patch 8**
taught the model-info lookup to read OpenRouter's published rate card; it
remains the case for a model whose prompt-length surcharge lands on a
boundary or component LiteLLM's map has no slot for, which is declined whole
Expand Down Expand Up @@ -165,7 +167,8 @@ def _usage_from_response_obj(response_obj):
counts (the final usage chunk's counts are folded into
``response_obj.usage`` by ``stream_chunk_builder``) and, on the egg-litellm
image, for cost as well: that reassembly rebuilds a fresh ``Usage`` and
stock drops ``cost`` / ``cost_details`` with it, which patch 10 restores.
older releases dropped ``cost`` / ``cost_details`` with it; 1.94.0 carries
``cost`` and patch 7 supplies ``cost_details``.
Under a stock LiteLLM the counts still arrive and ``_extract_cost`` returns
None — see the module docstring."""
if response_obj is None:
Expand Down Expand Up @@ -195,8 +198,8 @@ def _extract_cost(usage):
provider, so fall back to ``cost_details.upstream_inference_cost`` (what
the upstream provider will bill for the same request). Either way, the
number we record matches real spend on that turn. Returns None when no
positive cost is present — a provider that reports none, or a stock LiteLLM
whose chunk reassembly drops it (see ``_usage_from_response_obj``).
positive cost is present — a provider that reports none, or a LiteLLM whose
chunk reassembly drops it (see ``_usage_from_response_obj``).
Callers must treat None as "unknown", not "$0".

``_positive`` rejects non-finite values as well as non-positive ones: a
Expand Down Expand Up @@ -322,7 +325,7 @@ def _extract_estimated_cost(mcd):
reported a bill. It is an estimate, not a bill: the pricing map may lag the
provider's rates or lack cache-discount entries for a model. Returns None —
never 0.0 — when LiteLLM couldn't price the call, mirroring the billed-cost
"unknown ≠ free" discipline. On the egg-litellm image patch 11 supplies
"unknown ≠ free" discipline. On the egg-litellm image patch 8 supplies
OpenRouter's published rates for slugs the bundled map does not carry, so a
None here now means a genuinely unpriceable model (an inexpressible
prompt-length surcharge, or a provider with no live card to read) rather
Expand Down Expand Up @@ -376,7 +379,7 @@ def _extract_model(mcd):
#
# ``stream`` is included because it selects which of the two paths the cost on
# this line came through (see the module docstring), and it was the reason
# ``cost`` read null on every line before patch 10 — worth having next to the
# ``cost`` read null on every line before 1.94.0 and patch 7 — worth having next to the
# number rather than inferred, and worth keeping now that the null case is rare
# enough to need explaining when it happens. ``max_tokens`` and ``n`` are not
# sampling knobs but shape the generation, and are cheap to carry.
Expand Down Expand Up @@ -643,7 +646,7 @@ def _extract_request_params(mcd):

Source is ``model_call_details['optional_params']``: LiteLLM's
POST-mapping parameter set, i.e. the dict that becomes the upstream
request body. Verified against the pinned litellm 1.86.2 on the
request body. Verified against the pinned litellm 1.94.0 on the
``anthropic_messages`` route (the route every Claude Code agent request
takes), streaming and non-streaming — the streaming case matters most,
since that is essentially all real agent traffic.
Expand Down Expand Up @@ -845,7 +848,7 @@ def _record(self, mcd, response_obj):
# /v1/messages route (#3624): stock LiteLLM rewrites the
# caller's ``thinking`` block into a bucketed
# ``reasoning_effort``, but that bucket is a cap below the
# model default, so egg-litellm's patch 9 gates the
# model default, so egg-litellm's patch 6 gates the
# synthesis off by default. A missing key here means the
# request ran at the model's own reasoning depth, not that
# the field failed to record.
Expand Down
6 changes: 3 additions & 3 deletions config/litellm/drop_params_visibility.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

``drop_params`` exists so an unsupported parameter does not fail the whole
request, and that tradeoff is right. But dropping a parameter *changes
generation behaviour*, and in stock LiteLLM 1.86.2 it happens with no signal
generation behaviour*, and in stock LiteLLM it happens with no signal
at all: the branch that pops them is a bare loop with no logging. A
``reasoning_effort``, ``temperature`` or penalty set in a proxy config simply
never reaches the provider, and nothing in the logs or the response says so.
Expand All @@ -15,14 +15,14 @@
to notice (jwbron/egg#3620, #3624). One log line would have made it a
five-minute question.

Patch 7 fixes the OpenRouter false-negative specifically; this covers the rest.
Patch 4 fixes the OpenRouter false-negative specifically; this covers the rest.
A drop can still be *correct* and worth knowing about: ``poolside/laguna-s-2.1``
genuinely does not accept ``reasoning_effort``, so the knob is dropped on
purpose, and without this the operator has no way to learn why their config
line does nothing.

Mirrors jwbron/litellm#7 (merged into the fork the host proxy runs). The
cluster image pins stock 1.86.2, which predates it, hence this patch.
cluster image pins a stock release, which does not carry it, hence this patch.
"""

# Warn-once bookkeeping, keyed by (provider, model, sorted dropped param
Expand Down
4 changes: 2 additions & 2 deletions config/litellm/openrouter_capabilities.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
cost: ``supports_reasoning: true`` alone makes stock
``get_supported_openai_params`` admit ``thinking``, which Patch 2's notes
explain would forward an Anthropic-shaped block verbatim to a provider that
expects ``reasoning``. Patch 7 remains the only path by which a parameter
expects ``reasoning``. Patch 4 remains the only path by which a parameter
becomes admissible, and it admits exactly ``reasoning_effort``.
* **Tiered rate cards are translated only where LiteLLM can hold them.**
OpenRouter expresses a long-context surcharge as ``pricing.overrides`` — a
Expand Down Expand Up @@ -79,7 +79,7 @@
both, since one fetch serves both.
* ``LITELLM_OPENROUTER_PRICING=0`` disables only the pricing half, leaving the
parameter lookup running. For an operator who wants LiteLLM's bundled map to
be the sole authority on cost while keeping Patch 7's parameter fix.
be the sole authority on cost while keeping Patch 4's parameter fix.
* ``LITELLM_OPENROUTER_CAPABILITY_TTL`` seconds between refreshes
(default 3600). ``0`` disables caching and re-fetches on every lookup —
a debugging aid, not a production setting.
Expand Down
Loading
Loading