litellm: read OpenRouter capabilities live, and make drop_params visible

config/litellm-models.template.yaml

@@ -98,8 +98,14 @@ data:
             provider:
               order: [Alibaba]
               allow_fallbacks: false
-            # Closest knob to Opus xhigh — uncomment to opt into the
-            # deepest reasoning OpenRouter exposes for this model:
+            # Reasoning depth. MEASURE BEFORE UNCOMMENTING — on the models
+            # measured in #3624 this knob is a CAP BELOW the model's own
+            # default, not a ceiling above it. Mean reasoning tokens at
+            # max_tokens 16000, n=4: kimi-k3 3130 with no parameter vs 86 via
+            # `extra_body.reasoning.effort: high` (distributions do not
+            # overlap); glm-5.2 1689 vs 1516. Sending nothing is what gives
+            # these models full depth. Set it only for a model you have
+            # measured to reason MORE when explicitly asked:
             # reasoning:
             #   effort: "high"
       # Paired `[1m]` alias — absorbs Claude Code startup-probe
@@ -124,6 +130,8 @@ data:
             provider:
               order: [Alibaba]
               allow_fallbacks: false
+            # See the depth-vs-default measurement on the bare row above
+            # before uncommenting this.
             # reasoning:
             #   effort: "high"
 

config/litellm/Dockerfile

@@ -3,8 +3,10 @@
 # egg routes non-Claude agents (the cq-6 OpenRouter/Qwen pilot, #2799)
 # through this proxy: Claude Code -> gateway -> LiteLLM -> OpenRouter. The
 # stock image's Anthropic->OpenAI translation drops prompt-cache hits on
-# Qwen/DeepSeek and mis-streams reasoning models. patch_litellm_cache.py
-# closes the five gaps at build time (see that script for the diagnosis);
+# Qwen/DeepSeek, mis-streams reasoning models, silently drops params it does
+# not recognise, and manufactures a reasoning ceiling from the caller's
+# thinking budget. patch_litellm_cache.py closes the nine gaps at build time
+# (see that script for the diagnosis);
 # cost_callback.py surfaces the resulting cost/cache stats into the pod log
 # stream. Both mirror the host-side dotfiles setup that took Qwen cache hit
 # rate from 0% to ~99.99%.
@@ -21,6 +23,14 @@ FROM ghcr.io/berriai/litellm:v1.86.2
 USER root
 
 COPY patch_litellm_cache.py /egg/patch_litellm_cache.py
+# Staged, not baked directly into the patch script: these are whole modules the
+# script installs into every litellm tree (see NEW_MODULES). Keeping them real
+# files means they stay lintable and testable in the egg repo rather than
+# living as string literals — which is why none of them imports litellm at
+# module scope (tests/config/ imports all three directly).
+COPY openrouter_capabilities.py /egg/openrouter_capabilities.py
+COPY drop_params_visibility.py /egg/drop_params_visibility.py
+COPY anthropic_thinking_policy.py /egg/anthropic_thinking_policy.py
 RUN python3 /egg/patch_litellm_cache.py
 
 # Custom cost/cache logger, registered as `cost_callback.cost_logger` under

config/litellm/anthropic_thinking_policy.py

@@ -0,0 +1,125 @@
+"""Whether LiteLLM may synthesize ``reasoning_effort`` from ``thinking``.
+
+egg's primary route is ``/v1/messages``: Claude Code -> egg-gateway -> LiteLLM
+-> OpenRouter, carrying an Anthropic-shaped body with
+``thinking: {"type": "enabled", "budget_tokens": N}``. LiteLLM's Anthropic
+adapter (``_translate_thinking_to_openai``) turns that into an OpenAI body. For
+a Claude model it forwards ``thinking`` unchanged. For anything else —
+``is_anthropic_claude_model`` is a substring test for ``anthropic``/``claude``,
+so every OpenRouter slug egg routes falls here — it *replaces* the block with a
+bucketed ``reasoning_effort``: ``>=10000 -> "high"``, ``>=5000 -> "medium"``,
+``>=2000 -> "low"``. Nothing in ``litellm-models.yaml`` is involved; the value
+is manufactured per request from the caller's thinking budget.
+
+That synthesized value is not a floor, it is a **cap below the model default**.
+Measured directly against OpenRouter (``max_tokens: 16000``, n=4, mean
+reasoning tokens):
+
+===================================  ============  ===================
+Model                                no parameter  ``effort: "high"``
+===================================  ============  ===================
+``moonshotai/kimi-k3``               3130          340
+``z-ai/glm-5.2``                     1689          1090
+===================================  ============  ===================
+
+On kimi-k3 the distributions do not overlap. So sending the adapter's bucket
+costs roughly 9x the reasoning depth the model would have produced on its own.
+
+Historically this never mattered: the model-cost map did not carry these slugs,
+``OpenrouterConfig`` advertised no reasoning knobs, and ``drop_params`` silently
+discarded the synthesized param — which is precisely why these models have been
+running at full depth. Patch 7 makes the OpenRouter param gate accurate, which
+is right for an *operator-configured* ``reasoning_effort`` and wrong for this
+adapter-manufactured one: it would turn a knob nobody set into the effective
+setting, with no config file mentioning it and nothing in the logs (Patch 8
+only fires on drops, and this param would no longer be dropped).
+
+So Patch 9 gates the synthesis, defaulting it off. ``thinking`` stays out of
+the OpenAI body for non-Claude models and only an explicitly configured
+``reasoning_effort`` reaches the wire — exactly the property Patch 7 exists to
+restore, without the adapter's bucket riding along.
+
+The gate covers the *derived* value only. On an adaptive request
+(``thinking: {"type": "adaptive"}`` plus ``output_config: {"effort": ...}``)
+the caller states an effort outright; that is an instruction rather than a
+manufactured ceiling, and it still reaches the provider with this policy off.
+A ``thinking.summary`` request is suppressed along with the derived effort,
+because stock carries the summary only as a field of the ``reasoning_effort``
+dict — honouring it would mean sending the ceiling. There is no wire shape for
+"summary, no effort".
+
+Set ``LITELLM_ANTHROPIC_THINKING_TO_REASONING_EFFORT=1`` to restore stock
+behaviour (e.g. for a provider whose models do not reason unless asked, or
+after measuring the ``/v1/messages`` path for a specific model). A value that is
+neither a recognised on nor off spelling warns once and leaves the policy at its
+default, rather than being read as off — off is also the default, so an operator
+who typed ``=enabled`` would otherwise have no way to tell "ignored" apart from
+"working as configured", on the highest-impact knob in this changeset.
+"""
+
+import os
+
+ENV_VAR = "LITELLM_ANTHROPIC_THINKING_TO_REASONING_EFFORT"
+
+_TRUTHY = ("1", "true", "yes", "on")
+_FALSY = ("0", "false", "no", "off", "")
+
+# Values already complained about. Bounded by construction: the environment does
+# not change mid-process, so this holds at most one entry. Needed because this
+# is read once per translated request, and an unconditional warning would be one
+# WARNING line per request forever.
+_WARNED_VALUES: set[str] = set()
+
+
+def _log(level: str, message: str, *args: object) -> bool:
+    """Log via litellm's ``verbose_logger``, deferring the import.
+
+    Kept out of module scope so this file stays importable — and therefore unit
+    testable — where litellm is not installed. Never raises: a diagnostic must
+    not be able to break a request.
+
+    Returns whether the call completed without raising, not whether a record
+    reached a handler; see the same note in ``openrouter_capabilities._log``.
+    """
+    try:
+        from litellm._logging import verbose_logger
+
+        getattr(verbose_logger, level)(message, *args)
+        return True
+    except Exception:  # noqa: BLE001 - diagnostics must never break a request
+        return False
+
+
+def should_synthesize_reasoning_effort() -> bool:
+    """True when the adapter may derive ``reasoning_effort`` from ``thinking``.
+
+    Defaults to False: on every model egg routes, the derived value measurably
+    reduces reasoning depth relative to sending nothing at all.
+    """
+    raw = os.getenv(ENV_VAR)
+    if raw is None:
+        return False
+    value = raw.strip().lower()
+    if value in _TRUTHY:
+        return True
+    if value not in _FALSY and value not in _WARNED_VALUES:
+        # Recorded only once the emit did not raise, for the reason given in
+        # ``openrouter_capabilities._warn_env_once``: ``_log`` swallows its own
+        # failure, so recording first would let a logger that is not yet in
+        # place on the first request suppress the warning permanently.
+        if _log(
+            "warning",
+            "%s=%r is neither an on (%s) nor an off (%s) spelling; leaving "
+            "thinking -> reasoning_effort synthesis disabled, which is also the "
+            "default — set %s=1 if you meant to enable it.",
+            ENV_VAR,
+            raw,
+            ", ".join(_TRUTHY),
+            ", ".join(v for v in _FALSY if v),
+            ENV_VAR,
+        ):
+            _WARNED_VALUES.add(value)
+    return False
+
+
+__all__ = ["ENV_VAR", "should_synthesize_reasoning_effort"]

config/litellm/cost_callback.py

@@ -814,9 +814,18 @@ def _record(self, mcd, response_obj):
                     # (#3599). Top-level, not nested under ``call``, so an
                     # incident query can filter on it the same way it filters
                     # on model/role. Per line rather than once per session
-                    # because it is NOT session-stable: LiteLLM rewrites
-                    # ``thinking`` into a ``reasoning_effort`` bucket, so the
-                    # effective effort tracks the per-turn thinking budget.
+                    # because it is NOT session-stable: these are per-request
+                    # values, and a config change or an overlay edit takes
+                    # effect mid-session.
+                    #
+                    # Note ``reasoning_effort`` is normally ABSENT on egg's
+                    # /v1/messages route (#3624): stock LiteLLM rewrites the
+                    # caller's ``thinking`` block into a bucketed
+                    # ``reasoning_effort``, but that bucket is a cap below the
+                    # model default, so egg-litellm's patch 9 gates the
+                    # synthesis off by default. A missing key here means the
+                    # request ran at the model's own reasoning depth, not that
+                    # the field failed to record.
                     "request_params": request_params,
                     "call": {
                         "cost": cost,

config/litellm/drop_params_visibility.py

@@ -0,0 +1,124 @@
+"""Make ``drop_params`` say what it dropped.
+
+``drop_params`` exists so an unsupported parameter does not fail the whole
+request, and that tradeoff is right. But dropping a parameter *changes
+generation behaviour*, and in stock LiteLLM 1.86.2 it happens with no signal
+at all: the branch that pops them is a bare loop with no logging. A
+``reasoning_effort``, ``temperature`` or penalty set in a proxy config simply
+never reaches the provider, and nothing in the logs or the response says so.
+The config and the wire disagree, silently and indefinitely.
+
+That is not hypothetical here. Every OpenRouter slug egg routes is absent from
+LiteLLM's bundled model-cost map, so ``OpenrouterConfig`` advertised no
+reasoning knobs and every ``reasoning_effort: high`` in the operator overlay
+was discarded before the request body was built. It took a full investigation
+to notice (jwbron/egg#3620, #3624). One log line would have made it a
+five-minute question.
+
+Patch 7 fixes the OpenRouter false-negative specifically; this covers the rest.
+A drop can still be *correct* and worth knowing about: ``poolside/laguna-s-2.1``
+genuinely does not accept ``reasoning_effort``, so the knob is dropped on
+purpose, and without this the operator has no way to learn why their config
+line does nothing.
+
+Mirrors jwbron/litellm#7 (merged into the fork the host proxy runs). The
+cluster image pins stock 1.86.2, which predates it, hence this patch.
+"""
+
+# Warn-once bookkeeping, keyed by (provider, model, sorted dropped param
+# names) so a route that drops the same params on every request warns once
+# rather than once per call. Bounded so a long-lived proxy serving many models
+# cannot grow it without limit; on overflow the set is CLEARED rather than
+# frozen, because a frozen full set stops deduplicating and every subsequent
+# request warns again forever. Clearing costs one extra warning per key per
+# cycle and keeps both memory and log volume bounded.
+_MAX_WARNINGS = 1000
+_SEEN: set[tuple[str, str, tuple[str, ...]]] = set()
+
+
+def _log_warning(message: str, *args: object) -> bool:
+    """Log via litellm's ``verbose_logger``, deferring the import.
+
+    Kept out of module scope so this file stays importable where litellm is
+    not installed — which is what makes it unit testable in the egg repo, the
+    stated reason (``config/litellm/Dockerfile``) for keeping it a real file
+    rather than a string literal in the patch script.
+
+    Returns whether the call completed without raising — not whether a line
+    reached a handler, since a logger filtering the level away also returns
+    normally. That is enough for the warn-once bookkeeping below, whose failure
+    mode is an import or emit that *raised*: this module exists precisely so a
+    drop is not silent, and recording the dedup key ahead of a failed emit
+    would make that route silent for the life of the process.
+    """
+    try:
+        from litellm._logging import verbose_logger
+
+        verbose_logger.warning(message, *args)
+    except Exception:  # noqa: BLE001 - diagnostics must never break a request
+        return False
+    return True
+
+
+def warn_dropped_params(
+    unsupported_params: dict,
+    model: str | None,
+    custom_llm_provider: str | None,
+) -> None:
+    """Log once per (provider, model, param-set) when params are discarded.
+
+    Warns rather than debugs because the caller asked for something and did not
+    get it; at debug level it would be invisible in exactly the situation it
+    exists for. Never raises: a diagnostic must not be able to break a request.
+    """
+    try:
+        if not unsupported_params:
+            return
+        dropped = tuple(sorted(unsupported_params.keys()))
+        key = (custom_llm_provider or "", model or "", dropped)
+        if key in _SEEN:
+            return
+        # States what is known, and prescribes only under a condition the
+        # operator can check. The param most likely to be dropped on this
+        # deployment is ``reasoning_effort``, and it is frequently NOT in any
+        # config file: litellm's own Anthropic adapter synthesises it from the
+        # caller's `thinking` block on the /v1/messages route. An unconditional
+        # "edit config.yaml" would send that operator looking for a line that
+        # does not exist, and an unconditional "force it through
+        # `allowed_openai_params`" would turn a correct drop on a genuinely
+        # non-reasoning model into a provider-side error. So the remedy is
+        # offered gated on "if they came from this model's litellm_params",
+        # with the synthesized case named alongside it — the operator who has
+        # such a line gets the fix, and the one who does not is told why the
+        # drop is expected instead of being sent editing.
+        emitted = _log_warning(
+            "litellm.drop_params: dropped %s for model=%s provider=%s — the "
+            "provider does not advertise support for them, so they did not "
+            "reach it and whatever behaviour they were meant to control is "
+            "unchanged. If they came from this model's litellm_params in "
+            "config.yaml, remove them or override with `allowed_openai_params: "
+            "%s`. If they were synthesized from the request (e.g. "
+            "reasoning_effort derived from an Anthropic `thinking` block), the "
+            "drop is expected and the model ran at its own default.",
+            list(dropped),
+            model,
+            custom_llm_provider,
+            list(dropped),
+        )
+        # Recorded only once the emit did not raise. ``_log_warning`` swallows
+        # its own failure so a diagnostic cannot fail a request, and recording
+        # first would mean one failure on the *first* call — litellm's logger
+        # not yet in place, say — suppresses this route's warning forever,
+        # because every later call would find the key already there. The cost of
+        # the other ordering is one warning attempt per request until an emit
+        # succeeds, which is the right way round for a module whose whole job is
+        # to make a silent drop audible.
+        if emitted:
+            if len(_SEEN) >= _MAX_WARNINGS:
+                _SEEN.clear()
+            _SEEN.add(key)
+    except Exception:  # noqa: BLE001 - diagnostics must never break a request
+        pass
+
+
+__all__ = ["warn_dropped_params"]