Skip to content

openrouter: send prior-turn assistant reasoning on the request path - #11

Open
jwbron wants to merge 1 commit into
mainfrom
openrouter-reasoning-roundtrip
Open

openrouter: send prior-turn assistant reasoning on the request path#11
jwbron wants to merge 1 commit into
mainfrom
openrouter-reasoning-roundtrip

Conversation

@jwbron

@jwbron jwbron commented Jul 29, 2026

Copy link
Copy Markdown
Owner

The gap

On the /v1/messages route (Anthropic-format clients such as Claude Code, proxied at an
OpenRouter model), assistant reasoning from prior turns is never sent back to the model.

  1. The Anthropic adapter converts incoming thinking content blocks into
    assistant_message["thinking_blocks"]
    (llms/anthropic/experimental_pass_through/adapters/transformation.py, in
    _translate_anthropic_messages_to_openai).
  2. llms/openrouter/chat/transformation.py has no request-path consumer for that field. Its
    only reasoning line is response-path (reasoning -> reasoning_content on streaming deltas).
  3. reasoning_details appears nowhere under llms/ or litellm_core_utils/.

OpenAIGPTConfig.transform_request drops messages straight into the request body, so
thinking_blocks is transmitted to the provider as a field no one reads, and every historical
assistant turn arrives with its reasoning missing.

This is not egg-specific or fork-specific: it is present in stock BerriAI/litellm and predates
every change on this fork. PR #8 touched only get_supported_openai_params, which acts on
optional_params and cannot reach a message field.

Why it matters

For a model whose chat template re-renders prior thinking, this is a malformed history rather than
a lost optimisation. Poolside Laguna renders

'<think>' + message.reasoning|message.reasoning_content + '</think>'

for every previous assistant turn, so each one arrives as a literal empty <think></think>.
Poolside's model card warns that this degrades follow-up behaviour. Any OpenRouter reasoning model
with a template of that shape is affected.

The change

Map assistant thinking_blocks onto reasoning_content in OpenrouterConfig.transform_request,
and remove thinking_blocks so no unknown field is transmitted.

OpenRouter accepts reasoning, reasoning_content and reasoning_details interchangeably on an
assistant message and documents this for exactly this multi-turn tool-calling case; Poolside's
template reads message.reasoning / message.reasoning_content. The plain string form is used
because reasoning_details exists to carry encrypted or summarised blocks and the Anthropic
adapter produces neither.

Behaviour, all covered by tests:

  • multiple thinking blocks concatenate in order;
  • redacted_thinking blocks carry opaque data, not text, and contribute nothing;
  • a whitespace-only (or empty) result emits no field at all, rather than re-creating the empty
    <think></think> this exists to remove;
  • Anthropic signature values are not forwarded (they mean nothing to OpenRouter);
  • assistant messages only; user, system and tool messages pass through untouched;
  • the input list is not mutated (touched messages are shallow-copied);
  • fail soft: a block that cannot be parsed leaves its message exactly as it arrived, so the worst
    case is the current behaviour;
  • idempotent: applying twice is a no-op.

Direction

Worth stating explicitly, because the neighbouring code is easy to confuse with this: PR #4 is
response-path (provider -> client, how reasoning is streamed back). This is request-path
(client -> provider). They are adjacent, not the same thing.

Verification

Beyond the unit tests, verified end to end with a real Claude Code /v1/messages body (captured
from a live two-turn tool-calling exchange, assistant message carrying
['text', 'thinking', 'tool_use']) replayed through a LiteLLM proxy with the provider endpoint
captured:

outgoing assistant message
before ['role', 'thinking_blocks', 'tool_calls'], no reasoning field
after ['reasoning_content', 'role', 'tool_calls'], no thinking_blocks

On a live two-turn exchange against deepseek/deepseek-v4-pro, the reasoning restored to the
follow-up turn measured +28 tokens on an assistant turn that was 118 tokens without it (1.24x);
the magnitude scales with how much the model reasons.

tests/test_litellm/llms/openrouter/ passes in full (154 tests), including the 27 added here.

A separate upstream PR to BerriAI/litellm is a follow-up, not part of this.

The Anthropic adapter converts each assistant `thinking` content block into
an entry on `assistant_message["thinking_blocks"]`, but nothing on the
OpenRouter request path consumes that field: this module names reasoning
only on the response side (`reasoning` -> `reasoning_content` on streaming
deltas), and `reasoning_details` appears nowhere under `llms/` or
`litellm_core_utils/`. `OpenAIGPTConfig.transform_request` puts `messages`
straight into the request body, so `thinking_blocks` is transmitted as a
field no provider reads and every historical assistant turn reaches the
model with its reasoning missing.

For a model whose chat template re-renders prior thinking that is a
malformed history rather than a lost optimisation. Poolside Laguna renders
`'<think>' + message.reasoning|message.reasoning_content + '</think>'` per
previous assistant turn, so each one arrives as a literal empty
`<think></think>`; Poolside's model card warns this degrades follow-up
behaviour.

Map the blocks onto `reasoning_content` in `transform_request` and drop
`thinking_blocks` so no unknown field goes out. OpenRouter accepts
`reasoning`, `reasoning_content` and `reasoning_details` interchangeably
and documents this for exactly this multi-turn tool-calling case; the plain
string form is used because `reasoning_details` exists to carry encrypted or
summarised blocks and the adapter produces neither. Anthropic `signature`
values are not forwarded. `redacted_thinking` blocks carry no plaintext and
contribute nothing; a whitespace-only result emits no field at all rather
than re-creating the empty `<think></think>`. Assistant messages only, input
not mutated, and a block that cannot be parsed leaves its message untouched
rather than failing the request.

Verified against a real Claude Code /v1/messages body routed through the
proxy: before, the outgoing payload carried `thinking_blocks` and no
reasoning field; after, it carries `reasoning_content` and no
`thinking_blocks`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant