fix(together_ai): strip internal thinking fields from outbound messages, keep reasoning_content

[mateo-berri]

TLDR

Problem this solves:

• Outbound Together requests leak litellm-internal thinking_blocks and provider_specific_fields
• Nothing pins chat_template_kwargs passthrough or preserved-thinking replay

How it solves it:

• Strips internal fields from assistant messages before sending
• Keeps reasoning_content: Together consumes it for preserved thinking
• Regression tests pin passthrough, stripping, and reasoning replay

User Flow

Before: a developer chats with Together's GLM-5.2 in preserved-thinking mode through the proxy, and the request forwarded to Together carries litellm-internal junk fields

1. Their Anthropic-SDK app sends POST http://localhost:4000/v1/messages with a prior assistant turn (thinking + text blocks) and chat_template_kwargs: {"clear_thinking": false}
2. The reply is correct: the model recalls a secret that only existed in the replayed thinking
3. But the proxy's debug log shows the request sent to https://api.together.ai/v1/chat/completions carrying thinking_blocks on the assistant message (plus provider_specific_fields when an OpenAI-SDK client replays a LiteLLM response verbatim), undocumented fields Together happens to ignore today

After: the same chat works and the request Together receives is clean

1. Their Anthropic-SDK app sends the same POST http://localhost:4000/v1/messages with the replayed thinking and chat_template_kwargs: {"clear_thinking": false}
2. The reply is correct: the model still recalls the secret, proving reasoning_content still reaches Together
3. The proxy's debug log shows the assistant message sent to https://api.together.ai/v1/chat/completions with only content and reasoning_content, no litellm-internal fields

Relevant issues

Linear ticket

Resolves LIT-5967

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Both legs booted a live proxy with 2 uvicorn workers (no DB) hitting the real Together API, with qwen-hybrid = together_ai/Qwen/Qwen3.5-9B and glm-52 = together_ai/zai-org/GLM-5.2. Same five cases per leg; only the checked-out commit differs. The one commit since the proven tip, 8a61b28, only annotates test helpers, so it cannot change behavior and the proof stands

Before (merge base e4ff44f, port 39605)

Case 1: /v1/chat/completions kwargs passthrough (non-streaming)

curl -s http://localhost:39605/v1/chat/completions -H 'Authorization: Bearer sk-lit5967-qa' -H 'Content-Type: application/json' \
  -d '{"model":"qwen-hybrid","messages":[{"role":"user","content":"What is 17*23? Answer with just the number."}],"chat_template_kwargs":{"thinking":false}}'

With kwargs: content "391", message keys ['content','role'], no reasoning_content. Control (same body minus chat_template_kwargs): content "391" with reasoning_content present. Passthrough already worked here

Case 2: same, streaming

curl -s -N http://localhost:39605/v1/chat/completions -H 'Authorization: Bearer sk-lit5967-qa' -H 'Content-Type: application/json' \
  -d '{"model":"qwen-hybrid","messages":[{"role":"user","content":"What is 17*23? Answer with just the number."}],"chat_template_kwargs":{"thinking":false},"stream":true,"stream_options":{"include_usage":true}}'

With kwargs: 0 delta chunks carrying reasoning_content, answer "401" (thinking-off model artifact). Control: 359 reasoning deltas, answer "391"

Case 3: /v1/responses kwargs passthrough

curl -s http://localhost:39605/v1/responses -H 'Authorization: Bearer sk-lit5967-qa' -H 'Content-Type: application/json' \
  -d '{"model":"qwen-hybrid","input":"What is 17*23? Answer with just the number.","chat_template_kwargs":{"thinking":false}}'

With kwargs: output item types ['message']. Control: ['reasoning','message']

Case 4: /v1/messages preserved thinking (GLM-5.2)

curl -s http://localhost:39605/v1/messages -H 'Authorization: Bearer sk-lit5967-qa' -H 'Content-Type: application/json' \
  -d '{"model":"glm-52","max_tokens":2048,"messages":[{"role":"user","content":"Pick a secret two-digit number. Reply with ONLY the sum of its digits, nothing else."}]}'
# turn 2 replays the full assistant content array (thinking + text blocks) verbatim, adds the question and chat_template_kwargs
curl -s http://localhost:39605/v1/messages -H 'Authorization: Bearer sk-lit5967-qa' -H 'Content-Type: application/json' -d @case4_turn2_req.json

Turn 1 thinking picked secret 42, text "6". Turn 2 replied "42": the replayed thinking reached the model. But the proxy debug log's "POST Request Sent from LiteLLM" to https://api.together.ai/v1/chat/completions shows the outbound assistant message keys as ['content','reasoning_content','role','thinking_blocks']: thinking_blocks leaked

Case 5: /v1/chat/completions verbatim-history replay (GLM-5.2)

Turn 1 (same secret prompt) answered "11" with reasoning settling on 47. Turn 2 replayed the assistant message with reasoning_content, thinking_blocks, and provider_specific_fields exactly as a LiteLLM response carries them, plus "chat_template_kwargs":{"clear_thinking":false}:

curl -s http://localhost:39605/v1/chat/completions -H 'Authorization: Bearer sk-lit5967-qa' -H 'Content-Type: application/json' -d @case5_turn2_req.json

Reply "47", the correct secret. Outbound assistant message keys: ['content','provider_specific_fields','reasoning_content','role','thinking_blocks'] with provider_specific_fields: {'thinking_blocks': []}: all three litellm-internal keys leaked to Together

After (tip 1ba66b1, port 37398)

Case 1: /v1/chat/completions kwargs passthrough (non-streaming)

Same command on port 37398. With kwargs: content "391", keys ['content','role']. Control: "391" with reasoning_content. Unchanged

Case 2: same, streaming

With kwargs: 0 reasoning deltas, content "391". Control: 346 reasoning deltas. Unchanged

Case 3: /v1/responses kwargs passthrough

With kwargs: output item types ['message'], text "391". Control: ['reasoning','message']. Unchanged

Case 4: /v1/messages preserved thinking (GLM-5.2)

Turn 1 thinking picked 42, text "6". Turn 2 (same verbatim replay + {"clear_thinking": false}) replied "42", and the outbound body now shows the assistant message as

{'role': 'assistant', 'content': '6', 'reasoning_content': '1.  **Analyze the Request:** ... Let\'s pick 42. ...'}

thinking_blocks absent, provider_specific_fields absent, reasoning_content present and still consumed (the recall proves it)

Case 5: /v1/chat/completions verbatim-history replay (GLM-5.2)

Turn 1 picked 73, answered "10". Turn 2 with all three internal fields replayed replied "73", and the outbound assistant message carries only role, content, and reasoning_content alongside 'chat_template_kwargs': {'clear_thinking': False} in the body

Closing notes from the legs:

• Case 4 before-leg leak lacked provider_specific_fields; this PR leaves that path equally clean
• Qwen answers 401 with thinking disabled; model artifact, both legs' controls got 391
• Control streams carry hundreds of reasoning deltas; unrelated to this PR
• Functional passthrough already worked at the merge base; this PR fixes the leak and pins both behaviors

Type

🐛 Bug Fix
✅ Test

Caveats (if any)

Low

• Together silently ignores unknown chat_template_kwargs keys server-side
• LiteLLM passes them through for Together to validate, like tools in fix(together_ai): pass tools through for models missing from the registry #38265

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

• 1ba66b1 passes /live-pr-risk

[greptile-apps]

Greptile Summary

The PR sanitizes Together AI assistant-message payloads while preserving reasoning content used for replayed thinking.

• Removes LiteLLM-internal thinking and provider-specific fields before the provider request is built.
• Adds synchronous, asynchronous, and HTTP request-level regression coverage for preserved reasoning and parameter passthrough.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
litellm/llms/together_ai/chat/transformation.py	Adds provider-boundary sanitization for internal assistant fields while retaining reasoning_content across synchronous and asynchronous transformations.
tests/test_litellm/llms/together_ai/chat/test_together_ai_chat_transformation.py	Adds focused transformation and request-capture coverage for field stripping, reasoning replay, and chat-template passthrough.

_{Reviews (2): Last reviewed commit: "test(together_ai): annotate preserved-th..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 8a61b28. Configure here.}

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_together_chat_template_kwargs (8a61b28) with litellm_internal_staging (bb27bfd)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (bb22742) during the generation of this report, so bb27bfd was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩