test(e2e): cover Together AI reasoning, tool calls, template kwargs, and cost through a live proxy

[mateo-berri]

TLDR

Problem this solves:

• Together had no live e2e coverage, so a dropped tool call or stripped reasoning shipped silently
• The overhaul's new behaviors (reasoning_content, chat_template_kwargs, cost) were only unit-tested

How it solves it:

• 11 live tests on /chat/completions and /v1/messages against the real Together API
• The model is picked from the proxy's own cost map by capability flags, so the matrix does not rot
• Cost header and spend row are checked against the registry price, cached tokens included

User Flow

Before: a developer running an agent on a Together reasoning model through the gateway learns from their own users that tool calls stopped coming back, because nothing in CI exercised Together end to end

1. A change that breaks the Together request transform merges to staging
2. The scheduled e2e suite runs, collects zero Together tests, and stays green
3. The developer sends POST https://litellm-domain/v1/chat/completions with their Together model, a get_weather tool, and "What is the weather in Paris? Use the tool."
4. The 200 response has "tool_calls": null and the text content is a raw <|start|>assistant<|channel|>commentary... dump, so their agent loop stalls
5. They send POST https://litellm-domain/v1/messages with the same tool and get back only a thinking block, no tool_use block
6. They replay a prior assistant turn that carries reasoning_content and the model answers "I don't have that information."
7. They find out from a user report and file an issue

After: the same regression fails the scheduled e2e suite before it reaches the developer, and every request keeps behaving the way it did before the change

1. A change that breaks the Together request transform merges to staging
2. The scheduled e2e suite runs and fails test_tool_call_is_returned, test_tool_call_is_streamed, test_replayed_reasoning_content_reaches_together, and TestTogetherMessages::test_tool_use_block_is_returned, each naming what Together dropped
3. The developer sends POST https://litellm-domain/v1/chat/completions with their Together model, a get_weather tool, and "What is the weather in Paris? Use the tool."
4. The 200 response carries one tool_calls entry with an id, function.name: "get_weather", and arguments: {"location": "Paris"}, plus the model's reasoning_content
5. POST https://litellm-domain/v1/messages returns a tool_use block named get_weather with an id their tool_result can answer
6. Replaying an assistant turn that carries reasoning_content makes the model answer from it
7. The x-litellm-response-cost header on each response matches the registry price, and https://litellm-domain/ui/?page=logs shows that same spend

Relevant issues

None

Linear ticket

Resolves LIT-5971

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Shared setup, same for both legs. The proxy runs from the worktree with 2 uvicorn workers, TOGETHER_API_KEY and LITELLM_MASTER_KEY loaded from .env, and LITELLM_LOCAL_MODEL_COST_MAP=True so it prices with this branch's registry, and every call below hits api.together.ai for real

model_list:
  - model_name: together-placeholder
    litellm_params:
      model: together_ai/openai/gpt-oss-120b
      api_key: os.environ/TOGETHER_API_KEY

general_settings:
  store_model_in_db: true

set -a && source .env && set +a
LITELLM_LOCAL_MODEL_COST_MAP=True uv run python litellm/proxy/proxy_cli.py --config together_e2e_config.yaml --port 35102 --num_workers 2 --use_v2_migration_resolver
export LITELLM_PROXY_URL=http://localhost:35102

The broken-capability case applies this two-line change to litellm/llms/together_ai/chat/transformation.py before booting the proxy (strips reasoning_content from replayed assistant turns and drops tools from the request), then reverts it

-LITELLM_INTERNAL_ASSISTANT_FIELDS: Final = frozenset({"thinking_blocks", "provider_specific_fields"})
+LITELLM_INTERNAL_ASSISTANT_FIELDS: Final = frozenset({"thinking_blocks", "provider_specific_fields", "reasoning_content"})
         for param in _tool_params_to_drop(mapped_openai_params, model, drop_params):
             mapped_openai_params.pop(param)
+        mapped_openai_params.pop("tools", None)

Before (d0c527f)

The Together e2e matrix

1. uv run pytest tests/e2e/llm_translation -k together --collect-only -q -p no:cacheprovider
2. Output: no tests collected (156 deselected) in 0.13s, and ls tests/e2e/llm_translation | grep -i together prints nothing

A broken Together transform

1. Apply the two-line change above, boot the proxy, and run uv run pytest tests/e2e/llm_translation -k together -v -p no:cacheprovider
2. Output: no tests collected (156 deselected), so the suite stays green with tool calls and replayed reasoning gone

After (893482d)

The Together e2e matrix

1. uv run pytest tests/e2e/llm_translation/test_together_ai_e2e.py -v -p no:cacheprovider
2. Output:

collected 11 items
tests/e2e/llm_translation/test_together_ai_e2e.py::TestTogetherChatCompletions::test_reasoning_surfaces_as_reasoning_content PASSED [  9%]
tests/e2e/llm_translation/test_together_ai_e2e.py::TestTogetherChatCompletions::test_reasoning_streams_as_reasoning_content_deltas PASSED [ 18%]
tests/e2e/llm_translation/test_together_ai_e2e.py::TestTogetherChatCompletions::test_tool_call_is_returned PASSED [ 27%]
tests/e2e/llm_translation/test_together_ai_e2e.py::TestTogetherChatCompletions::test_tool_call_is_streamed PASSED [ 36%]
tests/e2e/llm_translation/test_together_ai_e2e.py::TestTogetherChatCompletions::test_tool_result_round_trip PASSED [ 45%]
tests/e2e/llm_translation/test_together_ai_e2e.py::TestTogetherChatCompletions::test_chat_template_kwargs_reach_together PASSED [ 54%]
tests/e2e/llm_translation/test_together_ai_e2e.py::TestTogetherChatCompletions::test_replayed_reasoning_content_reaches_together PASSED [ 63%]
tests/e2e/llm_translation/test_together_ai_e2e.py::TestTogetherChatCompletions::test_cost_header_and_spend_row_match_the_registry_price PASSED [ 72%]
tests/e2e/llm_translation/test_together_ai_e2e.py::TestTogetherMessages::test_tool_use_block_is_returned PASSED [ 81%]
tests/e2e/llm_translation/test_together_ai_e2e.py::TestTogetherMessages::test_tool_result_round_trip PASSED [ 90%]
tests/e2e/llm_translation/test_together_ai_e2e.py::TestTogetherMessages::test_streams_text_deltas PASSED [100%]
======================== 11 passed in 192.62s (0:03:12) ========================

3. cd tests/e2e && PYTHONPATH=. uv run python -m coverage_registry.collector --strict exits 0 with the 11 new together_ai cells covered

A broken Together transform

1. Apply the two-line change above, boot the proxy, and run uv run pytest tests/e2e/llm_translation/test_together_ai_e2e.py -k "tool_call_is_returned or tool_call_is_streamed or replayed_reasoning or tool_use_block_is_returned" -v -p no:cacheprovider (run at 55e3a97; 893482d only drops two docstrings on top of it)
2. Output:

collected 11 items / 7 deselected / 4 selected
tests/e2e/llm_translation/test_together_ai_e2e.py::TestTogetherChatCompletions::test_tool_call_is_returned FAILED [ 25%]
tests/e2e/llm_translation/test_together_ai_e2e.py::TestTogetherChatCompletions::test_tool_call_is_streamed FAILED [ 50%]
tests/e2e/llm_translation/test_together_ai_e2e.py::TestTogetherChatCompletions::test_replayed_reasoning_content_reaches_together FAILED [ 75%]
tests/e2e/llm_translation/test_together_ai_e2e.py::TestTogetherMessages::test_tool_use_block_is_returned FAILED [100%]
E       AssertionError: Together dropped the tool call: role='assistant' content='Searching web.<|start|>assistant<|channel|>commentary to=browser.search code<|message|>{"query": "current weather Paris", ...
E       AssertionError: stream carried no tool call deltas: [_StreamDelta(content='', reasoning_content=None, tool_calls=None), _StreamDelta(content='', reasoning_content='We', tool_calls=None), ...
E       AssertionError: the replayed reasoning_content never reached Together: role='assistant' content="I don't know." reasoning_content='The user is asking about their favorite color, but I don\'t have any information about them ...
E       AssertionError: expected one tool_use block, got [AnthropicContentBlock(type='thinking', text=None, id=None, name=None, thinking='The user asks: "What is the weather in Paris? Use the tool." ...
================== 4 failed, 7 deselected in 81.38s (0:01:21) ==================

3. Revert the change, reboot the proxy, and the full file passes again as in the matrix case

Type

✅ Test

Caveats (if any)

Medium

• Rerank stays uncovered: Together has no serverless rerank model, both known ones return 400 model_not_available
• TOGETHER_API_KEY reached the stage provider-keys secret after this PR's last CI run (litellm-ops PR 153, merged 2026-08-26), so this file hard-failed on stage until then
  • That is why buildkite/e2e-tests (litellm-e2e-pr build 133) is red: all 11 Together tests fail with 401 missing_api_key; the check is not required
  • Stage proof after the key landed: litellm-e2e-pr build 134 (same tip, same full-suite selection) passes all 11 Together tests, https://buildkite.com/berriai-1/litellm-e2e-pr/builds/134
  • Build 133's other two failures are unrelated: test_failed_chat_completions_error_span_attributes also fails on scheduled builds 68-70, and test_timeout_routes_to_fallback got an empty gpt-5.5 completion with finish_reason: length
• Stage proxies price with main's cost map, which lags staging, so a registry pick there can differ from a local run
  • Today both resolve to together_ai/openai/gpt-oss-120b
• Together's serverless gpt-oss-120b returns bursts of 503 service_unavailable_error (5 straight against api.together.ai directly in one run), which can outlast the harness's single 5xx rerun
  • The same burst hit 3 of 11 tests in one local run (two 503s and one read timeout) and none in the next three

Low

• Cached prompt tokens bill at $0 until fix(cost): apply Together AI cache read pricing and per-model registry rates #38280 lands; the cost test reads the registry's cache read rate either way
• Qwen/Qwen3.5-9B and MiniMaxAI/MiniMax-M3 are pinned on purpose; if Together retires either, those two tests fail loudly
• gpt-oss-120b sometimes leaks harmony text into content next to tool_calls; the tool tests do not assert on content
• The registry pick follows price, so a cheaper together_ai/ chat row with both capability flags that is dedicated-only on Together would move every registry-driven test onto it and fail them at POST /model/new
  • The next two cheapest rows today are Qwen/Qwen3.5-9B and MiniMaxAI/MiniMax-M3, both serverless

QA runbook

Prerequisites: TOGETHER_API_KEY in the proxy's environment, store_model_in_db: true, and LITELLM_LOCAL_MODEL_COST_MAP=True when running from this branch so the proxy prices with its own registry. Every test registers its own deployment with POST /model/new ({"model_name": "e2e-together-<marker>", "litellm_params": {"model": "<backend>", "api_key": "os.environ/TOGETHER_API_KEY"}}) and generates its own key with POST /key/generate, then deletes the deployment at the end. The registry-driven backend is the cheapest live together_ai/ chat row in GET /public/litellm_model_cost_map with both supports_function_calling and supports_reasoning, together_ai/openai/gpt-oss-120b today. Together charges for every call

• tests/e2e/llm_translation/test_together_ai_e2e.py::TestTogetherChatCompletions::test_reasoning_surfaces_as_reasoning_content - a reasoning model's thinking comes back as reasoning_content next to the answer

  • Register the registry-driven backend and generate a key
  • POST /v1/chat/completions with {"model": "<name>", "messages": [{"role": "user", "content": "What is 17 + 26? Answer with just the number."}], "max_tokens": 1024}
  • Expect 200 with a non-empty choices[0].message.reasoning_content and content containing 43
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/llm_translation/test_together_ai_e2e.py::TestTogetherChatCompletions::test_reasoning_streams_as_reasoning_content_deltas - the same thinking streams as delta.reasoning_content chunks before the answer

  • Same registration, then POST /v1/chat/completions with the same body plus "stream": true
  • Expect SSE chunks whose choices[0].delta.reasoning_content concatenate to non-empty text, delta.content concatenating to text containing 43, and a final [DONE]
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/llm_translation/test_together_ai_e2e.py::TestTogetherChatCompletions::test_tool_call_is_returned - a tool definition survives the transform and Together answers with a tool call

  • Same registration, then POST /v1/chat/completions with {"model": "<name>", "messages": [{"role": "user", "content": "What is the weather in Paris? Use the tool."}], "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a location.", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]}}}], "max_tokens": 512}
  • Expect 200 with exactly one choices[0].message.tool_calls entry carrying an id, function.name of get_weather, and JSON arguments whose location contains Paris
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/llm_translation/test_together_ai_e2e.py::TestTogetherChatCompletions::test_tool_call_is_streamed - the same tool call arrives as streamed delta.tool_calls chunks

  • Same registration, then the same tool request with "stream": true
  • Expect SSE chunks with choices[0].delta.tool_calls whose only function name is get_weather and whose arguments fragments concatenate to JSON with location containing Paris
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/llm_translation/test_together_ai_e2e.py::TestTogetherChatCompletions::test_tool_result_round_trip - replaying the assistant tool call turn (with its reasoning_content) plus a tool result gets an answer that uses the result

  • Same registration, run the non-streaming tool request above, and keep the assistant message it returned
  • POST /v1/chat/completions again with messages = the user prompt, the returned assistant message (content, reasoning_content, tool_calls), and {"role": "tool", "tool_call_id": "<returned id>", "content": "Paris: 22 degrees Celsius, clear skies, wind from the northwest at 9 km/h"}, same tools
  • Expect 200 with choices[0].message.content containing 22
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/llm_translation/test_together_ai_e2e.py::TestTogetherChatCompletions::test_chat_template_kwargs_reach_together - chat_template_kwargs is forwarded, proven by turning a model's default thinking off

  • Register together_ai/Qwen/Qwen3.5-9B and generate a key
  • Control: POST /v1/chat/completions with the arithmetic prompt and max_tokens: 1024, expect a non-empty reasoning_content
  • Treatment: the same body plus "chat_template_kwargs": {"enable_thinking": false}, expect no reasoning_content and content containing 43
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/llm_translation/test_together_ai_e2e.py::TestTogetherChatCompletions::test_replayed_reasoning_content_reaches_together - a replayed assistant turn's reasoning_content reaches Together instead of being stripped

  • Register together_ai/MiniMaxAI/MiniMax-M3 and generate a key
  • POST /v1/chat/completions with messages = {"role": "user", "content": "Remember this for later and reply with just OK."}, {"role": "assistant", "content": "OK.", "reasoning_content": "The user told me their favorite color is chartreuse. I must remember it."}, {"role": "user", "content": "What is my favorite color? Answer with one word."} and max_tokens: 512
  • Expect 200 with content containing chartreuse (the fact exists nowhere else in the conversation)
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/llm_translation/test_together_ai_e2e.py::TestTogetherChatCompletions::test_cost_header_and_spend_row_match_the_registry_price - the cost header and the spend log both equal the registry price for the usage Together reported

  • Register the registry-driven backend and generate a key, then read its row from GET /public/litellm_model_cost_map
  • POST /v1/chat/completions with the arithmetic prompt plus a unique suffix, max_tokens: 1024, and capture usage and the x-litellm-response-cost header
  • Expect the header within 1% of (prompt_tokens - cached_tokens) * input_cost_per_token + cached_tokens * (cache_read_input_token_cost or 0) + completion_tokens * output_cost_per_token
  • GET /spend/logs?api_key= until a row lands, expect custom_llm_provider of together_ai and spend within 1% of the header
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/llm_translation/test_together_ai_e2e.py::TestTogetherMessages::test_tool_use_block_is_returned - the same tool call works through the Anthropic-shaped endpoint

  • Register the registry-driven backend and generate a key
  • POST /v1/messages with {"model": "<name>", "max_tokens": 512, "tools": [{"name": "get_weather", "description": "Get the current weather for a location.", "input_schema": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]}}], "messages": [{"role": "user", "content": "What is the weather in Paris? Use the tool."}]}
  • Expect 200 with exactly one content block of type tool_use, named get_weather, carrying an id
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/llm_translation/test_together_ai_e2e.py::TestTogetherMessages::test_tool_result_round_trip - replaying the returned content blocks plus a tool_result gets an answer that uses the result

  • Same registration, run the /v1/messages tool request above, and keep the full content list it returned
  • POST /v1/messages again with messages = the user prompt, {"role": "assistant", "content": <returned content list>}, and {"role": "user", "content": [{"type": "tool_result", "tool_use_id": "<returned id>", "content": "Paris: 22 degrees Celsius, clear skies, wind from the northwest at 9 km/h"}]}, same tools
  • Expect 200 whose text blocks contain 22
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/llm_translation/test_together_ai_e2e.py::TestTogetherMessages::test_streams_text_deltas - /v1/messages streams incrementally and finishes cleanly

  • Same registration, then POST /v1/messages with {"model": "<name>", "max_tokens": 512, "stream": true, "messages": [{"role": "user", "content": "Count from 1 to 20, one number per line."}]}
  • Expect at least two content_block_delta events carrying text, that text containing 20, and a message_stop event
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

• 893482d passes /live-pr-risk

[greptile-apps]

Greptile Summary

Adds live Together AI proxy coverage for reasoning, tool use, multi-turn conversations, template arguments, streaming, and spend accounting.

• Registers eleven Together AI scenarios in the E2E coverage registry.
• Extends shared E2E request and response models for assistant reasoning, tool calls, Anthropic tool blocks, and model-cost metadata.
• Adds a proxy client helper for retrieving the public model cost map.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
tests/e2e/llm_translation/test_together_ai_e2e.py	Adds live Together AI coverage across chat-completions and Messages API behavior.
tests/e2e/coverage_registry/llm_conversational.yaml	Registers the eleven new Together AI E2E coverage cells.
tests/e2e/models.py	Extends E2E schemas for tool-call turns, template arguments, Anthropic tool names, and cost-map entries.
tests/e2e/proxy_client.py	Adds typed retrieval of the proxy model-cost registry.

_{Reviews (3): Last reviewed commit: "test(e2e): drop docstrings on the cost m..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 893482d. Configure here.}