Conversation
get_provider_model_info was a 16-branch elif chain, and every provider added to it pushed the function further past ruff's complexity ceiling. The ten providers whose model info is just a zero-arg config constructor now come from a lazy factory table, leaving explicit branches only for the cases that need a deferred import or the model name. Registers together_ai on the way through, so the provider can serve a capability baseline the way fireworks_ai already does.
Usage already understood DeepSeek's prompt_cache_hit_tokens and Anthropic's cache_read_input_tokens, but not a plain cached_tokens sitting at the top level of usage, which is what Together AI returns on its non-reasoning models. The count never reached prompt_tokens_details, so the cost calculator billed a warm prefix at the full input rate. The nested prompt_tokens_details value stays authoritative when a provider sends both.
…rent model map
Together shipped models faster than litellm's cost map could follow, and the provider
read an absent map entry as "no tool calling": every call to a model the map had not
caught up with lost tools, tool_choice, function_call and response_format, with only a
debug line to explain it. TogetherAIConfig now serves a provider-level capability
baseline, the way fireworks_ai does, so an unmapped or brand-new id keeps Together's
real feature set while an explicit map entry still wins.
Reasoning controls now work: reasoning_effort and thinking reach the models that
support them, reasoning_effort="none" becomes Together's reasoning={"enabled": false}
toggle, "minimal" lands on "low", and the Anthropic-shaped thinking param maps onto the
same toggle. OpenAI's boolean logprobs plus top_logprobs collapse into the integer
logprobs Together documents, and max_completion_tokens becomes max_tokens instead of
being silently ignored. The Together-native params (top_k, min_p, repetition_penalty,
echo, context_length_exceeded_behavior, safety_model, chat_template_kwargs, reasoning)
are now advertised as supported rather than only tolerated.
The model map gains the current serverless catalog with cached-input rates, so cache
hits bill at the cached rate: Kimi K3, Kimi K2.7 Code, Kimi K2.6, GLM-5.2,
DeepSeek-V4-Pro and V4-Pro-0813 and V4-Flash-0731, MiniMax M3, Nemotron 3 Ultra,
Inkling and Inkling Small, the Qwen3.5/3.6/3.7/3.8 line, Gemma 4 31B, Cogito v2.1,
LFM2.5, Muse Glimmer, Ternary Bonsai, and the multilingual-e5 embedding model. Existing
entries stay put; Llama 3.3 70B Turbo and Qwen2.5 7B Turbo pick up their published rates
and context windows, and Qwen3-235B-A22B-Instruct-2507-tput no longer bills output at
ten times its published rate.
Rounding it out: /v1/models discovery so `litellm.get_valid_models` can enumerate an
account's models, one source of truth for the api base and the four accepted key names,
and the embeddings column flipped on in the support matrix, which the provider has
routed since main.py:6218.
Greptile SummaryThis PR expands Together AI capability detection, parameter translation, model discovery, cached-token accounting, and the current pricing catalog
Confidence Score: 4/5The PR appears safe to merge, with only non-blocking cleanup needed for redundant source comments The changed provider translation, discovery, capability, and cached-token paths have focused regression coverage, and no concrete blocking failure remains Files Needing Attention: litellm/types/utils.py, litellm/llms/together_ai/chat.py, litellm/utils.py
|
| Filename | Overview |
|---|---|
| litellm/llms/together_ai/chat.py | Adds Together capability resolution, request translations, credential resolution, and model discovery; no blocking behavioral defect was established |
| litellm/types/utils.py | Normalizes flat cached-token usage while preserving nested values, but adds explanatory comments contrary to repository guidance |
| litellm/utils.py | Registers Together as a provider model-info factory and adds bundled pricing lookup support |
| model_prices_and_context_window.json | Refreshes Together model capabilities, context limits, cache rates, and token pricing |
| litellm/model_prices_and_context_window_backup.json | Keeps the bundled backup pricing catalog synchronized with the canonical Together model updates |
| tests/test_litellm/llms/together_ai/test_together_ai_chat_transformation.py | Covers capability resolution, parameter translations, API configuration, and both supported model-discovery payload shapes |
| tests/test_litellm/llms/together_ai/test_together_ai_cost_calculator.py | Verifies cached-prefix billing and representative updated Together catalog rates |
Reviews (1): Last reviewed commit: "feat(together_ai): capability baseline, ..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| ## FLAT `cached_tokens` MAPPING ## | ||
| # Some providers report cache hits at the top level of `usage` instead of nesting them | ||
| # under `prompt_tokens_details` (Together AI does this on its non-reasoning models), which | ||
| # would otherwise bill a cached prefix at the full input rate. The nested count is the more | ||
| # specific signal, so it wins when both are present. |
There was a problem hiding this comment.
This prose restates the adjacent cached-token precedence condition and must be maintained alongside the implementation; keep comments only where complex business logic genuinely requires explanation. The same pattern appears in the new Together parameter helpers and model-info factory.
Context Used: CLAUDE.md (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
get_model_params_and_category rewrote every together_ai model name into a parameter-count bucket before the cost lookup ran, so any model whose name carries a parameter count billed at its bucket rather than its own entry: Llama 3.3 70B Instruct Turbo charged the 41.1b-80b bucket's $0.90/1M against a published $1.04/1M, and Gemma 4 31B, Qwen3.5 9B, LFM2.5 8B, Muse Glimmer 30B and Ternary Bonsai 27B were all mispriced the same way, with their cached-input rates unreachable. The buckets now apply only to ids the map does not price, which is what the neighbouring replicate branch already does.
reasoning_effort="none" and the Anthropic-shaped thinking param both map onto Together's `reasoning` object, which is not an OpenAI parameter: handed to the SDK as a keyword it raised "AsyncCompletions.create() got an unexpected keyword argument 'reasoning'" before the request left the process. It now travels in extra_body, which the SDK spreads back into the request body, and merges with an extra_body the caller already set.
…published A missing cache_read_input_token_cost read as 0.0, so every cache hit on a model with no published cached-input rate was billed as free. Together AI reports a cache hit on every warm prefix, including on the models that publish no discount, so a repeated prompt there tracked at a fraction of what it costs. Cached tokens now fall back to the standard input rate, which is what the tiered-pricing path in the same module already does, and a published cached rate still wins.
Together documents https://api.together.ai/v1, and its own SDK and cURL examples use it; litellm defaulted to api.together.xyz, the legacy host, and hardcoded it in the rerank endpoint and the chat-template lookup too. Both hosts answer today, so api.together.xyz stays a recognized openai-compatible endpoint and an explicit api_base pointing there is still honored.
…t the wire body The module-scoped fixture wrote litellm.model_cost directly, which is a process-wide global the test-quality gate counts (TQ005); monkeypatch.setattr undoes it at teardown and drops the hand-rolled save and restore. The two boundary tests now fake at the HTTP layer and assert the JSON body Together would receive, rather than the keyword arguments handed to the SDK, so they read the bytes on the wire and no longer restate the call.
…e total Together AI reports completion_tokens_details with text_tokens equal to completion_tokens and reasoning_tokens nested inside it on Qwen3.6 Plus and Qwen3.7 Plus, while Qwen3.7 Max sends the disjoint shape. Text, audio, reasoning, image and video are each billed separately, so the overlapping shape charged the reasoning tokens a second time: a 140-token completion billed as 274, about 1.96x. The remainder branch now also runs when reasoning overlaps text, which is the completion-side twin of the cached-token guard the prompt side already had.
|
Independent confirmation that this is still broken on Reproduction from litellm.utils import supports_function_calling
supports_function_calling("zai-org/GLM-5.2", custom_llm_provider="together_ai") # False
supports_function_calling("deepseek-ai/DeepSeek-V4-Flash-0731", custom_llm_provider="together_ai") # False
supports_function_calling("Qwen/Qwen3.7-Plus", custom_llm_provider="together_ai") # FalseNone are in Why this is worse than a missing capability With the documented default Calling Together directly with the identical With For anyone hitting this before this PR lands: set Also worth noting that #18185 reported this in December 2025 and was closed with no comment and no fix. Your framing matches what we found exactly: |
|
Closing: the dedicated Together chat config landed in #38248; reasoning controls and the model map follow in queued PRs. Thanks for contributing! |
3 participants
TLDR
Problem this solves:
How it solves it:
User Flow
Before: a developer building an agent on a model Together launched this week gets no tool calls back, cannot turn thinking off, and sees the wrong spend
together_ai/moonshotai/Kimi-K3to their proxy config and restarttoolsarray holding oneget_weatherfunction, and "What is the weather in Paris? Use the tool."tool_callsarray, so their agent loop has nothing to executedrop_paramsthe same request fails outright with 500together_ai does not support parameters: ['tools']together_ai/Qwen/Qwen3.5-9Band send"reasoning_effort": "none"to skip thinking on a simple question; the answer still arrives behind 869 characters of chain of thought, and they pay for those tokensx-litellm-response-costresponse header: it is empty both times, so the calls land in their dashboard at zero spendtogether_ai/meta-llama/Llama-3.3-70B-Instruct-Turboand the header reads0.0042876every time, which is $0.90 per 1M tokens rather than the $1.04 Together publishes, and the responseusagecarries no cached-token count for their cache-hit dashboardtogether_ai/Qwen/Qwen3.7-Plus, which returns 140 completion tokens, and the spend recorded for that turn reflects 274 output tokensAfter: the same requests return tool calls, thinking is controllable, and the spend numbers match Together's published rates
together_ai/moonshotai/Kimi-K3to their proxy config and restarttoolsarray holding oneget_weatherfunction, and "What is the weather in Paris? Use the tool."finish_reason: "tool_calls"and atool_callsarray namingget_weatherwith{"location": "Paris"}, so the agent loop runs"response_format": {"type": "json_schema", ...}on the same model returns schema-shaped JSON instead of a 500together_ai/Qwen/Qwen3.5-9B,"reasoning_effort": "none"returns the answer with no chain of thought at all, and the same request without it still thinksx-litellm-response-costreads0.0141912and the second, hitting Together's warm prefix, reads0.0015444, a 9x drop that shows up in their dashboard0.00495456, matching Together's published $1.04 per 1M, and the responseusagenow reportsprompt_tokens_details.cached_tokensso their cache-hit dashboard is no longer stuck at zerotogether_ai/Qwen/Qwen3.7-Pluscharges the 140 completion tokens the model actually producedRelevant issues
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Both sides ran against the same live Together AI account, one proxy per side, with
LITELLM_LOCAL_MODEL_COST_MAP=Trueso the bundled map is the one under test.qa_config.yaml:Cases 3 and 4 send a 4.7k-token prefix twice so Together's automatic prefix cache can warm between calls. Cache hits are best-effort, so which call lands warm varies.
Before (66a89f5)
Tool calling on a model the cost map does not know
curl -sS localhost:4010/v1/chat/completions -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d @case1_tools.json | jq -c '{finish_reason: .choices[0].finish_reason, tool_calls: .choices[0].message.tool_calls, content_head: (.choices[0].message.content // "" | .[0:80])}'{"finish_reason":"stop","tool_calls":null,"content_head":"I don't have access to a weather tool or any real-time data sources, so I'm unab"}Turning thinking off on a hybrid model
curl -sS localhost:4010/v1/chat/completions -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model":"qwen3.5-9b","messages":[{"role":"user","content":"Is 9.11 bigger than 9.9? Answer in one short sentence."}],"reasoning_effort":"none","max_tokens":600}' | jq -c '{reasoning_chars: (.choices[0].message.reasoning_content // "" | length), content_head: (.choices[0].message.content // "" | .[0:60])}'{"reasoning_chars":869,"content_head":"No, 9.11 is smaller than 9.9."}Cost and cached input on Kimi K3
for i in 1 2; do curl -sS -D h.txt localhost:4010/v1/chat/completions -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d @case3_cache.json | jq -c .usage; grep -i '^x-litellm-response-cost:' h.txt; donex-litellm-response-cost:(empty) with{"completion_tokens":8,"prompt_tokens":4748,"total_tokens":4756,"prompt_tokens_details":{"cached_tokens":64}}x-litellm-response-cost:(empty) with{"completion_tokens":8,"prompt_tokens":4748,"total_tokens":4756,"prompt_tokens_details":{"cached_tokens":4748}}Published rate and flat cached tokens on Llama 3.3 70B
for i in 1 2; do curl -sS -D h.txt localhost:4010/v1/chat/completions -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d @case4_flat.json | jq -c '{usage, cached_via_details: .usage.prompt_tokens_details.cached_tokens}'; grep -i '^x-litellm-response-cost:' h.txt; donex-litellm-response-cost: 0.0042876with{"usage":{"completion_tokens":2,"prompt_tokens":4762,"total_tokens":4764,"cached_tokens":0},"cached_via_details":null}x-litellm-response-cost: 0.0042876with{"usage":{"completion_tokens":2,"prompt_tokens":4762,"total_tokens":4764,"cached_tokens":0},"cached_via_details":null}After (b94e4a7)
Tool calling on a model the cost map does not know
curl -sS localhost:4010/v1/chat/completions -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d @case1_tools.json | jq -c '{finish_reason: .choices[0].finish_reason, tool_calls: .choices[0].message.tool_calls, content_head: (.choices[0].message.content // "" | .[0:80])}'{"finish_reason":"tool_calls","tool_calls":[{"function":{"arguments":"{\"location\":\"Paris\"}","name":"get_weather"},"id":"get_weather_0","type":"function"}],"content_head":"I'll check the current weather in Paris for you."}Turning thinking off on a hybrid model
curl -sS localhost:4010/v1/chat/completions -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model":"qwen3.5-9b","messages":[{"role":"user","content":"Is 9.11 bigger than 9.9? Answer in one short sentence."}],"reasoning_effort":"none","max_tokens":600}' | jq -c '{reasoning_chars: (.choices[0].message.reasoning_content // "" | length), content_head: (.choices[0].message.content // "" | .[0:60])}'{"reasoning_chars":0,"content_head":"No, 9.9 is bigger than 9.11."}Cost and cached input on Kimi K3
for i in 1 2; do curl -sS -D h.txt localhost:4010/v1/chat/completions -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d @case3_cache.json | jq -c .usage; grep -i '^x-litellm-response-cost:' h.txt; donex-litellm-response-cost: 0.014191200000000001with{"completion_tokens":8,"prompt_tokens":4748,"total_tokens":4756,"prompt_tokens_details":{"cached_tokens":64}}x-litellm-response-cost: 0.0015444with{"completion_tokens":8,"prompt_tokens":4748,"total_tokens":4756,"prompt_tokens_details":{"cached_tokens":4748}}Published rate and flat cached tokens on Llama 3.3 70B
for i in 1 2; do curl -sS -D h.txt localhost:4010/v1/chat/completions -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d @case4_flat.json | jq -c '{usage, cached_via_details: .usage.prompt_tokens_details.cached_tokens}'; grep -i '^x-litellm-response-cost:' h.txt; donex-litellm-response-cost: 0.00495456with{"usage":{"completion_tokens":2,"prompt_tokens":4762,"total_tokens":4764,"prompt_tokens_details":{"cached_tokens":4608},"cached_tokens":4608},"cached_via_details":4608}x-litellm-response-cost: 0.00495456with{"usage":{"completion_tokens":2,"prompt_tokens":4762,"total_tokens":4764,"prompt_tokens_details":{"cached_tokens":4608},"cached_tokens":4608},"cached_via_details":4608}Llama 3.3 70B publishes no cached-input discount, so a warm call costs the same as a cold one. What changes is the rate, $1.04 per 1M rather than the bucket's $0.90, and the cached count reaching
prompt_tokens_details. The prefix was cold on both Before calls and warm on both After calls, which the counts show.Every model this branch adds also got its own live call: 21 of 24 answered, and each one billed at exactly its mapped rate, which is the check that the size buckets no longer shadow the entries. The three that did not answer are provider-side rather than litellm:
Kimi-K2.6is listed but not serverless on this account,LFM2.5-8B-A1Breturns 503 and no longer appears in/v1/modelsat all, and the pre-existingQwen3-235B-A22B-Instruct-2507-tputentry whose output rate this branch corrects is retired from serverless.Qwen3.6-Plus,Qwen3.7-PlusandQwen3.7-Maxare streaming-only and were confirmed over a stream.The reasoning double-count is the one fix with no proxy-visible before and after, because a streamed response flushes its headers before the usage chunk arrives, so
x-litellm-response-costis empty for streams on both sides. It shows up three other ways. On the wire,curl -N https://api.together.ai/v1/chat/completions -d '{"model":"Qwen/Qwen3.7-Plus", ..., "stream":true, "stream_options":{"include_usage":true}}'returns"completion_tokens":140beside"completion_tokens_details":{"reasoning_tokens":134,"text_tokens":140}, where Qwen3.7 Max sends the disjoint shape instead. In the per-model sweep, both Plus models went from a mismatched computed cost to a matching one. And the arithmetic is pinned by a unit test.Type
🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
Caveats (if any)
max_output_tokensomitted where Together publishes no capFinal Attestation