fix(cost): apply Together AI cache read pricing and per-model registry rates by mateo-berri · Pull Request #38280 · BerriAI/litellm

mateo-berri · 2026-08-25T23:47:10Z

TLDR

Problem this solves:

Registry-mapped Together models whose name carries a parameter size (Muse-Glimmer-30B, 33 models in the registry) were still priced by the size bucket, so their registry rates were ignored and cached tokens were billed at $0
Qwen3.7-Max registry prices lagged Together's current rates

How it solves it:

Skips the size-bucket rewrite when the model has a registry entry with an input price; unmapped and metadata-only models keep the bucket
Reprices Qwen3.7-Max to Together's live rates and adds the cache pricing regression tests
The cache read prices for the other 12 cached Together models landed on litellm_internal_staging through its registry audits (44c7cb2, 4456a44) while this PR was open, so the cost map delta here is Qwen3.7-Max only
No cache write cost added: Together does not charge for cache writes

User Flow

Before: a registry-mapped Together model with a size in its name bills the size bucket, so recorded spend ignores its registry rates and bills cache hits at $0

A developer sends POST https://litellm-domain/v1/chat/completions with model together_ai/meta-models/Muse-Glimmer-30B and a ~13k token prompt; the response header shows x-litellm-response-cost: 0.0106976 for 13356 prompt and 16 completion tokens, which is 13372 x $0.80/M (the 21.1B-41B bucket) against Together's $0.35/M input and $1.50/M output
They send the identical request again; the response usage now shows prompt_tokens_details.cached_tokens: 13344
The header reads x-litellm-response-cost: 0.0000224: the 12 uncached prompt tokens and the output at the bucket rate, the 13344 cached tokens at $0 while Together invoices them at $0.04/M
The usage page at https://litellm-domain/ui/?page=usage over-counts every cold prompt by 2.3x and records every cache hit at $0

After: the same requests bill Together's registry rates, cache hits included

A developer sends POST https://litellm-domain/v1/chat/completions with model together_ai/meta-models/Muse-Glimmer-30B and a ~13k token prompt; the response header shows x-litellm-response-cost: 0.0045726 for 12996 prompt and 16 completion tokens (12996 x $0.35/M + 16 x $1.50/M)
They send the identical request again; the response usage now shows prompt_tokens_details.cached_tokens: 12992
The header reads x-litellm-response-cost: 0.00054508: 4 uncached tokens at $0.35/M, 12992 cached tokens at $0.04/M, and 16 output tokens at $1.50/M
The usage page tracks the Together invoice

Relevant issues

Linear ticket

Resolves LIT-5972

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live proxy against the real Together API, one leg per commit: Before at 67c7b97 (the current merge base with litellm_internal_staging), After at dbadee7 (the PR tip). Each leg boots one proxy process with --num_workers 2 on its own random port with LITELLM_LOCAL_MODEL_COST_MAP=True and this config:

model_list:
  - model_name: deepseek-v4-flash
    litellm_params:
      model: together_ai/deepseek-ai/DeepSeek-V4-Flash-0731
      api_key: os.environ/TOGETHER_API_KEY
  - model_name: muse-glimmer-30b
    litellm_params:
      model: together_ai/meta-models/Muse-Glimmer-30B
      api_key: os.environ/TOGETHER_API_KEY

general_settings:
  master_key: sk-lit5972-test

Every payload carries a ~13k token system corpus plus a one-line user question (max_tokens: 16), salted per leg and per endpoint so each leg starts with a cold Together cache. The chat completions pair sends the same request twice: the first warms the cache, the identical second hits it. Together's published Muse-Glimmer-30B rates: input $0.35/M, cached input $0.04/M, output $1.50/M; the 21.1B-41B size bucket is $0.80/M in and out with no cache read price. Each line below is one request: the x-litellm-response-cost header followed by the response's usage block

send() { curl -s -D "$leg_$1.h" -o "$leg_$1.json" "http://127.0.0.1:$port$2" -H 'Authorization: Bearer sk-lit5972-test' -H 'Content-Type: application/json' -d @"$3"; }
send chat1 /v1/chat/completions muse_chat.json
send chat2 /v1/chat/completions muse_chat.json
send responses /v1/responses muse_responses.json
send messages /v1/messages muse_messages.json
send ds1 /v1/chat/completions ds_chat.json
send ds2 /v1/chat/completions ds_chat.json

Before (`67c7b97`, port 48430)

base chat1 cost=0.0106976 usage={"completion_tokens": 16, "prompt_tokens": 13356, "total_tokens": 13372, "prompt_tokens_details": {"cached_tokens": 0, "created_cache_tokens": 13344}}
base chat2 cost=2.24e-05 usage={"completion_tokens": 16, "prompt_tokens": 13356, "total_tokens": 13372, "prompt_tokens_details": {"cached_tokens": 13344, "created_cache_tokens": 0}}
base responses cost=0.0104096 usage={"input_tokens": 12996, "input_tokens_details": {"audio_tokens": null, "cached_tokens": 0, "text_tokens": null}, "output_tokens": 16, "output_tokens_details": null, "total_tokens": 13012, "cost": null}
base messages cost=0.0104096 usage={"input_tokens": 12996, "output_tokens": 16}
base ds1 cost=0.0019215000000000002 usage={"completion_tokens": 16, "prompt_tokens": 13693, "total_tokens": 13709, "completion_tokens_details": {"reasoning_tokens": 16}, "prompt_tokens_details": {"cached_tokens": 0}}
base ds2 cost=0.00041537999999999997 usage={"completion_tokens": 16, "prompt_tokens": 13693, "total_tokens": 13709, "completion_tokens_details": {"reasoning_tokens": 16}, "prompt_tokens_details": {"cached_tokens": 13692}}

Muse-Glimmer-30B bills the bucket on all three routes: chat1 is 13372 x $0.80/M, responses and messages are 13012 x $0.80/M, and chat2 bills its 13344 cached tokens at $0 (12 x $0.80/M + 16 x $0.80/M = $0.0000224). The x-litellm-response-cost-cache-read header is absent on chat2. DeepSeek-V4-Flash-0731 already bills registry rates and cache reads at this base (ds2 = 1 x $0.14/M + 13692 x $0.03/M + 16 x $0.28/M) because staging carries its cache price since 4456a44; its name has no size marker, so the bucket never applied to it

After (`dbadee7`, port 42191)

tip chat1 cost=0.0045726 usage={"completion_tokens": 16, "prompt_tokens": 12996, "total_tokens": 13012, "prompt_tokens_details": {"cached_tokens": 0, "created_cache_tokens": 12992}}
tip chat2 cost=0.0005450800000000001 usage={"completion_tokens": 16, "prompt_tokens": 12996, "total_tokens": 13012, "prompt_tokens_details": {"cached_tokens": 12992, "created_cache_tokens": 0}}
tip responses cost=0.004824599999999999 usage={"input_tokens": 13716, "input_tokens_details": {"audio_tokens": null, "cached_tokens": 0, "text_tokens": null}, "output_tokens": 16, "output_tokens_details": null, "total_tokens": 13732, "cost": null}
tip messages cost=0.0101216 usage={"input_tokens": 12636, "output_tokens": 16}
tip ds1 cost=0.0020223 usage={"completion_tokens": 16, "prompt_tokens": 14413, "total_tokens": 14429, "completion_tokens_details": {"reasoning_tokens": 16}, "prompt_tokens_details": {"cached_tokens": 0}}
tip ds2 cost=0.00043697999999999995 usage={"completion_tokens": 16, "prompt_tokens": 14413, "total_tokens": 14429, "completion_tokens_details": {"reasoning_tokens": 16}, "prompt_tokens_details": {"cached_tokens": 14412}}

$ grep -i '^x-litellm-response-cost' tip_chat2.h
x-litellm-response-cost: 0.0005450800000000001
x-litellm-response-cost-original: 0.0005450800000000001
x-litellm-response-cost-input: 1.400000000000034e-06
x-litellm-response-cost-output: 2.4e-05
x-litellm-response-cost-cache-read: 0.00051968

Muse-Glimmer-30B now bills the registry: chat1 is 12996 x $0.35/M + 16 x $1.50/M, chat2 bills 4 uncached tokens at $0.35/M, 12992 cached tokens at $0.04/M ($0.00051968 in the cache-read header) and 16 output tokens at $1.50/M, and responses is 13716 x $0.35/M + 16 x $1.50/M. DeepSeek is unchanged between the legs, as expected. The /v1/messages header still reads the bucket at the tip: that route prices the deployment by its alias muse-glimmer-30b, whose 30b trips the bucket fallback before the registry lookup runs (#38578, pre-existing and identical on the Before leg, see Caveats)

/v1/messages cost header prices the alias, not the model, for Together ([Bug]: /v1/messages prices Together deployments by their alias, so a size marker in the alias bills the size bucket #38578)
Together omits cached_tokens from the /v1/messages usage block on cold and warm sends
Streaming was not re-run this pass; it shares the same cost path

Type

🐛 Bug Fix

Caveats (if any)

Medium

On /v1/messages the response-cost header prices a Together deployment by its alias, so an alias carrying a size marker (muse-glimmer-30b) still bills the size bucket on that route, unchanged from before this PR; /v1/chat/completions and /v1/responses bill the registry. The alias is not a registry name, so the gate here cannot fix it; tracked as [Bug]: /v1/messages prices Together deployments by their alias, so a size marker in the alias bills the size bucket #38578 for the messages route's cost path

Low

Together models without registry pricing still price by size bucket, unchanged
33 registry-mapped Together models now bill registry rates, not buckets, so their recorded spend shifts on upgrade
Qwen3.7-Max recorded spend doubles to match Together's current $2.50/M input and $7.50/M output rates
Registry fields match the sync script in feat(models): add daily Together AI model registry sync script and workflow #38257, so future syncs stay clean

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR
abe9af6 passes /live-pr-risk
dbadee7 passes /live-pr-risk (merge of staging with the PR delta unchanged; the merge-ref check re-ran the top scenarios live on the merged tree, results above)

…y rates

codecov · 2026-08-25T23:50:56Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

greptile-apps · 2026-08-25T23:51:00Z

Greptile Summary

The PR corrects Together AI cost accounting for cached input tokens and registry-priced models.

Uses per-model registry pricing when a Together model has an input-token rate.
Retains size-bucket fallback for unmapped and metadata-only models.
Updates Qwen3.7-Max rates and adds cache-pricing regression coverage.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
litellm/cost_calculator.py	Selects Together registry pricing before falling back to parameter-size buckets; the previously flagged redundant comments have been removed.
litellm/llms/together_ai/cost_calculator.py	Adds a helper that detects whether a Together model has registry input pricing.
model_prices_and_context_window.json	Updates Qwen3.7-Max input, output, and cached-input rates.
litellm/model_prices_and_context_window_backup.json	Keeps the backup pricing registry synchronized with the primary registry.
tests/test_litellm/test_cost_calculator.py	Adds regression coverage for cached-token pricing and registry-versus-bucket selection.
tests/test_litellm/test_together_ai_model_metadata.py	Verifies Together cache metadata and representative model rates.

_{Reviews (4): Last reviewed commit: "Merge remote-tracking branch 'origin/lit..." | Re-trigger Greptile}

mateo-berri · 2026-08-25T23:56:29Z

@greptileai

mateo-berri · 2026-08-25T23:59:58Z

bugbot run

cursor

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

✅ Fixed: Registry skip zeros unpriced models
- has_together_registry_entry now only reports True when the registry row carries input_cost_per_token, output_cost_per_token, or tiered_pricing, so metadata-only Together rows still fall through to the size-bucket rewrite.

Or push these changes by commenting:

@cursor push 68d624d752

Preview (68d624d752)

diff --git a/litellm/llms/together_ai/cost_calculator.py b/litellm/llms/together_ai/cost_calculator.py
--- a/litellm/llms/together_ai/cost_calculator.py
+++ b/litellm/llms/together_ai/cost_calculator.py
@@ -19,9 +19,13 @@
 from litellm.types.utils import CallTypes
 
 
+_PRICING_KEYS: Final = ("input_cost_per_token", "output_cost_per_token", "tiered_pricing")
+
+
 def has_together_registry_entry(model: str, cost_map: Mapping[str, object]) -> bool:
     stripped: Final = model.removeprefix("together_ai/")
-    return f"together_ai/{stripped}" in cost_map
+    entry: Final = cost_map.get(f"together_ai/{stripped}")
+    return isinstance(entry, Mapping) and any(entry.get(key) is not None for key in _PRICING_KEYS)
 
 
 # Extract the number of billion parameters from the model name

diff --git a/tests/test_litellm/test_cost_calculator.py b/tests/test_litellm/test_cost_calculator.py
--- a/tests/test_litellm/test_cost_calculator.py
+++ b/tests/test_litellm/test_cost_calculator.py
@@ -3838,3 +3838,19 @@
     )
 
     assert cost == pytest.approx((23 + 15) * 9e-07, rel=1e-9)
+
+
+def test_completion_cost_together_metadata_only_registry_row_falls_back_to_size_bucket(_local_model_cost_map):
+    """Regression: Together registry rows that only carry capability metadata (no
+    input_cost_per_token/output_cost_per_token/tiered_pricing) must not short-circuit the
+    size-bucket rewrite. Otherwise models like togethercomputer/CodeLlama-34b-Instruct
+    resolve to $0 spend instead of the 21.1b-41b bucket rate."""
+
+    cost = completion_cost(
+        completion_response=_together_chat_response(
+            model="togethercomputer/CodeLlama-34b-Instruct", prompt_tokens=23, completion_tokens=15, cached_tokens=0
+        ),
+        custom_llm_provider="together_ai",
+    )
+
+    assert cost == pytest.approx((23 + 15) * 8e-07, rel=1e-9)

_{You can send follow-ups to the cloud agent here.}

mateo-berri · 2026-08-26T00:09:27Z

@greptileai

mateo-berri · 2026-08-26T00:11:51Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit abe9af6. Configure here.}

codspeed-hq · 2026-08-26T00:42:19Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_together_cache_pricing (dbadee7) with litellm_internal_staging (67c7b97)¹}

No successful run was found on litellm_internal_staging (852cb3a) during the generation of this report, so 67c7b97 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

…itellm_together_cache_pricing # Conflicts: # litellm/model_prices_and_context_window_backup.json # model_prices_and_context_window.json # tests/test_litellm/test_cost_calculator.py

mateo-berri · 2026-08-27T21:56:00Z

@greptileai

mateo-berri · 2026-08-27T22:02:48Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit dbadee7. Configure here.}

fix(cost): apply Together AI cache read pricing and per-model registr…

6fafb46

…y rates

greptile-apps Bot reviewed Aug 25, 2026

View reviewed changes

Comment thread litellm/cost_calculator.py Outdated

chore(cost): drop redundant together fallback comment

ac866e9

cursor Bot reviewed Aug 26, 2026

View reviewed changes

Comment thread litellm/llms/together_ai/cost_calculator.py Outdated

fix(cost): keep size buckets for Together registry rows without pricing

abe9af6

cursor Bot reviewed Aug 26, 2026

View reviewed changes

mateo-berri enabled auto-merge August 26, 2026 00:18

mateo-berri mentioned this pull request Aug 26, 2026

test(together_ai): regression suite across chat, responses, and messages surfaces #38283

Merged

8 tasks

mateo-berri mentioned this pull request Aug 26, 2026

test(e2e): cover Together AI reasoning, tool calls, template kwargs, and cost through a live proxy #38286

Merged

48 tasks

devin-ai-integration Bot mentioned this pull request Aug 26, 2026

fix(model_prices): rolling registry audit - verified models and rates for Novita, DeepInfra, W&B, Bedrock Sol, Gemini, Fireworks, Azure gpt-5.6, Mistral, Together #38207

Merged

5 tasks

Merge remote-tracking branch 'origin/litellm_internal_staging' into l…

dbadee7

…itellm_together_cache_pricing # Conflicts: # litellm/model_prices_and_context_window_backup.json # model_prices_and_context_window.json # tests/test_litellm/test_cost_calculator.py

cursor Bot reviewed Aug 27, 2026

View reviewed changes

mateo-berri mentioned this pull request Aug 27, 2026

[Bug]: /v1/messages prices Together deployments by their alias, so a size marker in the alias bills the size bucket #38578

Closed

1 task

yucheng-berri approved these changes Aug 27, 2026

View reviewed changes

mateo-berri merged commit dc1b847 into litellm_internal_staging Aug 27, 2026
82 checks passed

mateo-berri deleted the litellm_together_cache_pricing branch August 27, 2026 22:20

Uh oh!

Uh oh!

Conversation

mateo-berri commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Relevant issues

Linear ticket

Pre-Submission checklist

Delays in PR merge?

Screenshots / Proof of Fix

Before (67c7b97, port 48430)

After (dbadee7, port 42191)

Type

Caveats (if any)

Medium

Low

Final Attestation

Uh oh!

codecov Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

greptile-apps Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

Uh oh!

mateo-berri commented Aug 25, 2026

Uh oh!

mateo-berri commented Aug 25, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mateo-berri commented Aug 26, 2026

Uh oh!

mateo-berri commented Aug 26, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

codspeed-hq Bot commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

mateo-berri commented Aug 27, 2026

Uh oh!

mateo-berri commented Aug 27, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mateo-berri commented Aug 25, 2026 •

edited

Loading

Before (`67c7b97`, port 48430)

After (`dbadee7`, port 42191)

codecov Bot commented Aug 25, 2026 •

edited

Loading

greptile-apps Bot commented Aug 25, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

codspeed-hq Bot commented Aug 26, 2026 •

edited

Loading