Repository navigation
fix(cost): bill OpenRouter and Perplexity v1.1 decisions models - #45476
devin-ai-integration[bot] wants to merge 2 commits into
Conversation
Every decisions model OpenRouter serves except typesafe/jev-1.13, and Perplexity's pplx-decider-v1.1-27b, had no cost map row, so /v1/decisions and /v1/systemone calls through them returned no x-litellm-response-cost header and logged $0 spend. Add the 15 openrouter/ rows from OpenRouter's decisions catalog (prompt price, free output, max_input_tokens from context_length) and the perplexity/pplx-decider-v1.1-27b row from Perplexity's pricing page, which also lists v1 at $0.02/1M, so that row moves from $0.04/1M to $0.02/1M. The Together size-bucket branch of the cost calculator rewrote any model whose name contains "togethercomputer" to a bucket name even when the model had its own cost map row, which left openrouter/togethercomputer/tev1-4b-experimental unpriced; the guard now honours the model's own row too.
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 939ce3f. Configure here.
litellm-stampbot: risk lowApproved at Rules (
Judge (gpt-6-luna): low, probability of high 0.69 (high at or above 0.90)
Pushes do not re-run this check. Comment |
litellm-stampbot
Bot
added
risk:low
litellm-stampbot
left a comment
litellm-stampbot
left a comment
There was a problem hiding this comment.
Low risk by rule and by judgement at 939ce3f
This branch has not been deployed
2 participants
TLDR
Problem this solves:
pplx-decider-v1.1-27bhas none eitherx-litellm-response-costand log $0 spendopenrouter/togethercomputer/...is re-priced as a Together size bucketHow it solves it:
openrouter/rows from OpenRouter's decisions catalog (read 2026-10-08)perplexity/pplx-decider-v1.1-27bfrom Perplexity's pricing page (read 2026-10-08)perplexity/pplx-decider-v1-27bto the same page's $0.02/1M inputIntentional product change:
perplexity/pplx-decider-v1-27bbills $0.02/1M input instead of $0.04/1M, matching Perplexity's current pricing page (the page lists v1 and v1.1 at the same price), so spend on that model halvesUser Flow
Before: a team routing classification calls to an OpenRouter or Perplexity decisions model sees every one of them logged at $0, so budgets never move
openrouter/openai/gpt-6-luna-decisions(orperplexity/pplx-decider-v1.1-27b) to config.yaml and restarts the proxyusage.input_tokens: 378, but nox-litellm-response-costheaderspend: 0.0After: the same calls bill at the vendor's price, so the team's keys and budgets reflect them
openrouter/openai/gpt-6-luna-decisions(orperplexity/pplx-decider-v1.1-27b) to config.yaml and restarts the proxyusage.input_tokens: 378, andx-litellm-response-cost: 3.78e-05(378 tokens at $0.10/1M)spend: 3.78e-05Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Last updated: 939ce3f. QA and /live-pr-risk ran on c71b972, the last commit with a product diff
Both legs boot the same config on the same Postgres with two uvicorn workers, real OpenRouter and Perplexity keys, and
LITELLM_LOCAL_MODEL_COST_MAP=Trueso the proxy prices from the checkout's cost map. Before runs the merge base, After runs this PR's tip. Each case sends the same request through/v1/decisions(OpenAI shape) and/v1/systemone(TypeSafe shape), then reads the request back throughGET /spend/logs.openrouter/typesafe/jev-1.13already had a row and is the control: it must bill the same on both legsShared setup:
The response bodies below are trimmed to
id,model, andusage; the answers to the three questions came back on every callBefore (b1eae88)
openrouter-gpt-6-luna-decisions ->
openrouter/openai/gpt-6-luna-decisionscurl -s -D - -X POST http://127.0.0.1:36454/v1/decisions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d "$(oai openrouter-gpt-6-luna-decisions)" | tee /dev/stderr | jq -c '{id, model, usage}'curl -s "http://127.0.0.1:36454/spend/logs?request_id=2dd39dca-d1ec-4f03-9bf7-06af9da9c0b6" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c '.[] | {request_id, model, prompt_tokens, completion_tokens, spend}'curl -s -D - -X POST http://127.0.0.1:36454/v1/systemone -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d "$(s1 openrouter-gpt-6-luna-decisions)" | tee /dev/stderr | jq -c '{id, model, usage}'curl -s "http://127.0.0.1:36454/spend/logs?request_id=gen-dec-1791507579-W47sBWX3V38wxK7PtPwB" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c '.[] | {request_id, model, prompt_tokens, completion_tokens, spend}'openrouter-pplx-decider ->
openrouter/perplexity/pplx-decider-v1.1-27bcurl -s -D - -X POST http://127.0.0.1:36454/v1/decisions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d "$(oai openrouter-pplx-decider)" | tee /dev/stderr | jq -c '{id, model, usage}'curl -s "http://127.0.0.1:36454/spend/logs?request_id=a9cc5cd8-fa52-40ac-a0fb-cf7d1fa70baa" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c '.[] | {request_id, model, prompt_tokens, completion_tokens, spend}'curl -s -D - -X POST http://127.0.0.1:36454/v1/systemone -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d "$(s1 openrouter-pplx-decider)" | tee /dev/stderr | jq -c '{id, model, usage}'curl -s "http://127.0.0.1:36454/spend/logs?request_id=gen-dec-1791507591-dSacfBpPQ3Sb4aYQowID" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c '.[] | {request_id, model, prompt_tokens, completion_tokens, spend}'openrouter-clef-flash ->
openrouter/cloudflare/clef-flashcurl -s -D - -X POST http://127.0.0.1:36454/v1/decisions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d "$(oai openrouter-clef-flash)" | tee /dev/stderr | jq -c '{id, model, usage}'curl -s "http://127.0.0.1:36454/spend/logs?request_id=436b03ab-b206-441f-bba0-b4670050a359" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c '.[] | {request_id, model, prompt_tokens, completion_tokens, spend}'curl -s -D - -X POST http://127.0.0.1:36454/v1/systemone -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d "$(s1 openrouter-clef-flash)" | tee /dev/stderr | jq -c '{id, model, usage}'curl -s "http://127.0.0.1:36454/spend/logs?request_id=gen-dec-1791507595-3xcXFDYigXxMb3ngVR7L" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c '.[] | {request_id, model, prompt_tokens, completion_tokens, spend}'perplexity-decider-v1-1 ->
perplexity/pplx-decider-v1.1-27bcurl -s -D - -X POST http://127.0.0.1:36454/v1/decisions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d "$(oai perplexity-decider-v1-1)" | tee /dev/stderr | jq -c '{id, model, usage}'curl -s "http://127.0.0.1:36454/spend/logs?request_id=bf1af6d0-d999-46bd-905b-572c73335d30" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c '.[] | {request_id, model, prompt_tokens, completion_tokens, spend}'curl -s -D - -X POST http://127.0.0.1:36454/v1/systemone -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d "$(s1 perplexity-decider-v1-1)" | tee /dev/stderr | jq -c '{id, model, usage}'curl -s "http://127.0.0.1:36454/spend/logs?request_id=dbb281d1-b45a-465d-8b89-78abfe967c9e" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c '.[] | {request_id, model, prompt_tokens, completion_tokens, spend}'openrouter-jev ->
openrouter/typesafe/jev-1.13 (control, priced before this PR)curl -s -D - -X POST http://127.0.0.1:36454/v1/decisions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d "$(oai openrouter-jev)" | tee /dev/stderr | jq -c '{id, model, usage}'curl -s "http://127.0.0.1:36454/spend/logs?request_id=b1820e8c-3b76-4a64-8497-ec652dd8e88f" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c '.[] | {request_id, model, prompt_tokens, completion_tokens, spend}'curl -s -D - -X POST http://127.0.0.1:36454/v1/systemone -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d "$(s1 openrouter-jev)" | tee /dev/stderr | jq -c '{id, model, usage}'curl -s "http://127.0.0.1:36454/spend/logs?request_id=gen-dec-1791507602-hFKU15G3hYbjX3jyppea" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c '.[] | {request_id, model, prompt_tokens, completion_tokens, spend}'After (c71b972)
openrouter-gpt-6-luna-decisions ->
openrouter/openai/gpt-6-luna-decisionscurl -s -D - -X POST http://127.0.0.1:36454/v1/decisions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d "$(oai openrouter-gpt-6-luna-decisions)" | tee /dev/stderr | jq -c '{id, model, usage}'curl -s "http://127.0.0.1:36454/spend/logs?request_id=7d4673cd-533b-4c4d-9187-38421b7998b2" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c '.[] | {request_id, model, prompt_tokens, completion_tokens, spend}'curl -s -D - -X POST http://127.0.0.1:36454/v1/systemone -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d "$(s1 openrouter-gpt-6-luna-decisions)" | tee /dev/stderr | jq -c '{id, model, usage}'curl -s "http://127.0.0.1:36454/spend/logs?request_id=gen-dec-1791507930-nPIH11Qv6Q5zQm95QBd7" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c '.[] | {request_id, model, prompt_tokens, completion_tokens, spend}'openrouter-pplx-decider ->
openrouter/perplexity/pplx-decider-v1.1-27bcurl -s -D - -X POST http://127.0.0.1:36454/v1/decisions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d "$(oai openrouter-pplx-decider)" | tee /dev/stderr | jq -c '{id, model, usage}'curl -s "http://127.0.0.1:36454/spend/logs?request_id=0b6e6376-6045-432a-a757-c6a90a9037b6" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c '.[] | {request_id, model, prompt_tokens, completion_tokens, spend}'curl -s -D - -X POST http://127.0.0.1:36454/v1/systemone -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d "$(s1 openrouter-pplx-decider)" | tee /dev/stderr | jq -c '{id, model, usage}'curl -s "http://127.0.0.1:36454/spend/logs?request_id=gen-dec-1791507932-oNdzE1sb6pDqefO0JZeK" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c '.[] | {request_id, model, prompt_tokens, completion_tokens, spend}'openrouter-clef-flash ->
openrouter/cloudflare/clef-flashcurl -s -D - -X POST http://127.0.0.1:36454/v1/decisions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d "$(oai openrouter-clef-flash)" | tee /dev/stderr | jq -c '{id, model, usage}'curl -s "http://127.0.0.1:36454/spend/logs?request_id=d49c1ef0-986c-4b14-825e-d2aff436c079" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c '.[] | {request_id, model, prompt_tokens, completion_tokens, spend}'curl -s -D - -X POST http://127.0.0.1:36454/v1/systemone -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d "$(s1 openrouter-clef-flash)" | tee /dev/stderr | jq -c '{id, model, usage}'curl -s "http://127.0.0.1:36454/spend/logs?request_id=gen-dec-1791507934-N8F740szw0kbf3i70yDD" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c '.[] | {request_id, model, prompt_tokens, completion_tokens, spend}'perplexity-decider-v1-1 ->
perplexity/pplx-decider-v1.1-27bcurl -s -D - -X POST http://127.0.0.1:36454/v1/decisions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d "$(oai perplexity-decider-v1-1)" | tee /dev/stderr | jq -c '{id, model, usage}'curl -s "http://127.0.0.1:36454/spend/logs?request_id=a1d6c7a4-8373-4ba8-82c6-3e438e31fdbe" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c '.[] | {request_id, model, prompt_tokens, completion_tokens, spend}'curl -s -D - -X POST http://127.0.0.1:36454/v1/systemone -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d "$(s1 perplexity-decider-v1-1)" | tee /dev/stderr | jq -c '{id, model, usage}'curl -s "http://127.0.0.1:36454/spend/logs?request_id=62289c4e-7a57-4cb8-b5c5-453ccf41a59e" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c '.[] | {request_id, model, prompt_tokens, completion_tokens, spend}'openrouter-jev ->
openrouter/typesafe/jev-1.13 (control, priced before this PR)curl -s -D - -X POST http://127.0.0.1:36454/v1/decisions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d "$(oai openrouter-jev)" | tee /dev/stderr | jq -c '{id, model, usage}'curl -s "http://127.0.0.1:36454/spend/logs?request_id=85ad7fb7-95b8-4b07-aa19-8d87695dff96" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c '.[] | {request_id, model, prompt_tokens, completion_tokens, spend}'curl -s -D - -X POST http://127.0.0.1:36454/v1/systemone -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d "$(s1 openrouter-jev)" | tee /dev/stderr | jq -c '{id, model, usage}'curl -s "http://127.0.0.1:36454/spend/logs?request_id=gen-dec-1791507941-hje3LsaH3b0BULgOmHIL" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c '.[] | {request_id, model, prompt_tokens, completion_tokens, spend}'Observations from the legs:
/v1/systemonebody carriesusage.cost; Before still logged $0. Fixed hererequest_idis OpenRouter'sgen-dec-...id on/v1/systemone. Unchangedopenrouter/typesafe/jev-1.13bills 1.5456e-05 on both legs/live-pr-risk (c71b972 on main 79c3de4, merged tree 9aecc5355d)
Rig: base is main 79c3de4, head is main merged with c71b972, two uvicorn workers each on a shared Postgres, real OpenRouter, Perplexity, and Together keys, and a logging forwarder in front of each vendor on both legs so the outbound requests could be diffed. main had moved 24 commits past the branch point (6 touch the cost map, #45469 among them); the merge is clean with no duplicate keys
Verdict: no breaking change, one intentional backward-incompatible price change named in the TLDR
Breaking: none. Every
/v1/decisions,/v1/systemone, and Together/v1/chat/completionscell answered 200 on both legs with byte-identical recorded outbound requests (11 OpenRouter, 5 Perplexity, 2 Together per leg, one request per cell, same keys and sizes). The controlopenrouter/typesafe/jev-1.13bills 1.5456e-05 on both legs;together_ai/Qwen/Qwen3.5-9Bbills 3.97e-06 andtogether_ai/together/Tev1-4B-experimental7.56e-07 on bothBackward incompatible:
perplexity/pplx-decider-v1-27bbills 2e-08 per input token instead of 4e-08 (the same 298-token call: base 1.192e-05, head 5.96e-06). The TLDR names it and the approval on this PR is its sign-offChanges the PR intends, observed: the new rows bill (base 0.0; head
openrouter/openai/gpt-6-luna-decisions3.78e-05,openrouter/perplexity/pplx-decider-v1.1-27b5.96e-06,perplexity/pplx-decider-v1.1-27b5.96e-06,openrouter/cloudflare/clef-flash6.027e-06,openrouter/togethercomputer/tev1-4b-experimental1.3356e-05, which is 318 x 4.2e-08 and no longer a Together size bucket),GET /model/infoandGET /model_group/infocarrymode, the costs, andmax_input_tokensfor them (null before),GET /health?model=openrouter-tev1-4bgoes from 503 (a chat probe OpenRouter rejects) to 200 (a decisions probe), andPUT /lens/signalsaccepts the model (400 "Choose a System 1 model" before)Regression risk: the Together size-bucket path (a
together_ai/model with no cost map row) could not run live, since every chat model Together serves serverless today has a row.test_completion_cost_prefers_a_direct_row_over_the_together_size_bucketcovers it and an SDK probe on both trees pricestogether_ai/togethercomputer/CodeLlama-34b-Instructidentically, since the guard reads the same key twice for atogether_ai/-prefixed nameDependency graph: cost map rows read by
completion_costthrough/v1/decisions,/v1/systemone, and/spend/logs(verified live), by/model/infoand/model_group/info(verified live), by the/healthmode dispatch (verified live), by LensPUT /lens/signals(verified live), and byget_model_infoin the SDK (tested);has_together_registry_pricinghas one caller incost_calculator.py(tested and verified live); the barepplx-decider-v1.1-27bname resolves to the Perplexity provider identically on both trees (SDK probe)Not verified: the Together size-bucket path live (no serverless Together model without a row exists today); the Admin UI models page render (its data route
/model/infowas verified)Type
🐛 Bug Fix
Caveats (if any)
Low
Each row below stays as is because fixing it costs more than it buys; the judgment follows the caveat
mode: evaluationtodecisions) closed unmerged, so the 16 rows keepevaluation, the mode/healthand Lens read today. Switching them to a mode nothing reads would break both for these modelsclef-flash2.1e-08 vs Cloudflare's 9e-08). OpenRouter is who bills the call, so its price is the spend; the vendor's would overbillmax_input_tokenscomes from the catalog'scontext_length; the threerespan/rows omit it because the catalog says 0. Inventing a limit the catalog does not publish would reject valid promptsmax_output_tokens. The catalog'stop_provider.max_completion_tokens(8 fortev1-4b, 945000 forgpt-6-luna) was left out: the decisions API takes no output cap and no decisions code path reads the field, so pinning it only changes/model/info, and a product commit for that re-runs the whole proof loopopenrouter/~typesafe/jev-latestkeeps the catalog's tilde id, like the 18 existingopenrouter/~keys. It is the id OpenRouter routes by, so a cleaner key would never match a requestopenrouter/typesafe/jev-1.13row is untouched (max_input_tokens32000 vs catalog 64000,max_output_tokens28800 vs 57600). It bills correctly today and widening its limits is a separate one-row change outside this billing fixtogethercomputermodel name with its own row. That is the intended behavior, andopenrouter/togethercomputer/tev1-4b-experimentalis the only such key in the mapFinal Attestation