Skip to content

feat(spend): add net auto-router savings to the cost-optimization dashboard #35402

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
tin-berri wants to merge 24 commits into from
Closed

feat(spend): add net auto-router savings to the cost-optimization dashboard #35402

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
e30b7f2
feat(spend): add net auto-router savings to the cost-optimization das…
tin-berri Jul 31, 2026
4897f6e
fix(spend): price auto-router savings through litellm's cost engine a…
tin-berri Jul 31, 2026
0b9c503
test(spend): cover the savings fail-open paths
tin-berri Jul 31, 2026
eca16fa
refactor(spend): trim savings docstrings to what the code cannot say
tin-berri Jul 31, 2026
db15ef3
fix(spend): charge the cold-cache write to the model switch that caus…
tin-berri Aug 1, 2026
c46e96a
fix(spend): compare and price auto-router models as resolved identities
tin-berri Aug 1, 2026
59d66e4
feat(spend): derive the savings baseline from the router's own candid…
tin-berri Aug 1, 2026
e2ffae8
fix(ui): say what the auto-router savings are measured against
tin-berri Aug 1, 2026
6843ad0
fix(router): price baseline candidates under the provider their deplo…
tin-berri Aug 1, 2026
52ca688
refactor(spend): let the types carry the facts the tests already assert
tin-berri Aug 1, 2026
c507283
fix(router): keep the savings baseline out of the outbound provider r…
tin-berri Aug 1, 2026
e12415a
fix(router): derive the savings baseline per call instead of caching it
tin-berri Aug 1, 2026
e9b4f95
fix(ui): stop the savings legend and tab control moving between tabs
tin-berri Aug 1, 2026
669fd84
fix(ui): keep the savings header one shape across both tabs
tin-berri Aug 1, 2026
f66687b
fix(ui): lay the savings header out with the card's own slots
tin-berri Aug 1, 2026
d539fa6
fix(spend): charge the cold-cache write on the request that actually …
tin-berri Aug 1, 2026
68c56fe
feat(spend): give the complexity router a savings baseline from its h…
tin-berri Aug 1, 2026
11c6688
fix(router): price baseline candidates by base_model, not the deploym…
tin-berri Aug 1, 2026
6cf689d
chore: drop the built dashboard bundle from this branch
tin-berri Aug 1, 2026
5457640
chore: restore the built dashboard bundle to the branch point
tin-berri Aug 1, 2026
d0b59af
fix(spend): price the baseline on the whole request, not just its cac…
tin-berri Aug 1, 2026
bfe3413
docs(ui): say on the card that a cold first turn is undercounted
tin-berri Aug 1, 2026
10a2a29
Merge remote-tracking branch 'origin/litellm_internal_staging' into l…
tin-berri Aug 1, 2026
5410306
style: bring the savings changes under the strict and type-discipline…
tin-berri Aug 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions ...litellm_proxy_extras/migrations/20260731000000_add_autorouter_savings_spend/migration.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
-- AlterTable
ALTER TABLE "LiteLLM_DailyUserSpend" ADD COLUMN IF NOT EXISTS "autorouter_savings_spend" DOUBLE PRECISION NOT NULL DEFAULT 0.0;

-- AlterTable
ALTER TABLE "LiteLLM_DailyOrganizationSpend" ADD COLUMN IF NOT EXISTS "autorouter_savings_spend" DOUBLE PRECISION NOT NULL DEFAULT 0.0;

-- AlterTable
ALTER TABLE "LiteLLM_DailyEndUserSpend" ADD COLUMN IF NOT EXISTS "autorouter_savings_spend" DOUBLE PRECISION NOT NULL DEFAULT 0.0;

-- AlterTable
ALTER TABLE "LiteLLM_DailyAgentSpend" ADD COLUMN IF NOT EXISTS "autorouter_savings_spend" DOUBLE PRECISION NOT NULL DEFAULT 0.0;

-- AlterTable
ALTER TABLE "LiteLLM_DailyTeamSpend" ADD COLUMN IF NOT EXISTS "autorouter_savings_spend" DOUBLE PRECISION NOT NULL DEFAULT 0.0;

-- AlterTable
ALTER TABLE "LiteLLM_DailyTagSpend" ADD COLUMN IF NOT EXISTS "autorouter_savings_spend" DOUBLE PRECISION NOT NULL DEFAULT 0.0;
6 changes: 6 additions & 0 deletions litellm-proxy-extras/litellm_proxy_extras/schema.prisma
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -748,6 +748,7 @@ model LiteLLM_DailyUserSpend {
compression_saved_tokens BigInt @default(0)
compression_savings_spend Float @default(0.0)
prompt_caching_savings_spend Float @default(0.0)
autorouter_savings_spend Float @default(0.0)
spend Float @default(0.0)
api_requests BigInt @default(0)
successful_requests BigInt @default(0)
Expand Down Expand Up @@ -782,6 +783,7 @@ model LiteLLM_DailyOrganizationSpend {
compression_saved_tokens BigInt @default(0)
compression_savings_spend Float @default(0.0)
prompt_caching_savings_spend Float @default(0.0)
autorouter_savings_spend Float @default(0.0)
spend Float @default(0.0)
api_requests BigInt @default(0)
successful_requests BigInt @default(0)
Expand Down Expand Up @@ -816,6 +818,7 @@ model LiteLLM_DailyEndUserSpend {
compression_saved_tokens BigInt @default(0)
compression_savings_spend Float @default(0.0)
prompt_caching_savings_spend Float @default(0.0)
autorouter_savings_spend Float @default(0.0)
spend Float @default(0.0)
api_requests BigInt @default(0)
successful_requests BigInt @default(0)
Expand Down Expand Up @@ -849,6 +852,7 @@ model LiteLLM_DailyAgentSpend {
compression_saved_tokens BigInt @default(0)
compression_savings_spend Float @default(0.0)
prompt_caching_savings_spend Float @default(0.0)
autorouter_savings_spend Float @default(0.0)
spend Float @default(0.0)
api_requests BigInt @default(0)
successful_requests BigInt @default(0)
Expand Down Expand Up @@ -882,6 +886,7 @@ model LiteLLM_DailyTeamSpend {
compression_saved_tokens BigInt @default(0)
compression_savings_spend Float @default(0.0)
prompt_caching_savings_spend Float @default(0.0)
autorouter_savings_spend Float @default(0.0)
spend Float @default(0.0)
api_requests BigInt @default(0)
successful_requests BigInt @default(0)
Expand Down Expand Up @@ -917,6 +922,7 @@ model LiteLLM_DailyTagSpend {
compression_saved_tokens BigInt @default(0)
compression_savings_spend Float @default(0.0)
prompt_caching_savings_spend Float @default(0.0)
autorouter_savings_spend Float @default(0.0)
spend Float @default(0.0)
api_requests BigInt @default(0)
successful_requests BigInt @default(0)
Expand Down
8 changes: 7 additions & 1 deletion litellm/proxy/_types.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
field_validator,
model_validator,
)
from typing_extensions import Required, TypedDict
from typing_extensions import NotRequired, Required, TypedDict

from litellm._uuid import uuid
from litellm.constants import MCP_STDIO_ALLOWED_COMMANDS
Expand Down Expand Up @@ -3327,6 +3327,7 @@ class SpendLogsMetadata(TypedDict):
max_retries: Optional[int] # Max retries configured for this request
cost_breakdown: Optional[CostBreakdown] # Detailed cost breakdown (input_cost, output_cost, margin, discount, etc.)
compression_savings: CompressionSavingsMetadata | None
auto_router_savings_baseline_model: str | None # counterfactual model for the auto-router savings driver


class SpendLogsPayload(TypedDict):
Expand Down Expand Up @@ -4565,6 +4566,11 @@ class BaseDailySpendTransaction(TypedDict):
# cost-savings metrics (dollars, priced per request before aggregation)
compression_savings_spend: float
prompt_caching_savings_spend: float
# Not required: rows queued by a pod running the previous release, or replayed from
# the Redis buffer across an upgrade, carry no such key. Every reader coalesces a
# missing value to zero, so requiring it here would describe a shape the aggregation
# is explicitly tested against.
autorouter_savings_spend: NotRequired[float]

# request level metrics
spend: float
Expand Down
73 changes: 29 additions & 44 deletions litellm/proxy/db/db_spend_update_writer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1556,6 +1556,30 @@ async def _update_daily_spend(
# Get the table dynamically
table = getattr(batcher, table_name)

# Additive metrics that older queued rows may omit; one
# enumeration feeds both the create and the increment below
optional_metrics = {
field: value
for field, value in (
("cache_read_input_tokens", transaction.get("cache_read_input_tokens")),
(
"cache_creation_input_tokens",
transaction.get("cache_creation_input_tokens"),
),
("compression_saved_tokens", transaction.get("compression_saved_tokens")),
(
"compression_savings_spend",
transaction.get("compression_savings_spend"),
),
(
"prompt_caching_savings_spend",
transaction.get("prompt_caching_savings_spend"),
),
("autorouter_savings_spend", transaction.get("autorouter_savings_spend")),
)
if value is not None
}

# Common data structure for both create and update
common_data = {
entity_id_field: entity_id,
Expand All @@ -1572,30 +1596,9 @@ async def _update_daily_spend(
"api_requests": transaction["api_requests"],
"successful_requests": transaction["successful_requests"],
"failed_requests": transaction["failed_requests"],
**optional_metrics,
}

# Add cache-related fields if they exist
if "cache_read_input_tokens" in transaction:
common_data["cache_read_input_tokens"] = transaction.get(
"cache_read_input_tokens", 0
)
if "cache_creation_input_tokens" in transaction:
common_data["cache_creation_input_tokens"] = transaction.get(
"cache_creation_input_tokens", 0
)
if "compression_saved_tokens" in transaction:
common_data["compression_saved_tokens"] = transaction.get(
"compression_saved_tokens", 0
)
if "compression_savings_spend" in transaction:
common_data["compression_savings_spend"] = transaction.get(
"compression_savings_spend", 0
)
if "prompt_caching_savings_spend" in transaction:
common_data["prompt_caching_savings_spend"] = transaction.get(
"prompt_caching_savings_spend", 0
)

if entity_type == "tag" and "request_id" in transaction:
common_data["request_id"] = transaction.get("request_id")

Expand All @@ -1607,30 +1610,9 @@ async def _update_daily_spend(
"api_requests": {"increment": transaction["api_requests"]},
"successful_requests": {"increment": transaction["successful_requests"]},
"failed_requests": {"increment": transaction["failed_requests"]},
**{field: {"increment": value} for field, value in optional_metrics.items()},
}

# Add cache-related fields to update if they exist
if "cache_read_input_tokens" in transaction:
update_data["cache_read_input_tokens"] = {
"increment": transaction.get("cache_read_input_tokens", 0)
}
if "cache_creation_input_tokens" in transaction:
update_data["cache_creation_input_tokens"] = {
"increment": transaction.get("cache_creation_input_tokens", 0)
}
if "compression_saved_tokens" in transaction:
update_data["compression_saved_tokens"] = {
"increment": transaction.get("compression_saved_tokens", 0)
}
if "compression_savings_spend" in transaction:
update_data["compression_savings_spend"] = {
"increment": transaction.get("compression_savings_spend", 0)
}
if "prompt_caching_savings_spend" in transaction:
update_data["prompt_caching_savings_spend"] = {
"increment": transaction.get("prompt_caching_savings_spend", 0)
}

if entity_type == "tag" and "request_id" in transaction:
update_data["request_id"] = transaction.get("request_id")

Expand Down Expand Up @@ -1886,6 +1868,8 @@ async def _common_add_spend_log_transaction_to_daily_transaction(
custom_llm_provider=payload.get("custom_llm_provider", None),
compression_saved_tokens=compression_saved_tokens,
cache_read_input_tokens=cache_read_input_tokens,
baseline_model=_metadata.get("auto_router_savings_baseline_model"),
usage_object=usage_obj,
Comment thread
cursor[bot] marked this conversation as resolved.
)

daily_transaction = BaseDailySpendTransaction(
Expand All @@ -1907,6 +1891,7 @@ async def _common_add_spend_log_transaction_to_daily_transaction(
compression_saved_tokens=compression_saved_tokens,
compression_savings_spend=savings_spend.compression,
prompt_caching_savings_spend=savings_spend.prompt_caching,
autorouter_savings_spend=savings_spend.autorouter,
Comment thread
greptile-apps[bot] marked this conversation as resolved.
)
return daily_transaction
except Exception as e:
Expand Down
4 changes: 4 additions & 0 deletions litellm/proxy/db/db_transaction_queue/daily_spend_update_queue.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,10 @@ def get_aggregated_daily_spend_update_transactions(
payload.get("prompt_caching_savings_spend", 0) or 0
) + daily_transaction.get("prompt_caching_savings_spend", 0)

daily_transaction["autorouter_savings_spend"] = (
payload.get("autorouter_savings_spend", 0) or 0
) + daily_transaction.get("autorouter_savings_spend", 0)

else:
aggregated_daily_spend_update_transactions[_key] = deepcopy(payload)
return aggregated_daily_spend_update_transactions
Expand Down
1 change: 1 addition & 0 deletions litellm/proxy/litellm_pre_call_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,7 @@ def parse_cache_control(cache_control):
"applied_policies",
"policy_sources",
"routing_decision",
"auto_router_savings_baseline_model",
INTERNAL_CALL_ORIGIN_METADATA_KEY,
"standard_logging_object",
"proxy_server_request",
Expand Down
9 changes: 9 additions & 0 deletions litellm/proxy/management_endpoints/common_daily_activity.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,9 @@ def compression_savings_spend(self) -> float: ...
@property
def prompt_caching_savings_spend(self) -> float: ...

@property
def autorouter_savings_spend(self) -> float: ...

@property
def api_requests(self) -> int: ...

Expand Down Expand Up @@ -135,6 +138,7 @@ class _GroupingSetsRow(SimpleNamespace):
compression_saved_tokens: int | None
compression_savings_spend: float | None
prompt_caching_savings_spend: float | None
autorouter_savings_spend: float | None
api_requests: int | None
successful_requests: int | None
failed_requests: int | None
Expand All @@ -158,6 +162,7 @@ def update_metrics(existing_metrics: SpendMetrics, record: DailySpendRecord) ->
existing_metrics.compression_saved_tokens += record.compression_saved_tokens or 0
existing_metrics.compression_savings_spend += record.compression_savings_spend or 0
existing_metrics.prompt_caching_savings_spend += record.prompt_caching_savings_spend or 0
existing_metrics.autorouter_savings_spend += record.autorouter_savings_spend or 0
existing_metrics.api_requests += record.api_requests or 0
existing_metrics.successful_requests += record.successful_requests or 0
existing_metrics.failed_requests += record.failed_requests or 0
Expand Down Expand Up @@ -590,6 +595,7 @@ def _build_aggregated_sql_query(
SUM(compression_saved_tokens)::bigint AS compression_saved_tokens,
SUM(compression_savings_spend)::float AS compression_savings_spend,
SUM(prompt_caching_savings_spend)::float AS prompt_caching_savings_spend,
SUM(autorouter_savings_spend)::float AS autorouter_savings_spend,
SUM(api_requests)::bigint AS api_requests,
SUM(successful_requests)::bigint AS successful_requests,
SUM(failed_requests)::bigint AS failed_requests
Expand Down Expand Up @@ -732,6 +738,7 @@ def _record_to_spend_metrics(record: _GroupingSetsRow) -> SpendMetrics:
compression_saved_tokens=record.compression_saved_tokens or 0,
compression_savings_spend=record.compression_savings_spend or 0,
prompt_caching_savings_spend=record.prompt_caching_savings_spend or 0,
autorouter_savings_spend=record.autorouter_savings_spend or 0,
api_requests=record.api_requests or 0,
successful_requests=record.successful_requests or 0,
failed_requests=record.failed_requests or 0,
Expand Down Expand Up @@ -986,6 +993,7 @@ async def get_daily_activity(
total_compression_saved_tokens=metadata_metrics.compression_saved_tokens,
total_compression_savings_spend=metadata_metrics.compression_savings_spend,
total_prompt_caching_savings_spend=metadata_metrics.prompt_caching_savings_spend,
total_autorouter_savings_spend=metadata_metrics.autorouter_savings_spend,
page=page,
total_pages=-(-total_count // page_size), # Ceiling division
has_more=(page * page_size) < total_count,
Expand Down Expand Up @@ -1075,6 +1083,7 @@ async def get_daily_activity_aggregated(
total_compression_saved_tokens=aggregated["totals"].compression_saved_tokens,
total_compression_savings_spend=aggregated["totals"].compression_savings_spend,
total_prompt_caching_savings_spend=aggregated["totals"].prompt_caching_savings_spend,
total_autorouter_savings_spend=aggregated["totals"].autorouter_savings_spend,
page=1,
total_pages=1,
has_more=False,
Expand Down
6 changes: 6 additions & 0 deletions litellm/proxy/schema.prisma
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -748,6 +748,7 @@ model LiteLLM_DailyUserSpend {
compression_saved_tokens BigInt @default(0)
compression_savings_spend Float @default(0.0)
prompt_caching_savings_spend Float @default(0.0)
autorouter_savings_spend Float @default(0.0)
spend Float @default(0.0)
api_requests BigInt @default(0)
successful_requests BigInt @default(0)
Expand Down Expand Up @@ -782,6 +783,7 @@ model LiteLLM_DailyOrganizationSpend {
compression_saved_tokens BigInt @default(0)
compression_savings_spend Float @default(0.0)
prompt_caching_savings_spend Float @default(0.0)
autorouter_savings_spend Float @default(0.0)
spend Float @default(0.0)
api_requests BigInt @default(0)
successful_requests BigInt @default(0)
Expand Down Expand Up @@ -816,6 +818,7 @@ model LiteLLM_DailyEndUserSpend {
compression_saved_tokens BigInt @default(0)
compression_savings_spend Float @default(0.0)
prompt_caching_savings_spend Float @default(0.0)
autorouter_savings_spend Float @default(0.0)
spend Float @default(0.0)
api_requests BigInt @default(0)
successful_requests BigInt @default(0)
Expand Down Expand Up @@ -849,6 +852,7 @@ model LiteLLM_DailyAgentSpend {
compression_saved_tokens BigInt @default(0)
compression_savings_spend Float @default(0.0)
prompt_caching_savings_spend Float @default(0.0)
autorouter_savings_spend Float @default(0.0)
spend Float @default(0.0)
api_requests BigInt @default(0)
successful_requests BigInt @default(0)
Expand Down Expand Up @@ -882,6 +886,7 @@ model LiteLLM_DailyTeamSpend {
compression_saved_tokens BigInt @default(0)
compression_savings_spend Float @default(0.0)
prompt_caching_savings_spend Float @default(0.0)
autorouter_savings_spend Float @default(0.0)
spend Float @default(0.0)
api_requests BigInt @default(0)
successful_requests BigInt @default(0)
Expand Down Expand Up @@ -917,6 +922,7 @@ model LiteLLM_DailyTagSpend {
compression_saved_tokens BigInt @default(0)
compression_savings_spend Float @default(0.0)
prompt_caching_savings_spend Float @default(0.0)
autorouter_savings_spend Float @default(0.0)
spend Float @default(0.0)
api_requests BigInt @default(0)
successful_requests BigInt @default(0)
Expand Down
Loading
Loading