feat(spend): surface per-request auto-router savings to logging callbacks

[tin-berri]

TLDR

Problem this solves:

• Auto-router savings only reached the dashboard's daily rollup tables
• Logging callbacks (Datadog, GCS, S3) never saw the per-request figure
• Customers could not forward savings to their own cost systems

How it solves it:

• Compute the figure once when the logging payload is built
• Stamp it as autorouter_savings beside cost_breakdown on the payload
• Spend writers read the recorded value instead of recomputing
• Internal sub-calls and caller-forged values are never stamped

User Flow

Before: a platform team routing traffic through an auto-router cannot ship the reported savings into Datadog, because the per-request records their callbacks receive carry no savings figure

1. They send POST https://litellm-domain/v1/chat/completions with "model": "auto_router1"
2. The response comes back served by the cheaper routed model
3. Their Datadog callback receives the request record with the routing decision and the request cost, but no savings figure anywhere on it
4. The savings exist only on https://litellm-domain/ui/?page=cost-optimization and the /auto_router/benchmarks endpoint, which their pipeline does not read

After: the same request record now carries the savings figure, so their existing pipeline forwards it with zero changes

1. They send POST https://litellm-domain/v1/chat/completions with "model": "auto_router1"
2. The response comes back served by the cheaper routed model
3. Their Datadog callback receives the same record, now with "autorouter_savings": 0.0123931 alongside the routing decision and cost breakdown
4. The spend log row at https://litellm-domain/ui/?page=logs carries the same number, and the dashboard totals equal the sum of the per-request figures

Relevant issues

• Customer support ticket (docs) Update vertex.md old code example #7736: cost savings data is stored stand-alone, so existing callbacks cannot pick it up and push it to Datadog / archival / cost systems

Linear ticket

Resolves LIT-5973

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Rig: proxy on localhost:4620 with real Postgres, an auto-router (auto_router1, tiers nano/mini/5.4) over an instrumented upstream with explicit per-token prices, and a probe callback that appends every standard logging payload it receives to a JSONL file. Expected per-request figure at these prices: baseline gpt-5.4 minus routed gpt-5-nano on 4242 in / 137 out = 0.0123931

Before (9821b45)

/v1/chat/completions: callback payload has no savings field

1. for n in 1 2 3; do curl -s http://127.0.0.1:4620/v1/chat/completions -H "Authorization: Bearer sk-rig-1234" -d '{"model": "auto_router1", "messages": [{"role": "user", "content": "repro probe unique turn N"}]}'; done
2. Probe JSONL for each auto-routed request: {"model_group": "auto_router1", "routing_decision": {..., "savings_baseline_model": "openai/gpt-5.4", ...}, "savings_fields": {}}
3. Spend log metadata keys include routing_decision and cost_breakdown but no savings key: savings keys anywhere: ['compression_savings'] (null)

The figure exists, only in the rollup tables

1. psql -c 'select autorouter_savings_spend from "LiteLLM_DailyUserSpend"' returns 0.0371793 (3 x 0.0123931)
2. psql -c 'select sum(saved_spend) from "LiteLLM_AutoRouterSession"' returns 0.0371793

After (a6c67c1)

/v1/chat/completions: callback receives the figure

1. Same three curls as Before
2. Probe JSONL per auto-routed request: {"model_group": "auto_router1", "routing_decision_present": true, "savings_fields": {"autorouter_savings": 0.0123931}}
3. Spend log metadata per row: autorouter_savings: 0.0123931 | routed: gpt-5-nano
4. Rollup and turn table both advanced by exactly 3 x 0.0123931 (recorded value equals what the writer previously computed, same figure to the 7th digit)

/v1/messages

1. curl -s http://127.0.0.1:4620/v1/messages -H "Authorization: Bearer sk-rig-1234" -d '{"model": "auto_router1", "max_tokens": 100, "messages": [{"role": "user", "content": "messages surface probe"}]}'
2. Probe JSONL: {"call_type": "anthropic_messages", "model_group": "auto_router1", "savings": {"autorouter_savings": 0.0123931}}

/v1/responses

1. curl -s http://127.0.0.1:4620/v1/responses -H "Authorization: Bearer sk-rig-1234" -d '{"model": "auto_router1", "input": "responses surface probe"}'
2. Probe JSONL: {"call_type": "aresponses", "model_group": "auto_router1", "savings": {"autorouter_savings": 0.0123931}}
3. Spend log row: aresponses -> autorouter_savings: 0.0123931

Internal classifier calls and forged values stay excluded

1. Every classifier sub-call in the probe shows "savings_fields": {"autorouter_savings": null}
2. A request sent with "metadata": {"autorouter_savings": 999.0} lands in the spend log with the real computed 0.0123931, the forged value discarded
3. A direct non-routed request ("model": "gpt-5-mini") records autorouter_savings: None

Type

🆕 New Feature

Caveats (if any)

• Upstream was an instrumented stub with real map prices; savings math itself is unchanged and already live-verified on staging
• OpenTelemetry exports metadata.* keys only, so this top-level field needs a follow-up OTel attribute if wanted there
• Two pre-existing local test failures in test_datadog.py (missing DD_API_KEY env) fail identically at the merge base

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Medium Risk
Touches spend/cost reporting: a wrong stamp would misreport savings in callbacks and daily rollups. Safeguards exclude internal calls and caller-forged metadata.

Overview
Adds autorouter_savings to the standard logging payload so logging callbacks (Datadog, GCS, S3, etc.) receive the same per-request auto-router savings that previously lived only in daily rollups.

The figure is computed once when the payload is built (lazy proxy import so SDK-only installs skip it). Spend writers prefer that recorded value over recomputation so logs, turn tables, and rollups cannot disagree; older rows without the field still recompute. Internal classifier/shadow-eval calls are not stamped, and client-supplied autorouter_savings in request metadata is overwritten. None means not auto-routed; 0.0 is a real routed figure.

^{Reviewed by Cursor Bugbot for commit a6c67c1. Bugbot is set up for automated code reviews on this repo. Configure here.}

[tin-berri]

@greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

Score: 4/5

What works well:

• Compute-once, thread-through architecture is the right call. Computing at payload build time and reading the recorded value in spend writers eliminates the rollup/per-request disagreement that would have existed if each writer recomputed independently.
• Security properties are solid: internal sub-calls gated by INTERNAL_CALL_ORIGIN_METADATA_KEY, forged caller values discarded (test_caller_forged_autorouter_savings_is_discarded proves this), ReadOnly annotation signals intent to downstream code.
• Backward compatibility handled properly: rows written before the field shipped recompute via the same helper, so no migration needed.
• SDK-only safety: the lazy import in _autorouter_savings_for_payload with a broad except Exception is intentional and correct — savings must never fail request logging.
• Tests cover the invariants that matter: None vs 0.0 distinction, boolean rejection, recorded value surviving a corrupt usage object, internal call exclusion, fallback recomputation.

What holds it back from 5/5:

1. autorouter_savings is not in the create_dummy_standard_logging_payload comment or any schema sync target — the diff adds autorouter_savings=None to the dummy payload correctly, but if there is a schema.d.ts sync check in CI (the PR checklist has it unchecked), this field may need to be reflected there too.

2. _proxy_llm_router() imports the module-level llm_router variable at call time. If the proxy replaces its router after startup, calls during the transition window get the old instance. This is pre-existing behavior inherited from the spend writer path, but the new autorouter_savings_for_logging_payload code introduces a second call site for this pattern where the impact is slightly higher (logging payload is on the hot path).

3. The _get_spend_logs_metadata "early return" branch (when metadata is None) now stamps autorouter_savings into the returned dict, but the SpendLogsMetadata TypedDict has autorouter_savings as ReadOnly[float | None] — the value written is whatever the caller passes, not necessarily from the logging payload. This is fine because the callers in spend_tracking_utils.py thread the value through correctly, but it is a subtle dependency that is not enforced by the type system.

None of these are blockers — the core design is sound and the feature delivers exactly what the description promises.

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit a6c67c1. Configure here.}

[greptile-apps]

Greptile Summary

This PR computes auto-router savings while constructing the standard logging payload, exposes the value to callbacks, and carries the same recorded figure into spend persistence.

• Adds a per-request autorouter_savings field to standard logging and spend-log metadata.
• Centralizes auto-router savings calculation and lets database writers prefer the recorded value over recomputation.
• Excludes internal calls and overwrites caller-provided metadata values.
• Adds focused savings propagation, fallback, and forgery tests.

Confidence Score: 4/5

The PR appears safe to merge, with one non-blocking metadata-mutation cleanup needed to match repository conventions.

The savings value is consistently computed, propagated, and preferred by spend writers, while the only accepted concern is the mutable construction of the new metadata field.

Files Needing Attention: litellm/proxy/spend_tracking/spend_tracking_utils.py

Important Files Changed

Filename	Overview
litellm/litellm_core_utils/litellm_logging.py	Computes auto-router savings during standard payload construction and exposes the result as a callback field.
litellm/proxy/spend_tracking/savings.py	Extracts reusable per-request savings calculation and gives valid recorded values precedence during rollup calculation.
litellm/proxy/spend_tracking/spend_tracking_utils.py	Propagates the standard payload value into spend metadata, but the new assignment violates the repository's immutability guidance.
litellm/proxy/db/db_spend_update_writer.py	Passes recorded auto-router savings from spend metadata into both persistence calculation paths.
litellm/types/utils.py	Extends the standard logging callback contract with the optional-value auto-router savings field.
litellm/proxy/_types.py	Extends spend-log metadata typing with the recorded auto-router savings field.
tests/test_litellm/proxy/spend_tracking/test_savings.py	Covers recorded-value precedence, legacy recomputation, invalid values, and internal-call exclusion.
tests/test_litellm/proxy/spend_tracking/test_spend_tracking_utils.py	Verifies propagation into serialized spend metadata and rejection of caller-forged values.

_{Reviews (1): Last reviewed commit: "feat(spend): surface per-request auto-ro..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 85.41667% with 7 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/litellm_core_utils/litellm_logging.py	58.33%	5 Missing ⚠️
litellm/proxy/spend_tracking/savings.py	93.75%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[tin-berri]

Checked each of the three "holds it back from 5/5" items in #37894 (comment) against the repo and this PR's CI. Evidence below — none required a code change:

1. schema.d.ts sync — the check exists and already ran on this PR. .github/workflows/check-ui-api-types.yml ("Verify schema.d.ts matches the proxy OpenAPI spec") triggers on any litellm/proxy/ or litellm/types/ change — both are touched here — and it is green on a6c67c1 (run). The generated schema.d.ts derives from the proxy OpenAPI spec, which these logging TypedDicts do not alter, so there is nothing to reflect. (create_dummy_standard_logging_payload has no field-list comment to update; the diff already adds autorouter_savings=None to it, as you noted.)

2. _proxy_llm_router() call-time import — this is the codebase-wide established pattern, not a new risk: rg "from litellm.proxy.proxy_server import llm_router" finds 20+ function-scope call sites (prometheus.py, SlackAlerting/slack_alerting.py, spend_management_endpoints.py, llm_passthrough_endpoints.py x5, ...). A function-scope from ... import llm_router re-reads the module attribute on every call, so a router replaced after startup is picked up by the very next call; only a call already in flight keeps the instance it read — true of any reference, and harmless here because the router is used solely for _effective_model_info pricing lookups, which fall back to public map rates. Importing once at module load would be the actual bug (it would freeze the startup-time instance permanently).

3. Early-return provenance not type-enforced — this matches the field's design precedent exactly: cost_breakdown, litellm_overhead_time_ms, and litellm_call_id are threaded through _get_spend_logs_metadata the same way — extracted from the standard logging payload by the caller in get_logging_payload — and provenance is not type-enforced for any of them. For this field it is enforced behaviorally: test_caller_forged_autorouter_savings_is_discarded (tests/test_litellm/proxy/spend_tracking/test_spend_tracking_utils.py) fails if the stamped value ever comes from anywhere other than the logging payload. Passing the whole payload into _get_spend_logs_metadata instead would diverge from every sibling field's threading and grow the diff for no behavioral change.

The inline P2 on the clean_metadata assignment is answered in its own thread.

@greptileai given the above, please reply here with your updated confidence score and why.

[greptile-apps]

All three points hold up against the evidence:

1. schema.d.ts sync — confirmed green on the CI run linked. StandardLoggingPayload and SpendLogsMetadata are TypedDicts that don't feed the proxy OpenAPI spec, so the check passing is the correct outcome, not a miss.

2. _proxy_llm_router() call-time import — the 20+ existing call-time import sites (prometheus, slack alerting, passthrough endpoints) establish this as the repo's deliberate pattern for reading a module-level singleton that can be replaced after startup. A module-load import would be the bug here. No concern.

3. Provenance — ReadOnly[float | None] on SpendLogsMetadata signals intent at the type level (same as I'd want), and the behavioral guarantee is in test_caller_forged_autorouter_savings_is_discarded, which fails if get_logging_payload ever lets a client-supplied value through. That's the right place to enforce it. Threading the whole payload into _get_spend_logs_metadata for this one field would diverge from every sibling field and add diff for no behavioral change.

The implementation is defensive throughout (None on any exception in the savings path, _numeric_savings rejecting booleans, INTERNAL_CALL_ORIGIN_METADATA_KEY guard, recorded-value precedence surviving an unparseable usage object), the None/0.0 distinction is correct and tested, and the field is properly excluded from client-writable buckets.

Updated score: 5/5. The PR is ready for maintainer review.

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_autorouter_savings_callbacks (a6c67c1) with litellm_internal_staging (91f2382)}