fix(proxy): estimate auto-router baseline costs from durable cache history

[tin-berri]

TLDR

Problem this solves:

• Expired prefixes can produce false reported losses
• Zero savings must retain both nonzero costs
• Unknown estimates must not distort savings percentages

How it solves it:

• Match cache prefixes, availability times and TTLs
• Preserve observed costs before routing diverges
• Publish replayable estimates with explicit coverage

User Flow

Before: a conversation can report losses from unsupported baseline cache reads

1. Send POST http://localhost:47998/v1/messages through the auto-router with a session ID and cache controls
2. Continue after changing the cached prefix or letting it expire
3. Read the session costs: the changed prefix reports −$0.05672550, and expiry reports −$0.03224500

After: each comparison requires observed costs or usable prefix history

1. Send POST http://localhost:47999/v1/messages through the auto-router with a session ID and cache controls
2. Continue after changing the cached prefix or letting it expire
3. Read the session costs: baseline-identical turns preserve both costs and $0 savings; the expired five-minute prefix reports +$0.06131975, while unavailable one-hour history produces an unknown comparison

Relevant issues

• Replace conversation-shape assumptions with recorded cache-prefix accounting
• Keep known-zero turns covered and preserve signed savings
• Provider fast-mode and geography pricing shipped separately in fix(proxy): preserve Anthropic pricing modifiers in router savings #41341
• Product documentation: litellm-docs #1575

Changes

• Capture observations without pre-dispatch database or token-count calls
• Replay late observations and atomically correct request, session and daily estimates
• Preserve actual billed spend while affected estimates await replay
• Carry covered actual/baseline costs through the API, dashboard and CLI
• Preserve upstream Azure metadata and logging changes during rebase

Affected release

Linear ticket

Pre-Submission checklist

• Focused Python checks: 1,038 passed, including real PostgreSQL
• Dashboard checks: 67 passed; TypeScript check passed
• Full make check, static scanners and cost-map guard passed
• Scope remains baseline estimation; provider pricing shipped separately
• Current-tip hosted CI, final coverage and reviewer checks complete

Screenshots / Proof of Fix

Fresh paid native-message requests on the current base and tip, with isolated PostgreSQL, unique prefixes and uncached suffixes. Every streamed request ended with message_stop

The final revision only removes misplaced product documentation; runtime and test files are byte-identical to the locally tested e103981. The requests below were rerun after that revision. The expiry case resumes a session seeded on the identical runtime before the documentation move, after more than five minutes

First-turn equality and cold-write pricing after the five-minute window passed. Gateway token counting exceeded the three-second deadline on seed requests, so the one-hour cases remain unknown. This run does not establish a successful live warm-prefix hit; the matching-prefix cases pass controlled regression tests

Shared setup and reproducible requests

Run each revision with the same isolated PostgreSQL database, gateway credentials and model configuration. The before proxy uses port 47998; after uses 47999. Apply the schema migrations before starting after

model_list:
  - model_name: sonnet
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/BASELINE_GATEWAY_KEY
      api_base: os.environ/BASELINE_GATEWAY_URL
  - model_name: opus
    litellm_params:
      model: anthropic/claude-opus-5
      api_key: os.environ/BASELINE_GATEWAY_KEY
      api_base: os.environ/BASELINE_GATEWAY_URL
  - model_name: baseline-cache-router
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        tiers: {SIMPLE: [sonnet], MEDIUM: [sonnet], COMPLEX: [opus], REASONING: [opus]}
        classifier_type: heuristic
        classification_mode: every_request
        session_affinity: false
        keyword_tier_rules:
          - {keywords: [USE_OPUS], tier: COMPLEX}
          - {keywords: [USE_SONNET], tier: SIMPLE}
general_settings:
  master_key: os.environ/LOCAL_PROXY_KEY
  database_url: os.environ/DATABASE_URL
router_settings:
  num_retries: 0

Generate a fresh payload for each side and TTL. Keep the session ID and cached block unchanged for the follow-up; append an assistant turn and a user turn requesting USE_SONNET. For the changed-prefix case, prepend a fresh UUID inside the cached block. For five-minute expiry, wait at least 320 seconds before sending the follow-up

import json, uuid
run = uuid.uuid4().hex
ttl = "1h"  # repeat with "5m"
body = {
    "model": "baseline-cache-router", "litellm_session_id": run,
    "stream": True, "max_tokens": 16,
    "messages": [{"role": "user", "content": [
        {"type": "text", "text": run + "\n" +
         "A stable example for comparing recorded request cache costs.\n" * 900,
         "cache_control": {"type": "ephemeral", "ttl": ttl}},
        {"type": "text", "text": "USE_OPUS. Reply only OK. " + run}
    ]}]
}
open("request.json", "w").write(json.dumps(body))

curl -sS -N "$PROXY_URL/v1/messages"   -H "Authorization: Bearer $LOCAL_PROXY_KEY"   -H 'Content-Type: application/json' -H 'anthropic-version: 2023-06-01'   --data-binary @request.json
curl -sS "$PROXY_URL/spend/logs?request_id=$REQUEST_ID"   -H "Authorization: Bearer $LOCAL_PROXY_KEY"
curl -sS "$PROXY_URL/auto_router/session?session_id=$SESSION_ID"   -H "Authorization: Bearer $LOCAL_PROXY_KEY"

Allow asynchronous accounting to flush before reading costs. The request cases below isolate this run. The five-minute initial request uses stream: false; all other requests stream

Before (b946d12ffdbd3a8379a78ed835bada22b9096207)

Baseline-identical first turn, nonstreaming

1. Send the payload to POST http://localhost:47998/v1/messages; request msg_bdrk_bldf47nnemk3yrryrjnwbkxiupthzmhuojwnaa7pitfdspy57mkq completed
2. Read GET http://localhost:47998/spend/logs?request_id=msg_bdrk_bldf47nnemk3yrryrjnwbkxiupthzmhuojwnaa7pitfdspy57mkq: actual $0.10216125, savings $+0.00000000

Baseline-identical first turn, streaming

1. Send the payload to POST http://localhost:47998/v1/messages; request msg_bdrk_zxm5ntidn6h3utg567yoxeoy6sejdnpphrvwglwj6xopuw3efc7q completed
2. Read GET http://localhost:47998/spend/logs?request_id=msg_bdrk_zxm5ntidn6h3utg567yoxeoy6sejdnpphrvwglwj6xopuw3efc7q: actual $0.16294500, savings $+0.00000000

Matching one-hour prefix

1. Send the payload to POST http://localhost:47998/v1/messages; request msg_011CfDMZbczbz3fBG9rpWgeT completed
2. Read GET http://localhost:47998/spend/logs?request_id=msg_011CfDMZbczbz3fBG9rpWgeT: actual $0.06521800, savings $-0.05664300

Changed one-hour prefix

1. Send the payload to POST http://localhost:47998/v1/messages; request msg_011CfDMadF17pGRnvJfDttby completed
2. Read GET http://localhost:47998/spend/logs?request_id=msg_011CfDMadF17pGRnvJfDttby: actual $0.06533400, savings $-0.05672550

Expired five-minute prefix

1. Send the payload to POST http://localhost:47998/v1/messages; request msg_011CfDMwktxzMvt2suVbYRsH completed
2. Read GET http://localhost:47998/spend/logs?request_id=msg_011CfDMwktxzMvt2suVbYRsH: actual $0.04085250, savings $-0.03224500

After (ed40241d26f6a23048c51d39bbb862a5e3a92de0)

Baseline-identical first turn, nonstreaming

1. Send the payload to POST http://localhost:47999/v1/messages; request msg_bdrk_zxdz7r2x5dh2sze3ctjz2e7jem2oq2bbifyfjsggalkjslh3yc3a completed
2. Read GET http://localhost:47999/spend/logs?request_id=msg_bdrk_zxdz7r2x5dh2sze3ctjz2e7jem2oq2bbifyfjsggalkjslh3yc3a: actual $0.10222375, savings $+0.00000000, identical_baseline_path, baseline $0.10222375

Baseline-identical first turn, streaming

1. Send the payload to POST http://localhost:47999/v1/messages; request msg_bdrk_zj3mwsxxqic6rbi3es2ofnrurxaa2skjcxwolbayyp6gajguexlq completed
2. Read GET http://localhost:47999/spend/logs?request_id=msg_bdrk_zj3mwsxxqic6rbi3es2ofnrurxaa2skjcxwolbayyp6gajguexlq: actual $0.16304500, savings $+0.00000000, identical_baseline_path, baseline $0.16304500

Matching one-hour prefix

1. Send the payload to POST http://localhost:47999/v1/messages; request msg_011CfDNfwDzajr5TpXPhfYAN completed
2. Read GET http://localhost:47999/spend/logs?request_id=msg_011CfDNfwDzajr5TpXPhfYAN: actual $0.06527800, savings unknown, token_count_timeout; comparison costs are null

Changed one-hour prefix

1. Send the payload to POST http://localhost:47999/v1/messages; request msg_011CfDNgx1A614LdpAJq2GSs completed
2. Read GET http://localhost:47999/spend/logs?request_id=msg_011CfDNgx1A614LdpAJq2GSs: actual $0.06539400, savings unknown, history_unavailable; comparison costs are null

Expired five-minute prefix

1. Send the payload to POST http://localhost:47999/v1/messages; request msg_011CfDNhHJGK1iuq64KZ5CLj completed
2. Read GET http://localhost:47999/spend/logs?request_id=msg_011CfDNhHJGK1iuq64KZ5CLj: actual $0.04087900, savings $+0.06131975, cache_prefix_cold, baseline $0.10219875

Python additions: 1,544 test lines / 2,442 runtime lines = 0.63:1

Type

Bug Fix

Caveats

Medium

• Estimates hold recorded prompts, output usage and timing fixed
• Unsupported semantics remain unknown after routing diverges
• Provider eviction and unrecorded traffic remain unobservable
• Spend logging must stay enabled for durable publication
• Gateway count timeouts limited this run's live coverage
• Current-tip interactive client and dashboard screenshots remain unverified

Low

• One-hour natural expiry uses controlled-clock tests
• Hosted checks and current-tip bot reviews are pending
• Prior Veria resource-consumption findings await reassessment

Final Attestation

• All current-tip checks and interactive-client verification complete; do not merge

[tin-berri]

bugbot run

Please review baseline cache prefix matching, expiry, retry uncertainty, and savings coverage on this draft's current commit

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_autorouter_baseline_cache (ed40241) with main (b946d12)}

[cursor]

Stale Bugbot comment from a previous run.

[codecov]

Codecov Report

❌ Patch coverage is 83.08157% with 168 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/db/baseline_accounting.py	58.36%	107 Missing ⚠️
litellm/proxy/db/db_spend_update_writer.py	48.00%	26 Missing ⚠️
litellm/proxy/hooks/autorouter_baseline_cache.py	90.30%	16 Missing ⚠️
...itellm/proxy/spend_tracking/baseline_accounting.py	95.34%	6 Missing ⚠️
litellm/proxy/spend_tracking/savings.py	93.54%	4 Missing ⚠️
litellm/proxy/db/daily_spend_bulk_upsert.py	83.33%	3 Missing ⚠️
...llm/proxy/client/cli/commands/statusline_script.py	92.59%	2 Missing ⚠️
litellm/llms/anthropic/prompt_cache_prediction.py	99.35%	1 Missing ⚠️
litellm/llms/custom_httpx/llm_http_handler.py	50.00%	1 Missing ⚠️
litellm/proxy/common_request_processing.py	66.66%	1 Missing ⚠️
... and 1 more

📢 Thoughts on this report? Let us know!

[tin-berri]

@greptileai Please review bec8a42, including baseline prefix matching, expiry, estimate coverage, and reservation cleanup after estimator failures

[tin-berri]

bugbot run

Please review bec8a42 for baseline cache prefix matching, expiry, estimate coverage, and reservation cleanup after estimator failures

[greptile-apps]

Confidence Score: 5/5

The reviewed code appears safe to merge, with no new findings and no outstanding previous findings.

Findings

1.  Critical-path database queries ▶

Summary

This PR replaces conversation-shape-based auto-router savings estimates with durable cache-prefix observations and replayable accounting.

• Records cache-prefix history and baseline observations through the deferred spend pipeline.
• Atomically reconciles request, session, and daily savings estimates while preserving billed spend.
• Adds explicit estimate coverage, actual cost, baseline cost, and unknown-reason fields across APIs, CLI, and dashboard views.
• Extends Anthropic cache-plan parsing, token counting, streaming usage accumulation, and model capability metadata.
• Adds aligned Prisma schemas, forward migrations, and regression coverage for the new durable state.

_{Reviews (17) · Last reviewed commit: "fix(proxy): estimate auto-router baselin..."}

[cursor]

Stale Bugbot comment from a previous run.

[tin-berri]

The quoted documentation rule explicitly exempts README.md; this change updates the existing router README with estimator behavior and coordination Redis requirements

[tin-berri]

@greptileai Please review 146d1ab, including atomic reservation recovery, active versus terminal retries, original-error preservation, and documented unavailable prefix counts

[tin-berri]

bugbot run
Please review 146d1ab after the reservation lifecycle redesign, including storage recovery, retry ownership, partial streams, and failure telemetry

[cursor]

Stale Bugbot comment from a previous run.

[tin-berri]

bugbot run
Please review 0e4f2c7, including terminal failure cleanup after retry exhaustion, retained attempt uncertainty, and original error telemetry

[tin-berri]

@greptileai Please review 0e4f2c7, which retires baseline reservations at the proxy terminal failure boundary while preserving per-attempt retry uncertainty

[cursor]

Stale Bugbot comment from a previous run.

[tin-berri]

@greptileai Please review 1250ee5, including explicit invalidation cutoffs, late success ordering, and prompt recovery after terminal failures

[tin-berri]

bugbot run
Please review 1250ee5, including explicit invalidation cutoffs, late success ordering, and prompt recovery after terminal failures

[cursor]

Stale Bugbot comment from a previous run.

[tin-berri]

@greptileai Please review 64610c1: duplicate tag attribution is fixed through the shared identity resolver, and the handwritten catalog schema is updated.

[tin-berri]

bugbot run

Please review 64610c1: duplicate tag attribution now uses one shared resolver, with nonzero PostgreSQL regression coverage and a killed multiplicity mutation.

[tin-berri]

@veria-ai Please review 64610c1, including asynchronous baseline observations, bounded counting, retained-history retirement, and shared identity deduplication for daily attribution.

[cursor]

Stale Bugbot comment from a previous run.

[tin-berri]

@greptileai Please review e103981 after rebasing onto current main, focusing on logging finalization and trusted spend metadata integration

[tin-berri]

bugbot run Please review e103981 after the rebase, especially logging finalization, baseline accounting, and preservation of upstream metadata fields

[tin-berri]

@veria-ai Please review e103981 after the rebase and reassess previous findings against the current baseline accounting and logging integration

[greptile-apps]

Comments Outside Diff

These findings sit on lines the diff does not cover, so they could not be posted inline. Each one leaves this list once its file changes.

•  Product documentation is misplaced litellm/router_strategy/complexity_router/README.md:17 ▶

  This section documents customer-facing LiteLLM behavior in-tree, violating the requirement to use litellm-docs before merging.

[cursor]

Stale Bugbot comment from a previous run.

[tin-berri]

@greptileai Please review ed40241. Removed the product documentation from the implementation README and moved it into the litellm-docs guide

[tin-berri]

bugbot run Please review ed40241. This revision only removes product documentation from the implementation README; runtime code is unchanged

[tin-berri]

@veria-ai Please review ed40241 and reassess earlier findings. This revision removes misplaced documentation without changing runtime code from e103981

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit ed40241. Configure here.}

[tin-berri]

@greptileai review