fix(usage): preserve daily spend by public model by tin-berri · Pull Request #42587 · BerriAI/litellm

tin-berri · 2026-09-22T21:25:45Z

TLDR

Problem this solves:

Usage merges public model names sharing one deployment

How it solves it:

Keep each public model’s daily totals separate
Preserve stable PTU charges across model renames

Intentional product change: Usage shows N/A when there are no successful requests to average, and explains UTC daily totals

User Flow

Before: two public names can appear as one model in Usage

Send POST https://litellm-domain/v1/chat/completions as the same key using direct-model, then alternate-model, backed by one deployment
Open https://litellm-domain/ui/usage/ and select the request date
One model row contains both requests; the other name is absent

After: Usage retains the separate public names and their totals

Send the same two requests using direct-model and alternate-model
Open https://litellm-domain/ui/usage/ and select the request date
Each model row contains its own spend, tokens and requests

Relevant issues

Fix the daily model breakdown while preserving total spend
Keep the migration and PTU compatibility changes together
Evaluation billing is tracked independently in LIT-9199

Changes

Include model_group in all six daily queue, bulk-upsert and database identities
Synchronize three Prisma schemas and replace six unique indexes without rewriting historical rows
Preserve stable PTU IDs across renames and concurrent writes
Explain zero-request averages and UTC usage days in the dashboard

Linear ticket

Resolves LIT-9198

Pre-Submission checklist

Meaningful regressions: 94 added test lines / 97 implementation lines
357 targeted Python tests and 61 dashboard tests pass independently
make check, discovery/shard guards and migration scanner pass locally
This PR addresses one defect and can merge independently
Fresh CI, coverage and reviews pending on this split commit

Type

Bug Fix

Caveats (if any)

Severe

Drain buffers and stop old writers before index replacement
Index construction blocks writes; schedule a maintenance window
Rollback requires reconciling rows before restoring old indexes

Medium

Historical merged model attribution cannot be reconstructed

Low

Fresh live proof is being captured on the split commit
Previous combined CI exposed unrelated dependency and ROI-fixture failures

tin-berri · 2026-09-22T21:30:09Z

@greptileai Please review the current commit for attribution correctness, migration safety, and preservation of provisioned-throughput billing across renames

tin-berri · 2026-09-22T21:30:10Z

bugbot run

Please review the current commit for daily usage attribution, database identity changes, and provisioned-throughput rename safety

codspeed · 2026-09-22T21:32:20Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_daily_spend_model_group (c6669ba) with main (5724117)}

greptile-apps · 2026-09-22T21:34:27Z

Confidence Score: 5/5

The reviewed changes appear safe to merge, with no outstanding previous findings or accepted new defects from the current-tip rebase.

Summary

This PR separates daily usage by public model group, preserves stable PTU charging identities, and attributes shadow-evaluation costs to the initiating administrator. It also updates the dashboard’s internal-usage presentation and supporting billing integrations.

Expands six daily-spend identities and matching bulk upserts to include model_group.
Projects shadow-router and judge-call billing onto the evaluation creator while retaining internal-call metadata.
Keeps PTU flat-cost rollups stable across public-model renames.
Updates usage and evaluation UI copy, dates, and unavailable-average rendering.
Adds regression coverage across spend queues, billing callbacks, evaluation ownership, PTU rollups, and dashboard components.

_{Reviews (8) · Last reviewed commit: "fix(usage): separate router costs and bi..."}

cursor

Stale Bugbot comment from a previous run.

tin-berri · 2026-09-22T21:36:10Z

CodSpeed compared different CPUs and an older base. The resolver is unchanged; same-environment A/B measured 2.01µs versus 2.08µs, below its 10% threshold

codecov · 2026-09-22T21:36:28Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

tin-berri · 2026-09-22T21:43:56Z

@greptileai Please review 1a2decb, including the corrected global rollup fixture and regression coverage proving legacy NULL/empty totals remain included

tin-berri · 2026-09-22T21:43:58Z

bugbot run

Please review 1a2decb; live usage APIs and added regression coverage confirm legacy NULL/empty totals remain included

cursor

Stale Bugbot comment from a previous run.

tin-berri · 2026-09-22T21:50:06Z

The same glibc CVE fails on main's unchanged base image. Resolving this requires upgrading the pinned base image separately

tin-berri · 2026-09-22T23:34:58Z

@greptileai Please review the revised admin-owned evaluation accounting, including preserved routing identity, billing exports, and sidecar replay

tin-berri · 2026-09-22T23:35:02Z

bugbot run

Please review the revised evaluation billing ownership and confirm sampled-user accounting and routing remain correctly isolated

cursor

Stale Bugbot comment from a previous run.

tin-berri · 2026-09-23T00:11:59Z

@greptileai Please review d5220d5: creator model budgets now load per sample, and the router test injects its observer before construction

tin-berri · 2026-09-23T00:12:01Z

bugbot run

Please review d5220d5 for creator budget ownership, refreshed configuration, deleted creators, and callback context propagation

tin-berri · 2026-09-23T00:12:03Z

CodeQL 9134 flags existing API-token SHA256; passwords use salted scrypt. SARIF confirms the token path, and all 14 helper regressions pass

cursor

Stale Bugbot comment from a previous run.

tin-berri · 2026-09-23T00:43:14Z

@greptileai Please review current tip 0197df1: tests now match implementation size, retaining creator ownership, production controls, and replay coverage

tin-berri · 2026-09-23T00:43:15Z

bugbot run

Please review current tip 0197df1; consolidated tests preserve ownership regressions, financial exports, creator budgets, and production controls

cursor

Stale Bugbot comment from a previous run.

tin-berri · 2026-09-23T01:36:19Z

@greptileai Please review current tip 5015c2c for model group isolation, admin billing, coordinated migration, PTU rename safety and Usage clarity

tin-berri · 2026-09-23T01:36:20Z

bugbot run
Please review 5015c2c for separate router accounting, initiating-admin ownership, migration compatibility, primary PTU reads and zero-request Usage displays

cursor

Stale Bugbot comment from a previous run.

veria-ai · 2026-09-23T01:46:38Z

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

tin-berri · 2026-09-23T01:46:48Z

@greptileai Please review current tip e5deff1 after rebasing onto current main. The two conflicts combined test imports; 195 spend tests and 577 auto-router/logging tests pass. Daily public model grouping, PTU identity, and initiating-admin evaluation billing remain the scope. Fresh live replay and CI are running.

tin-berri · 2026-09-23T01:46:49Z

bugbot run

Please review current tip e5deff1 after the rebase.

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit e5deff1. Configure here.}

tin-berri · 2026-10-03T07:43:15Z

Superseded by #44357 for public-model usage. Evaluation billing is isolated in #44356; both replacement PRs can merge independently

tin-berri requested a review from a team September 22, 2026 21:25

tin-berri requested review from ryan-crabbe-berri and yuneng-berri as code owners September 22, 2026 21:25

tin-berri had a problem deploying to e2e-changed September 22, 2026 21:25 — with GitHub Actions Error

cursor Bot reviewed Sep 22, 2026

View reviewed changes

Comment thread ...itellm_proxy_extras/migrations/20260922000000_daily_spend_model_group_identity/migration.sql

tin-berri force-pushed the litellm_daily_spend_model_group branch from 0bbf28d to 1a2decb Compare September 22, 2026 21:43

tin-berri deployed to e2e-changed September 22, 2026 21:43 — with GitHub Actions Active

cursor Bot reviewed Sep 22, 2026

View reviewed changes

tin-berri force-pushed the litellm_daily_spend_model_group branch from 1a2decb to 11a5b06 Compare September 22, 2026 23:34

tin-berri changed the title ~~fix(usage): preserve model groups in daily spend aggregation~~ fix(shadow-eval): charge evaluation spend to the initiating admin Sep 22, 2026

tin-berri had a problem deploying to e2e-changed September 22, 2026 23:34 — with GitHub Actions Error

cursor Bot reviewed Sep 22, 2026

View reviewed changes

greptile-apps Bot reviewed Sep 22, 2026

View reviewed changes

Comment thread litellm/litellm_core_utils/internal_call_metadata.py Outdated

Comment thread tests/test_litellm/integrations/test_shadow_eval_logger.py Outdated

tin-berri force-pushed the litellm_daily_spend_model_group branch from 11a5b06 to d5220d5 Compare September 23, 2026 00:12

tin-berri had a problem deploying to e2e-changed September 23, 2026 00:12 — with GitHub Actions Error

cursor Bot reviewed Sep 23, 2026

View reviewed changes

Comment thread litellm/litellm_core_utils/internal_call_metadata.py Outdated

cursor Bot reviewed Sep 23, 2026

View reviewed changes

tin-berri force-pushed the litellm_daily_spend_model_group branch from 0197df1 to 5015c2c Compare September 23, 2026 01:35

tin-berri changed the title ~~fix(shadow-eval): charge evaluation spend to the initiating admin~~ fix(usage): separate router costs and bill evaluations to admins Sep 23, 2026

cursor Bot reviewed Sep 23, 2026

View reviewed changes

tin-berri force-pushed the litellm_daily_spend_model_group branch from 5015c2c to e5deff1 Compare September 23, 2026 01:46

veria-ai Bot reviewed Sep 23, 2026

View reviewed changes

Comment thread litellm/proxy/auth/user_api_key_auth.py Outdated

tin-berri had a problem deploying to e2e-changed September 23, 2026 01:46 — with GitHub Actions Error

cursor Bot reviewed Sep 23, 2026

View reviewed changes

tin-berri force-pushed the litellm_daily_spend_model_group branch from e5deff1 to f5098eb Compare September 25, 2026 22:49

tin-berri had a problem deploying to e2e-changed September 25, 2026 22:49 — with GitHub Actions Error

tin-berri force-pushed the litellm_daily_spend_model_group branch from f5098eb to 7d0b72f Compare September 26, 2026 03:28

tin-berri had a problem deploying to e2e-changed September 26, 2026 03:28 — with GitHub Actions Error

tin-berri force-pushed the litellm_daily_spend_model_group branch from 7d0b72f to 93157d3 Compare October 3, 2026 07:13

tin-berri had a problem deploying to e2e-changed October 3, 2026 07:13 — with GitHub Actions Error

fix(usage): preserve daily spend by public model

c6669ba

tin-berri force-pushed the litellm_daily_spend_model_group branch from 93157d3 to c6669ba Compare October 3, 2026 07:34

tin-berri requested a deployment to e2e-changed October 3, 2026 07:34 — with GitHub Actions Waiting

tin-berri changed the title ~~fix(usage): separate router costs and bill evaluations to admins~~ fix(usage): preserve daily spend by public model Oct 3, 2026

This was referenced Oct 3, 2026

fix(shadow-eval): bill the creator and enforce their budgets #44356

Open

fix(usage): preserve daily spend by public model #44357

Open

tin-berri had a problem deploying to e2e-changed October 3, 2026 07:42 — with GitHub Actions Error

tin-berri closed this Oct 3, 2026

Conversation

tin-berri commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Relevant issues

Changes

Linear ticket

Pre-Submission checklist

Type

Caveats (if any)

Severe

Medium

Low

Uh oh!

tin-berri commented Sep 22, 2026

Uh oh!

tin-berri commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codspeed Bot commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

greptile-apps Bot commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tin-berri commented Sep 22, 2026

Uh oh!

codecov Bot commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

tin-berri commented Sep 22, 2026

Uh oh!

tin-berri commented Sep 22, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tin-berri commented Sep 22, 2026

Uh oh!

tin-berri commented Sep 22, 2026

Uh oh!

tin-berri commented Sep 22, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tin-berri commented Sep 23, 2026

Uh oh!

tin-berri commented Sep 23, 2026

Uh oh!

tin-berri commented Sep 23, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tin-berri commented Sep 23, 2026

Uh oh!

tin-berri commented Sep 23, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tin-berri commented Sep 23, 2026

tin-berri commented Sep 22, 2026 •

edited

Loading

tin-berri commented Sep 22, 2026 •

edited

Loading

codspeed Bot commented Sep 22, 2026 •

edited

Loading

greptile-apps Bot commented Sep 22, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

codecov Bot commented Sep 22, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

veria-ai Bot commented Sep 23, 2026 •

edited

Loading