Skip to content

revert(spend-logs): drop the endTime backfill migration for spend log timestamps - #37875

Merged
yuneng-berri merged 2 commits into
litellm_internal_stagingfrom
litellm_/revert-pr-37554-migration-5af8ea
Aug 21, 2026
Merged

revert(spend-logs): drop the endTime backfill migration for spend log timestamps#37875
yuneng-berri merged 2 commits into
litellm_internal_stagingfrom
litellm_/revert-pr-37554-migration-5af8ea

Conversation

@yuneng-berri

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • The backfill rewrites every historical spend log row at upgrade
  • On a large table that rewrite outlives the prisma command timeout
  • The upgrade then rolls back and the gateway never finishes booting
  • Cost scales with spend history, so the biggest deployments hurt most

How it solves it:

User Flow

Before: an operator upgrading a gateway that has a large spend history watches the deploy fail, and nothing in the failure names the size of their data as the cause

  1. They deploy the new gateway image against their existing database
  2. Startup applies the pending database migrations, one of which rewrites every historical spend log row in a single statement
  3. On a spend history in the hundreds of millions of rows that rewrite runs longer than the 60 second prisma command timeout, the default
  4. The statement rolls back and startup aborts, so GET https://litellm-domain/health/liveliness never answers and no request is served
  5. They have to raise LITELLM_PRISMA_COMMAND_TIMEOUT and redeploy before the gateway will come up at all

After: the same upgrade boots in its usual window no matter how much spend history the database holds

  1. They deploy the new gateway image against their existing database
  2. Startup applies the pending database migrations, none of which rewrite historical rows
  3. Migration time no longer scales with the size of the spend log, so startup finishes as it did on every prior upgrade
  4. GET https://litellm-domain/health/liveliness answers and POST https://litellm-domain/v1/chat/completions serves traffic normally
  5. GET https://litellm-domain/spend/logs?start_date=&end_date=&summarize=false shows pre-upgrade rows carrying the upgrade-time write stamp, unchanged from what feat(spend-logs): add lifecycle timestamps #37361 gave them

Relevant issues

Reverts #37554

Linear ticket

Notes for the reviewer

The risk being backed out is not a newly observed failure. #37554 documented it in its own Caveats and body: the UPDATE seq-scans the table and rewrites every matching heap tuple, prisma runs it under LITELLM_PRISMA_COMMAND_TIMEOUT, and a table where it needs longer times out, rolls back, and fails the boot until an operator raises that variable for the one deploy that performs the backfill. Trading a guaranteed boot for more honest historical timestamps is the wrong side of that trade for large deployments, so the backfill comes out

#37554 shipped in v1.99.0-dev.2, so any database that booted that build has 20260819000000_backfill_spend_log_timestamps recorded as applied while this branch no longer ships the directory. The v2 resolver's applied-but-unknown check compares each unknown migration's timestamp strictly against the newest migration this build ships, and the newest is still 20260819000000 (shadow_eval_max_budget), so the deleted name does not clear that bar and the check stays quiet. Those databases keep whatever the backfill already wrote, which is the correct value anyway; nothing is un-backfilled by this revert

schema.prisma is untouched, matching #37554, which only added data-only SQL and changed no schema. No test is added or removed: the change is the deletion of a raw SQL file and the repo has no harness that executes migration SQL against a database

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests (deletion of a raw SQL migration, no harness executes migration SQL)
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Pending, to be replaced with the live run before review

Type

🚄 Infrastructure

Caveats (if any)

  • Historical rows keep the upgrade-time write stamp from feat(spend-logs): add lifecycle timestamps #37361
  • Databases already on v1.99.0-dev.2 keep the backfill they already ran
  • Export checkpointing on write time is inaccurate again for old rows
  • A large-table backfill needs an out-of-band mechanism if still wanted

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

… timestamps

Reverts #37554, which added 20260819000000_backfill_spend_log_timestamps
@yuneng-berri
yuneng-berri requested a review from a team August 21, 2026 18:46
@greptile-apps

greptile-apps Bot commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR removes the spend-log timestamp backfill migration so upgrades no longer rewrite the entire historical spend-log table during startup.

  • Deletes the data-only update that copied endTime into created_at and updated_at.
  • Preserves the timestamp columns and leaves historical rows with their existing write timestamps.
  • Avoids migration runtime scaling with the size of the spend-log table.

Confidence Score: 5/5

The PR appears safe to merge because the removed data-only backfill avoids the documented startup timeout while its timestamp tradeoff is explicit.

The remaining timestamp behavior is intentionally documented, and the deleted migration’s equal timestamp does not trigger the custom resolver’s strictly-newer applied-migration check.

Important Files Changed

Filename Overview
litellm-proxy-extras/litellm_proxy_extras/migrations/20260819000000_backfill_spend_log_timestamps/migration.sql Deletes the unbounded historical timestamp backfill; no concrete unacknowledged correctness or migration-resolution failure was established.

Reviews (1): Last reviewed commit: "Merge remote-tracking branch 'origin/lit..." | Re-trigger Greptile

@codecov

codecov Bot commented Aug 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@yuneng-berri
yuneng-berri merged commit f6c19ea into litellm_internal_staging Aug 21, 2026
70 of 72 checks passed
@yuneng-berri
yuneng-berri deleted the litellm_/revert-pr-37554-migration-5af8ea branch August 21, 2026 19:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants