Skip to content

fix(proxy): enforce the migration check by default and stop building SpendLogs indexes in migrations (rc/1.104.0) - #44206

Merged
yuneng-berri merged 1 commit into
rc/1.104.0from
litellm_rc_1_104_0_enforce_migration_check_default
Oct 2, 2026
Merged

yuneng-berri merged 1 commit into
rc/1.104.0from
litellm_rc_1_104_0_enforce_migration_check_default

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • A failed boot migration let the proxy start on a stale schema
  • Two SpendLogs index migrations block writes for the whole index build
  • CREATE INDEX CONCURRENTLY fails outright on a partitioned SpendLogs table

How it solves it:

  • ENFORCE_PRISMA_MIGRATION_CHECK now defaults to true
  • Opt out with ENFORCE_PRISMA_MIGRATION_CHECK=false or --no-enforce_prisma_migration_check
  • The standalone migration entrypoint reads the same opt-out values as the proxy flag
  • Both SpendLogs index migrations are now SELECT 1
  • Operators build the indexes online by hand per the SpendLogs index runbook

Files changed

File What changed and why
litellm/proxy/proxy_cli.py --enforce_prisma_migration_check becomes a --flag/--no-flag Click option that defaults to true and still reads ENFORCE_PRISMA_MIGRATION_CHECK, so a failed boot migration exits 1 unless the operator opts out
litellm/proxy/prisma_migration.py The standalone migration entrypoint parses ENFORCE_PRISMA_MIGRATION_CHECK with Click's boolean rules, so false, 0, no and off opt out in both entrypoints and anything else enforces
litellm-proxy-extras/.../20260823000000_add_spend_logs_api_key_starttime_index/migration.sql SELECT 1, the api_key, startTime index is built online by hand
litellm-proxy-extras/.../20260831120001_spend_logs_litellm_call_id_index/migration.sql SELECT 1, the litellm_call_id index is built online by hand

User Flow

Before: an operator upgrades a large deployment and new pods either hang on an index build or serve traffic on a half-migrated schema

  1. They roll out the new image against their existing Postgres
  2. The boot migration starts CREATE INDEX on LiteLLM_SpendLogs, inserts block, and on a partitioned table the concurrent variant fails with 0A000
  3. Once the migration fails, the pod logs Database migration failed but continuing startup and starts serving anyway
  4. POST https://litellm-domain/v1/chat/completions returns 500s for anything that touches the missing column or index

After: the same upgrade either comes up on a complete schema or stops and says why

  1. They roll out the new image against their existing Postgres
  2. The two index migrations run instantly as no-ops, the column migration still runs, and no write is blocked
  3. If a migration still fails, the pod logs Database setup failed after multiple retries. The proxy cannot start safely and exits 1 instead of serving
  4. They build the two indexes online following the runbook, at a time of their choosing
  5. POST https://litellm-domain/v1/chat/completions keeps working throughout

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Shared setup: the proxy is pointed at a Postgres that is not there, so the boot migration fails every retry. The legacy resolver is used because the v2 resolver treats an unreachable database as unrecoverable and exits 2 before this check is reached

export DATABASE_URL=postgresql://u:p@127.0.0.1:1/db LITELLM_MASTER_KEY=sk-1234

Before (5663d07)

  1. python litellm/proxy/proxy_cli.py --skip_server_startup --use_legacy_migration_resolver; echo exit=$?
  2. Output ends with the proxy carrying on after the failed migration:
LiteLLM Proxy: Database migration failed but continuing startup. Set --enforce_prisma_migration_check or ENFORCE_PRISMA_MIGRATION_CHECK=true to exit on failure.
exit=0

After (84e8577)

default

  1. python litellm/proxy/proxy_cli.py --skip_server_startup --use_legacy_migration_resolver; echo exit=$?
  2. Output ends with the proxy refusing to start:
LiteLLM Proxy: Database setup failed after multiple retries. The proxy cannot start safely. Please check your database connection and migration status.
exit=1

explicit opt-out

  1. ENFORCE_PRISMA_MIGRATION_CHECK=false python litellm/proxy/proxy_cli.py --skip_server_startup --use_legacy_migration_resolver; echo exit=$?
  2. Output ends with the old behavior, now clearly labeled as opted out:
LiteLLM Proxy: Database migration failed but continuing startup because ENFORCE_PRISMA_MIGRATION_CHECK is disabled. The schema may be behind the code.
exit=0

Type

Bug Fix

Caveats (if any)

Severe

  • Deployments relying on a failed migration being ignored now exit 1 at boot. Set ENFORCE_PRISMA_MIGRATION_CHECK=false to keep the old behavior
  • Upgrades from 1.103.x no longer get the two SpendLogs indexes built for them. Follow the runbook to build them online

Medium

  • A _prisma_migrations row for either rewritten migration that ran its step but never got finished_at no longer matches the shipped checksum, so the v2 resolver stops with its P3009 guidance instead of finishing the row. Verify the index exists, then run prisma migrate resolve --applied <migration_name>
  • The legacy drift repair (_resolve_all_migrations, only reached with --use_legacy_migration_resolver after prisma migrate deploy fails) still diffs the live database against schema.prisma, so on a database that never got those two indexes it can emit their plain CREATE INDEX and build them with the write lock the runbook avoids. Kept out of this PR to keep the backport minimal; operators on that path build the indexes first
  • Pre-existing and unchanged here: under the legacy resolver a drift script that fails to apply after prisma migrate deploy succeeded is logged and setup still reports success
  • Pre-existing and unchanged here: when DATABASE_URL is set but no Prisma CLI or module is available, startup skips database setup with a warning and enforcement never runs

Low

  • schema.prisma still declares both indexes, so prisma migrate diff keeps reporting them as drift until they are built by hand
  • With the default v2 resolver most migration failures already exit 2 before this check, so the flip mainly affects the legacy resolver and prisma db push paths

Taxonomy audit

  • C1-C7: this is a default flip. It is the point of the PR, the opt-out is ENFORCE_PRISMA_MIGRATION_CHECK=false or --no-enforce_prisma_migration_check, and the Severe caveat spells out who is affected
  • X1-X5: an unset or blank ENFORCE_PRISMA_MIGRATION_CHECK is treated as true on purpose, only Click's false spellings opt out; the test table pins each spelling
  • R: migrations only lose DDL, no data rewrite, check_migrations_no_data_rewrites.py passes on all 189 migrations
  • T1-T5: the two assert lines removed from test_proxy_cli.py moved into the parameterized table unchanged in meaning; every row asserts both the exit path and the setup_database call
  • H: no new comments or Optional, the four Ruff findings in test_proxy_cli.py are on the base branch

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/6c17bcd2090f4977a0f95ef6f57911a5
Open in Devin Desktop: https://app.devin.ai/desktop/session/6c17bcd2090f4977a0f95ef6f57911a5?variant=devin
Requested by: @yassin-berriai

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@greptile-apps

greptile-apps Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[Critical risk] Changes database migration enforcement and removes index creation.

The PR appears safe to merge with the documented operator-managed index build.

Summary

The PR makes migration failure stop proxy startup by default and replaces two SpendLogs index migrations with no-ops.

  • Operators can opt out of enforcement and must build the indexes separately.
  • The new tests cover the default, explicit flags, and supported environment opt-out values.

Reviews (8) · Last reviewed commit: "fix(proxy): enforce the migration check ..."

Comment thread litellm/proxy/proxy_cli.py
@mateo-berri

Copy link
Copy Markdown
Contributor

@greptileai

@devin-ai-integration
devin-ai-integration Bot force-pushed the litellm_rc_1_104_0_enforce_migration_check_default branch from 9555738 to ef75765 Compare October 2, 2026 17:20
@mateo-berri

Copy link
Copy Markdown
Contributor

@greptileai

1 similar comment
@mateo-berri

Copy link
Copy Markdown
Contributor

@greptileai

@devin-ai-integration
devin-ai-integration Bot force-pushed the litellm_rc_1_104_0_enforce_migration_check_default branch from ef75765 to ea5cf56 Compare October 2, 2026 17:32
@mateo-berri

Copy link
Copy Markdown
Contributor

@greptileai

…SpendLogs indexes in migrations

ENFORCE_PRISMA_MIGRATION_CHECK now defaults to true in both the proxy CLI and the standalone migration entrypoint, with ENFORCE_PRISMA_MIGRATION_CHECK=false or --no-enforce_prisma_migration_check as the opt-out. The two LiteLLM_SpendLogs index migrations become SELECT 1 so operators build those indexes online by hand.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration
devin-ai-integration Bot force-pushed the litellm_rc_1_104_0_enforce_migration_check_default branch from ea5cf56 to 84e8577 Compare October 2, 2026 17:34
@mateo-berri

Copy link
Copy Markdown
Contributor

@greptileai

@devin-ai-integration devin-ai-integration Bot changed the title fix(proxy): enforce the migration check by default (rc/1.104.0) fix(proxy): enforce the migration check by default and stop building SpendLogs indexes in migrations (rc/1.104.0) Oct 2, 2026
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

Intentional per the SpendLogs index runbook: operators build both indexes online by hand on this line, matching #44205 (5/5). Legacy drift path is in Caveats

@mateo-berri

Copy link
Copy Markdown
Contributor

@greptileai

@mateo-berri

Copy link
Copy Markdown
Contributor

@greptileai

@yuneng-berri
yuneng-berri merged commit e012755 into rc/1.104.0 Oct 2, 2026
8 checks passed
@yuneng-berri
yuneng-berri deleted the litellm_rc_1_104_0_enforce_migration_check_default branch October 2, 2026 19:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants