Repository navigation
fix(proxy): enforce the migration check by default and stop building SpendLogs indexes in migrations (rc/1.104.0) - #44206
Conversation
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
|
|
|
9555738 to
ef75765
Compare
1 similar comment
ef75765 to
ea5cf56
Compare
…SpendLogs indexes in migrations ENFORCE_PRISMA_MIGRATION_CHECK now defaults to true in both the proxy CLI and the standalone migration entrypoint, with ENFORCE_PRISMA_MIGRATION_CHECK=false or --no-enforce_prisma_migration_check as the opt-out. The two LiteLLM_SpendLogs index migrations become SELECT 1 so operators build those indexes online by hand. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
ea5cf56 to
84e8577
Compare
|
Intentional per the SpendLogs index runbook: operators build both indexes online by hand on this line, matching #44205 (5/5). Legacy drift path is in Caveats |
TLDR
Problem this solves:
CREATE INDEX CONCURRENTLYfails outright on a partitioned SpendLogs tableHow it solves it:
ENFORCE_PRISMA_MIGRATION_CHECKnow defaults to trueENFORCE_PRISMA_MIGRATION_CHECK=falseor--no-enforce_prisma_migration_checkSELECT 1Files changed
litellm/proxy/proxy_cli.py--enforce_prisma_migration_checkbecomes a--flag/--no-flagClick option that defaults to true and still readsENFORCE_PRISMA_MIGRATION_CHECK, so a failed boot migration exits 1 unless the operator opts outlitellm/proxy/prisma_migration.pyENFORCE_PRISMA_MIGRATION_CHECKwith Click's boolean rules, sofalse,0,noandoffopt out in both entrypoints and anything else enforceslitellm-proxy-extras/.../20260823000000_add_spend_logs_api_key_starttime_index/migration.sqlSELECT 1, theapi_key, startTimeindex is built online by handlitellm-proxy-extras/.../20260831120001_spend_logs_litellm_call_id_index/migration.sqlSELECT 1, thelitellm_call_idindex is built online by handUser Flow
Before: an operator upgrades a large deployment and new pods either hang on an index build or serve traffic on a half-migrated schema
CREATE INDEXonLiteLLM_SpendLogs, inserts block, and on a partitioned table the concurrent variant fails with0A000Database migration failed but continuing startupand starts serving anywayAfter: the same upgrade either comes up on a complete schema or stops and says why
Database setup failed after multiple retries. The proxy cannot start safelyand exits 1 instead of servingPre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Screenshots / Proof of Fix
Shared setup: the proxy is pointed at a Postgres that is not there, so the boot migration fails every retry. The legacy resolver is used because the v2 resolver treats an unreachable database as unrecoverable and exits 2 before this check is reached
Before (5663d07)
python litellm/proxy/proxy_cli.py --skip_server_startup --use_legacy_migration_resolver; echo exit=$?After (84e8577)
default
python litellm/proxy/proxy_cli.py --skip_server_startup --use_legacy_migration_resolver; echo exit=$?explicit opt-out
ENFORCE_PRISMA_MIGRATION_CHECK=false python litellm/proxy/proxy_cli.py --skip_server_startup --use_legacy_migration_resolver; echo exit=$?Type
Bug Fix
Caveats (if any)
Severe
ENFORCE_PRISMA_MIGRATION_CHECK=falseto keep the old behaviorMedium
_prisma_migrationsrow for either rewritten migration that ran its step but never gotfinished_atno longer matches the shipped checksum, so the v2 resolver stops with its P3009 guidance instead of finishing the row. Verify the index exists, then runprisma migrate resolve --applied <migration_name>_resolve_all_migrations, only reached with--use_legacy_migration_resolverafterprisma migrate deployfails) still diffs the live database againstschema.prisma, so on a database that never got those two indexes it can emit their plainCREATE INDEXand build them with the write lock the runbook avoids. Kept out of this PR to keep the backport minimal; operators on that path build the indexes firstprisma migrate deploysucceeded is logged and setup still reports successDATABASE_URLis set but no Prisma CLI or module is available, startup skips database setup with a warning and enforcement never runsLow
schema.prismastill declares both indexes, soprisma migrate diffkeeps reporting them as drift until they are built by handprisma db pushpathsTaxonomy audit
ENFORCE_PRISMA_MIGRATION_CHECK=falseor--no-enforce_prisma_migration_check, and the Severe caveat spells out who is affectedENFORCE_PRISMA_MIGRATION_CHECKis treated as true on purpose, only Click's false spellings opt out; the test table pins each spellingcheck_migrations_no_data_rewrites.pypasses on all 189 migrationsassertlines removed fromtest_proxy_cli.pymoved into the parameterized table unchanged in meaning; every row asserts both the exit path and thesetup_databasecallOptional, the four Ruff findings intest_proxy_cli.pyare on the base branchFinal Attestation
Link to Devin session: https://app.devin.ai/sessions/6c17bcd2090f4977a0f95ef6f57911a5
Open in Devin Desktop: https://app.devin.ai/desktop/session/6c17bcd2090f4977a0f95ef6f57911a5?variant=devin
Requested by: @yassin-berriai