Skip to content

fix(proxy): fail the standalone prisma migration entrypoint on migration errors - #37692

Merged
yassin-berriai merged 6 commits into
litellm_internal_stagingfrom
devin/lit-5385-migration-job-exits-0-on-failed-migration-by-default-so
Aug 20, 2026
Merged

fix(proxy): fail the standalone prisma migration entrypoint on migration errors#37692
yassin-berriai merged 6 commits into
litellm_internal_stagingfrom
devin/lit-5385-migration-job-exits-0-on-failed-migration-by-default-so

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • Failed migration Job still exits 0
  • GitOps then deploys against a stale schema
  • Failed prisma generate also exits 0

How it solves it:

  • Migration entrypoint enforces the check by default
  • prisma generate failure returns its exit code
  • ENFORCE_PRISMA_MIGRATION_CHECK=false restores the old leniency
  • In-proxy startup default stays lenient

User Flow

Before: a platform engineer upgrading the gateway with ArgoCD sees the sync go green even though the database was never migrated, then the app serves errors against an out-of-date schema

  1. They bump the gateway chart version and let ArgoCD sync in an air-gapped cluster
  2. The pre-upgrade migration Job logs a database error and finishes as Succeeded, exit code 0
  3. ArgoCD marks the whole app Healthy and rolls out the new gateway pods
  4. They hit the new release, for example POST https://litellm-domain/v1/chat/completions, and get 500s from endpoints whose columns do not exist yet
  5. Only after they explicitly set ENFORCE_PRISMA_MIGRATION_CHECK=true and re-sync does the Job fail and stop the rollout

After: the same sync stops at the migration Job, so nothing gets deployed against a stale schema

  1. They bump the gateway chart version and let ArgoCD sync in an air-gapped cluster
  2. The pre-upgrade migration Job logs the database error and finishes as Failed, exit code 1
  3. ArgoCD reports the sync as Degraded and does not roll out the new gateway pods, so the old release keeps serving traffic on the schema it matches
  4. They fix the database access, re-sync, and the Job succeeds before the new pods start
  5. Anyone who wants the previous behavior can set ENFORCE_PRISMA_MIGRATION_CHECK=false on the Job and get exit 0 on a failed migration again

Relevant issues

Linear ticket

Resolves LIT-5385

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Shared setup: point the migration entrypoint at an unreachable Postgres so prisma migrate deploy fails, and print the exit code the Job would report

export DATABASE_URL='postgresql://litellm:wrongpw@127.0.0.1:5599/nope'
export DISABLE_SCHEMA_UPDATE=false
export LITELLM_MASTER_KEY=sk-1234

Before (3a31331)

Failed migration, default settings

  1. Run the migration entrypoint the Helm Job runs
uv run --no-sync python litellm/proxy/prisma_migration.py; echo "exit code: $?"
  1. Observed output, the migration fails yet the Job reports success
Datasource "client": PostgreSQL database "nope", schema "public" at "127.0.0.1:5599"
2026-08-20 20:12:17,703 - litellm_proxy_extras - INFO - Running prisma migrate deploy
2026-08-20 20:12:24,049 - litellm_proxy_extras - INFO - prisma db error: Error: P1001: Can't reach database server at `127.0.0.1`:`5599`
Please make sure your database server is running at `127.0.0.1`:`5599`.
LiteLLM Proxy: Using default (v1) migration resolver.
LiteLLM Proxy: Database migration failed but continuing startup. Set --enforce_prisma_migration_check or ENFORCE_PRISMA_MIGRATION_CHECK=true to exit on failure.
LiteLLM: Setup complete. Skipping server startup as requested.
exit code: 0

Failed migration, ENFORCE_PRISMA_MIGRATION_CHECK=false

  1. Not applicable before this change: leniency was the only behavior, shown by the case above

After (da75cd5)

Failed migration, default settings

  1. Run the same command
uv run --no-sync python litellm/proxy/prisma_migration.py; echo "exit code: $?"
  1. Observed output, the Job now fails so the deploy is blocked
Datasource "client": PostgreSQL database "nope", schema "public" at "127.0.0.1:5599"
2026-08-20 20:12:25,510 - litellm_proxy_extras - INFO - Running prisma migrate deploy
2026-08-20 20:12:31,696 - litellm_proxy_extras - INFO - prisma db error: Error: P1001: Can't reach database server at `127.0.0.1`:`5599`
Please make sure your database server is running at `127.0.0.1`:`5599`.
LiteLLM Proxy: Using default (v1) migration resolver.
LiteLLM Proxy: Database setup failed after multiple retries. The proxy cannot start safely. Please check your database connection and migration status.
exit code: 1

Failed migration, ENFORCE_PRISMA_MIGRATION_CHECK=false

  1. Opt back into the old behavior
ENFORCE_PRISMA_MIGRATION_CHECK=false uv run --no-sync python litellm/proxy/prisma_migration.py; echo "exit code: $?"
  1. Observed output, unchanged leniency
Datasource "client": PostgreSQL database "nope", schema "public" at "127.0.0.1:5599"
2026-08-20 20:13:29,818 - litellm_proxy_extras - INFO - Running prisma migrate deploy
2026-08-20 20:13:36,293 - litellm_proxy_extras - INFO - prisma db error: Error: P1001: Can't reach database server at `127.0.0.1`:`5599`
Please make sure your database server is running at `127.0.0.1`:`5599`.
LiteLLM Proxy: Using default (v1) migration resolver.
LiteLLM Proxy: Database migration failed but continuing startup. Set --enforce_prisma_migration_check or ENFORCE_PRISMA_MIGRATION_CHECK=true to exit on failure.
LiteLLM: Setup complete. Skipping server startup as requested.
exit code: 0

Type

🐛 Bug Fix

Caveats (if any)

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/d6ef92e6d1ea4b39ae7aabb14b4dc2a2
Requested by: @yassin-berriai

yassin-berriai and others added 2 commits August 20, 2026 20:04
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@greptile-apps

greptile-apps Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR makes the standalone Prisma migration entrypoint fail by default when migration or client-generation steps fail, while retaining explicit opt-out behavior

  • Defaults the standalone entrypoint to enforcing migration checks
  • Propagates a failed prisma generate exit status
  • Adds focused tests for enforced and lenient execution paths

Confidence Score: 5/5

The PR appears safe to merge

No blocking failure remains

Important Files Changed

Filename Overview
litellm/proxy/prisma_migration.py Refactors the standalone entrypoint into a testable main function and propagates migration and Prisma generation failures unless enforcement is explicitly disabled
tests/test_litellm/proxy/test_prisma_migration.py Covers default enforcement, explicit opt-out, Prisma generation exit propagation, and migration failure propagation

Reviews (2): Last reviewed commit: "Merge remote-tracking branch 'origin/lit..." | Re-trigger Greptile

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@codecov

codecov Bot commented Aug 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.44444% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
litellm/proxy/prisma_migration.py 94.44% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@codspeed-hq

codspeed-hq Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing devin/lit-5385-migration-job-exits-0-on-failed-migration-by-default-so (c958ad5) with litellm_internal_staging (e07a712)1

Open in CodSpeed

Footnotes

  1. No successful run was found on litellm_internal_staging (22e8b45) during the generation of this report, so e07a712 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…evin/lit-5385-migration-job-exits-0-on-failed-migration-by-default-so
@yassin-berriai

Copy link
Copy Markdown
Contributor

@greptileai please review the current head 63052c3b8e. Your score is anchored to the earlier commit da75cd5fb0, so it predates the latest changes

…evin/lit-5385-migration-job-exits-0-on-failed-migration-by-default-so
@yassin-berriai
yassin-berriai merged commit 7b20828 into litellm_internal_staging Aug 20, 2026
69 checks passed
@yassin-berriai
yassin-berriai deleted the devin/lit-5385-migration-job-exits-0-on-failed-migration-by-default-so branch August 20, 2026 23:20
shudonglin pushed a commit to rayward-external/litellm that referenced this pull request Aug 21, 2026
Genuine upstream breakage from this sync's PR BerriAI#37692 (made the
standalone entrypoint's prisma generate failure fatal by default
instead of log-only), colliding with a real non-root permission
constraint: prisma-python's generate() unconditionally re-copies
schema.prisma into the installed package and chmod's the copy even
when content already matches, and chmod requires owning the
destination file, which no arbitrary runtime uid ever does for a file
baked into the image at build time.

Both runtime images already bake the generated client from this same
schema.prisma at build time, so regenerating it at container start was
always redundant work. Add LITELLM_PRISMA_CLIENT_PREBAKED to skip the
runtime entrypoint's prisma generate call when set, and set it in both
Dockerfile and docker/Dockerfile.non_root.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants