Skip to content

fix(proxy): keep a failed prisma generate from failing the migration entrypoint - #37947

Merged
yuneng-berri merged 1 commit into
litellm_internal_stagingfrom
litellm_/image-scans-failing-92a7a1
Aug 22, 2026
Merged

fix(proxy): keep a failed prisma generate from failing the migration entrypoint#37947
yuneng-berri merged 1 commit into
litellm_internal_stagingfrom
litellm_/image-scans-failing-92a7a1

Conversation

@yuneng-berri

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • Migration Job exits 1 after applying every migration
  • Only the redundant post-migration prisma generate failed
  • It cannot write site-packages as non-root
  • Blocks rollouts the Job was meant to gate

How it solves it:

  • A failed prisma generate no longer sets the exit code
  • Migration failures stay fatal, unchanged
  • image-scan now watches the entrypoint it exercises

User Flow

Before: a platform engineer upgrading the gateway on a cluster that runs the migrations Job as a non-root uid sees the sync fail even though the database migrated cleanly

  1. They set securityContext.runAsNonRoot: true on the chart and let ArgoCD sync
  2. The pre-upgrade migration Job logs All migrations have been successfully applied. and then a PermissionError on .../site-packages/prisma/schema.prisma
  3. The Job finishes as Failed, exit code 1, despite the schema being fully created
  4. ArgoCD reports the sync Degraded and never rolls out the new pods, so the release is stuck on a database that is already migrated
  5. Their only way forward is ENFORCE_PRISMA_MIGRATION_CHECK=false, which also gives up catching genuinely failed migrations

After: the same sync completes, and a genuinely failed migration still stops it

  1. They set securityContext.runAsNonRoot: true on the chart and let ArgoCD sync
  2. The pre-upgrade migration Job logs All migrations have been successfully applied. and a warning that the client baked at image build time is being used
  3. The Job finishes as Succeeded, exit code 0
  4. ArgoCD rolls out the new pods against the migrated schema
  5. If the database is genuinely unreachable, the Job still exits 1 and still blocks the rollout, with no need to touch ENFORCE_PRISMA_MIGRATION_CHECK

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Shared setup: a Postgres on a --internal (no egress) Docker network, and the migration entrypoint run as an arbitrary non-root uid in GID 0, which is what OpenShift restricted-v2 assigns. The entrypoint under test is bind-mounted over the one in a released image so both sides run against an identical bake

docker network create --internal secfixnet-ab
docker run -d --name pgab --network secfixnet-ab -e POSTGRES_PASSWORD=pw -e POSTGRES_DB=litellm postgres:16-alpine

Before (7a1afa1)

Migration succeeds, entrypoint reports failure

  1. Run the entrypoint the Helm migrations Job runs, as uid 12345:0
docker run --rm --network secfixnet-ab --user 12345:0 -e DATABASE_URL=postgresql://postgres:pw@pgab:5432/litellm -e LITELLM_MASTER_KEY=sk-offline-migration-test -e DISABLE_SCHEMA_UPDATE=false -v /tmp/prisma_migration.before.py:/app/litellm/proxy/prisma_migration.py:ro -w /app --entrypoint python ghcr.io/berriai/litellm:v1.97.0-rc.1 litellm/proxy/prisma_migration.py; echo "EXIT CODE: $?"
  1. Observed output, the schema is applied and the Job still fails
  All migrations have been successfully applied.
  litellm_proxy_extras - INFO - prisma migrate deploy completed
  litellm_proxy_extras - INFO - Post-migration sanity check completed
  PermissionError: [Errno 13] Permission denied: '/app/.venv/lib/python3.13/site-packages/prisma/schema.prisma'
EXIT CODE: 1
  1. Confirm the database really did migrate
docker exec pgab psql -U postgres -d litellm -tAc "SELECT count(*) FROM information_schema.tables WHERE table_schema='public';"
71

Unreachable database, default settings

  1. Point the same entrypoint at a database that is not there
docker run --rm --network secfixnet-ab --user 12345:0 -e DATABASE_URL=postgresql://postgres:pw@127.0.0.1:5599/nope -e LITELLM_MASTER_KEY=sk-offline-migration-test -e DISABLE_SCHEMA_UPDATE=false -v /tmp/prisma_migration.before.py:/app/litellm/proxy/prisma_migration.py:ro -w /app --entrypoint python ghcr.io/berriai/litellm:v1.97.0-rc.1 litellm/proxy/prisma_migration.py; echo "EXIT CODE: $?"
Database setup failed after multiple retries. The proxy cannot start safely.
EXIT CODE: 1

After (9f79218)

Migration succeeds, entrypoint reports success

  1. Recreate Postgres so the run starts from an empty database
docker rm -f pgab && docker run -d --name pgab --network secfixnet-ab -e POSTGRES_PASSWORD=pw -e POSTGRES_DB=litellm postgres:16-alpine
docker exec pgab psql -U postgres -d litellm -tAc "SELECT count(*) FROM information_schema.tables WHERE table_schema='public';"
0
  1. Run the same command against the fixed entrypoint
docker run --rm --network secfixnet-ab --user 12345:0 -e DATABASE_URL=postgresql://postgres:pw@pgab:5432/litellm -e LITELLM_MASTER_KEY=sk-offline-migration-test -e DISABLE_SCHEMA_UPDATE=false -v /tmp/prisma_migration.after.py:/app/litellm/proxy/prisma_migration.py:ro -w /app --entrypoint python ghcr.io/berriai/litellm:v1.97.0-rc.1 litellm/proxy/prisma_migration.py; echo "EXIT CODE: $?"
  All migrations have been successfully applied.
  'prisma generate' exited 1; continuing with the client baked at image build time.
EXIT CODE: 0
  1. Confirm the same schema landed
docker exec pgab psql -U postgres -d litellm -tAc "SELECT count(*) FROM information_schema.tables WHERE table_schema='public';"
71

Unreachable database, default settings

  1. Point the fixed entrypoint at a database that is not there
docker run --rm --network secfixnet-ab --user 12345:0 -e DATABASE_URL=postgresql://postgres:pw@127.0.0.1:5599/nope -e LITELLM_MASTER_KEY=sk-offline-migration-test -e DISABLE_SCHEMA_UPDATE=false -v /tmp/prisma_migration.after.py:/app/litellm/proxy/prisma_migration.py:ro -w /app --entrypoint python ghcr.io/berriai/litellm:v1.97.0-rc.1 litellm/proxy/prisma_migration.py; echo "EXIT CODE: $?"
Database setup failed after multiple retries. The proxy cannot start safely.
EXIT CODE: 1
  1. Confirm the documented opt-out still works
docker run --rm --network secfixnet-ab --user 12345:0 -e DATABASE_URL=postgresql://postgres:pw@127.0.0.1:5599/nope -e LITELLM_MASTER_KEY=sk-offline-migration-test -e DISABLE_SCHEMA_UPDATE=false -e ENFORCE_PRISMA_MIGRATION_CHECK=false -v /tmp/prisma_migration.after.py:/app/litellm/proxy/prisma_migration.py:ro -w /app --entrypoint python ghcr.io/berriai/litellm:v1.97.0-rc.1 litellm/proxy/prisma_migration.py; echo "EXIT CODE: $?"
EXIT CODE: 0

Type

🐛 Bug Fix

Caveats (if any)

  • The refresh still runs, only its exit code stops counting
  • Source checkouts still rely on it; CircleCI ignores its status
  • image-scan gating this entrypoint is new, expect it on more PRs

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

…entrypoint

The standalone migration entrypoint re-runs `prisma generate` after the
migration completes. That refresh writes into the installed prisma package in
site-packages, which an arbitrary non-root uid cannot do, and which no uid can
do under a read-only root filesystem. Both are supported configurations of the
migrations Job: helm/litellm-helm/tests/migrations-job_tests.yaml asserts
runAsNonRoot, runAsUser and readOnlyRootFilesystem all render.

The write has always failed there, but the failure used to be swallowed. Making
migration failures fatal turned it into a hard exit 1, so a Job that applied
every migration correctly now reports Failed and blocks the rollout it was
supposed to gate.

The refresh is redundant in the shipped images: every Dockerfile generates the
client at build time from the same baked schema, copies it into the runtime
stage, and asserts it resolves there. It stays load-bearing only for a source
checkout, where CircleCI runs the entrypoint under `set +e` and ignores the exit
code anyway. So the call stays and only its exit code stops propagating;
migration failures are still fatal.

image-scan never ran on the change that introduced this, because its path filter
did not list the entrypoint it exercises. Add prisma_migration.py and
entrypoint.sh so the non-root offline migration test gates them from now on.
@yuneng-berri
yuneng-berri requested a review from a team August 22, 2026 17:49
@codecov

codecov Bot commented Aug 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@greptile-apps

greptile-apps Bot commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR makes post-migration Prisma client-generation failures non-fatal while preserving fatal migration failures

  • Logs failed prisma generate attempts and continues with the client baked into shipped images
  • Adds regression coverage for enforced and non-enforced configurations
  • Triggers image-scan validation when the runtime migration entrypoint changes

Confidence Score: 5/5

The PR appears safe to merge because shipped images bake the Prisma client and genuine migration failures remain fatal

The changed entrypoint suppresses only the redundant post-migration generation result, while the migration call still fails before generation when enforcement detects a database migration error

Important Files Changed

Filename Overview
litellm/proxy/prisma_migration.py Treats post-migration client regeneration as best-effort while leaving migration enforcement unchanged
tests/test_litellm/proxy/test_prisma_migration.py Verifies generation failures return success under both environment configurations and migration failures still propagate
.github/workflows/image-scan.yml Adds the production entrypoint and migration script to image-scan pull-request triggers

Reviews (1): Last reviewed commit: "fix(proxy): keep a failed prisma generat..." | Re-trigger Greptile

@codspeed-hq

codspeed-hq Bot commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing litellm_/image-scans-failing-92a7a1 (9f79218) with litellm_internal_staging (7a1afa1)

Open in CodSpeed

@yuneng-berri
yuneng-berri enabled auto-merge (squash) August 22, 2026 18:35

@mateo-berri mateo-berri left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

@yuneng-berri
yuneng-berri merged commit deab367 into litellm_internal_staging Aug 22, 2026
75 of 76 checks passed
@yuneng-berri
yuneng-berri deleted the litellm_/image-scans-failing-92a7a1 branch August 22, 2026 18:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants