fix(docker): resolve LiteLLM startup failure on self-hosted dedicated deployments - #129
Merged
Merged
Conversation
…s run offline for any uid (BerriAI#34325) * fix(docker): bake non_root prisma engines at /opt/prisma so migrations run offline for any uid The non_root image baked the prisma CLI and engines under /app/.cache and used the CLI's default (library) engine mode. Prisma stopped baking the library engine, so `prisma migrate deploy` fell back to downloading it at startup, which needs network egress and a writable cache. Under an arbitrary non-root uid (OpenShift restricted-v2), an air-gapped network, or a readOnlyRootFilesystem, that download fails and the proxy starts on an empty schema while every DB endpoint returns 500. The migration entrypoint exits 0 on that failure, so a default-uid `docker run` with network never surfaced it Bake to /opt/prisma, a fixed world-readable path no cache mount shadows, and pin PRISMA_CLI_PATH plus PRISMA_CLI_QUERY_ENGINE_TYPE=binary so the baked binary engine is used directly, matching Dockerfile and Dockerfile.database. A build-time guard asserts the binary query engine is present, so a future prisma change that stops baking it fails the image build instead of silently degrading migrations Adds docker/test_offline_migration.sh, run from image-scan, which migrates a fresh Postgres with no egress as a non-root uid and asserts the schema was created, the case a default-uid `docker run` with network cannot catch * test(docker): move the offline migration check into a gated pytest and stop pinning XDG_CACHE_HOME at the read-only bake The offline migration check lived in docker/ as a shell script. It now lives in tests/proxy_migration_tests/ as a pytest gated on LITELLM_IMAGE, matching the sibling schema-migration test gated on DATABASE_URL, and image-scan invokes it with pytest instead of bash. It also asserts the migration entrypoint's exit code alongside the table count, so a crash or a container-startup failure fails loudly rather than only surfacing as a low table count Runtime XDG_CACHE_HOME pointed at /opt/prisma/.cache, which is baked a+rX with no write, so any XDG-aware library writing a cache at runtime would be denied for every uid. Leave it unset so it falls back to $HOME/.cache (/app/.cache, created here and owned by the runtime uid), matching Dockerfile and Dockerfile.database which never pin XDG at runtime. A second test guards against a future edit pointing a cache or home var back at the read-only bake
mateo-di
force-pushed
the
sc-566140/restore-prisma-binary-targets
branch
from
August 12, 2026 12:43
aff3eb7 to
26c3a72
Compare
image-scan.yml only ever triggered on main/litellm_internal_staging/ litellm_oss_branch/litellm_** — carto/main was never in the list, so this workflow (including the offline-migration regression test just cherry-picked from upstream) has never run automatically on a CARTO PR since the fork's default branch became carto/main.
mateo-di
marked this pull request as ready for review
August 12, 2026 13:14
mateo-di
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Since #121 (Upstream Sync: LiteLLM v1.92.0), the
non_rootruntime image can't apply Prisma migrations in any environment without internet egress:ded-k8sand any egress-restricted Self-Hosted deployment.prisma migrate deployfails trying to downloadlibquery_engine.so.nodefrombinaries.prisma.sh, logs "Database migration failed but continuing startup" (non-fatal), and the pod comes up healthy with zero schema — every keyed AI-api request then 401s withThe table public.LiteLLM_VerificationToken does not exist.Root cause
The sync's restructured
docker/Dockerfile.non_rootdroppedPRISMA_CLI_BINARY_TARGETSfrom the runtime-stageENVblock, silently reverting a prior CARTO fix for the same failure class (arm64 schema-engine fetching). But that env var was never the durable fix — Prisma's own platform detection is "arch-blind" on the Wolfi base, so relying on it to resolve a cached engine by name is inherently fragile.Upstream had already fixed this properly, just not in the release CARTO happened to sync to: BerriAI#33853 and BerriAI#34325 (merged 2026-07-18 and 2026-07-23) bake the Prisma CLI and engines at a fixed path,
/opt/prisma, and resolve them by exact path (PRISMA_CLI_PATH,PRISMA_CLI_QUERY_ENGINE_TYPE=binary) instead of by Prisma's name-based platform guess. Their commit message describes the exact same symptom independently: "fell back to a nodeenv Node download that crashes on Wolfi (libatomic.so.1)." Both PRs merged to upstreammainbeforev1.92.0published, butv1.92.0's tag doesn't include them — BerriAI backports fixes to separatestable/X.Y.xbranches (see BerriAI#34670), and this fix landed onstable/1.93.x, not on thev1.92.0lineage CARTO's sync picked.What changed
Cherry-picked upstream's actual fix (BerriAI#34325) rather than carrying a bespoke patch, so the fork stays aligned with upstream's approach:
docker/Dockerfile.non_root— as upstream merged it, minus CARTO's now-supersededPRISMA_CLI_BINARY_TARGETS/PRISMA_BINARY_CACHE_DIRlines (upstream's path-based resolution replaces what those were working around).tests/proxy_migration_tests/test_offline_image_migration.py— upstream's own regression test, which reproduces this exact failure (fresh DB, internal-only no-egress network, arbitrary non-root uid) and asserts the schema was actually created..github/workflows/image-scan.yml— wires that test into CI, plus one CARTO-specific addition: addedcarto/mainto the trigger'sbranches:list. Without it, this workflow (including this new regression test, and the existing vulnerability scan) never runs automatically on a CARTO PR, because it only ever triggered onmain/litellm_internal_staging/litellm_oss_branch— never oncarto/main, our actual default branch. This gap predates this fix entirely.This is the only CARTO-specific customization needed. Adopting upstream's
/opt/prismamechanism as-is fully resolves this for both amd64 and arm64 — CARTO's old arm64-specific workaround is now obsolete, since path-based resolution doesn't depend on Prisma's arch-blind detection at all.Note:
v1.95.0(the target of the currently-open sync PR #128) already includes this fix upstream — so once #128 merges, a future sync would carry it forward correctly. Landing this now unblocks the current Self-Hosted release cycle without waiting on that sync.Verification
Passed at every layer:
--no-cache, offline against a fresh Postgres on a network with no route to the internet: clean startup, nobinaries.prisma.shcalls.test_offline_image_migration.pysuite, run locally against the built image: both tests pass.Image ScanCI on this PR: both offline-migration tests pass (the CI run is red only on an unrelated CVE-scan finding, not this change)./v1/modelsrequest returns 200 with the full model list — noVerificationToken401.AI-generated code disclosure
This fix (investigation, the cherry-pick, conflict resolution, and all verification above) was done by Claude Code. Flagging for extra reviewer scrutiny given the Dockerfile/build-tooling nature of the change.