Skip to content

fix(docker): pin prisma caches to a UID-independent path so air-gapped and arbitrary-UID deploys work - #33700

Closed
ianalloway wants to merge 1 commit into
BerriAI:litellm_oss_daily_2026_07_16from
ianalloway:fix_airgap_prisma_cache
Closed

fix(docker): pin prisma caches to a UID-independent path so air-gapped and arbitrary-UID deploys work#33700
ianalloway wants to merge 1 commit into
BerriAI:litellm_oss_daily_2026_07_16from
ianalloway:fix_airgap_prisma_cache

Conversation

@ianalloway

@ianalloway ianalloway commented Jul 17, 2026

Copy link
Copy Markdown

Relevant issues

Fixes #33365

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • My PR passes all CI/CD checks (e.g., lint, format, unit tests). Every check is green except osv-scan, which fails on the base branch as well (mcp 1.26.0 in uv.lock has three High advisories; build(deps): bump mcp to 1.28.1 to clear osv-scan High findings #33591 tracks the bump and this PR does not touch uv.lock)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes). Scored 5/5

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Repro environment: docker network create --internal airgap (containers can reach each other, zero egress), postgres:16-alpine on it, and the litellm image with DATABASE_URL set and a minimal config. --user 12345:12345 simulates a Kubernetes arbitrary-UID securityContext

Published image (ghcr.io/berriai/litellm:main-latest), arbitrary UID, air-gapped:

$ docker run -d --name airgap-before --network airgap --user 12345:12345 -e DATABASE_URL=... ghcr.io/berriai/litellm:main-latest --config /app/config.yaml --port 4000
$ docker logs airgap-before
2026-07-17 13:59:15 - litellm_proxy_extras - INFO - prisma db error: Traceback (most recent call last):
FileNotFoundError: [Errno 2] No such file or directory: '/.cache/prisma-python/binaries/5.4.2/ac9d7041ed77bcc8a8dbd2ab6616b39013829574'
  File ".../prisma/cli/prisma.py", line 78, in ensure_cached
    cache_dir.mkdir(parents=True)
PermissionError: [Errno 13] Permission denied: '/.cache'
$ docker ps -a --filter name=airgap-before --format '{{.Status}}'
Exited (3) 4 seconds ago

Same image built with this PR, same command, arbitrary UID, air-gapped:

$ docker logs airgap-fixed-uid | grep -iE "migrate deploy completed|Uvicorn running"
2026-07-17 14:32:47,271 - litellm_proxy_extras - INFO - prisma migrate deploy completed
INFO:     Uvicorn running on http://0.0.0.0:4000 (Press CTRL+C to quit)
$ docker exec airgap-fixed-uid python3 -c "import urllib.request; print(urllib.request.urlopen('http://localhost:4000/health/liveliness').read().decode())"
"I'm alive!"

Root regression check (patched image, default user, air-gapped): migrations complete and the proxy serves identically. The remote model-cost-map fetch logs a warning and falls back to the local backup, which is the expected offline behavior

Type

🐛 Bug Fix

Changes

Prisma resolves every cache location relative to $HOME: the python client looks for engines in ~/.cache/prisma-python/binaries/<version>/<hash> and the node CLI keeps its engines in ~/.cache/prisma. The published root images bake these caches into /root/.cache and rely on the process running as root with HOME=/root. Kubernetes deployments that set an arbitrary runAsUser (standard pod hardening, and the default on OpenShift) get HOME=/ instead, so prisma misses the baked caches, tries to mkdir /.cache and re-download the toolchain, and the pod dies; in an air-gapped cluster the download can never succeed, which is the failure in #33365 (and earlier #4915). The same $HOME dependence is why the reporter's traceback shows nodeenv downloading Node into /.cache/prisma-python/nodeenv

The fix pins the caches to a fixed, UID-independent path at build time and runtime: XDG_CACHE_HOME and PRISMA_BINARY_CACHE_DIR are set to /opt/prisma-cache in both stages of Dockerfile and docker/Dockerfile.database, prisma generate in the builder populates that path, the runtime stage copies it and makes it world-readable (a+rX). /opt is used rather than /app/.cache deliberately: deployments with readOnlyRootFilesystem mount an emptyDir over /app/.cache, which would shadow baked-in engines (the old comment in Dockerfile.database documents that trap). The non_root image already ships equivalent env pinning; this brings the two root images in line

No application code changes. Runtime behavior for existing root deployments is unchanged (verified below); arbitrary-UID and air-gapped deployments go from crash-looping to serving

QA runbook

  1. docker network create --internal airgap and start postgres on it
  2. Run the image on that network with --user 12345:12345, DATABASE_URL pointing at the postgres, and any minimal config
  3. Before this change the container exits within seconds with PermissionError: [Errno 13] Permission denied: '/.cache' after failing to find /.cache/prisma-python/binaries/...; after it, migrations run and the proxy serves on :4000 with no egress
  4. Repeat without --user (root) to confirm the default path still works

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

@ianalloway
ianalloway requested review from a team and Copilot July 17, 2026 14:35
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@greptile-apps

greptile-apps Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes Prisma cache resolution for arbitrary-UID and air-gapped container deployments by pinning the cache to a fixed, UID-independent path (/opt/prisma-cache) instead of relying on $HOME/.cache, which breaks when Kubernetes sets a non-root runAsUser.

  • XDG_CACHE_HOME and PRISMA_BINARY_CACHE_DIR are set to /opt/prisma-cache in both the builder and runtime stages of both Dockerfiles, so prisma generate writes binaries there at build time and the runtime process finds them without attempting any downloads.
  • The old COPY --from=builder /root/.cache/prisma ... pair is replaced by a single COPY --from=builder /opt/prisma-cache /opt/prisma-cache, followed by chmod -R a+rX to make the cache world-readable for any runtime UID.
  • No application code is changed; the fix is purely at the Docker image layer and is consistent with how the non-root variant already handles this.

Confidence Score: 5/5

Safe to merge — the change is confined to Dockerfile build instructions, is verified end-to-end by the author in an air-gapped, arbitrary-UID environment, and is consistent with the approach already used in the non-root image variant.

Both Dockerfiles apply the fix symmetrically and correctly: the ENV vars are set before prisma generate in the builder stage so the cache is written to the right path, the full cache directory is then copied to the runtime stage, and chmod -R a+rX ensures readability under any UID. The /opt/prisma-cache path deliberately avoids the /app/.cache emptyDir shadowing trap that was already documented in the prior comment block. No application code is touched.

No files require special attention.

Important Files Changed

Filename Overview
Dockerfile Sets XDG_CACHE_HOME and PRISMA_BINARY_CACHE_DIR to /opt/prisma-cache in both builder and runtime stages, copies the cache from builder, and applies a+rX permissions — correctly addressing the UID-independent path requirement.
docker/Dockerfile.database Identical fix to Dockerfile: pins Prisma caches to /opt/prisma-cache in both stages; no divergence from the main Dockerfile's approach.

Reviews (1): Last reviewed commit: "fix(docker): pin prisma caches to a UID-..." | Re-trigger Greptile

@codecov

codecov Bot commented Jul 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@ianalloway

Copy link
Copy Markdown
Author

Closing this in favor of #33853, which shipped an equivalent fix (baking the prisma CLI and engines at a fixed /opt path so migrations run for any uid offline) and has already merged and been backported to the stable 1.90.x and 1.91.x lines. #34325 followed up with the non_root variant. The default branch now resolves the prisma cache at /opt/prisma independent of $HOME, which is what this PR was after, so there is nothing left here to add. Thanks to the folks who landed those

@ianalloway ianalloway closed this Jul 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants