fix(migrations): recover from an interrupted Prisma toolchain install

[yassin-berriai]

TLDR

Problem this solves:

• Killed Node toolchain install breaks migrations permanently
• Prisma skips reinstalling a half-installed Node cache
• The 60s migration timeout was not overridable

How it solves it:

• Bootstrap the toolchain under its own generous timeout
• Delete a cache directory left without a node binary
• Route all 14 hardcoded timeouts through one configurable helper

Relevant issues

Linear ticket

Resolves LIT-5165

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Both captures ran on the same rig against the same Postgres 16 container. Before runs are at ad79b314c5 (the base this branch forked from); after runs are at 02a22b41de, the first commit on this branch. The current commit amends only the finite-value guard on the timeout overrides onto that tree, which does not touch any path these captures exercise. Each run prints the package directory it actually loaded so the two sides cannot be confused, and the base tree was checked to contain zero occurrences of the new symbol before the before-run

A cold machine where installing the Node toolchain takes 75s

This is the reported failure. prisma migrate deploy carries the whole toolchain install inside its 60s budget, gets killed, and the four-attempt retry loop cannot help because every attempt starts the same install from scratch

Before, on ad79b314c5:

$ PYTHONPATH=$BASE/litellm-proxy-extras python run_migrations.py
package under test: $BASE/litellm-proxy-extras/litellm_proxy_extras
nodeenv cache dir: $SCRATCH/ne-slow exists= False has bin/node= False
INFO litellm_proxy_extras: Attempt 1 timed out
INFO litellm_proxy_extras: Attempt 2 timed out
INFO litellm_proxy_extras: Attempt 3 timed out
INFO litellm_proxy_extras: Attempt 4 timed out
setup_database returned: False in 281.0s

$ psql -tAc "select count(*) from information_schema.tables
             where table_schema='public' and table_name like 'LiteLLM%'"
0

After, on this branch, same 75s install, same empty database:

$ PYTHONPATH=$BRANCH/litellm-proxy-extras python run_migrations.py
package under test: $BRANCH/litellm-proxy-extras/litellm_proxy_extras
nodeenv cache dir: $SCRATCH/ne-slow exists= False has bin/node= False
INFO litellm_proxy_extras: Preparing the Prisma CLI toolchain (timeout 600.0s)
INFO litellm_proxy_extras: Prisma CLI toolchain ready
INFO litellm_proxy_extras: prisma migrate deploy stdout: Prisma schema loaded from schema.prisma
setup_database returned: True in 93.2s

$ psql -tAc "select count(*) from information_schema.tables
             where table_schema='public' and table_name like 'LiteLLM%'"
68

The 84 seconds of toolchain install land inside the bootstrap budget, and migrate deploy then runs against a warm toolchain and finishes well inside its own 60s

A cache directory left behind by an install that was already killed

This is the state a pod is stuck in after the first failure: the cache directory exists, but the node binary was never written. Prisma tests that directory for existence alone, so it skips reinstalling and fails on the missing binary forever

$ mkdir -p $SCRATCH/ne-run/bin $SCRATCH/ne-run/lib   # what a killed install leaves

Before, on ad79b314c5:

$ PYTHONPATH=$BASE/litellm-proxy-extras python run_migrations.py
package under test: $BASE/litellm-proxy-extras/litellm_proxy_extras
nodeenv cache dir: $SCRATCH/ne-run exists= True has bin/node= False
FileNotFoundError: [Errno 2] No such file or directory: '$SCRATCH/ne-run/bin/node'
FileNotFoundError: [Errno 2] No such file or directory: '$SCRATCH/ne-run/bin/node'
FileNotFoundError: [Errno 2] No such file or directory: '$SCRATCH/ne-run/bin/node'
FileNotFoundError: [Errno 2] No such file or directory: '$SCRATCH/ne-run/bin/node'
setup_database returned: False in 10.0s

$ psql -tAc "select count(*) ... like 'LiteLLM%'"
0

After, on this branch, from the identical starting state:

$ PYTHONPATH=$BRANCH/litellm-proxy-extras python run_migrations.py
package under test: $BRANCH/litellm-proxy-extras/litellm_proxy_extras
nodeenv cache dir: $SCRATCH/ne-run exists= True has bin/node= False
WARNING litellm_proxy_extras: Node toolchain at $SCRATCH/ne-run has no node, so a previous
        install was interrupted. Removing it so it can be reinstalled.
INFO litellm_proxy_extras: Preparing the Prisma CLI toolchain (timeout 600.0s)
INFO litellm_proxy_extras: Prisma CLI toolchain ready
INFO litellm_proxy_extras: prisma migrate deploy stdout: Prisma schema loaded from schema.prisma
setup_database returned: True in 17.9s

$ psql -tAc "select count(*) ... like 'LiteLLM%'"
68

A real proxy starting against a real database

Fresh database, migrations applied by the proxy itself at startup, then a real request billed to a real provider

$ python litellm/proxy/proxy_cli.py --config litellm/proxy/dev_config.yaml --port 4000
INFO litellm_proxy_extras: Preparing the Prisma CLI toolchain (timeout 600.0s)
INFO litellm_proxy_extras: Prisma CLI toolchain ready
Applying migration `20250326162113_baseline`
Applying migration `20250326171002_add_daily_user_table`
... (every committed migration applied)

$ curl -s http://localhost:4000/health/readiness
{"status": "healthy", "db": "connected"}

$ curl -s -X POST http://localhost:4000/key/generate \
    -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
    -d '{"models":["gpt-5.5"],"key_alias":"toolchain-proof"}'
sk-WJadJDv...

$ curl -s -X POST http://localhost:4000/v1/chat/completions \
    -H "Authorization: Bearer sk-WJadJDv..." -H "Content-Type: application/json" \
    -d '{"model":"gpt-5.5","messages":[{"role":"user","content":"Reply with the single word: migrated"}]}'
model: gpt-5.5
content: migrated

$ psql -tAc "select model, total_tokens, round(spend::numeric,6)
             from \"LiteLLM_SpendLogs\" order by \"startTime\" desc limit 1"
openai/gpt-5.5|32|0.000635

Type

🐛 Bug Fix

Changes

The Prisma CLI is a Node program, and its first invocation in a fresh container installs a private Node runtime and npm-installs the CLI itself. That one-time install used to share the 60s budget that bounds each individual migration command, so on a cold or slow machine it was killed before it could finish

A killed install does not correct itself. The installer leaves its cache directory behind, and Prisma decides whether to install by testing that directory for existence alone, so every later attempt skips the install and then fails on a node binary that was never written. The existing four-attempt retry loop is powerless against this: each attempt re-enters the same resolution, sees the same directory, and fails the same way. One slow start turns into a pod that never migrates again

A new litellm_proxy_extras/prisma_toolchain.py separates the two concerns. heal_incomplete_nodeenv_cache() deletes a cache directory that exists without a node binary (Scripts/node.exe on Windows) so an interrupted install becomes recoverable, and ensure_prisma_toolchain() runs the install as its own step under its own budget before any migration command. It never raises; a toolchain that cannot be prepared is reported and stepped over so the real Prisma command produces the real error

Both budgets are now overridable, so an operator can widen them without waiting for a release. LITELLM_PRISMA_BOOTSTRAP_TIMEOUT bounds the toolchain install and defaults to 600s. LITELLM_PRISMA_COMMAND_TIMEOUT bounds each individual Prisma command and defaults to 60s, unchanged from today. A value that is not a positive number is ignored with a warning rather than silently disabling the timeout

Every previously hardcoded timeout now goes through one helper instead of fourteen literals. Two of those sites (generating and resolving the baseline migration) were at 30s and are now at 60s. That is deliberate: it is a loosening, the only cost is up to 30 extra seconds before a rare failure is reported, and 30s for a baseline migrate diff against a large database was thin to begin with. One knob is a simpler contract than two

An override is honoured only when it parses as a finite positive number. This one is worth spelling out, because it is the same failure this PR exists to fix arriving through the door this PR installs: inf and NaN parse as perfectly good floats and survive a plain positivity check, since every NaN comparison is False and infinity is not less than or equal to zero, and subprocess treats either as no deadline at all

$ python -c 'import subprocess, time
t = time.monotonic()
subprocess.run(["python", "-c", "import time; time.sleep(3)"], timeout=float("inf"))
print("timeout=inf -> returned after %.1fs (timeout NEVER fired)" % (time.monotonic() - t))'
timeout=inf -> returned after 3.0s (timeout NEVER fired)

So LITELLM_PRISMA_COMMAND_TIMEOUT=inf, or a fat-fingered 1e400 that overflows to infinity, would have silently removed the timeout from every Prisma subprocess. The guard is math.isfinite rather than a string check precisely because of that overflow case, which a check on the raw text would miss

The cache directory is resolved the supported way, from PRISMA_NODEENV_CACHE_DIR first since that is what Prisma itself reads, then from Prisma's own config.nodeenv_cache_dir behind a guarded import because litellm-proxy-extras declares no runtime dependencies, and finally from the documented default. There is no hardcoded path

ensure_prisma_toolchain() runs unconditionally rather than behind a "does the toolchain look complete" predicate. Measured cost is 2.4s with the toolchain already warm, and it is paid only on starts that actually run migrations, since ProxyExtrasDBManager.setup_database is the only caller. A predicate would have to couple to two separate Prisma internals to be correct, and because use_global_node defaults to true the nodeenv cache directory does not exist at all on a machine that already has node, so the predicate would fire on every start in exactly the environments that never needed it

nodeenv_cache_dir() returns Optional[Path] rather than Path | None deliberately. This package declares requires-python = ">=3.9" in its own pyproject.toml and is publishable standalone, and a return annotation is evaluated at definition time, so Path | None is a TypeError on 3.9. The neighbouring utils.py already uses Optional[str]. The root pyproject.toml requiring 3.10 does not govern where this package can be installed

The regression tests live in tests/proxy_migration_tests/, alongside the other migration tests, rather than in the conventionally-named tests/litellm-proxy-extras/, which is referenced by no CI job. Worth stating plainly rather than overclaiming: .circleci/config.yml runs that directory in its schema_migration_check job against a live Postgres, but CircleCI posts no per-job checks on pull requests in this repository, so I cannot confirm from outside that the job executed. What is demonstrable is that the whole directory passes locally against a live Postgres, 23 passed and 2 skipped. Anyone with Buildkite access can settle the rest in seconds. They drive real subprocesses through a fake prisma on PATH rather than patching anything, so setup_database is exercised end to end. Every mutation of the fix is caught: reverting the heal to Prisma's own existence-only rule kills 3 tests, running the bootstrap on the per-command budget kills 2, reintroducing a single hardcoded literal kills 1, dropping the bootstrap call kills 1, and healing unconditionally kills 1

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[yassin-berriai]

@greptileai please review

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

The PR makes Prisma migrations recover from interrupted Node toolchain installation and allows operators to configure bounded bootstrap and command timeouts.

• Adds incomplete nodeenv-cache healing and a separately timed Prisma bootstrap step.
• Routes Prisma subprocess timeouts through finite-positive environment-variable validation.
• Adds end-to-end regression coverage for cache recovery, bootstrap ordering, timeout isolation, and timeout overrides.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
litellm-proxy-extras/litellm_proxy_extras/prisma_toolchain.py	Adds recoverable Prisma toolchain bootstrapping, cache healing, and finite-positive timeout parsing; the previously reported timeout and annotation concerns are resolved or invalid.
litellm-proxy-extras/litellm_proxy_extras/utils.py	Bootstraps the Prisma toolchain before migration setup and consistently replaces fixed subprocess deadlines with the configurable command timeout.
litellm-proxy-extras/litellm_proxy_extras/replica_identity.py	Routes the replica-identity Prisma subprocess through the shared configurable timeout.
tests/proxy_migration_tests/test_prisma_toolchain.py	Adds fully typed regression tests covering interrupted caches, bootstrap behavior, timeout validation, and migration invocation ordering.

_{Reviews (2): Last reviewed commit: "fix(migrations): recover from an interru..." | Re-trigger Greptile}

[greptile-apps]

Optional annotation uses legacy syntax

The new nodeenv_cache_dir API uses Optional[Path] instead of the repository-standard Path | None, adding an avoidable style inconsistency and unnecessary Optional import.

Rule Used: In this repo, prefer str | None (PEP 604 union s... (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[yassin-berriai]

@greptileai correct catch on non-finite overrides. Fixed in 9f7b0a0 with math.isfinite, so inf, NaN and 1e400 now fall back to the default.

[greptile-apps]

Good fix — math.isfinite correctly rejects all three edge cases (inf, NaN, and 1e400 which overflows to inf on conversion), whereas the earlier > 0 guard would have silently accepted inf and let it propagate as a timeout value. The implementation in _timeout_from_env now looks correct.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[yassin-berriai]

@greptileai the non-finite override finding is fixed; please re-review at head 9f7b0a0 so the score reflects the current commit rather than 02a22b4.