fix(proxy-extras): retry P3009 when a peer already recovered the named migration row

[devin-ai-integration]

TLDR

Problem this solves:

• With 3+ replicas booting together, one can exit on startup
• It says "Migration completion could not be verified" after a peer already fixed it

How it solves it:

• Read back the exact ledger row P3009 named, by name and start time
• If a peer already rolled it back or finished it, log and retry

User Flow

Before: an operator scales a fresh deployment to 3 replicas and one replica crash-loops at startup

1. The operator starts 3 replicas against one empty database at the same time
2. Two replicas hit a deadlock on the same migration and one of them rolls the failed attempt back
3. The third replica, which saw the failure before that rollback, logs "Migration completion could not be verified. LiteLLM startup has stopped." and exits

After: the same rollout comes up with every replica healthy

1. The operator starts 3 replicas against one empty database at the same time
2. Two replicas hit a deadlock on the same migration and one of them rolls the failed attempt back
3. The third replica logs that a peer already recovered the migration, retries, and finishes startup

Screenshots / Proof of Fix

Root cause: when replica C gets P3009 naming a deadlocked row, it reads that row's logs to decide whether to recover. The read only matched rows with finished_at IS NULL AND rolled_back_at IS NULL, so if replica B rolled the row back first, C read nothing and stopped. A plain "any unresolved row for this name" check is not enough either, because B's retry can already have inserted a fresh in-flight row with the same name. P3009 prints the failed row's start time (The <name> migration started at <ts> failed), so both ledger reads now key on migration_name and started_at = <ts>::timestamptz

P3009 for (name, ts):
  row (name, ts) unresolved, logs contain "deadlock detected" -> roll back, retry  (unchanged)
  row (name, ts) rolled back or finished                        -> log info, retry   (new, no rollback issued)
  otherwise (no marker, no row, no DATABASE_URL, no psycopg, DB error) -> same RuntimeError as today

I checked the stderr format and the timestamp match against real Prisma 5.4.2 (the CLI the image ships) and Postgres 16: P3009 prints one started at line per unresolved row with a variable fraction (23:30:00 UTC, 23:31:00.500 UTC, 23:20:56.439594 UTC), and started_at = '<ts>'::timestamptz matches only the reported row, including after it is rolled back and a fresh in-flight row with the same name is inserted

Before (b94eaa6)

New regression tests on main

1. uv run --no-sync pytest tests/unit/litellm_proxy_extras/test_litellm_proxy_extras_utils.py -k TestConcurrentP3009Recovery -p no:cacheprovider -q
2. Output:

FFF.                                                                     [100%]
E           RuntimeError: Migration completion could not be verified. LiteLLM startup has stopped.
E           Prisma migration history (migration name and start time):
E           Error: P3009
E           The `20260415120000_health_check_latest_per_model_index` migration started at 2026-10-02 23:20:56.439594 UTC failed
FAILED tests/unit/litellm_proxy_extras/test_litellm_proxy_extras_utils.py::TestConcurrentP3009Recovery::test_a_p3009_row_a_peer_already_recovered_is_retried[rolled-back]
FAILED tests/unit/litellm_proxy_extras/test_litellm_proxy_extras_utils.py::TestConcurrentP3009Recovery::test_a_p3009_row_a_peer_already_recovered_is_retried[rolled-back-beside-a-fresh-in-flight-row]
FAILED tests/unit/litellm_proxy_extras/test_litellm_proxy_extras_utils.py::TestConcurrentP3009Recovery::test_a_p3009_row_a_peer_already_recovered_is_retried[finished]
3 failed, 1 passed, 96 deselected in 4.33s

fresh-3 pooling e2e on main

1. Image built with docker build --label org.opencontainers.image.revision=b94eaa65c3213bd79015f8e3b3f52c7d44a03c53 -t litellm-docker-database:main -f docker/Dockerfile.database ., Postgres 16 container on port 55432
2. LITELLM_MIGRATION_TESTS=1 LITELLM_MIGRATION_TEST_IMAGE=litellm-docker-database:main MIGRATION_TEST_ADMIN_URL=postgresql://postgres:postgres@127.0.0.1:55432/postgres MIGRATION_TEST_CONTAINER_ADMIN_URL=postgresql://postgres:postgres@host.docker.internal:55432/postgres MIGRATION_TEST_OUTPUT=<run dir> PYTHONPATH=tests/e2e pytest tests/e2e/migrations/test_pooling.py -k "fresh-3" -vv --tb=short -o addopts= --reruns=0, 5 times
3. 5 of 5 passed on this box, so the flake did not reproduce locally; the unit test above is the deterministic repro

After (dec1a31)

New regression tests on the PR

1. Same command as Before
2. Output: 5 passed, 96 deselected in 3.41s, and the whole mapped file is 101 passed
3. Mutation check: with the new peer-recovered if block deleted, the same command gives FFF. and 3 failed, 1 passed: the rolled-back, rolled-back-beside-a-fresh-in-flight-row, and finished cases all go red
4. Second mutation check: with the started_at condition dropped from the recovered-row query (name only), beside-a-recovered-earlier-attempt goes red, so an older rolled-back attempt of the same migration cannot mask a genuinely failed one
5. Existing tests unchanged and green: litellm-proxy-extras/tests/test_setup_database_fail_fast.py 22 passed, including the P3009 deadlock, empty-logs, unreadable-ledger, and non-deadlock cases, and test_an_interrupted_migration_without_confirmation_stops passes in the mapped file

fresh-3 pooling e2e on the PR head

1. Image built from 7e562d9 (later commits only touch tests and Final annotations) with docker build --label org.opencontainers.image.revision=7e562d936df41b6b47c9ed5bbb20fb974c645d64 -t litellm-docker-database:pr -f docker/Dockerfile.database ., same Postgres container
2. LITELLM_MIGRATION_TESTS=1 LITELLM_MIGRATION_TEST_IMAGE=litellm-docker-database:pr MIGRATION_TEST_ADMIN_URL=postgresql://postgres:postgres@127.0.0.1:55432/postgres MIGRATION_TEST_CONTAINER_ADMIN_URL=postgresql://postgres:postgres@host.docker.internal:55432/postgres MIGRATION_TEST_OUTPUT=<run dir> PYTHONPATH=tests/e2e pytest tests/e2e/migrations/test_pooling.py -k "fresh-3" -vv --tb=short -o addopts= --reruns=0, run 5 times in a row
3. 5 of 5 passed (1 passed, 3 deselected each, 40.83s, 42.85s, 39.16s, 44.20s, 41.57s)
4. No replica log in either leg contained P3009 or a deadlock, so the race did not fire locally on main or on the PR; the unit tests above cover the race deterministically

Type

🐛 Bug Fix

Caveats (if any)

Low

• If P3009 stderr ever lacks the started at line, the deadlock rollback is skipped and startup stops as before
  • Prisma always prints it today, verified on 5.4.2
• The fresh-3 flake did not reproduce on this box at main, 0 of 5

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/5d9e27bce61e474497c2316aabb474d5
Open in Devin Desktop: https://app.devin.ai/desktop/session/5d9e27bce61e474497c2316aabb474d5?variant=devin

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[greptile-apps]

Confidence Score: 5/5

[Medium risk] Refactors migration error handling and adds P3009 retry logic.

The current head appears safe to merge, with the previous annotation requirement satisfied

Summary

The PR retries startup when a peer has already recovered the exact migration attempt named by P3009. Both ledger reads match the migration name and start time

• The latest commit adds the remaining required Final annotations without changing behavior
• The earlier annotation finding is fully addressed
• The regression tests include an older recovered attempt beside the named unresolved attempt and still require startup to stop
• No new actionable issues or rule violations were found

_{Reviews (3) · Last reviewed commit: "fix(proxy-extras): annotate the migratio..."}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[devin-ai-integration]

@greptileai please review the current head 766573a

[devin-ai-integration]

@greptileai please review the current head dec1a31