test(e2e): make dynamic model provisioning robust on split deployments

[mubashir1osmani]

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Both failures were diagnosed from the 2026-07-09 stage e2e run (image 1.0.0-main.20260709072624, split control/data-plane deployment). The fix is in this branch at f1ffbe6d94.

Root cause 1, model_id collision. The harness pinned model_info.id = model_name, so re-registering a fixed-name deployment collided on the model_id unique constraint. The backend surfaced it as a prisma.errors.UniqueViolationError swallowed into the generic 500. Reproduced and fixed live against a local proxy (curl to localhost:4000); the 500 body matches the stage error byte-for-byte:

BEFORE (harness pins model_info.id = model_name -> re-register collides)
create #1 (pinned id): HTTP 200
create #2 (pinned id): HTTP 500   <-- collision
  {"error":{"message":"{'error': 'Failed to add model to db. Check your server logs for more details.'}","type":"auth_error","param":"None","code":"500"}}

AFTER (no pinned id -> proxy assigns a unique model_id)
create #1 (no id): HTTP 200
create #2 (no id): HTTP 200   <-- both succeed, distinct model_ids

Root cause 2, split-plane propagation. /model/new succeeded on the control plane (the stage backend logged 20x POST /model/new -> 200 OK), but the gateway had not reloaded the model from the DB when the test called it, so /chat, /embeddings, /v1/messages, /ocr, /responses returned Invalid model name passed. This only shows on a split deployment; a monolithic proxy shares the process, which is why it passes locally. create_model now polls the data-plane /v1/models until the model is servable before returning. Verified against a local proxy that create-then-immediately-invoke still works (the rust OCR suite registers each provider via /model/new then calls /ocr):

test_rust_ocr_response[mistral]                     PASSED
test_rust_ocr_response[azure-ai]                    PASSED
test_rust_ocr_response[azure-document-intelligence] PASSED
test_rust_ocr_response[vertex-mistral]              PASSED

The definitive end-to-end proof is the next stage e2e run going green on the suites that were red purely because of provisioning (batches, embeddings, responses, messages, image, rerank, audio, ocr); that run happens automatically after this merges to staging.

Type

🐛 Bug Fix

✅ Test

Changes

create_model no longer assumes /model/new makes a deployment instantly callable. It polls the data-plane /v1/models until the new model_name is listed, then returns, and fails loudly with a clear message if the model never becomes servable within poll_timeout (a real propagation or STORE_MODEL_IN_DB reload problem, surfaced at the source instead of as a downstream Invalid model name passed). In the monolithic case the model is present on the first poll, so this adds a single request.

create_model also stops setting model_info.id, so the proxy assigns a unique model_id per deployment. Every suite except batches already used unique_marker() names and never collided; batches register fixed names (openai-batch, azure-batch, vertex-batch, bedrock-batch), so a leftover row from a crashed or evicted prior run made the next run's create collide on the id constraint and errored every test_batch_lifecycle case at setup. With a proxy-assigned id, re-registration is collision-free.

_await_model_servable remembers the last /v1/models poll result and, when it never returned a Success (a 5xx on the data plane or a network partition rather than the model simply not being listed yet), appends it to the timeout AssertionError so a split-deployment failure names the real cause instead of always pointing at propagation/reload.

Tests: test_e2e_gateway.py gains coverage that create_model waits for data-plane visibility, fails loudly when the model never appears, and surfaces the last data-plane error when /v1/models keeps erroring; it also asserts create_model no longer pins model_info.id. The typed fake Transport now serves /v1/models (and can return a canned error) so a regression in the wait or error-surfacing behavior fails there rather than in a live stage run.

Link to Devin session: https://app.devin.ai/sessions/226a1a920e6d4e1196a3cd3384140721
Requested by: @mubashir1osmani

[greptile-apps]

Greptile Summary

This PR hardens e2e test model provisioning against two real failures observed in split control/data-plane stage runs: a model_id unique-constraint collision (fixed by dropping the pinned model_info.id) and a race between /model/new returning and the data-plane gateway reloading its model list (fixed by polling /v1/models until the model is servable before create_model returns).

• create_model no longer sets model_info.id, letting the proxy assign a unique ID per registration so fixed-name batch deployments can be re-registered without colliding on a leftover DB row.
• _await_model_servable polls the data-plane /v1/models to a configurable deadline and raises a clear AssertionError (including the last non-Success poll result) when the model never appears, replacing the opaque downstream "Invalid model name passed" error.
• _RecordingTransport in test_e2e_gateway.py gains a /v1/models handler with configurable servable_after_gets and models_error; four new unit tests cover the wait, timeout, and error-surfacing paths deterministically without live network calls.

Confidence Score: 5/5

All changes are confined to the tests/e2e directory and fix two well-documented provisioning bugs with no production code touched.

The two root causes are clearly diagnosed, the fixes are minimal and targeted, all existing test assertions are updated to reflect correct behavior (not weakened), and the four new unit tests verify the wait/timeout/error-surface paths deterministically. No custom rules are violated.

No files require special attention.

Important Files Changed

Filename	Overview
tests/e2e/e2e_gateway.py	create_model now omits pinned model_info.id and polls /v1/models until servable; _await_model_servable surfaces the last non-Success poll result in its AssertionError.
tests/e2e/models.py	ModelInfoBody.id made optional (None default); new ModelListEntry and ModelsListResponse models added for /v1/models polling.
tests/e2e/test_e2e_gateway.py	_RecordingTransport gains /v1/models handling with configurable servable_after_gets and models_error; four new tests cover wait-for-servable, loud failure, and error surface behavior; existing tests updated to reflect removed model_id pinning.

_{Reviews (2): Last reviewed commit: "test(e2e): surface last /v1/models error..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_e2e_split_plane_provisioning (391f579) with litellm_internal_staging (1d9a86e)}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ mubashir1osmani
❌ Mubashir Osmani

---

Mubashir Osmani seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[devin-ai-integration]

@greptileai