test(e2e): bound the post-/model/new servable wait at 40s

[mubashir1osmani]

TLDR

Problem this solves:

• A broken model reload cost 120s per deployment before failing
• That deadline is the spend-log read-back budget, not a propagation budget
• Suites creating many models stalled for minutes on a config problem
• A slow final /v1/models poll could still overrun the new overall budget

How it solves it:

• Bound first /v1/models listing at 40s (vs the old 120s spend poll_timeout)
• Then require continuous listing for 30s (product default DB reload / add_deployment interval) so multi-worker gateways finish syncing before create_model returns
• Cap each readiness poll at 5s, clamped to remaining deadline
• Unit-test first listing, continuous DB-sync window, miss reset, and request-timeout clamp

Companion ops PR

BerriAI/litellm-ops#110 removes stage's proxy_config_reload_interval_seconds: 5 override so stage uses the product default (30s). This harness PR owns the wait with a fixed 40s budget and does not read that setting

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• Greptile: fixed harness timeout is intentional (not derived from proxy general_settings); see resolved threads. Not blocking on bot score for this contract

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Both runs against a live proxy (python litellm/proxy/proxy_cli.py --config litellm/proxy/dev_config.yaml), driving the real harness call path. The wait is timed against a model name the data plane never serves, which is what a genuinely broken reload looks like from the harness

Before, with the old budget (poll_timeout=120s, poll_interval=5s):

failed after 121.4s
message: model 'model-that-will-never-be-listed' was created but never became servable
on the data plane within 120.0s of /model/new (control/data-plane propagation or
STORE_MODEL_IN_DB reload issue)

After, with this PR's bound (model_servable_timeout=40s, model_servable_interval=2s, model_servable_request_timeout=5s):

failed after 38.6s
message: model 'model-that-will-never-be-listed' was created but never became servable
on the data plane within 40.0s of /model/new (control/data-plane propagation or
STORE_MODEL_IN_DB reload issue)

Happy path create_model returned in 2.31s on the same proxy (model listed on the first poll). Unit coverage for the clamp and deadline arithmetic:

$ cd tests/e2e && python3 -m pytest test_proxy_client_model_servable.py -q
......                                                                   [100%]
6 passed in 0.01s

Confirming the happy path still works on a live proxy:

$ curl -s -X POST http://localhost:4000/model/new -H "Authorization: Bearer sk-1234" \
    -d '{"model_name":"divergence-probe-15688","litellm_params":{"model":"openai/gpt-4o-mini","api_key":"os.environ/OPENAI_API_KEY"}}'
# model_id f07ee01e-c544-474a-bd7b-24a6360de55b

$ curl -s http://localhost:4000/v1/models -H "Authorization: Bearer sk-1234" | ...
True

$ curl -s -X POST http://localhost:4000/chat/completions -H "Authorization: Bearer sk-1234" \
    -d '{"model":"divergence-probe-15688","messages":[{"role":"user","content":"hi"}],"max_tokens":5}'
http=200
{"id":"chatcmpl-E6khX4oOxfqkWYcxqXRfljjA9Nx4a","model":"divergence-probe-15688",
 "choices":[{"finish_reason":"length","index":0,"message":{"content":"Hello! How can I",...

Type

Test

Changes

_await_model_servable polled /v1/models to poll_timeout, which exists for eventually-consistent read-backs like spend rows. Model readiness is a different wait: after /model/new, the data plane must list the model before callers can invoke it. Sharing the spend read-back budget meant a broken reload was absorbed as a two-minute stall per model rather than reported

Fixed harness constants: model_servable_timeout (40s) for first listing, model_servable_db_sync_seconds (30s, product default proxy_config_reload_interval_seconds) of continuous listing after first sight, model_servable_interval (2s), model_servable_request_timeout (5s, clamped to remaining). Not read from live proxy config. A single /v1/models success is not enough: stage gateway runs multiple workers, and only the writer reloads immediately; peers/workers sync on the add_deployment job every 30s

The poll loop is extracted as await_servable, a pure function over an injected clock/sleep and a list_models callback that takes the per-poll request timeout, returning a Servable | NotServable union. tests/e2e/test_proxy_client_model_servable.py covers the happy path, multi-poll appearance, deadline give-up, the request-timeout clamp regression, and failed-read reporting without a live proxy

Scope note: this changes only how long the harness waits before reporting. It does not make a model propagate faster, and it does not fix any test that fails after the wait succeeds; those have a separate root cause on the proxy side and are tracked outside this PR. Reviewers evaluating this should read it as latency-of-failure, not as a fix for model-propagation failures

QA runbook

• tests/e2e/test_proxy_client_model_servable.py::test_clamps_request_timeout_to_remaining_deadline - when remaining budget is less than the 5s request cap, the poll receives the remaining budget so wall time never exceeds the overall timeout

  • Run cd tests/e2e && python3 -m pytest test_proxy_client_model_servable.py -q and expect 6 passed
  • Sanity check: mutating await_servable to pass the full request cap (ignoring remaining) fails this test

• ProxyClient._await_model_servable - a model the data plane never lists fails in ~40s instead of ~120s, with the diagnostic preserved

  • Start a proxy: python litellm/proxy/proxy_cli.py --config litellm/proxy/dev_config.yaml (needs STORE_MODEL_IN_DB=True and a reachable DATABASE_URL)
  • POST /model/new with the master key and a real deployment; expect the returned model id and a 200 from an immediate /chat/completions against that model name, proving the happy path still resolves on the first poll
  • Call ProxyClient._await_model_servable("model-that-will-never-be-listed") and time it; expect an AssertionError in ~40s naming the model and the 40.0s budget
  • Sanity check: 40s is a deliberate harness middle ground (not proxy general_settings) and is not hand-wavey or potentially flaky

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Greptile Summary

The PR bounds and strengthens the E2E model-readiness wait.

• Adds separate first-listing and continuous DB-sync polling phases.
• Clamps each /v1/models request and sleep to the active phase deadline.
• Rejects a first listing that arrives after the configured deadline.
• Adds per-request GET timeout overrides through both HTTP transport implementations.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
tests/e2e/proxy_client.py	Adds bounded model-readiness polling with continuous-listing validation and completes the previously requested deadline safeguards.
tests/e2e/transport.py	Plumbs an optional per-call GET timeout through the transport protocol, split router, and concrete HTTP transport.

_{Reviews (8): Last reviewed commit: "fix(e2e): reject first listing that retu..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

.

[mubashir1osmani]

@greptile-apps review it

[mubashir1osmani]

@greptileai

[mubashir1osmani]

@greptile-apps review it

[mubashir1osmani]

@greptileai

[mubashir1osmani]

@greptile-apps review, give me a new score

[mubashir1osmani]

@greptileai

Timeout is a fixed harness constant (40s), not derived from proxy general_settings or PROXY_CONFIG_RELOAD_INTERVAL_SECONDS. That is intentional: the e2e suite owns how long create_model waits for /v1/models, as a middle ground between first-poll happy path and the 120s spend poll_timeout.

[greptile-apps]

Fair point — a test harness being opinionated about its own timing contract is a legitimate design choice, and "the suite owns how long create_model waits" is a coherent position.

My concern was specifically: the PR description justifies the 40s bound as "one full default reload interval plus headroom," which ties the value semantically to the proxy's reload interval. When that reasoning is embedded in the constant, a reader configuring the proxy above 30s might not realize the harness has its own fixed contract and will see unexplained failures. The concern is documentation, not correctness.

A one-line comment on model_servable_timeout making this explicit would resolve it:

    # Fixed harness budget: suits default proxy_config_reload_interval_seconds (30 s).
    # The e2e suite owns this deadline; it is not read from proxy general_settings.
    model_servable_timeout: float = 40.0

If you consider this out of scope or already obvious from the module context, I'm fine closing this thread.

[mubashir1osmani]

Companion ops change: BerriAI/litellm-ops#110 drops stage's proxy_config_reload_interval_seconds: 5 override so stage keeps the product default (30s). This PR owns the post-/model/new wait as a fixed harness 40s budget and does not read that setting.

[mubashir1osmani]

@greptile-apps

[ryan-crabbe-berri]

@greptileai re review

[mubashir1osmani]

@greptileai

Please re-review. Latest tip rejects first listings whose response arrives after the first-listing deadline (started + 40s), so a late /v1/models return cannot open the 30s continuous DB-sync phase.