test(e2e): harden stage flakes for batches, UI, and MCP

[mubashir1osmani]

Relevant issues

Stage e2e job failed 5 tests: azure/vertex unified batch lifecycle, UI teamless key create, and both MCP key-access denials

Linear ticket

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

These are harness-only changes; proof is the stage e2e job going green after this lands. Local basedpyright on tests/e2e was 0 errors before commit

Type

Test

Changes

Stage e2e showed five failures that were harness/infra, not product regressions

test_batch_lifecycle[azure-unified] failed create with Azure model_not_found for the retired deployment gpt-4.1-mini-batch, while azure-encoded / model_param / provider_fallback all passed with gpt-5.4-mini-batch. The suite registers fixed names like azure-batch via /model/new and deletes them on teardown; when teardown misses, the next run load-balances the managed-file (unified) path across the pile of leftovers, including a deployment still pointing at the old Azure model. Encoding and model-param paths pin credentials differently and keep working. Batch model names now carry a per-process unique suffix so this run never shares a router entry with a stale peer

test_batch_lifecycle[vertex_ai-unified] failed a later retrieve with 500 All connection attempts failed after create already succeeded. Retrieve now retries 500s the same way cancel already did

test_create_teamless_key_offers_proxy_scope_and_persists failed with Playwright net::ERR_ABORTED on the first /ui/api-keys/ navigation; the sibling create-team and both edit cases passed. Navigation now retries that abort race

Both MCP key-access tests failed because the granted key saw no tools. The suite depends on the compose-only mcp-upstream:8090 service; stage k8s does not run it, so the permitted-key control is empty and denial never gets a fair check. The suite skips when that upstream is not TCP-reachable, and still runs under docker-compose (or when E2E_MCP_UPSTREAM_URL points at a live host)

QA runbook

• batches/test_batches_e2e.py::test_batch_lifecycle[azure-unified] - managed-file azure batch create succeeds against the current Global Batch deployment

  • POST /model/new with model_name a unique azure-batch-* and litellm_params model azure/gpt-5.4-mini-batch (needs AZURE_API_BASE / AZURE_API_KEY and a gpt-5.4-mini-batch Global Batch deployment)
  • Upload a purpose=batch file with target_model_names set to that model name and a JSONL body.model matching it
  • POST /v1/batches with that file id and expect 200 with a managed batch id (base64 litellm_proxy payload)
  • Sanity check: re-registering the same public name twice without unique suffixes can leave a second deployment; create then intermittently hits the wrong Azure model

• batches/test_batches_e2e.py::test_batch_lifecycle[vertex_ai-unified] - managed vertex batch create then retrieve survive a single provider 500

  • Register a unique vertex-batch-* deployment, upload with target_model_names, create the batch
  • GET /v1/batches/{id} twice with a short pause; if the first returns 500 connection failed, the second should still return the created batch
  • Sanity check: a permanent provider failure still fails the test after the retry budget

• management/test_key_models_dropdown_e2e.py::TestKeyModelsDropdownUI::test_create_teamless_key_offers_proxy_scope_and_persists - teamless create modal offers All Proxy Models and persists all-proxy-models

  • Open http://localhost:4000/ui/api-keys/ (or stage UI) as admin, click + Create New Key
  • Open Models, expect All Proxy Models present and All Team Models absent, submit with All Proxy Models
  • GET /key/info on the created key and expect models == ["all-proxy-models"] and team_id null
  • Sanity check: a single aborted first navigation should not fail the case after a retry

• mcp/test_mcp_key_access_e2e.py::TestMcpKeyWithoutAccessIsDenied - key without MCP grant is denied tools when the math upstream is up

  • With mcp-upstream running (compose service or E2E_MCP_UPSTREAM_URL), register it via POST /v1/mcp/server
  • Mint a key with object_permission.mcp_servers=[server_id] and one without; list tools and expect add/multiply only on the granted key
  • Call add(3,4) with the denied key and expect 403 access_denied
  • Sanity check: without the upstream host, the suite skips rather than failing product coverage for missing harness infra

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Greptile Summary

This PR hardens five flaky stage e2e tests by fixing harness/infrastructure issues rather than product regressions. All changes are confined to tests/e2e.

• Batch model uniqueness (capabilities.py, test_batches_e2e.py): Each test process now registers batch deployments under a per-run unique suffix (via unique_marker()), preventing a stale leftover from a previous run from polluting the router and misrouting the next unified-path create. retrieve_batch() gains the same 500-retry budget that cancel_batch() already had.
• UI navigation retry (test_key_models_dropdown_e2e.py): The _goto_api_keys helper retries up to three times on ERR_ABORTED / interrupted navigation, which is an SPA auth-redirect race on stage.
• MCP infra skip (mcp/conftest.py): A session-scoped autouse fixture TCP-probes the MCP upstream and skips the entire suite when it is unreachable, replacing hard failures with an informative skip message.

Confidence Score: 4/5

Safe to merge — all changes are test-harness only and improve stage reliability without touching any production code path.

Every change is in tests/e2e and addresses a real environmental flake (stale deployments, provider 500s, SPA navigation races, missing infra). The retry logic in retrieve_batch and _goto_api_keys is narrow and mirrors existing patterns. Two minor style points flagged, no production code touched.

capabilities.py for the bare next() constants; test_batches_e2e.py for the retry-constant naming.

Important Files Changed

Filename	Overview
tests/e2e/batches/capabilities.py	Adds per-run unique suffix to batch model names via unique_marker() to prevent stale deployment collisions; exports OPENAI_BATCH_MODEL / AZURE_BATCH_MODEL constants so tests reference the live names.
tests/e2e/batches/test_batches_e2e.py	Adds retrieve_batch() helper with 500-retry logic mirroring cancel_batch(); all hardcoded model-name strings replaced with OPENAI_BATCH_MODEL / AZURE_BATCH_MODEL constants.
tests/e2e/management/test_key_models_dropdown_e2e.py	Extracts _goto_api_keys() with up to 3 retries for ERR_ABORTED / interrupted navigation races; both _open_create_key_modal and _open_key_edit_form now use it.
tests/e2e/mcp/conftest.py	Adds session-scoped autouse fixture that TCP-probes the MCP upstream and skips the whole suite when it is not reachable, so missing infra doesn't produce spurious failures.

_{Reviews (1): Last reviewed commit: "test(e2e): harden stage flakes for batch..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_e2e_stage_green_fixes (4a17be4) with litellm_internal_staging (010b200)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (a2614b1) during the generation of this report, so 010b200 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩