test(e2e): migrate access-control and inference-endpoint regression tests

[mubashir1osmani]

Relevant issues

Migrates the access-control and non-chat inference-endpoint checks from the standalone litellm-regression-tests suite onto the in-repo e2e harness under tests/e2e/, so a regression in either surface fails in CI here rather than only in the external suite

Linear ticket

N/A

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests); basedpyright, py_compile, and pytest collection are green locally, and the live e2e run needs the deployed proxy plus provider keys
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

These are live e2e tests that run against the deployed proxy with real provider keys and cost real money, exactly as a consumer would hit the endpoints. To exercise them once a proxy is up:

uv run pytest tests/e2e/access_control -v
uv run pytest tests/e2e/llm_translation/test_responses_e2e.py \
  tests/e2e/llm_translation/test_messages_e2e.py \
  tests/e2e/llm_translation/test_embeddings_endpoint_e2e.py \
  tests/e2e/llm_translation/test_rerank_e2e.py \
  tests/e2e/llm_translation/test_audio_speech_e2e.py \
  tests/e2e/llm_translation/test_image_generation_e2e.py -v

The rerank case registers a cohere/rerank-v3.5 deployment, so COHERE_API_KEY must be present in the proxy environment; the others use OPENAI_API_KEY and ANTHROPIC_API_KEY, which the gateway config already relies on. I will attach the run output against the staging proxy

Type

✅ Test

Changes

access_control/ is a new suite that asserts the gateway's authorization and error-shape contract. A key limited to one model is denied a 403 key_model_access_denied when it calls another, a key scoped to allowed_routes=["llm_api_routes"] is forbidden a 403 from a management route, and a syntactically valid request naming a non-existent model is rejected with a 400 before any provider is called. The source suite asserted a 401 for the disallowed-model case against an older proxy; the live contract is now a 403 (litellm/proxy/auth/auth_checks.py), so the guard tracks current behavior, and the original unknown-route smoke is replaced by the stronger route-permission check

llm_translation/ gains one file per non-chat inference endpoint: /v1/responses, /v1/messages, /embeddings, /v1/rerank, /v1/audio/speech, and /v1/images/generations. Rather than hardcoding models into the gateway config, each test registers the deployment it needs through /model/new (with the provider key passed as an os.environ/... reference the proxy resolves), drives real provider traffic, parses the provider-native body with a typed model and asserts it carries real content instead of just a 200, then deletes the model on teardown. All HTTP goes through the shared transport, request and response bodies are typed pydantic models in models.py and the suite client, and the two new typed model-management bodies plus the allowed_routes key field are the only additions to shared harness code

The guardrail and rate_limits cases from the source suite are intentionally out of scope: the rate limiter is already covered by router/test_rate_limiter.py, and the guardrail case is deferred until a guardrail is wired into the e2e gateway config

[greptile-apps]

Greptile Summary

This PR migrates access-control and non-chat inference endpoint regression tests from the external litellm-regression-tests suite into the in-repo tests/e2e/ harness, so regressions surface in CI here rather than only in the external suite. It also refactors test_custom_pricing_e2e.py to provision models dynamically via /model/new instead of reading from the static gateway config, making each test self-contained.

• New access_control/ suite asserts the 403 + key_model_access_denied shape for per-key model allow-list violations, the 403 + "not allowed to call this route" shape for allowed_routes enforcement, and the 400 + valid JSON shape for requests naming a non-existent model.
• New llm_translation/ endpoint tests cover /v1/responses, /v1/messages, /embeddings, /v1/rerank, /v1/audio/speech, and /v1/images/generations — each registers its deployment through /model/new with an os.environ/ key reference, drives real provider traffic, parses the response with a typed Pydantic model, and deletes the deployment on teardown.
• test_custom_pricing_e2e.py is refactored to provision its own deployments rather than depend on a statically configured custom-priced-flash model in the gateway YAML, eliminating the config-file coupling and making the isolation check self-contained.

Confidence Score: 4/5

Safe to merge after fixing the missing is_ok import in endpoints_client.py; all other changes are additive test infrastructure.

The delete_model teardown helper in endpoints_client.py calls is_ok(result) without importing it from e2e_http. On the happy path (deletion succeeds) this branch is never reached. When deletion fails — network error, 500, or model already gone — Python raises NameError: name 'is_ok' is not defined instead of the intended warning, crashing teardown with a misleading error that hides what actually went wrong.

tests/e2e/llm_translation/endpoints_client.py — the is_ok import is missing.

Important Files Changed

Filename	Overview
tests/e2e/llm_translation/endpoints_client.py	New client for non-chat inference endpoints; is_ok is used in delete_model but not imported from e2e_http, causing a NameError in teardown when model deletion fails.
tests/e2e/access_control/test_access_control_e2e.py	New access-control suite asserting 403 for model-access denial, 403 for route-permission denial, and 400 for unknown models — well-structured with clear error messages.
tests/e2e/llm_translation/test_custom_pricing_e2e.py	Refactored to dynamically provision models via /model/new instead of reading from static gateway config; rates are now constants rather than parsed from YAML, removing the config-file dependency.
tests/e2e/models.py	Adds LiteLLMParamsBody, ModelInfoBody, ModelNewBody, ModelNewResponse, ModelDeleteBody, and allowed_routes field to KeyGenerateBody; all additions are clean and well-documented.
tests/e2e/llm_translation/test_rerank_e2e.py	New live e2e for /v1/rerank against Cohere; registers deployment at runtime and asserts scored results within the requested top_n.
tests/e2e/llm_translation/test_responses_e2e.py	New live e2e for /v1/responses against OpenAI; parses the typed response body and asserts non-empty text.
tests/e2e/llm_translation/test_messages_e2e.py	New live e2e for the Anthropic Messages endpoint via the gateway; asserts role and non-empty text.
tests/e2e/llm_translation/test_embeddings_endpoint_e2e.py	New live e2e for /embeddings; asserts a non-empty, non-zero vector from the OpenAI embedding model.
tests/e2e/llm_translation/test_audio_speech_e2e.py	New live e2e for /v1/audio/speech; asserts an audio content-type and non-empty binary body.
tests/e2e/llm_translation/test_image_generation_e2e.py	New live e2e for /v1/images/generations; asserts at least one image item with either url or b64_json.
tests/e2e/llm_translation/conftest.py	Adds endpoints_client session fixture alongside the existing client fixture.
tests/e2e/access_control/access_control_client.py	New client for access-control suite; clean delegation to the shared gateway/transport layer.
tests/e2e/access_control/conftest.py	Minimal conftest that wires up the AccessControlClient session fixture for the new suite.
tests/e2e/CLAUDE.md	Updated suite directory descriptions to include the new access_control/ suite and the expanded llm_translation/ scope.

_{Reviews (2): Last reviewed commit: "test(e2e): provision custom-pricing depl..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mubashir1osmani]

@greptile-apps