Repository navigation
test(e2e): add the endpoint x provider x auth mode matrix - #42372
devin-ai-integration[bot] wants to merge 7 commits into
Conversation
… and auth modes One scenario per non-management inference endpoint, run over every catalog provider that serves it and every way the proxy can hold that provider's secret (os.environ reference, literal value, stored credential). Cells carry exact coverage registry ids and the OpenAI and Anthropic cells join the record/replay lane. The provider edge stops recording or owing the proxy's scheduled GET /v1/models discovery, and ProxyClient gains register_models so the matrix pays the propagation budget once per run instead of once per cell Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
|
bugbot run |
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| for route, name in credentials.items(): | ||
| proxy.create_credential( | ||
| CredentialCreateBody(credential_name=name, credential_values=dict(credential_values(providers[route]))) | ||
| ) | ||
| try: |
There was a problem hiding this comment.
If a later credential creation fails, earlier credentials bypass cleanup and leave stale rows that can break deterministic replay reruns.
There was a problem hiding this comment.
Fixed in 09f01a8 with the same ExitStack change: cleanup is registered per resource as soon as it exists, not after the loop
| """Provider catalog behind test_endpoint_matrix_e2e.py. | ||
|
|
||
| One `Provider` row per route the matrix drives. Adding a provider is adding a row | ||
| here (its credential fields, its backend model per endpoint family, and the edge | ||
| mount when the provider edge can record and replay it) plus the registry cells | ||
| those combinations claim. The test module never names a provider. | ||
|
|
||
| `E2E_MATRIX_PROVIDERS` and `E2E_MATRIX_AUTH_MODES` narrow the default of every | ||
| catalog row and every auth mode; an unknown name fails collection instead of | ||
| quietly selecting nothing. | ||
| """ |
There was a problem hiding this comment.
These docstrings, including similar ones in the matrix test, violate the repository directive limiting comments to complex logic, tool directives, or TODOs. This requirement must be satisfied before merging.
Context Used: AGENTS.md (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| _deployment_body(key, providers[key[0]], credentials[key[0]] if key[2] == "stored_credential" else None) | ||
| for key in keys | ||
| ) | ||
| model_ids: Final = proxy.register_models(bodies) |
There was a problem hiding this comment.
Missing secret fails every matrix cell
Medium Severity
credential_values() raises from the shared deployments fixture as soon as any selected provider is missing an env var. Because every cell depends on that fixture, env_ref cells and unrelated providers error at setup too, instead of only the inline and stored-credential cells for the provider whose secret is absent.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit e4d9c22. Configure here.
There was a problem hiding this comment.
Fixed in 09f01a8: markers allocated inside module-scoped fixtures now key off the module node, so a full recording replays a narrowed -k or E2E_MATRIX_* run
There was a problem hiding this comment.
Correction: 09f01a8 makes a missing secret fail only that provider's inline and stored_credential cells by name; env_ref cells and other providers still register
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
bugbot run |
| for route, name in credentials.items(): | ||
| proxy.create_credential( | ||
| CredentialCreateBody(credential_name=name, credential_values=dict(credential_values(providers[route]))) | ||
| ) | ||
| try: |
There was a problem hiding this comment.
When multiple stored-credential providers are selected, credentials are created before the cleanup try begins. If a later provider is missing a required environment variable or its /credentials request fails, setup raises and leaves the credentials created for earlier providers in the shared proxy database. Those stale credentials can contaminate later e2e runs. Start the protected scope before the loop or register cleanup immediately after each successful creation.
There was a problem hiding this comment.
Fixed in 09f01a8: each credential and model registers its delete on an ExitStack the moment it is created, so partial setup tears down
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…ule fixture markers Register credential and model cleanup as each one succeeds, so a failure partway through deployments setup leaves nothing behind. A provider whose secret is absent now fails only its inline and stored_credential cells by variable name; its os.environ/ cells and every other provider still run. Module-scoped fixtures allocate deterministic markers against the module node, not whichever cell pytest happens to run first, so a full recording replays a narrowed -k or E2E_MATRIX_* selection Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
bugbot run |
…rovider credentials Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
bugbot run |
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
bugbot run |
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 43a7968. Configure here.
This branch is waiting to be deployed
1 participant
TLDR
Problem this solves:
/v1/chat/completionskeeps living on/v1/messagesor/v1/responsesHow it solves it:
Providercatalog entry, no new test codeUser Flow
Before: a developer whose stored-credential Anthropic deployment answers on
/v1/chat/completionsbut fails on/v1/messageshas no test that would have caught it before release{"credential_name":"cred-1","credential_values":{"api_key":"..."},"credential_info":{}}and gets{"success":true}{"model_name":"m-1","litellm_params":{"model":"anthropic/claude-haiku-4-5","litellm_credential_name":"cred-1"}}and get back amodel_info.id{"model":"m-1","messages":[...]}and gets assistant text back/v1/chat/completionsAfter: the same regression turns exactly one cell red in CI,
messages-nonstream-anthropic-stored_credential, before it ships{"credential_name":"cred-1","credential_values":{"api_key":"..."},"credential_info":{}}and gets{"success":true}{"model_name":"m-1","litellm_params":{"model":"anthropic/claude-haiku-4-5","litellm_credential_name":"cred-1"}}and get back amodel_info.id{"model":"m-1","messages":[...]}and gets assistant text back{"role":"assistant","content":[{"type":"text","text":"..."}]}Relevant issues
Affected release
Linear ticket
Pre-Submission checklist
uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Screenshots / Proof of Fix
Shared setup: a live proxy on
http://localhost:4000with Postgres andSTORE_MODEL_IN_DB=True, provider keys in the environment (OPENAI_API_KEY,ANTHROPIC_API_KEY,GEMINI_API_KEY,VERTEXAI_CREDENTIALS,VERTEXAI_PROJECT,AWS_BEARER_TOKEN_BEDROCK,TOGETHER_API_KEY). Master key, virtual keys, and provider keys are redacted below. The case walked by hand is thestored_credentialcell family for Anthropic, which is the same shape every cell in the matrix takes: create the credential, register a deployment that names it, generate a virtual key, then call the endpoint as a customer would. The whole run below was captured on video at the tip and posted in Slack: https://berriaillm.slack.com/archives/D0BE7TV8KSQ/p1790066286931849Before (30d8b12)
The proxy answers, but nothing asserts it across endpoints, providers, and auth modes
ls tests/e2e/llm_translation/ | grep -c endpoint_matrix(in a worktree at the merge base)python -m pytest tests/e2e/llm_translation/test_endpoint_matrix_e2e.py -qPOST /credentials,POST /model/new,POST /key/generate, and the three inference calls listed under After behave identically at this commit, so a translation bug on one of/v1/messages,/v1/chat/completions, or/v1/responsesfor a stored credential would only be found by whichever endpoint-specific test happened to cover that exact provider and auth modeAfter (43a7968)
Stored credential, Anthropic, three endpoints through the proxy
curl -X POST http://localhost:4000/credentials -H "Authorization: Bearer $MASTER" -d '{"credential_name":"proof-anthropic-9716","credential_values":{"api_key":"<ANTHROPIC_API_KEY>"},"credential_info":{}}'curl -X POST http://localhost:4000/model/new -H "Authorization: Bearer $MASTER" -d '{"model_name":"proof-claude-9716","litellm_params":{"model":"anthropic/claude-haiku-4-5","litellm_credential_name":"proof-anthropic-9716"}}'curl -X POST http://localhost:4000/key/generate -H "Authorization: Bearer $MASTER" -d '{"models":[]}'returns a fresh virtual keysk-...fCJgcurl -X POST http://localhost:4000/v1/chat/completions -H "Authorization: Bearer <virtual key>" -d '{"model":"proof-claude-9716","messages":[{"role":"user","content":"Reply with one short friendly sentence."}],"max_tokens":64}'curl -X POST http://localhost:4000/v1/messages -H "Authorization: Bearer <virtual key>" -d '{"model":"proof-claude-9716","messages":[{"role":"user","content":"Reply with one short friendly sentence."}],"max_tokens":64}'curl -X POST http://localhost:4000/v1/responses -H "Authorization: Bearer <virtual key>" -d '{"model":"proof-claude-9716","input":"Reply with one short friendly sentence."}'The matrix does the above for every cell, live
uv run pytest tests/e2e/llm_translation/test_endpoint_matrix_e2e.py -m "not replayable" -q(Gemini, Vertex, Bedrock, Together: 75 cells, real providers)E2E_FIXTURE_MODE=record E2E_FIXTURE_DIR=$HOME/e2e-matrix-fixtures-tip uv run pytest tests/e2e/llm_translation/test_endpoint_matrix_e2e.py -m replayable -q(OpenAI, Anthropic: 60 cells, real providers through the edge)OPENAI_API_KEY=sk-bogus ANTHROPIC_API_KEY=sk-ant-bogus E2E_FIXTURE_MODE=replay E2E_FIXTURE_DIR=$HOME/e2e-matrix-fixtures-tip uv run pytest tests/e2e/llm_translation/test_endpoint_matrix_e2e.py -m replayable -q(bogus provider keys prove zero egress)Mutation: a deployment holding a wrong literal key turns exactly its cells red
OPENAI_API_KEY=sk-bogus E2E_MATRIX_PROVIDERS=openai E2E_MATRIX_AUTH_MODES=inline uv run pytest tests/e2e/llm_translation/test_endpoint_matrix_e2e.py -k "chat_completions-nonstream or messages-nonstream or responses-nonstream" -qType
✅ Test
Caveats
Medium
ANTHROPIC_API_KEYandVERTEXAI_CREDENTIALS, so 78inlineandstored_credentialcells fail by name (openai inline auth needs OPENAI_API_KEY in the test environment) while all 45env_refcells and all 12 Anthropicinlineandstored_credentialcells pass; the conversational matrix already on main errors the same way 90 times. BerriAI/project-releaser#262 hands the podOPENAI_API_KEY,GEMINI_API_KEY,VERTEXAI_PROJECTandTOGETHER_API_KEYfrom the Secret the gateway already mounts.AWS_BEARER_TOKEN_BEDROCKis in no cluster Secret (the gateway uses IRSA), so the 14 Bedrockinlineandstored_credentialcells stay red there until that key is added to the stagelitellm-provider-keyssecretLow
GET /v1/modelsdiscovery with 599 instead of recording it; only the exact empty-query path is filtered,GET /v1/models?purpose=batchstill records and replaystest_vertex_chat_vision_describes_image,test_rust_ocr_response[mistral]on a Mistral 429,test_vertex_prompt_caching_reads_cache) and errored the 90 conversational matrix cells onOPENAI_API_KEY; the main cadence rerun at litellm 25af172 (litellm-e2e-pr build 691) fails the same vision and OCR tests, so those are red on main independently of this branchllm.messages.openai.basic.{nonstream,stream}.works,llm.responses.anthropic.basic.stream.works) landed on main via test(e2e): add conversational matrix across chat, messages and responses #42359 while this PR was open, so the merge keeps main's rows and the matrix cells cover them through theircoversmarkers rather than duplicate rows;collector.py --strictstill exits 0 with zero orphans/credentialsmanagement calls now go to the control plane in split deployments (CONTROL_PLANE_PREFIXES); a test that expected them on a gateway replica would have been getting 404s from the stage-mirror stack alreadyLive PR risk
Verdict: no breaking or backward incompatible change found. Every changed symbol lives in the
tests/e2eharness and every dependent is another e2e test, so the A/B is the same harness run at the merge base and at the tip against the same live proxy and the same real providersBreaking: none observed
Backward incompatible: none.
deterministic_markerstill returns the same value for every per-test call site; only markers drawn inside a module- or class-scoped fixture change owner, and no recorded bundle on main holds one of those (see Regression risk).ProviderEdgestill records and replays every path except the exact empty-queryGET /v1/models.ProxyClient.register_modelsis new and only the matrix calls it.CONTROL_PLANE_PREFIXESgains/credentials; the existing entries and the replica fan-out for inference routes are unchangedRegression risk: a replayable test that drew a marker inside a module- or class-scoped fixture would get a different marker at head than the bundle recorded at base holds, and miss its fixture on the first replay after merge. The four modules on main that draw markers inside such fixtures (
router/test_auto_router_regressions_e2e.py,access_control/test_model_access_group_e2e.py,quota_management/spend_tracking/test_key_attribution_e2e.py,quota_management/budgets/test_model_access_group_budget_e2e.py) carry noreplayablemarker and no edge mount, so their markers never enter a bundle; the scheduled recorder re-records the replayable lane from scratch anywayDependency graph, base 30d8b12 vs head 43a7968, same live proxy (port 4000, split control plane 4001 and replicas 4010/4011):
deterministic_marker/marker_owner(tests/e2e/fixture_mode.py): every replayable e2e test. Verified live:-m replayablerecord lane overtests/e2eat head, 174 passed, 1 failed, 4 skipped; the one failure istest_reliability_cache_e2e.py::test_exact_cache_returns_cached, which fails identically at base against this proxy (nox-litellm-cache-keyon the second call, pre-existing). Replay of that bundle at head: 164 passed, 11 failed; the same 10 spend-log tests plus the cache test. The 10 spend-log failures reproduce at base: recording the same five spend tests from the base worktree into a fresh bundle and replaying them fails the same 10 nodes (10 failed, 12 passed), because a replayed request writes no fresh spend row for the replayed call id while the assertion reads the live database. Pre-existing, unrelated to this diffProviderEdgescheduled-discovery filter (tests/e2e/provider_edge.py): every edge-wired test. Verified live in the record and replay lanes above, plus the three new unit cases intests/e2e/test_provider_edge.pyCONTROL_PLANE_PREFIXES/credentials(tests/e2e/transport.py): every stored-credential e2e test on the stage-mirror job. Verified live on the local split deployment (60 stored and inline cells above) and pinned bytests/e2e/test_proxy_client.py::test_split_deployment_reads_management_routes_back_from_the_control_plane; the hosted stage-mirror job on this tip is waiting for a LiteLLM Engineering approval of thee2e-changedenvironment, which I cannot grantProxyClient.register_models(tests/e2e/proxy_client.py): the matrixdeploymentsfixture only. Verified live in every lane abovetests/e2e/coverage_registry/collector.py --strictexits 0 with zero orphans at headNot verified: the failure-mode leg (paused Postgres, concurrent bursts) and the packaging leg did not run; the diff moves no runtime pin and adds no production code path, so neither leg has a target. Bedrock
inlineandstored_credentialcells were verified locally withAWS_BEARER_TOKEN_BEDROCKbut cannot be verified on the hosted runner until that key exists thereQA runbook
tests/e2e/llm_translation/test_endpoint_matrix_e2e.py::TestEndpointMatrix::test_endpoint_answers_through_every_provider_and_auth_mode[---<auth_mode>] - one cell per endpoint x provider x auth mode; the proxy answers the endpoint with a real, non-empty result through a deployment whose secret is held the way
<auth_mode>says. NeedsSTORE_MODEL_IN_DB=Trueand the provider variables named inendpoint_matrix.py(OPENAI_API_KEY,ANTHROPIC_API_KEY,GEMINI_API_KEY,VERTEXAI_CREDENTIALSplusVERTEXAI_PROJECT,AWS_BEARER_TOKEN_BEDROCK,TOGETHER_API_KEY); a missing variable fails only that provider'sinlineandstored_credentialcells by name rather than skipping, while itsos.environ/cells and every other provider still run.E2E_MATRIX_PROVIDERSandE2E_MATRIX_AUTH_MODESnarrow the run; an unknown name fails at collectionstored_credentialonly:curl -X POST http://localhost:4000/credentials -H "Authorization: Bearer sk-1234" -d '{"credential_name":"cred-1","credential_values":{"api_key":"<provider key>"},"credential_info":{}}'and expect{"success":true,...}curl -X POST http://localhost:4000/model/new -H "Authorization: Bearer sk-1234" -d '{"model_name":"m-1","litellm_params":{"model":"<backend from endpoint_matrix.py>", <"api_key":"os.environ/OPENAI_API_KEY"> or <"api_key":"<literal>"> or <"litellm_credential_name":"cred-1">, ...provider params}}'and expect amodel_info.idcurl -X POST http://localhost:4000/key/generate -H "Authorization: Bearer sk-1234" -d '{"models":[]}'and use the returned key below"model":"m-1":/v1/chat/completionsand/v1/completionsexpect non-emptychoices[0]text;/v1/messagesexpectsrole: assistantand a non-empty text block;/v1/responsesexpectsstatus: completedand non-empty output text;/v1/embeddings(with"encoding_format":"float") expects a vector longer than one element with a non-zero component;/v1/audio/speechexpects non-empty audio bytes;/v1/audio/transcriptionsexpects non-empty text;/v1/images/generationsand/v1/images/editsexpect adata[0]withb64_jsonorurl;/v1/moderationsexpects oneresults[0]with acategoriesobject. For thestreamcells, expect at least one content delta and a terminal eventPOST /model/deletewith the id, thenDELETE /credentials/cred-1; the fixture registers each delete the moment the resource exists, so a failure partway through setup still tears down what it madeOPENAI_API_KEYin the pytest process only and runE2E_MATRIX_PROVIDERS=openai,anthropic ... -k chat_completions-nonstream; expect exactlyopenai-inlineandopenai-stored_credentialred withneeds OPENAI_API_KEY, the other four green (2 failed, 4 passed)tests/e2e/test_fixture_mode.py::TestDeterministicMarker::test_module_fixture_markers_belong_to_the_module_not_its_first_test - a marker drawn inside a module-scoped fixture is keyed by the module node, not by whichever cell pytest runs first, so a full recording replays a narrowed
-korE2E_MATRIX_*selection. Mutation: withmarker_owner()reverted tocurrent_test_key()the test fails, restored it passes (18 passed)-m replayable, 60 passed), then replay only-k embeddings-nonstream-openai-inlinewithOPENAI_API_KEY=sk-bogus; expect 1 passed with zero provider callstests/e2e/test_provider_edge.py (the three new cases) - the edge keys fixtures by query string, keeps
GET /v1/models?purpose=batchas ordinary traffic, and neither records nor demands the proxy's scheduled empty-queryGET /v1/modelsrefresh_model_infojob to fire; expect noget-openai-v1-models.jsonin the bundletests/e2e/test_proxy_client.py::TestSplitDeployment::test_split_deployment_reads_management_routes_back_from_the_control_plane -
/credentials/<name>resolves to the control plane in a split deployment, like/key/infoand/project/info, while/v1/modelsstill fans out to the gateway replicasstored_credentialmatrix cell against a split deployment (LITELLM_CONTROL_PLANE_URLandLITELLM_PROXY_REPLICA_URLSset); expect the credential create and delete to return 200 instead of 404Final Attestation
Note
Low Risk
Changes are confined to e2e tests, docs, and test harness helpers; no production proxy or gateway logic is modified.
Overview
Adds a parametrized e2e endpoint matrix (
endpoint_matrix.py+test_endpoint_matrix_e2e.py) that exercises ten inference routes across six catalog providers and three credential shapes (env_ref, inline, stored), with registrycoversids andE2E_MATRIX_PROVIDERS/E2E_MATRIX_AUTH_MODESfilters. OpenAI and Anthropic cells are edge-wired and@pytest.mark.replayable; deployments register once per module via newProxyClient.register_modelsbatch API.Fixture/replay plumbing:
deterministic_marker()keys offmarker_owner()so module-scoped setup (matrix deployments) gets stable markers independent of which parametrized cell runs first; tests lock that in. The provider edge rejects the proxy’s scheduled empty-queryGET /v1/modelswith 599 so record/replay neither captures nor owes that housekeeping traffic.Harness routing:
/credentialsis treated as a control-plane path in split deployments; coverage YAML and CONTRIBUTING document the matrix and replay lane membership.Reviewed by Cursor Bugbot for commit 43a7968. Bugbot is set up for automated code reviews on this repo. Configure here.
Link to Devin session: https://app.devin.ai/sessions/d756e8154a4149e3a6d779d13b2324e0
Open in Devin Desktop: https://app.devin.ai/desktop/session/d756e8154a4149e3a6d779d13b2324e0?variant=devin
Requested by: @mateo-berri