test(e2e): replace custom endpoints_client with provider SDK clients - #34358
Open
mateo-berri wants to merge 2 commits into
Open
test(e2e): replace custom endpoints_client with provider SDK clients#34358mateo-berri wants to merge 2 commits into
mateo-berri wants to merge 2 commits into
Conversation
The llm_translation suite drove /v1/responses, /v1/messages, /embeddings, /v1/images/generations, /v1/moderations and /v1/audio/* through a bespoke endpoints_client wrapper that no customer runs. Tests now call the proxy the way customers do: the OpenAI SDK for the OpenAI-compatible surface and the Anthropic SDK for /v1/messages, wired through a session-scoped sdk fixture (sdk_clients.py) that points both SDKs at the proxy with a virtual key. Endpoints no official SDK covers keep the shared typed transport: rerank moves onto ProxyClient (RerankBody/RerankResponse in models.py) and the passthrough header test parses with the shared AnthropicMessagesResponse model. Files that only used endpoints_client for model registration now use the proxy fixture directly. endpoints_client.py is deleted; anthropic joins the e2e-dev dependency group so the lint env resolves the SDK imports. Resolves LIT-4577
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Contributor
mateo-berri
marked this pull request as ready for review
July 23, 2026 06:28
Contributor
Greptile SummaryReplaces the bespoke endpoint test client with provider SDK clients and the shared typed proxy transport.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains in the changes eligible for this follow-up review.
|
| Filename | Overview |
|---|---|
| tests/e2e/llm_translation/sdk_clients.py | Adds proxy-configured OpenAI and Anthropic SDK factories with retries disabled and the shared request timeout. |
| tests/e2e/llm_translation/conftest.py | Replaces the deleted endpoint-client fixture with a session-scoped SDK client fixture. |
| tests/e2e/proxy_client.py | Adds rerank support to the shared typed proxy client. |
| tests/e2e/models.py | Adds typed rerank request and response models used by the shared transport. |
| pyproject.toml | Adds the pinned Anthropic SDK to the e2e development dependency group. |
| tests/e2e/CONTRIBUTING.md | Updates local test commands to install the e2e dependency group and documents provider-SDK use in the harness. |
| tests/e2e/CLAUDE.md | Records the provider-SDK exception to the shared-transport convention for LLM endpoint tests. |
Reviews (2): Last reviewed commit: "docs(e2e): name the e2e-dev group in the..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
The llm_translation suite needs the e2e-dev dependency group at collection time (websockets for the realtime folder, now also the anthropic SDK for the sdk fixture). make bootstrap installs the group, but the documented pytest command did not name it, so a default dev-group environment failed collection. Naming the group on uv run makes the command work from any environment state
Contributor
Author
|
Generated by Claude Code |
Contributor
Author
|
CI note: frontend-lint is red on the merge ref, but the failure comes from the base branch, not this diff. The three flagged files (models-and-endpoints/layout.tsx, llm-credentials/page.tsx, vertexCredentialsUpload.ts) were added on litellm_internal_staging by #34327 with antd imports, and the antd no-restricted-imports rule landed separately in #34341, so the base tip fails frontend-lint by itself. This PR changes no files under ui/ (the previous run at 76bf0cd passed frontend-lint before the base moved). Will update the branch and let CI re-run once the base is green again Generated by Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
Problem this solves:
endpoints_client.pywrapperHow it solves it:
endpoints_client.pyis deleted; a smallsdkfixture builds the SDK clientsProxyClienton the shared typed transportRelevant issues
Linear ticket
Resolves LIT-4577
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
The change under test is the client itself: the point of LIT-4577 is that customers reach the proxy through the OpenAI and Anthropic SDKs, so curl cannot exercise what changed here. The migrated tests are exactly the customer flow (SDK client, virtual key, live proxy, real provider APIs, real spend), so the proof is a live run of them at commit 76bf0cd against a proxy started with
store_model_in_db: true, Postgres, Redis, and real provider keysThose 11 tests cost real Anthropic and Gemini money and cover both SDK clients end to end: OpenAI SDK nonstream, tool calling, and store/metadata/previous_response_id on /v1/responses, and Anthropic SDK nonstream, raw-response cost header, SSE streaming, and tool use on /v1/messages, with spend-log read-back and Redis TTL checks intact
The remaining cells could not be exercised from this sandbox for credential reasons only, all surfaced correctly as typed SDK errors: the OpenAI key returns
insufficient_quota, the AWS token is invalid for Bedrock, the Azure AI resource has noclaude-haiku-4-5deployment, and there is no Cohere key, Vertex service account, or enterprise license (needed byallowed_passthrough_routesin the pre-existing passthrough-headers test). They need one QA pass from an environment with full keys; the QA runbook below covers themGates run locally at the same commit:
make lint-e2e-basedpyrightreports 0 errors,check_e2e_no_raw_requestspasses, the full suite collects 87 tests, andpython -m coverage_registry.collector --strictaccepts everycoversmarkerType
🧹 Refactoring
✅ Test
Changes
tests/e2e/llm_translation/sdk_clients.pyis new: a frozenSdkClientsdataclass whoseopenai(key)/anthropic(key)builders return real SDK clients pointed atLITELLM_PROXY_URLwith the harness request timeout and retries disabled, exposed as the session-scopedsdkfixture in the suite conftest. Failures now surface as the SDK's own typed exceptions, which is what an end user seesMigrated to the OpenAI SDK:
test_responses_e2e.py(create, stream events,with_raw_responsefor thex-litellm-call-idheader, typedFunctionToolParamtools, vision input parts),test_responses_metadata_e2e.py(store, metadata,previous_response_id),test_embeddings_endpoint_e2e.py,test_audio_speech_e2e.py(with_raw_responseandwith_streaming_responsefor the chunked-transfer assertions),test_audio_transcriptions_e2e.py,test_moderations_e2e.py,test_image_generation_e2e.pyMigrated to the Anthropic SDK:
test_messages_e2e.py(create,with_raw_responsefor thex-litellm-response-costheader,stream=Trueraw events,ToolParamtools),test_messages_azure_foundry_e2e.py(including thex-litellm-rustheader check on a raw streamed response),test_credential_messages_e2e.py, and both mid-conversation-system files. The mid-conversation files cast therole: "system"reminder turn toMessageParambecause the SDK type only admits user/assistant while that wire shape is exactly what is under test; the SDK serializes the dict verbatimRerank has no official OpenAI/Anthropic SDK, so
ProxyClient.rerankplusRerankBody/RerankResponseinmodels.pyreplace the wrapper method andtest_rerank_e2e.pyuses them.test_ocr_rust_e2e.py,test_custom_pricing_e2e.py,test_cache_control.py, andtest_passthrough_headers_e2e.pyonly used the wrapper for model registration or shared pydantic models, so they now use theproxyfixture andmodels.pyimports directly and their assertions are unchangedendpoints_client.pyis deleted.anthropic==0.84.0joins thee2e-devdependency group (matching thecigroup pin) so the lint CI env resolves the new imports;openaiis already a main dependency.tests/e2e/CLAUDE.mdandCONTRIBUTING.mddocument the SDK exception to the shared-transport rule. Follow-up commit 0e34c18 updates the documented suite run commands intests/e2e/CONTRIBUTING.mdto name thee2e-devgroup (uv run --group e2e-dev pytest ...) so a default dev-group environment collects the suiteQA runbook
Environment prerequisites: a proxy with
store_model_in_db: true, Postgres, Redis, and keys for OpenAI, Anthropic, AWS Bedrock, Azure AI (with aclaude-haiku-4-5deployment), Vertex, Cohere, and Mistral depending on the cell. Every test registers its own deployment through /model/new and deletes it on teardown, so no static model config is needed. In the snippets below,keyis a fresh virtual key from /key/generate and both SDKs are constructed withbase_url="http://localhost:4000"and the virtual key asapi_keyOpenAI(base_url="http://localhost:4000", api_key=key).responses.create(model=m, input="reply with one word", instructions="You are a helpful assistant")returns non-emptyoutput_textstream=Trueyieldsresponse.output_text.deltaevents and ends withresponse.completedtools=[get_weather function tool]the output contains afunction_callitem whose arguments parse to alocationx-litellm-call-idand polling /spend/logs by the response id lands a row with spend > 0 for gpt-4o-miniresponses.createwithstore=Trueand a uniquemetadata.session_id, then a second call withprevious_response_idset to the first id; both return textredis-cli --scan --pattern "*<marker>*"then TTL on each key; expect no TTL of -1Anthropic(base_url="http://localhost:4000", api_key=key).messages.create(model=m, max_tokens=64, messages=[{"role": "user", "content": "reply with one word"}])returns an assistant message with textmessages.with_raw_response.create(...)carries a positivex-litellm-response-costheader, and the priced /spend/logs row for the key matches it within 1% with non-zero token countsmessages.create(..., stream=True)yieldscontent_block_deltaevents and reachesmessage_stopget_weathertool the response content contains atool_useblockazure_ai/claude-haiku-4-5deployment; streamed tool use must carry acontent_block_startwith atool_useblockE2E_EXPECT_RUST=1, the streamed raw response must carryx-litellm-rust: truelitellm_credential_name, then the Anthropic SDK basic call returns assistant textsystemblock markedcache_controland a cache-marked user turn, repeated until usage shows both cache_read and cache_creation tokens > 0{"role": "system"}reminder turn, an assistant turn, and a fresh user turn; on a flagged model (e.g. claude-sonnet-5 or claude-opus-4-8) usage.cache_read_input_tokens must be at least the full primed prefixembeddings.create(model=m, input="Say this is a test!")returns a non-empty vector with at least one non-zero component for each of the three backendsaudio.speech.with_raw_response.create(model=m, voice="alloy", input="Hello!")has an audio content-type and non-empty bytesaudio.speech.with_streaming_response.create(...)with a longer input showstransfer-encoding: chunked, no content-length, and non-zero streamed bytesaudio.transcriptions.create(model=m, file=(name, wav_bytes, "audio/wav"))with the realtime suite's weather WAV returns a transcript containing "weather"moderations.create(model=m, input=<violent text>)returns flagged=true with at least one true category; the benign sentence returns flagged=falseimages.generate(model=m, prompt="Draw a cute cat", n=1, size="1024x1024")returns data with a url or b64_json for both backendscurl -X POST http://localhost:4000/v1/rerank -H "Authorization: Bearer $KEY" -d '{"model": m, "query": "What is the capital of the United States?", "documents": [...4 docs...], "top_n": 3}'returns at most 3 results with a relevance_score on the top hitFinal Attestation