test(e2e): vendor API strategy coverage across endpoints

[mubashir1osmani]

TLDR

Problem this solves:

• Vendor API strategy suite had gaps across many endpoints
• Several vendor endpoints had no live e2e coverage at all
• Registry rows existed with no covering test

How it solves it:

• Adds e2e suites for the uncovered vendor endpoints
• Extends existing suites with input-validation and contract cases
• Adds the harness plumbing those suites need
• Fills the matching coverage registry rows

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Needs filling in before review. This section must show curl commands and their output against a live proxy hitting real provider APIs, per the repo's contributing rules, and a pytest run does not satisfy it

Type

✅ Test

Changes

Vendor API strategy e2e coverage across endpoints that previously had none or had only happy-path cases. New suites: bedrock native (converse and invoke, including input validation), the chat completions streaming SSE contract, files and batches contract, image edits, realtime over HTTP, responses retrieve, search, vector stores, an OpenAI moderation category matrix, chat auth headers, and team daily activity. Existing suites gain input-validation and contract cases: audio speech, audio transcriptions, embeddings, image generation, messages, moderations, OCR, and responses

Harness changes those need: e2e_http gains error-classification helpers (is_client_error, is_auth_denied, assert_client_error, assert_not_server_error, assert_error_or_server_known) and mid-stream disconnect handling so a Bedrock early close surfaces as a typed result rather than a raw exception; transport, models, endpoints_client, guardrails_client, and spend_e2e_client gain the routes and typed bodies the new suites drive. The coverage registry gains the matching rows plus new LlmEndpoint values (vector_stores, search, ocr, bedrock_native)

The google-native generateContent and Prometheus queue-time coverage that was briefly on this branch now lives in its own PR, #34650, so the two can be reviewed independently. That PR is based directly on litellm_internal_staging and does not depend on this one

QA runbook

Needs filling in before review: one entry per test added, with the manual steps a reviewer can follow against a live proxy. The suites awaiting entries are

• tests/e2e/access_control/test_chat_auth_headers_e2e.py
• tests/e2e/guardrails/test_openai_moderation_category_matrix_e2e.py
• tests/e2e/llm_translation/test_bedrock_native_e2e.py
• tests/e2e/llm_translation/test_chat_completions_sec_vulnerability_e2e.py
• tests/e2e/llm_translation/test_chat_stream_contract_e2e.py
• tests/e2e/llm_translation/test_files_batches_contract_e2e.py
• tests/e2e/llm_translation/test_image_edits_e2e.py
• tests/e2e/llm_translation/test_model_matrix_smoke_e2e.py
• tests/e2e/llm_translation/test_realtime_http_e2e.py
• tests/e2e/llm_translation/test_responses_retrieve_e2e.py
• tests/e2e/llm_translation/test_search_e2e.py
• tests/e2e/llm_translation/test_vector_stores_e2e.py
• tests/e2e/quota_management/spend_tracking/test_team_daily_activity_e2e.py

plus the cases added to the eight existing suites listed under Changes

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Greptile Summary

Adds broad live vendor API coverage and the supporting typed e2e harness, client routes, credential resolution, and coverage-registry entries

• Adds endpoint and contract suites for auth, guardrails, Bedrock-native calls, chat streaming, files, batches, image edits, realtime, responses retrieval, search, vector stores, and team activity
• Strengthens input-validation coverage across existing conversational and non-conversational endpoint suites
• Resolves environment-backed search credentials and represents mid-stream disconnects as typed test results
• Tightens the previously permissive search, vector-store, and validation-helper assertions

Confidence Score: 5/5

The PR appears safe to merge

No blocking failure remains

Important Files Changed

Filename	Overview
litellm/router_utils/search_api_router.py	Resolves environment-referenced search API keys and bases before provider dispatch
tests/e2e/e2e_http.py	Adds reusable error classifiers and typed handling for interrupted streaming responses
tests/e2e/llm_translation/test_search_e2e.py	Adds live search lifecycle and validation coverage with substantive result assertions
tests/e2e/llm_translation/test_vector_stores_e2e.py	Adds vector-store lifecycle, file indexing, search, and strict invalid-input coverage
tests/e2e/guardrails/test_openai_moderation_category_matrix_e2e.py	Exercises moderation blocking across chat, Messages, and Responses API surfaces
tests/e2e/coverage_registry/schema.py	Extends registry endpoint and capability literals for the newly covered surfaces

_{Reviews (2): Last reviewed commit: "fix(e2e): address greptile and veria rev..." | Re-trigger Greptile}

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

[mubashir1osmani]

Addressed the greptile points: validation now requires 4xx, search asserts non-empty results, invalid vector store id no longer wildcards failures.

[mubashir1osmani]

@greptile-apps

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing mubashir1osmani:litellm_e2e_vendor_api_testing_strategy (b4898b4) with litellm_internal_staging (fa56283)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (b5cfc2c) during the generation of this report, so fa56283 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩