revert: "test(e2e): vendor API strategy coverage across endpoints" (#34649)

[yuneng-berri]

TLDR

Problem this solves:

• test(e2e): vendor API strategy coverage across endpoints #34649 merged with its proof-of-fix section unfilled
• Its QA runbook was left as a to-do list, not steps
• 13 new suites claim coverage nobody verified live

How it solves it:

• Reverts dcb4e50 in full, restoring the pre-PR e2e tree
• Keeps every later change to the four shared files
• Registry rows drop with the suites they described

Relevant issues

Reverts #34649

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

This PR touches nothing under litellm/, so there is no proxy behavior to demonstrate against a live provider. Being straight about that rather than pasting an unrelated curl: the thing that needs proving here is that removing those suites leaves the rest of tests/e2e intact, so that is what I checked, at 472dd27

The revert is a byte-exact mirror of the merge commit. #34649 was 2601 insertions and 94 deletions across 31 files; this branch against litellm_internal_staging is 94 insertions and 2601 deletions across the same 31 files

$ git diff --stat origin/litellm_internal_staging | tail -1
 31 files changed, 94 insertions(+), 2601 deletions(-)

27 of those 31 paths restore to the pre-PR blob exactly, so an empty diff against the merge commit's parent is the whole result

$ git diff 487074f6 -- $(comm -23 pr_paths.txt later_paths.txt | tr '\n' ' ')
$ echo $?
0

The other four were touched by commits that landed after #34649, so they were auto-merged and are where a sloppy revert would quietly eat someone else's work. All four kept it. e2e_http.py still has the bounded transient-retry transport from #35824 with every call site attached, and llm_nonconversational.yaml still has the legacy text /completions row from #34431

$ grep -c 'request_with_retry' tests/e2e/e2e_http.py
9
$ grep -c 'llm.completions.openai.basic.nonstream.works' tests/e2e/coverage_registry/llm_nonconversational.yaml
1

Nothing left dangling. The five error-classification helpers the revert removes from e2e_http.py have no surviving callers, and no registry row references the LlmEndpoint or LlmCapability values it removes

$ grep -rn -E '\b(is_client_error|is_auth_denied|assert_client_error|assert_not_server_error|assert_error_or_server_known)\b' tests/e2e/ --include='*.py'
$ grep -rn -E '\b(vector_stores|ocr|bedrock_native|input_sanitization|input_validation|multi_turn)\b' tests/e2e/coverage_registry/*.yaml
$

The suite still collects, the registry still validates against its schema, and both e2e gates are clean

$ python -m pytest tests/e2e --collect-only -q | tail -1
477/479 tests collected (2 deselected) in 1.19s

$ python -m pytest tests/e2e/coverage_registry/test_collector.py -q | tail -1
13 passed, 1 warning in 0.41s

$ python tests/code_coverage_tests/check_e2e_no_raw_requests.py
tests/e2e raw HTTP client check passed

$ basedpyright tests/e2e
0 errors, 0 warnings, 0 notes

$ ruff check tests/e2e
All checks passed!

Type

✅ Test

Changes

Reverts dcb4e50 in full. The 13 suites it added come out, along with the cases it appended to eight existing suites, the error-classification helpers and mid-stream disconnect handling it added to e2e_http.py, the routes and typed bodies it added to transport, models, endpoints_client, guardrails_client, and spend_e2e_client, and the matching coverage registry rows plus the vector_stores, ocr, and bedrock_native LlmEndpoint values and the input_sanitization, input_validation, and multi_turn LlmCapability values

Four files came back through an auto-merge because later commits also touched them, and in each case the later work is preserved: the transient-retry transport in e2e_http.py from #35824, and the legacy text /completions coverage in schema.py, endpoints_client.py, and llm_nonconversational.yaml from #34431. test_image_edits_e2e.py returns to the version #34476 shipped, and its registry row returns to that PR's wording

The reason for the revert is that #34649 shipped with the two sections that would have made it reviewable left unwritten. Its own Proof of Fix section said it needed filling in before review, and its QA runbook listed the 13 suites still awaiting entries instead of giving any. Nothing in it was exercised against a live proxy, so the coverage the registry rows now assert is unverified, and a registry that overstates coverage is worse than one that admits a gap. Re-landing it suite by suite, each with a runbook a reviewer can follow, is the cheaper path

QA runbook

This PR adds no tests, so there is no per-test runbook to write. What a reviewer should confirm is that the suites it removes are gone and the rest is untouched

• The revert is complete and preserves later work

  • git diff --stat origin/litellm_internal_staging shows 31 files, 94 insertions, 2601 deletions, the exact inverse of test(e2e): vendor API strategy coverage across endpoints #34649
  • grep -rn 'request_with_retry' tests/e2e/e2e_http.py still shows the bounded retry from test(e2e): retry provider-transient statuses at the transport with bounded backoff #35824 and its call sites
  • grep 'llm.completions.openai' tests/e2e/coverage_registry/llm_nonconversational.yaml still shows the legacy /completions row from test(e2e): cover legacy text /completions endpoint #34431
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• The surviving suite is self-consistent

  • pytest tests/e2e --collect-only -q collects with no import errors
  • pytest tests/e2e/coverage_registry/test_collector.py passes, so no registry row points at a file this PR deleted
  • basedpyright tests/e2e reports zero errors
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Greptile Summary

Reverts the vendor-strategy E2E expansion, deleting its suites, shared harness additions, and matching coverage-registry entries while retaining later changes in overlapping files.

• Removes endpoint, authorization, validation, guardrail, spend, and provider-matrix E2E cases.
• Restores shared E2E models and clients to their earlier interfaces.
• Removes the stream-consumption disconnect handler from the shared HTTP harness.

Confidence Score: 4/5

The PR needs the streaming disconnect regression fixed before merging; the broad loss of regression coverage should also be reconsidered or replaced with verified tests.

Streaming response consumption can now raise an uncaught requests exception after the request-level handler has completed, while numerous endpoint and security-sensitive regression assertions are removed without equivalent surviving coverage.

Files Needing Attention: tests/e2e/e2e_http.py, tests/e2e/access_control/test_chat_auth_headers_e2e.py, tests/e2e/guardrails/test_openai_moderation_category_matrix_e2e.py, tests/e2e/guardrails/guardrails_client.py

Important Files Changed

Filename	Overview
tests/e2e/e2e_http.py	Removes shared error helpers and mid-stream exception conversion, allowing streaming transport exceptions to escape callers.
tests/e2e/access_control/test_chat_auth_headers_e2e.py	Deletes the complete malformed and missing Authorization-header E2E matrix without replacement.
tests/e2e/guardrails/test_openai_moderation_category_matrix_e2e.py	Deletes moderation-category coverage across chat, messages, and responses.
tests/e2e/coverage_registry/llm_nonconversational.yaml	Removes the registry claims corresponding to deleted endpoint and input-validation suites while preserving the later image-edits row.
tests/e2e/coverage_registry/schema.py	Removes endpoint and capability literals in sync with the deleted registry rows.
tests/e2e/guardrails/guardrails_client.py	Restores the narrower Gemini-only helper interface but introduces an overlong changed line.

Comments Outside Diff (1)

1. tests/e2e/access_control/test_chat_auth_headers_e2e.py

   Removed endpoint regression coverage

   This revert deletes the only E2E assertions and registry claims for malformed authorization headers and numerous validation, streaming, guardrail, provider, and management behaviors. Regressions in those contracts can now pass the E2E gates until another test or production use exposes them.

   Rule Used: What: Flag any modifications to existing tests and... (source)

   Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

_{Reviews (1): Last reviewed commit: "Merge remote-tracking branch 'origin/lit..." | Re-trigger Greptile}

[greptile-apps]

Streaming disconnects escape transport

When a provider or network connection terminates while iter_lines() is consuming a streaming response, the removed exception handler allows the requests exception to escape, aborting the E2E test instead of returning a typed StreamingResponse with the partial stream and error details.

[greptile-apps]

Restored line exceeds limit

The restored LiteLLMParamsBody(...) statement exceeds the repository's 120-character Python line limit and should be split across lines to preserve the required formatting convention.

Suggested change

	LiteLLMParamsBody(model="gemini/gemini-2.5-flash", api_key="os.environ/GEMINI_API_KEY"),
	LiteLLMParamsBody(
	model="gemini/gemini-2.5-flash",
	api_key="os.environ/GEMINI_API_KEY",
	),

Context Used: CLAUDE.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!