test(e2e): cover 12 non-core LLM coverage registry cells

[ryan-crabbe-berri]

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live proof is pending: the environment this was authored in has no provider credentials, so the new tests have not yet been run against real providers. They will be run 5x with cooldowns against a live proxy (real OpenAI, Bedrock, and Vertex calls, real cost) and the command plus output posted here with the commit hash before requesting maintainer review. The run command is

cd tests/e2e
uv run pytest llm_translation/test_embeddings_endpoint_e2e.py llm_translation/test_audio_speech_e2e.py \
  llm_translation/test_audio_transcriptions_e2e.py llm_translation/test_moderations_e2e.py \
  "batches/test_batches_e2e.py::TestOpenAIFiles" -q

Static validation already done at 9076968: full-suite collection is clean (338 tests), python -m coverage_registry.collector --strict exits 0 with Non-Core LLMs at 36/50 (was 24/50) and overall registry coverage at 54.8% (was 51.9%), and make lint-e2e-basedpyright reports zero errors

Wiring was additionally verified at 60d52a1 against a live proxy running this branch (worktree code, port 41230, shared dev DB). The transcription path ran fully live against a real provider: registering groq/whisper-large-v3 through /model/new and posting the wav fixture as multipart with a freshly generated virtual key returned {"text":" What is the weather in Paris?"} with HTTP 200, confirming the new multipart plumbing, the route, and the test's weather assertion against a real transcript. Every other new test was then run against that proxy and failed only at the provider credential boundary (OpenAI and Bedrock key errors, expired Vertex ADC, and the transcription route passing through the provider's 401), with no harness-level failures. The audio speech route also round-tripped to the real Groq API (their side reports playai-tts decommissioned), and the streamed error path surfaced status and body exactly as designed

Type

✅ Test

Changes

Covers 12 Non-Core LLM coverage-registry cells. Four cells were already asserted by existing tests that predate the registry and only gain their @pytest.mark.covers marker: openai embeddings, openai image generation, openai TTS, and cohere rerank (the registry rows already pointed their source: at these tests). A fifth, llm.embeddings.openai.basic.nonstream.cost_logged, is intentionally dual-marked onto the existing test_embedding_writes_nonzero_spend_row in the spend-tracking suite, which already asserts exactly that behavior, rather than duplicating it

New tests: bedrock (amazon.titan-embed-text-v2:0) and vertex (gemini-embedding-2) embeddings; streaming TTS, which asserts chunked transfer encoding so a buffered body cannot pass as a stream; audio transcriptions, uploading the realtime suite's spoken-weather wav fixture as multipart; a moderations flag/pass pair (only the flagging test claims the cell); and files list plus retrieve in the batches suite, where list uses a raw provider upload because GET /v1/files returns the org's raw ids while retrieve uses a model-encoded upload because the route re-encodes the id it was called with

Harness additions, all backward compatible: e2e_http.upload now takes any form model and an optional file_content_type (the batches path is byte-for-byte unchanged), a stream_binary primitive plus BinaryStream result for binary chunked responses threaded through the Transport protocol, transcribe and moderations methods on EndpointsClient, and retrieve_file/list_files on BatchClient

A separate first commit fixes client.gateway to client.proxy in test_messages_mid_conversation_system_native_providers_e2e.py. That attribute never existed on EndpointsClient, so the helper raised AttributeError at runtime and failed the tests/e2e basedpyright zero-error CI gate for any PR touching e2e files, including this one

Found while writing the transcription test and filed as LIT-4653 (not fixed here): the /v1/audio/transcriptions route resolves its server-default model from general_settings.moderation_model, a copy-paste from the moderations route

Still uncovered in Non-Core LLMs (14 cells), deferred for credential availability: the four azure_openai cells, cohere embeddings, together_ai rerank, black_forest_labs image generation, and the nvidia_riva/soniox P2 smoke cells. llm.embeddings.anthropic.basic.nonstream.works looks unimplementable as written since Anthropic has no embeddings API; the registry README already lists it as a prune candidate

QA runbook

Marker-only changes (no test-body change, behavior already reviewed when these tests landed): test_embeddings_returns_vector, test_image_generation_returns_image, test_audio_speech_returns_audio, test_rerank_scores_top_n, and the added second marker on test_embedding_writes_nonzero_spend_row

• tests/e2e/llm_translation/test_embeddings_endpoint_e2e.py::TestEmbeddingsEndpoint::test_bedrock_embeddings_returns_vector - a Bedrock Titan deployment registered at runtime returns a real embedding vector
  • POST /model/new with the master key, model bedrock/amazon.titan-embed-text-v2:0, aws_region_name us-west-2 (needs ambient AWS credentials with Titan model access)
  • POST /embeddings with a fresh key and that model name and expect a non-empty vector with at least one non-zero component
  • DELETE the deployment via /model/delete
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/llm_translation/test_embeddings_endpoint_e2e.py::TestEmbeddingsEndpoint::test_vertex_embeddings_returns_vector - a Vertex embedding deployment registered at runtime returns a real vector
  • POST /model/new with model vertex_ai/gemini-embedding-2, vertex_project os.environ/VERTEXAI_PROJECT, vertex_location us-central1 (needs Vertex ADC on the proxy host)
  • POST /embeddings with a fresh key and expect a non-empty, non-zero vector; delete the deployment after
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/llm_translation/test_audio_speech_e2e.py::TestAudioSpeech::test_audio_speech_streams_audio_chunks - streamed TTS arrives as a chunked audio stream, not a buffered body
  • POST /model/new with model openai/gpt-4o-mini-tts
  • POST /v1/audio/speech with a sentence-long input, consuming the response as a stream
  • Expect status 200, an audio content-type, transfer-encoding: chunked on the response (no content-length), and non-zero audio bytes
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/llm_translation/test_audio_transcriptions_e2e.py::TestAudioTranscriptions::test_audio_transcriptions_returns_text - a spoken weather question uploaded as multipart comes back transcribed
  • POST /model/new with model openai/gpt-4o-mini-transcribe
  • POST /v1/audio/transcriptions as multipart with file=llm_translation/realtime/fixtures/weather_question_24k.wav and model=<the deployment>
  • Expect a JSON body whose text is non-empty and contains the word weather (case-insensitive)
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/llm_translation/test_moderations_e2e.py::TestModerations::test_moderations_flags_violent_content - clearly violent text is flagged with at least one category tripped
  • POST /model/new with model openai/omni-moderation-latest
  • POST /v1/moderations with a violent-threat input and expect results[0].flagged true with at least one category true
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/llm_translation/test_moderations_e2e.py::TestModerations::test_moderations_passes_benign_content - benign text is not flagged
  • POST /v1/moderations with an everyday benign sentence against the same deployment kind and expect results[0].flagged false
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/batches/test_batches_e2e.py::TestOpenAIFiles::test_uploaded_file_appears_in_list - a freshly uploaded file shows up in GET /v1/files
  • POST /v1/files as multipart with a unique-marker jsonl, purpose batch, routed raw to the openai provider
  • GET /v1/files and expect the returned id in the data array with purpose batch; delete the file after
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/batches/test_batches_e2e.py::TestOpenAIFiles::test_retrieve_round_trips_metadata - GET /v1/files/{id} round-trips id, purpose, and filename
  • POST /v1/files as multipart with a unique-marker jsonl and the batch model query param
  • GET /v1/files/{returned id} and expect the same id, purpose batch, and filename batch_input.jsonl; delete the file after
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

Environment prerequisites for a manual run: OPENAI_API_KEY in the proxy environment for every openai-route test, ambient AWS credentials with Bedrock Titan access for the bedrock test, Vertex ADC plus VERTEXAI_PROJECT for the vertex test, and COHERE_API_KEY only if re-running the pre-existing rerank test. The files list test additionally needs files_settings with an openai entry in the proxy config for the raw-provider /v1/files routes, the same prerequisite the suite's existing raw-upload tests already carry

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Greptile Summary

This PR expands live end-to-end coverage for non-core LLM features. The main changes are:

• Adds embeddings, speech, transcription, moderation, and file API tests
• Adds binary streaming and reusable multipart upload support
• Adds typed endpoint and batch client methods
• Adds coverage-registry markers to existing tests
• Fixes the messages test to use the existing proxy transport

Confidence Score: 5/5

This looks safe to merge.

• Streamed responses are closed on successful, error, and exception paths.
• The new transport methods and type signatures are consistent across implementations and callers.
• No blocking issue remains in the updated code.

Important Files Changed

Filename	Overview
tests/e2e/e2e_http.py	Adds binary response streaming, bounded error details, and configurable multipart file content types.
tests/e2e/transport.py	Extends each transport implementation with binary streaming and generalized multipart uploads.
tests/e2e/llm_translation/endpoints_client.py	Adds typed client methods for streamed speech, transcription, and moderation.
tests/e2e/llm_translation/test_audio_speech_e2e.py	Adds streamed speech coverage and includes provider response details in failure output.
tests/e2e/llm_translation/test_audio_transcriptions_e2e.py	Adds a multipart transcription test using the existing spoken-weather fixture.
tests/e2e/llm_translation/test_moderations_e2e.py	Adds live moderation tests for violent and benign input.
tests/e2e/batches/test_batches_e2e.py	Adds OpenAI file list and metadata retrieval coverage.

_{Reviews (2): Last reviewed commit: "fix(e2e): close streamed TTS response on..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[ryan-crabbe-berri]

@greptileai re review

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_e2e_noncore_llm_cells (6fc8af6) with litellm_internal_staging (2b2ae4c)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (7dd0541) during the generation of this report, so 2b2ae4c was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[yucheng-berri]

vertex_ai/gemini-embedding-2

are you sure this is a legit embedding? looks sus to me

[yucheng-berri]

needs assert result.content_length is None for (no content-length)

Expect status 200, an audio content-type, transfer-encoding: chunked on the response (no content-length), and non-zero audio bytes

[ryan-crabbe-berri]

https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/gemini/embedding-2

not the imposter

[ryan-crabbe-berri]

needs assert result.content_length is None for (no content-length)

Expect status 200, an audio content-type, transfer-encoding: chunked on the response (no content-length), and non-zero audio bytes

fixed