test(e2e): assert provider error shape instead of pinned prose

[yuneng-berri]

TLDR

Problem this solves:

• Three e2e cells are red without any behavior changing
• Two pinned provider error prose that both providers reworded
• One needs stack config the ephemeral e2e stack never ships

How it solves it:

• Pin the machine-readable error shape, not the prose
• Assert the 400 relays the provider's own rejection
• Skip the bedrock web search cell with its reason recorded

User Flow

Before: an engineer reading the e2e run sees three red cells and cannot tell which ones mean the gateway broke

1. They open the e2e run and see test_failed_chat_completions_error_span_attributes failing on assert 'API key is invalid.' == 'invalid x-api-key'
2. They see test_missing_file_returns_error failing because the 400 body says "This model does not support the format you provided." and names neither "file" nor "audio"
3. They see test_web_search_server_tool_is_served failing on a 400 that tells them to enable web search interception
4. Nothing on screen separates "a provider reworded a string" from "the gateway regressed", so all three get triaged by hand

After: the same run is green, and the one cell that cannot run says why

1. They open the e2e run and the two error-shape cells pass, because the assertions now key off the stable error type rather than the sentence a provider happens to ship this week
2. A real regression still turns them red: swallowing the provider's reason, or answering 500 instead of 400, fails both
3. The bedrock web search cell reports as skipped with the reason naming the two config entries the stack is missing
4. They can act on the run directly instead of re-deriving which failures were product bugs

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Captured at 481ab07ca7. Both providers were hit directly, and the transcription case was reproduced through a live proxy on port 4071

Anthropic's auth error, which the otel cell used to pin as invalid x-api-key:

curl -s https://api.anthropic.com/v1/messages -H "x-api-key: sk-ant-invalid-key-for-error-probe" -H "anthropic-version: 2023-06-01" -H "content-type: application/json" -d '{"model":"claude-haiku-4-5","max_tokens":8,"messages":[{"role":"user","content":"hi"}]}'

{"type":"error","error":{"type":"authentication_error","message":"API key is invalid."},"request_id":null}

The authentication_error type the cell now pins is present; the sentence it used to pin is not

The same error as it reaches the span, through litellm's own mapping:

litellm.AuthenticationError: AnthropicException - {"type":"error","error":{"type":"authentication_error","message":"API key is invalid."},"request_id":null}

An empty upload through a live proxy, which the transcription cell used to assert named "file" or "audio":

curl -s -w "\n<<HTTP_STATUS:%{http_code}>>\n" http://127.0.0.1:4071/v1/audio/transcriptions -H "Authorization: Bearer sk-probe-1234" -F "model=e2e-transcribe-probe" -F "file=@empty.wav;type=audio/wav"

{"error":{"message":"litellm.BadRequestError: OpenAIException - This model does not support the format you provided.No fallback model group found for original model_group=e2e-transcribe-probe. Fallbacks=[{'claude-haiku-4-5': ['xai/grok-4-1-fast-non-reasoning']}, {'gpt-5.5': ['xai/grok-4-1-fast-non-reasoning']}]. Received Model Group=e2e-transcribe-probe\nAvailable Model Group Fallbacks=None\nError doing the fallback: litellm.BadRequestError: OpenAIException - This model does not support the format you provided.No fallback model group found for original model_group=e2e-transcribe-probe. Fallbacks=[{'claude-haiku-4-5': ['xai/grok-4-1-fast-non-reasoning']}, {'gpt-5.5': ['xai/grok-4-1-fast-non-reasoning']}]","type":"invalid_request_error","param":"messages","code":"400"}}
<<HTTP_STATUS:400>>

Status is 400, OpenAIException and invalid_request_error are both present, and neither "file" nor "audio" appears anywhere. Running the old and new assertions against these exact payloads: the old ones fail, reproducing the run, and the new ones pass

The bedrock cell's upstream error is unchanged and still by design, so its proof is the stack config rather than a request. The ephemeral stack declares callbacks: [arize_phoenix, datadog, smtp_email, prometheus, otel] and no search_tools block, so neither prerequisite in that module's docstring is present

Type

✅ Test

Caveats

• Bedrock web search coverage is paused, not restored
• Unskipping it needs interception plus a search backend in the stack
• The transcription cell's name says missing file; it uploads an empty one

QA runbook

• tests/e2e/logging/test_otel_trace_e2e.py::TestOtelTraceCompleteness::test_failed_chat_completions_error_span_attributes - a provider auth failure reaches the span as the provider's own error object, whole and parseable
  • Point a deployment at anthropic with a deliberately wrong upstream key, so the call clears proxy auth and fails at the provider
  • Send POST http://localhost:4000/v1/chat/completions against it and expect a 401 back
  • Open the trace in Jaeger and read the model-call span's error.message; expect AnthropicException followed by the provider's JSON
  • Expect that JSON to parse and carry "type": "authentication_error" with a non-empty message, and expect otel.status_description to carry the same untruncated text
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/llm_translation/test_audio_transcriptions_e2e.py::TestAudioTranscriptions::test_missing_file_returns_error - an unusable audio upload is refused with the provider's own reason rather than a swallowed or 500 failure
  • Register a deployment on openai/gpt-4o-mini-transcribe (needs OPENAI_API_KEY)
  • POST http://localhost:4000/v1/audio/transcriptions as multipart with that model and a zero-byte empty.wav
  • Expect 400, and expect the body to carry both OpenAIException and invalid_request_error
  • Note the body also carries router fallback diagnostics; that text is expected and is not what the assertions key off
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/llm_translation/test_bedrock_web_search_server_tool_e2e.py::TestBedrockWebSearchServerTool::test_web_search_server_tool_is_served - skipped here, so the check is that the skip is honest rather than hiding a product bug
  • Confirm the proxy config has no websearch_interception callback and no search_tools entry
  • Register a bedrock invoke deployment on us.anthropic.claude-haiku-4-5-20251001-v1:0 and POST http://localhost:4000/v1/messages with a web_search_20250305 tool
  • Expect a 400 naming the tool type and pointing at web search interception, which is the documented behavior for that config
  • Add the callback and a search backend from the module docstring, remove the skip, and expect the cell to pass; that is the unskip criterion
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Greptile Summary

This PR makes two provider-error e2e assertions depend on stable machine-readable error shape rather than mutable provider prose, and skips the Bedrock web-search test because its current stack lacks prerequisites.

• Validates empty transcription failures using the relayed provider exception and invalid-request type.
• Validates Anthropic trace errors using the provider authentication type and a non-empty parseable message.
• Unconditionally skips Bedrock native web-search interception coverage pending stack configuration.

Confidence Score: 4/5

The PR appears safe to merge, but the unconditional Bedrock test skip should be made configuration-aware so capable environments retain this coverage.

The error assertion changes preserve the relevant machine-readable contracts, while the sole non-blocking concern is that an unconditional skip disables Bedrock web-search interception coverage on every target stack.

Files Needing Attention: tests/e2e/llm_translation/test_bedrock_web_search_server_tool_e2e.py

Important Files Changed

Filename	Overview
tests/e2e/llm_translation/test_audio_transcriptions_e2e.py	Replaces prose-sensitive empty-upload assertions with status, provider-exception, and machine-readable error-type checks.
tests/e2e/llm_translation/test_bedrock_web_search_server_tool_e2e.py	Unconditionally skips the sole Bedrock native web-search interception test, preventing it from running even against configured stacks.
tests/e2e/logging/test_otel_trace_e2e.py	Replaces a mutable Anthropic message assertion with provider error-type and non-empty-message validation while retaining parseability and span-fidelity checks.

_{Reviews (1): Last reviewed commit: "Merge remote-tracking branch 'origin/lit..." | Re-trigger Greptile}

[greptile-apps]

Unconditional skip disables valid coverage

If the target stack has the interception callback and search backend configured, this unconditional marker still skips the sole test for the Bedrock native web-search interception path, leaving that integration untested and its coverage cell reported as skipped.

Rule Used: What: Flag any modifications to existing tests and... (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!