test(e2e): cover /v1/responses OpenAI vision and Anthropic basic

[devin-ai-integration]

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Captured against a live proxy (docker compose stack) hitting real OpenAI and Anthropic APIs. The proof was captured at commit d79e2b9; the branch was later rebased onto litellm_internal_staging with byte-identical test content (head 685948a)

The two new tests pass:

$ curl -fs http://localhost:4000/health/liveliness
"I'm alive!"

$ uv run --no-sync pytest llm_translation/test_responses_e2e.py -v -k 'vision or anthropic'
llm_translation/test_responses_e2e.py::TestResponses::test_responses_vision_describes_image PASSED [ 50%]
llm_translation/test_responses_e2e.py::TestResponses::test_responses_anthropic_returns_completion PASSED [100%]
2 passed, 4 deselected

The same behavior driven by hand through the live proxy, showing the vision request actually reads the image and the Anthropic translation returns real Responses-shaped text:

$ curl -s http://localhost:4000/v1/responses -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
  -d '{"model":"gpt-5.5","input":[{"role":"user","content":[
        {"type":"input_text","text":"What animal is shown in this image? Answer in one word"},
        {"type":"input_image","image_url":"https://upload.wikimedia.org/wikipedia/commons/3/3a/Cat03.jpg"}]}]}'
# output[].content[].text -> "Cat"

$ curl -s http://localhost:4000/v1/responses -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
  -d '{"model":"claude-haiku-4-5","input":"reply with one word"}'
# output[].content[].text -> "Hello"

Type

✅ Test

Changes

This is PR 3 of a stacked series adding e2e coverage for /v1/responses. Its base (#33835) has since merged into litellm_internal_staging, so this PR now targets staging directly and the diff is only the two new tests plus the typed multimodal input they need

It closes two registry rows that were previously uncovered

llm.responses.openai.vision.nonstream.works
llm.responses.anthropic.basic.nonstream.works

test_responses_vision_describes_image registers a vision-capable openai/gpt-4o deployment at runtime, sends a multimodal /v1/responses request (an input_text part plus an input_image part pointing at a stable public cat image), and asserts the returned text names the animal (cat/feline), so a model that did not actually see the image fails the test rather than passing on any non-empty answer

test_responses_anthropic_returns_completion registers anthropic/claude-haiku-4-5 and drives a non-streaming /v1/responses call, parsing the body as ResponsesResult and asserting .text.strip() is non-empty, which proves the Responses to Anthropic translation returns valid Responses-shaped output with real text rather than a 200 with an empty or mistranslated body

To support multimodal input without raw dicts or Any, endpoints_client.py widens ResponsesRequest.input from str to a typed union str | list[ResponsesInputMessage], where each message's content is a list of a composed part union (input_text and input_image variants sharing a type discriminant). The existing str path keeps working for the basic, streaming, cost, and tool tests. A responses_vision(...) helper sends the multimodal shape

QA runbook

Prerequisites: OPENAI_API_KEY and ANTHROPIC_API_KEY in tests/e2e/.env, then from tests/e2e/ run docker compose up -d and confirm curl -fs http://localhost:4000/health/liveliness returns ok

• tests/e2e/llm_translation/test_responses_e2e.py::TestResponses::test_responses_vision_describes_image - a multimodal /v1/responses call to a vision model returns text that actually describes the supplied image

  • curl -s http://localhost:4000/v1/responses -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"model":"gpt-5.5","input":[{"role":"user","content":[{"type":"input_text","text":"What animal is shown in this image? Answer in one word"},{"type":"input_image","image_url":"https://upload.wikimedia.org/wikipedia/commons/3/3a/Cat03.jpg"}]}]}'
  • Expect a 200 whose output[].content[].text says Cat (case-insensitive), proving the model read the image and not just any non-empty reply
  • Sanity check: the assertion matches image-specific keywords, so it fails if the model did not see the image

• tests/e2e/llm_translation/test_responses_e2e.py::TestResponses::test_responses_anthropic_returns_completion - a non-streaming /v1/responses call backed by Anthropic returns valid Responses-shaped output with real text

  • curl -s http://localhost:4000/v1/responses -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"model":"claude-haiku-4-5","input":"reply with one word"}'
  • Expect a 200 whose output[].content[].text is non-empty real text, proving the Responses to Anthropic translation round-trips
  • Sanity check: an empty or mistranslated body fails the test rather than passing on the status code

• uv run --no-sync pytest llm_translation/test_responses_e2e.py -v -k 'vision or anthropic' passes both, and PYTHONPATH=. python -m coverage_registry.collector --strict reports the two rows as covered with no unknown markers

Video of the live run and curl proof

https://app.devin.ai/attachments/7e22b083-6bda-4325-bcda-9c64b37543ec/rec-2504f162-ba2f-4f5d-a61e-3771a45253f2-edited.mp4

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/b54b82329e57464f908f57e73f6a766c
Requested by: @ishaan-berri

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

This PR adds two e2e tests for the /v1/responses endpoint — one that drives a multimodal (vision) request through openai/gpt-4o and asserts the response describes a known cat image, and one that drives a basic non-streaming request through anthropic/claude-haiku-4-5 and asserts non-empty Responses-shaped output. To support typed multimodal input, ResponsesRequest.input is widened from str to str | list[ResponsesInputMessage] and a new responses_vision() helper is added in endpoints_client.py.

• Vision test registers an openai/gpt-4o deployment at runtime, sends a text + image URL payload to /v1/responses, and checks that the reply text contains \"cat\" or \"feline\".
• Anthropic test registers anthropic/claude-haiku-4-5, calls the existing responses() helper, and verifies .text.strip() is non-empty — confirming the Responses↔Anthropic translation round-trips correctly.
• Both tests follow the existing pattern (runtime model registration, resources.defer cleanup, require_successful_call) and are confined entirely to tests/e2e/.

Confidence Score: 4/5

Both files are confined to the tests/e2e directory with no production code changes, and the new helpers follow the established test patterns.

The vision test assertion checks only two keywords ("cat" and "feline"), so valid single-word model answers like "kitten" or "tabby" would cause a spurious failure. The test has passed in practice, but the narrow keyword list leaves a small non-determinism window that could surface in CI against a different model snapshot.

tests/e2e/llm_translation/test_responses_e2e.py — the vision keyword assertion at lines 172–175

Important Files Changed

Filename	Overview
tests/e2e/llm_translation/endpoints_client.py	Adds typed multimodal input models and a responses_vision() helper; the image_url field is a plain string which matches the Responses API's input_image schema
tests/e2e/llm_translation/test_responses_e2e.py	Adds two e2e tests: vision image description (OpenAI gpt-4o) and Anthropic basic completion; assertions are tight but the keyword list for the vision test could miss edge-case valid answers

_{Reviews (1): Last reviewed commit: "test(e2e): cover /v1/responses OpenAI vi..." | Re-trigger Greptile}

[greptile-apps]

Vision assertion keyword coverage is narrow

The assertion only matches "cat" or "feline", so it would fail if the model replies with a different valid single-word answer for that image such as "kitten", "kitty", or a breed name like "persian" or "tabby". Prompt-compliance ("answer in one word") reduces this risk in practice, but extending the keyword list or anchoring the assertion with a broader substring check would make the test more robust against non-deterministic model output.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_e2e_responses_vision_anthropic (685948a) with litellm_internal_staging (a1fb07f)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (7a42f25) during the generation of this report, so a1fb07f was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩