test(e2e): spendlog cost for streaming /v1/messages via responses bridge

[yassin-berriai]

Relevant issues

Linear ticket

Resolves LIT-4546

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Captured at commit fdec1afd79 against a live proxy running the published ghcr.io/berriai/litellm:main-latest image, hitting the real OpenAI Responses API and costing real dollars.

Bring the suite's compose stack up (namespaced so it does not collide with other local proxies), with OPENAI_API_KEY in tests/e2e/.env:

docker compose -p e2espd -f docker-compose.yml -f spend-override.yml up -d
curl -fs http://localhost:4630/health/liveliness
"I'm alive!"

Run the new test against it:

LITELLM_PROXY_URL=http://localhost:4630 python -m pytest \
  tests/e2e/quota_management/spend_tracking/test_spend_tracking_e2e.py::test_streaming_messages_via_responses_bridge_tracks_spend -v

============================= test session starts ==============================
platform darwin -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0
collected 1 item
tests/e2e/.../test_streaming_messages_via_responses_bridge_tracks_spend PASSED [100%]
============================== 1 passed in 11.59s ==============================

The same behavior by hand against that live proxy. Generate a scoped key, then POST a streaming /v1/messages request (anthropic format) to the responses-bridged deployment openai-responses-codex -> openai/gpt-5.3-codex, a Responses-only OpenAI model:

curl -sS -N -D - http://localhost:4630/v1/messages \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"model":"openai-responses-codex","max_tokens":64,"stream":true,
       "messages":[{"role":"user","content":"reply with exactly one word final-proof"}]}'

HTTP/1.1 200 OK
x-litellm-call-id: 3e3ff690-44d2-4712-862c-2202a81d559b
x-litellm-model-group: openai-responses-codex
content-type: text/event-stream; charset=utf-8

event: message_start
data: {"type": "message_start", "message": {"id": "msg_be6a1a88-...", "model": "gpt-5.3-codex", ...}}
event: content_block_delta
data: {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "final"}}
... (more text_delta events) ...
event: message_delta
data: {"type": "message_delta", "delta": {"stop_reason": "end_turn"}, "usage": {"input_tokens": 18, "output_tokens": 11}}
event: message_stop

Poll /spend/logs for that key once the batch write lands; exactly one row appears, carrying a nonzero cost and token counts, the openai provider, and the /v1/messages billing identity:

curl -s "http://localhost:4630/spend/logs?api_key=$KEY" -H "Authorization: Bearer sk-1234"

{
  "request_id": "chatcmpl-21ee0c0f-dca0-406b-a468-25f8eb9a0063",
  "model": "openai/gpt-5.3-codex",
  "custom_llm_provider": "openai",
  "call_type": "anthropic_messages",
  "spend": 0.0001855,
  "prompt_tokens": 18,
  "completion_tokens": 11,
  "total_tokens": 29,
  "status": "success"
}

custom_llm_provider: openai on a gpt-5.3-codex row that is only reachable via /v1/responses, under call_type: anthropic_messages, is the combination that proves the Responses adapter served the /v1/messages request. A chat-completions bridge would have failed at OpenAI on an endpoint the model does not expose, so the test cannot pass via the wrong path.

The rest of the tests/e2e/quota_management/spend_tracking/ suite was run against the same stack and stays green, except two pre-existing gemini tests that fail only because this box has no GEMINI_API_KEY; the gemini-2.5-flash deployment then 401s and the router falls back to openai/gpt-5.5 per the compose config, tripping their "gemini-2.5-flash" in model assertion. That is an environment gap, not a regression from this change, and it is unrelated to the responses-bridge path this PR covers.

Type

✅ Test

Changes

Existing in-suite spend coverage only drives streaming /chat/completions; nothing exercised /v1/messages through the OpenAI Responses adapter. In litellm any custom_llm_provider == "openai" model called on /v1/messages is bridged through LiteLLMMessagesToResponsesAPIHandler (the anthropic-messages -> Responses adapter) unless use_chat_completions_url_for_anthropic_messages is set, so the switch is provider-based rather than a per-model mode flag.

The new test test_streaming_messages_via_responses_bridge_tracks_spend creates a scoped key, POSTs /v1/messages with stream: true to a Responses-only OpenAI deployment, consumes the whole SSE stream, then polls /spend/logs to a deadline and asserts exactly one costed row for the key with nonzero prompt and completion tokens that sum to the total, custom_llm_provider == "openai", and a call_type that keeps the /v1/messages billing identity. Picking a Responses-only model (gpt-5.3-codex) makes the assertion path unambiguous, since the row could not have come from the chat-completions bridge.

Supporting changes are a streaming /v1/messages method on the shared Gateway (so other suites get the route for free) and on the spend suite client, the openai-responses-codex deployment added to the inline docker-compose.yml config and the suite's driver-model registration, a coverage-registry row quota_management.spend_tracking.messages_bridge.logs_cost, and the matching messages_bridge entry in the spend_tracking variant vocab. A follow-up commit adds call_type and custom_llm_provider to the _summarize diagnostic detail so a failed assertion prints the fields it asserts on.

QA runbook

• tests/e2e/quota_management/spend_tracking/test_spend_tracking_e2e.py::test_streaming_messages_via_responses_bridge_tracks_spend - a fully consumed streaming /v1/messages request bridged to the OpenAI Responses adapter writes exactly one SpendLogs row with cost and token counts, attributed to the calling key
  • Bring up the stack (docker compose -f docker-compose.yml up -d) with OPENAI_API_KEY in tests/e2e/.env; the inline config already registers openai-responses-codex -> openai/gpt-5.3-codex
  • Generate a key: curl -X POST http://localhost:4000/key/generate -H "Authorization: Bearer sk-1234" -d '{"models":[]}'
  • POST /v1/messages with that key, "model":"openai-responses-codex", "stream":true, and a short message; read the SSE stream to message_stop and expect a 200 with content-type: text/event-stream
  • Poll GET /spend/logs?api_key=<key> until one row appears; expect spend > 0, nonzero prompt_tokens and completion_tokens summing to total_tokens, custom_llm_provider == "openai", and call_type == "anthropic_messages"
  • Sanity check: this test makes sense to add and is not hand-wavey (it asserts a real cost and token arithmetic on a Responses-only model, so the wrong bridge cannot make it pass) or potentially flaky

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[yassin-berriai]

@greptileai

[greptile-apps]

Greptile Summary

This PR adds an e2e test that verifies spend tracking for streaming /v1/messages requests bridged through litellm's anthropic-messages → OpenAI Responses adapter, and addresses two findings from the previous review round (the _summarize diagnostic now exposes call_type and custom_llm_provider, and the PR description now includes a live proof run of the correct test).

• New test (test_streaming_messages_via_responses_bridge_tracks_spend): creates a scoped key, POSTs a streaming Anthropic-format /v1/messages request to a Responses-only OpenAI deployment (gpt-5.3-codex), consumes the SSE stream, then polls /spend/logs and asserts exactly one costed row with nonzero tokens, custom_llm_provider == "openai", and call_type containing "anthropic_messages" — making the bridge path unambiguous.
• Supporting additions: messages_stream route on Gateway and SpendClient, openai-responses-codex deployment in docker-compose.yml and DRIVER_MODELS, a P1 coverage-registry entry, and the messages_bridge variant added to the CLAUDE.md taxonomy.

Confidence Score: 5/5

All changes are confined to the tests/e2e/ directory and do not touch any production code paths.

The diff is entirely test infrastructure — a new e2e test, a shared gateway method, docker-compose and conftest additions, and registry/doc updates. Previous review findings (missing fields in _summarize, stale proof of fix) are addressed in this head commit. The test logic is well-isolated (scoped key per test, polling to a deadline, unambiguous bridge-path verification via a Responses-only model), and no production code is modified.

No files require special attention.

Important Files Changed

Filename	Overview
tests/e2e/quota_management/spend_tracking/test_spend_tracking_e2e.py	Adds test_streaming_messages_via_responses_bridge_tracks_spend and expands _summarize to include call_type and custom_llm_provider; assertions are well-formed and use a scoped key for isolation.
tests/e2e/quota_management/spend_tracking/spend_e2e_client.py	Adds messages_stream method delegating to gateway.messages_stream; mirrors the existing chat_stream pattern cleanly.
tests/e2e/e2e_gateway.py	Adds messages_stream route method to Gateway using the existing transport.stream helper; no functional concerns.
tests/e2e/docker-compose.yml	Registers openai-responses-codex → openai/gpt-5.3-codex deployment required by the new test; reads OPENAI_API_KEY from env consistently with other OpenAI deployments.
tests/e2e/quota_management/spend_tracking/conftest.py	Adds openai-responses-codex to DRIVER_MODELS following the exact same pattern as existing entries; registration/cleanup logic is unchanged.
tests/e2e/coverage_registry/quota_management.yaml	Adds a P1 coverage entry quota_management.spend_tracking.messages_bridge.logs_cost that accurately identifies the source file and rationale.
tests/e2e/CLAUDE.md	Adds messages_bridge to the <spend_tracking> variant vocabulary; a minor documentation update consistent with the new coverage entry.

_{Reviews (3): Last reviewed commit: "test(e2e): surface call_type and provide..." | Re-trigger Greptile}

[greptile-apps]

Proof in PR description does not cover this test

The PR's "Screenshots / Proof of Fix" section describes and demonstrates MCP key-access tests (tests/e2e/mcp/test_mcp_key_access_e2e.py) — a completely different suite. The command shown (python -m pytest tests/e2e/mcp/ -v) and the collected items (2 MCP tests) have no relation to this new spend-tracking test for the /v1/messages bridge. No evidence is supplied that test_streaming_messages_via_responses_bridge_tracks_spend passes against a live proxy, which is the claimed proof-of-fix for LIT-4547.

Rule Used: What: Ensure that any PR claiming to fix an issue ... (source)

[greptile-apps]

Greptile Summary

This PR adds an e2e spend-tracking test that validates a streaming /v1/messages request served by an OpenAI Responses-only model (openai/gpt-5.3-codex) is correctly billed through litellm's anthropic-messages → Responses adapter bridge, producing exactly one costed spend row with the right call_type, custom_llm_provider, and token arithmetic.

• Adds test_streaming_messages_via_responses_bridge_tracks_spend with assertions covering spend attribution, call_type identity, provider, model, and token arithmetic.
• Registers openai-responses-codex as a new driver model in the docker-compose config, conftest.py DRIVER_MODELS, and coverage registry.
• Extends Gateway and SpendClient with messages_stream helpers that post AnthropicMessagesBody to /v1/messages.

Confidence Score: 3/5

The code changes are internally consistent and follow established patterns, but the PR cannot be verified from the supplied proof.

The "Screenshots / Proof of Fix" section and the "Changes" narrative in the PR description document MCP authorization tests that do not exist in the changed files — they appear to have been pasted from a different PR. No evidence is provided that the actual new test has ever been run successfully against a live proxy. The test's correctness depends on a chain of conditions — the responses bridge aggregating the SSE stream correctly, the spend row landing with call_type containing anthropic_messages, and custom_llm_provider being openai — none of which are demonstrated.

tests/e2e/quota_management/spend_tracking/test_spend_tracking_e2e.py — the new test has no attached proof of a successful run.

Important Files Changed

Filename	Overview
tests/e2e/quota_management/spend_tracking/test_spend_tracking_e2e.py	Adds test_streaming_messages_via_responses_bridge_tracks_spend; logic is well-structured but the PR provides no evidence this test actually passes — the proof section documents MCP tests, not this one.
tests/e2e/docker-compose.yml	Adds openai-responses-codex deployment (openai/gpt-5.3-codex) to the compose config; consistent with existing model entries and pattern.
tests/e2e/quota_management/spend_tracking/conftest.py	Adds openai-responses-codex to DRIVER_MODELS using the same registration pattern as existing entries.
tests/e2e/quota_management/spend_tracking/spend_e2e_client.py	Adds messages_stream helper wrapping AnthropicMessagesBody; follows the same pattern as the existing chat_stream method.
tests/e2e/e2e_gateway.py	Adds messages_stream gateway method that posts to /v1/messages; straightforward extension of the existing streaming pattern.
tests/e2e/coverage_registry/quota_management.yaml	Adds the messages_bridge.logs_cost coverage entry with accurate source/rationale.
tests/e2e/CLAUDE.md	Adds messages_bridge to the spend_tracking variant taxonomy; reformats surrounding items for line-length consistency.

Comments Outside Diff (1)

1. tests/e2e/quota_management/spend_tracking/test_spend_tracking_e2e.py, line 125-199 (link)

   Proof of fix is for a different feature entirely

   The PR's "Screenshots / Proof of Fix" section and "Changes" description document MCP authorization tests (tests/e2e/mcp/test_mcp_key_access_e2e.py), which do not appear anywhere in the changed files. No evidence is provided that test_streaming_messages_via_responses_bridge_tracks_spend actually passes — the spend-log polling, the bridged_costed_row predicate, and the call_type / custom_llm_provider assertions are all untested by the supplied proof. Per the team's rule requiring PRs to include evidence that the addressed issue is resolved, the submitted screenshots fail to satisfy that bar for this change.

   Rule Used: What: Ensure that any PR claiming to fix an issue ... (source)

   Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

_{Reviews (2): Last reviewed commit: "test(e2e): spendlog cost for streaming /..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[yassin-berriai]

@greptileai please review the current head fdec1af

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_e2e_messages_bridge_spend (fdec1af) with litellm_internal_staging (561b679)}

[yassin-berriai]

Superseded by #33753, which carries the identical e2e change rebased onto current litellm_internal_staging. Continuing the work there