test(e2e): spendlog cost for streaming /v1/messages via responses bridge

[yassin-berriai]

Relevant issues

Linear ticket

Resolves LIT-4546

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Captured at commit f57971f604 against a live proxy running the published ghcr.io/berriai/litellm:main-latest image, hitting the real OpenAI Responses API and costing real dollars

Bring the suite's compose stack up (namespaced so it does not collide with other local proxies) with OPENAI_API_KEY in tests/e2e/.env, remapping the proxy off 4000 so it does not clash with another local proxy:

docker compose -p e2elit4546 -f docker-compose.yml -f override-lit4546.yml up -d
curl -fs http://localhost:4546/health/liveliness
"I'm alive!"

Run the new test against it:

LITELLM_PROXY_URL=http://localhost:4546 python -m pytest \
  tests/e2e/quota_management/spend_tracking/test_spend_tracking_e2e.py::test_streaming_messages_via_responses_bridge_tracks_spend -v

quota_management/spend_tracking/test_spend_tracking_e2e.py::test_streaming_messages_via_responses_bridge_tracks_spend PASSED [100%]
============================== 1 passed in 6.64s ===============================

The same behavior by hand: generate a scoped key, POST a streaming /v1/messages request (anthropic format) to the responses-bridged deployment openai-responses-codex -> openai/gpt-5.3-codex, a Responses-only OpenAI model, consume the whole SSE stream:

curl -sS -N -D - http://localhost:4546/v1/messages \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"model":"openai-responses-codex","max_tokens":64,"stream":true,
       "messages":[{"role":"user","content":"reply with exactly one word proofrun"}]}'

HTTP/1.1 200 OK
x-litellm-call-id: 7deeaf2b-6a8a-4643-8452-0a928a164c52
x-litellm-model-group: openai-responses-codex
content-type: text/event-stream; charset=utf-8

event: message_start
data: {"type": "message_start", "message": {"id": "msg_45e78402-...", "model": "gpt-5.3-codex", ...}}
... (text_delta events) ...
event: message_delta
data: {"type": "message_delta", "delta": {"stop_reason": "end_turn"}, "usage": {"input_tokens": 13, "output_tokens": 6}}
event: message_stop

Poll /spend/logs for that key once the batch write lands; exactly one row appears, carrying a nonzero cost and token counts, the openai provider, and the /v1/messages billing identity:

curl -s "http://localhost:4546/spend/logs?api_key=$KEY" -H "Authorization: Bearer sk-1234"

{
  "request_id": "chatcmpl-0307bd44-ba8a-43cb-a329-7af6e58f4e36",
  "model": "openai/gpt-5.3-codex",
  "custom_llm_provider": "openai",
  "call_type": "anthropic_messages",
  "spend": 0.00010675,
  "prompt_tokens": 13,
  "completion_tokens": 6,
  "total_tokens": 19,
  "status": "success"
}

custom_llm_provider: openai on a gpt-5.3-codex row that is only reachable via /v1/responses, under call_type: anthropic_messages, is the combination that proves the Responses adapter served the /v1/messages request. A chat-completions bridge would have failed at OpenAI on an endpoint the model does not expose, so the test cannot pass via the wrong path

Independent end-to-end run (Devin)

Devin independently checked out the branch, brought up the namespaced e2e compose stack, and ran the new test against a live proxy hitting the real OpenAI Responses API for real money. The test passed (1 passed in 15.93s), and the by-hand repro produced exactly one costed /spend/logs row for the scoped key: spend $0.00018375, custom_llm_provider openai, call_type anthropic_messages, model openai/gpt-5.3-codex, and prompt_tokens 17 + completion_tokens 11 == total_tokens 28, with the streaming /v1/messages response coming back 200 text/event-stream and ending in message_stop

The test passing:

The single costed /spend/logs row it asserts on:

Full-resolution screen recording: https://raw.githubusercontent.com/yassin-berriai/litellm-pr-media/main/pr33753-lit4546/e2e_run.mp4

Type

✅ Test

Changes

Existing in-suite spend coverage only drives streaming /chat/completions; nothing exercised /v1/messages through the OpenAI Responses adapter. In litellm any custom_llm_provider == "openai" model called on /v1/messages is bridged through LiteLLMMessagesToResponsesAPIHandler (the anthropic-messages -> Responses adapter) unless use_chat_completions_url_for_anthropic_messages is set, so the switch is provider-based rather than a per-model mode flag

The new test test_streaming_messages_via_responses_bridge_tracks_spend creates a scoped key, POSTs /v1/messages with stream: true to a Responses-only OpenAI deployment, consumes the whole SSE stream, then polls /spend/logs to a deadline and asserts exactly one costed row for the key with nonzero prompt and completion tokens that sum to the total, custom_llm_provider == "openai", and a call_type that keeps the /v1/messages billing identity. Picking a Responses-only model (gpt-5.3-codex) makes the assertion path unambiguous, since the row could not have come from the chat-completions bridge

Supporting changes are a streaming /v1/messages method on the shared Gateway (so other suites get the route for free) and on the spend suite client, the openai-responses-codex deployment added to the inline docker-compose.yml config and the suite's driver-model registration, a coverage-registry row quota_management.spend_tracking.messages_bridge.logs_cost, and the matching messages_bridge entry in the spend_tracking variant vocab. The _summarize spend-row detail also gains call_type and custom_llm_provider so a failed assertion prints the two fields it asserts on

QA runbook

• tests/e2e/quota_management/spend_tracking/test_spend_tracking_e2e.py::test_streaming_messages_via_responses_bridge_tracks_spend - a fully consumed streaming /v1/messages request bridged to the OpenAI Responses adapter writes exactly one SpendLogs row with cost and token counts, attributed to the calling key
  • Bring up the stack (docker compose -f docker-compose.yml up -d) with OPENAI_API_KEY in tests/e2e/.env; the inline config already registers openai-responses-codex -> openai/gpt-5.3-codex
  • Generate a key: curl -X POST http://localhost:4000/key/generate -H "Authorization: Bearer sk-1234" -d '{"models":[]}'
  • POST /v1/messages with that key, "model":"openai-responses-codex", "stream":true, and a short message; read the SSE stream to message_stop and expect a 200 with content-type: text/event-stream
  • Poll GET /spend/logs?api_key=<key> until one row appears; expect spend > 0, nonzero prompt_tokens and completion_tokens summing to total_tokens, custom_llm_provider == "openai", and call_type == "anthropic_messages"
  • Sanity check: this test makes sense to add and is not hand-wavey (it asserts a real cost and token arithmetic on a Responses-only model, so the wrong bridge cannot make it pass) or potentially flaky

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Low Risk
Test-only changes (e2e clients, registry, docs); no proxy or spend-tracking production code modified.

Overview
Adds end-to-end coverage that a fully consumed streaming /v1/messages call against an OpenAI Responses-only deployment (openai-responses-codex → gpt-5.3-codex) produces exactly one costed spend row with correct token math, custom_llm_provider: openai, and a anthropic_messages call_type (messages billing identity, not chat).

Supporting harness changes: messages_stream on ProxyClient and SpendClient, the codex deployment in spend suite driver models, a messages_bridge coverage-registry row and CLAUDE.md variant vocab, and call_type / custom_llm_provider in spend-row failure summaries.

^{Reviewed by Cursor Bugbot for commit 48659a6. Bugbot is set up for automated code reviews on this repo. Configure here.}

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[yassin-berriai]

@greptileai

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

This PR adds end-to-end spend-tracking coverage for streaming /v1/messages requests bridged through the OpenAI Responses adapter, verifying that a fully consumed SSE stream from a Responses-only model (gpt-5.3-codex) produces exactly one costed spend row with the correct call_type, custom_llm_provider, and token arithmetic.

• New test test_streaming_messages_via_responses_bridge_tracks_spend: creates a scoped key, POSTs a streaming /v1/messages to the openai-responses-codex deployment, consumes the full SSE stream, then polls /spend/logs with a two-stage assertion — all costed rows must be bridged (bridged == costed), and exactly one bridged row must exist. This addresses the filter-consistency issue from a prior review round.
• Harness additions: messages_stream route added to ProxyClient and a matching wrapper on SpendClient, both using self.proxy correctly; openai-responses-codex registered in DRIVER_MODELS; coverage registry entry and CLAUDE.md vocab updated.

Confidence Score: 5/5

Test-only change with no modifications to production proxy or spend pipeline code; safe to merge.

Every changed file is confined to tests/e2e/. The two previously flagged issues — a stale self.gateway reference in SpendClient.messages_stream and a filter-consistency gap between the polling predicate and the costed list — are both resolved in this head: the method correctly calls self.proxy.messages_stream(...), and the test now asserts bridged == costed before checking len(bridged) == 1, which catches any extra non-bridged costed rows. No production code is touched.

No files require special attention.

Important Files Changed

Filename	Overview
tests/e2e/quota_management/spend_tracking/test_spend_tracking_e2e.py	Adds test_streaming_messages_via_responses_bridge_tracks_spend with well-structured two-stage assertion: all costed rows must be bridged (bridged == costed), then exactly one bridged row is required — addressing the filter-consistency concern from a prior review. Also adds call_type and custom_llm_provider to _summarize diagnostics.
tests/e2e/quota_management/spend_tracking/spend_e2e_client.py	Adds messages_stream wrapper on SpendClient that correctly delegates to self.proxy.messages_stream(...) — consistent with every other method in the class and the proxy: ProxyClient dataclass field.
tests/e2e/proxy_client.py	Adds messages_stream route to the shared ProxyClient, delegating to transport.stream("/v1/messages", ...) — follows the same pattern as chat_stream.
tests/e2e/quota_management/spend_tracking/conftest.py	Registers openai-responses-codex → openai/gpt-5.3-codex in DRIVER_MODELS, providing the Responses-only deployment the new test targets.
tests/e2e/coverage_registry/quota_management.yaml	Adds quota_management.spend_tracking.messages_bridge.logs_cost registry entry at tier P1 with correct source, exercised_on, and rationale fields.
tests/e2e/CLAUDE.md	Adds messages_bridge to the <spend_tracking> variant vocabulary and re-wraps the existing token list — documentation-only change.

_{Reviews (8): Last reviewed commit: "test(e2e): spendlog cost for streaming /..." | Re-trigger Greptile}

[greptile-apps]

Greptile Summary

Adds a new live e2e test (test_streaming_messages_via_responses_bridge_tracks_spend) that verifies streaming /v1/messages requests served by an OpenAI-provider model are correctly bridged through the Anthropic-messages → Responses adapter and produce a costed spend row. Supporting changes include a messages_stream route on Gateway and SpendClient, the openai-responses-codex deployment in the docker-compose config, a matching driver_models entry, and a registry row for the new coverage path.

• New Gateway.messages_stream and SpendClient.messages_stream delegate cleanly through transport.stream(\"/v1/messages\", ...), consistent with the existing chat_stream pattern.
• The test asserts SSE stream integrity, exactly one costed spend row, call_type containing \"anthropic_messages\", custom_llm_provider == \"openai\", and correct token arithmetic.
• _summarize gains call_type and custom_llm_provider fields, improving failure diagnostics across the whole suite.

Confidence Score: 4/5

All changes are confined to the e2e test suite with no production code touched; safe to merge.

The test logic, gateway additions, and docker-compose stanza are consistent with existing patterns. The only concern is a minor filter mismatch in the new test where costed collects all spend > 0 rows rather than matching the call_type guard the predicate uses.

tests/e2e/quota_management/spend_tracking/test_spend_tracking_e2e.py — the costed filter on line 168 should mirror the predicate's call_type guard.

Important Files Changed

Filename	Overview
tests/e2e/quota_management/spend_tracking/test_spend_tracking_e2e.py	Adds the new messages bridge test with a slightly looser costed filter than the polling predicate; also widens _summarize.
tests/e2e/e2e_gateway.py	Adds messages_stream method to Gateway, consistent with existing chat_stream pattern.
tests/e2e/quota_management/spend_tracking/conftest.py	Adds openai-responses-codex to DRIVER_MODELS with the same auto-registration pattern as existing drivers.
tests/e2e/quota_management/spend_tracking/spend_e2e_client.py	Adds messages_stream to SpendClient, mirroring the chat_stream shape.
tests/e2e/docker-compose.yml	Adds the openai-responses-codex deployment stanza, consistent with existing model entries.
tests/e2e/coverage_registry/quota_management.yaml	Adds messages_bridge.logs_cost coverage entry with accurate source and rationale.
tests/e2e/CLAUDE.md	Reformats spend_tracking vocabulary to include messages_bridge.

_{Reviews (2): Last reviewed commit: "test(e2e): spendlog cost for streaming /..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[yassin-berriai]

CI note for the reviewer: the two red checks are both base/environment, not this diff (which is tests/e2e only).

• `osv-scan`: a CVE in the base branch lockfile, published after the last dependency bump. It fails identically on every open PR right now and no dependency is changed in this diff (`git diff base...HEAD -- uv.lock` is empty). The bump belongs in its own PR on staging
• `misc / Run tests`: the single failure is `tests/test_litellm/test_gpt_realtime_mode.py::test_get_model_info_reports_realtime_mode` (`assert 'chat' == 'realtime'`). This diff touches no realtime code or `get_model_info`. The test is order-dependent on the model cost map source: it passes under `LITELLM_LOCAL_MODEL_COST_MAP=True` and fails when the shard resolves the remote map first. It is failing on other open PRs (e.g. test(e2e): guard destructive spend-log truncate behind an explicit opt-in #33751, fix(proxy): bill partial streamed spend when the client disconnects mid-stream #33736) at the same time, so it is a staging-side flake, not a regression here

Everything else is green and Greptile is 5/5 on the head commit

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_lit4546_e2e_messages_bridge_spend (48659a6) with litellm_internal_staging (66dea7d)}

[yassin-berriai]

@greptileai please review the current head 12a5305. The follow-up commit reconciles the costed-row selection with the polling predicate by sharing one is_bridged_costed guard, and asserts costed == bridged so a stray costed row under a different call_type fails loud (keeping the exactly-one-costed-row guarantee).

[yassin-berriai]

@greptileai please review the current head cc419c0. Rebased onto current staging to adopt the Gateway to ProxyClient rename from #33750 (the previous head no longer merged cleanly). No behavior change: messages_stream now lives on ProxyClient and the spend client calls self.proxy; the test and its assertions are unchanged.

[yucheng-berri]

bugbot run

[yassin-berriai]

@greptileai good catch, fixed. The stale self.gateway in SpendClient.messages_stream now calls self.proxy.messages_stream(...) to match the ProxyClient rename. Please review the current head b9ca047. Verified locally: basedpyright tests/e2e is clean (0 errors) and the committed tree has no remaining self.gateway reference.

[yassin-berriai]

@greptileai please review the current head 325462e. Rebased again onto current staging to resolve a conflict from #33831 (which added a native messages/count_tokens to ProxyClient and the AnthropicMessagesResponse import); dropped my now-duplicate AnthropicMessagesBody import and kept messages_stream. No behavior change; basedpyright tests/e2e delta vs staging is zero.

[yassin-berriai]

@greptileai please review the current head babb8bf. Rebased onto current staging now that #33839 (load-suite migration to ProxyClient) has merged, clearing the base-drift e2e basedpyright failure. No change to this PR's diff beyond the rebase.

[yassin-berriai]

@greptileai please review the current head 48659a6. Rebased onto current staging after #33837 removed the bundled tests/e2e/docker-compose.yml; resolved the modify/delete by dropping this PR's edit to that file. The openai-responses-codex deployment is still registered by the suite via DRIVER_MODELS (conftest), so the test is unaffected. All .py files are byte-identical to the previously 5/5 head.

[yucheng-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 48659a6. Configure here.}