fix(streaming): use provider-reported usage cost for OpenRouter streams

[mateo-berri]

Relevant issues

Port of #16162 by @dhruvyad, rebased onto litellm_internal_staging with conflicts resolved. All code credit goes to the original author; the commit keeps their authorship. Extension of #13653

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

This environment has no OpenRouter key, so here are the exact steps to verify against a live proxy

1. Add an openrouter/openai/gpt-5.2 model backed by OPENROUTER_API_KEY to litellm/proxy/dev_config.yaml
2. python litellm/proxy/proxy_cli.py --config litellm/proxy/dev_config.yaml --detailed_debug --reload --use_v2_migration_resolver 2>&1 | tee litellm.log
3. curl -s http://localhost:4000/v1/chat/completions -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"model": "openrouter/openai/gpt-5.2", "messages": [{"role": "user", "content": "hi"}], "stream": true, "stream_options": {"include_usage": true}}'
4. Open http://localhost:4000/ui/?page=logs and check that the spend for the request matches the cost OpenRouter reports on the usage chunk instead of a token-based estimate; before this change the usage chunk that OpenRouter sends after the finish_reason chunk was dropped and spend was estimated from tokens

Type

🐛 Bug Fix
✅ Test

Changes

OpenRouter sends a final streaming chunk carrying usage data, including a provider-reported cost field, after the finish_reason chunk. The stream handler used to raise StopIteration as soon as a post-finish chunk arrived, so that usage chunk never reached the assembled response and cost tracking fell back to token-based estimates

return_processed_chunk_logic now returns post-finish chunks that carry usage instead of raising, ChunkProcessor accumulates usage.cost across chunks, calculate_total_usage preserves the cost from the latest usage chunk, and the assembled response propagates the provider cost into _hidden_params["additional_headers"]["llm_provider-x-litellm-response-cost"], which is where the cost calculator already looks for provider-reported cost. ModelResponseStream.__init__ also sets the coerced Usage object explicitly so a dict or BaseModel usage kwarg survives pydantic init

One deviation from the original PR: the usage chunk extraction in _calculate_usage_per_chunk moved to a small _extract_usage_chunk helper because the extra branches pushed the function over the C901 complexity budget on this repo

---

Note

Medium Risk
Touches core streaming termination and spend logging paths; behavior change for any provider that sends usage after finish, though scoped to preserving usage/cost rather than new billing logic.

Overview
Fixes streaming spend for providers (notably OpenRouter) that emit a usage chunk with cost after finish_reason. That chunk was previously dropped when the handler stopped the stream, so logging fell back to token-based cost estimates.

Streaming handler: After the last content chunk, post-finish chunks that carry usage are yielded instead of ending the iterator. Incoming chunk usage is attached to the model response, preserved in _hidden_params, and calculate_total_usage keeps cost from the latest usage (including dict-shaped usage). On stream completion, usage.cost is copied into _hidden_params["additional_headers"]["llm_provider-x-litellm-response-cost"] so the existing cost calculator prefers the provider value.

Chunk assembly: ChunkProcessor threads cost through per-chunk usage aggregation (with a small _extract_usage_chunk helper). ModelResponseStream explicitly assigns the coerced Usage so usage passed as dict/BaseModel survives init.

Tests cover cost in assembled usage, dict usage regression, async streaming after finish, and hidden-params propagation.

^{Reviewed by Cursor Bugbot for commit 214130e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[mateo-berri]

@greptileai

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR fixes streaming cost tracking for providers (notably OpenRouter) that emit a usage chunk containing a provider-reported cost field after the finish_reason chunk. Previously, return_processed_chunk_logic raised StopIteration on any post-finish chunk, so the usage chunk was silently dropped and billing fell back to token-based estimates.

• return_processed_chunk_logic now returns post-finish chunks that carry usage instead of raising, allowing the chunk to be accumulated in self.chunks via the normal path.
• ChunkProcessor._calculate_usage_per_chunk and calculate_total_usage are updated to thread cost through chunk aggregation, and _propagate_usage_cost_to_hidden_params copies it into _hidden_params["additional_headers"]["llm_provider-x-litellm-response-cost"] so the existing cost calculator prefers the provider value.
• ModelResponseStream.__init__ adds an explicit self.usage = usage_to_set after super().__init__() to prevent Pydantic coercion from discarding the cost field when usage is passed as a dict or BaseModel.

Confidence Score: 4/5

• The core cost-propagation fix is implemented correctly and well-tested by three of the four new tests; one async test has an assertion that conflicts with how the streaming handler strips and skips usage-only chunks.
• The production fix — accumulating the post-finish usage chunk in self.chunks, extracting cost via _calculate_usage_per_chunk, and copying it to _hidden_params in _propagate_usage_cost_to_hidden_params — is logically sound and end-to-end correct. test_openrouter_streaming_cost_propagates_to_hidden_params and the chunk-builder test validate the critical path well. However, test_openrouter_streaming_cost_after_finish_reason asserts that the post-finish usage chunk is visible in the async for output, but the async path in __anext__ strips usage from outgoing chunks, finds the remainder empty (finish_reason=None, delta.content=""), and issues continue — so the chunk is never yielded and that assertion would fail.
• tests/test_litellm/litellm_core_utils/test_streaming_handler.py — specifically test_openrouter_streaming_cost_after_finish_reason and its assertion on collected_chunks.

Important Files Changed

Filename	Overview
litellm/litellm_core_utils/streaming_handler.py	Replaces raise StopIteration after finish_reason with a usage-aware return so OpenRouter's post-finish cost chunk is captured; adds _propagate_usage_cost_to_hidden_params to copy usage.cost into _hidden_params["additional_headers"] for the cost calculator. Core fix is correct; see test comment for a potential assertion gap in the async test.
litellm/litellm_core_utils/streaming_chunk_builder_utils.py	Extracts _extract_usage_chunk helper to reduce complexity, threads cost through UsagePerChunk and _calculate_usage_per_chunk, and preserves it in the assembled Usage object via setattr. Since cost is a declared Pydantic field on Usage, it survives the subsequent Usage(**returned_usage.model_dump()) round-trip correctly.
litellm/types/utils.py	Adds explicit self.usage = usage_to_set after super().__init__() to prevent Pydantic coercion from discarding the cost field on a dict/BaseModel usage kwarg.
litellm/types/litellm_core_utils/streaming_chunk_builder_utils.py	Adds cost: Optional[float] to the UsagePerChunk TypedDict to carry provider-reported cost through chunk aggregation.
tests/test_litellm/litellm_core_utils/test_streaming_chunk_builder_utils.py	Adds test_cost_field_in_usage_chunks covering ChunkProcessor.calculate_usage with a cost-bearing usage chunk; assertions are correct and the test exercises the right code path.
tests/test_litellm/litellm_core_utils/test_streaming_handler.py	Adds four new tests; three are sound, but test_openrouter_streaming_cost_after_finish_reason asserts that the post-finish usage chunk is visible in collected_chunks, which conflicts with the async path's strip-and-skip logic for empty chunks — the assertion would fail.

_{Reviews (6): Last reviewed commit: "fix(streaming): preserve provider cost w..." | Re-trigger Greptile}

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Dict usage drops provider cost
  • Replaced the attribute-only hasattr(latest_usage_chunk, "cost") check with a dict-vs-object aware lookup (using .get("cost") for dicts and getattr(..., "cost", None) otherwise) so provider cost is preserved for both dict- and object-shaped usage chunks, and added a regression test with dict-shaped usage.

_{You can send follow-ups to the cloud agent here.}

[CLAassistant]

All committers have signed the CLA.

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 214130e. Configure here.}

[veria-ai]

Low: Provider cost disclosure

usage.cost is serialized back to streaming clients when they request usage chunks, so an authenticated client can learn provider-reported per-request cost even when include_cost_in_streaming_usage is false. Keep the provider cost in hidden metadata for spend tracking, but only add it to the public Usage object when cost disclosure is explicitly enabled.

[mateo-berri]

The include_cost_in_streaming_usage flag gates a different thing: whether litellm injects its own calculated cost into streaming usage chunks. Every usage site (stream_chunk_builder in litellm/main.py, the responses streaming iterators, the pass-through cost injection) calls logging_obj._response_cost_calculator behind that flag. It has never governed passthrough of provider-reported cost, so there is no configuration boundary being crossed here

For provider-reported cost specifically, the non-streaming path already returns it to the same authenticated client today: litellm always requests usage: {include: true} from OpenRouter (see add_usage_parameter in litellm/llms/openrouter/chat/transformation.py), the raw body's usage.cost survives Usage(**response_object["usage"]) in convert_dict_to_response.py, and cost is a declared field on the Usage model. This PR brings streaming to parity with that existing non-streaming behavior, which also matches what OpenRouter's own API returns for the identical request. The value disclosed is the cost of the client's own request, not data belonging to another tenant or the proxy operator's margin

Spend tracking does not depend on the public field either way; the cost calculator reads _hidden_params["additional_headers"]["llm_provider-x-litellm-response-cost"], which this PR populates via _propagate_usage_cost_to_hidden_params. Suppressing usage.cost on streaming responses while non-streaming responses keep returning it would add an inconsistency without adding a boundary

[veria-ai]

PR overview

This pull request updates OpenRouter streaming usage handling to rely on provider-reported usage cost when building streaming usage chunks. The touched streaming chunk builder code affects how usage and cost data are carried through streamed responses.

There is one open security concern: provider-reported request cost can still be serialized to authenticated streaming clients that request usage chunks, even when public cost disclosure is disabled. The impact is limited to disclosure of per-request provider cost rather than direct access control or code execution, but it conflicts with the intended configuration boundary for hiding cost data. No issues have been fixed yet, so the PR still needs an adjustment to keep cost available only for internal spend tracking unless explicitly enabled for clients.

Open issues (1)

• Low: Provider cost disclosure — litellm/litellm_core_utils/streaming_chunk_builder_utils.py:807

Fixed/addressed: 0 · PR risk: 4/10

[greptile-apps]

Post-finish usage chunk never appears in collected_chunks

chunk3 has a non-null usage so it is returned by chunk_creator, but the async iteration path in __anext__ immediately strips usage from the outgoing chunk, rebuilds it via model_response_creator, and then calls is_model_response_stream_empty. The rebuilt chunk has choices=[StreamingChoices(finish_reason=None, delta=Delta(content=""))] and usage=None; because content="" is not considered meaningful by _has_meaningful_content and finish_reason is None, is_model_response_stream_empty returns True and the chunk is continue'd — never yielded to the caller.

As a result collected_chunks contains only chunk1 and chunk2 (both with usage=None), so usage_chunks is empty and assert len(usage_chunks) > 0 raises AssertionError.

The underlying cost-tracking fix is correct (chunk3 is saved to self.chunks via model_copy() and its cost flows through stream_chunk_builder → _propagate_usage_cost_to_hidden_params), but this test is asserting user-visible behavior that the current async strip-and-skip path does not produce. The assertion should instead inspect the assembled response's hidden params or usage.cost, mirroring test_openrouter_streaming_cost_propagates_to_hidden_params.