fix(proxy): request stream usage upstream by default and strip it from client streams - #35290
Merged
mateo-berri merged 4 commits intoJul 31, 2026
Merged
Conversation
…m client streams Streamed chat completions that did not opt into stream_options.include_usage were logged with tiktoken estimates over the visible text, so hidden reasoning tokens (billed as output by OpenAI-compatible providers) were never counted and SpendLogs could undercount output tokens by 90%+ on reasoning models. The proxy now injects include_usage upstream for /v1/chat/completions streams by default and strips the injection artifacts (the final usage chunk and the empty prompt-filter chunk) from the client-facing SSE stream, so accounting uses provider-billed usage while the client-visible stream stays byte-identical to today. always_include_stream_usage keeps its existing semantics: true forwards the usage chunk to clients as before, and an explicit false now acts as a kill switch that disables the upstream injection for OpenAI-compatible backends that reject stream_options.
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Contributor
Greptile SummaryThis PR requests upstream stream usage by default while preserving the client-visible streaming contract.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| litellm/proxy/common_request_processing.py | Adds support-gated stream usage injection and neutralizes untrusted strip-marker values; the previously reported unsupported-provider issue is addressed. |
| litellm/proxy/proxy_server.py | Filters automatically induced empty usage artifacts and removes the internal marker from queue request bodies; the previously reported client-controlled stripping issue is addressed. |
| litellm/types/utils.py | Registers the strip marker as an internal LiteLLM parameter so it is not forwarded as a provider parameter. |
| tests/test_litellm/proxy/test_common_request_processing.py | Adds focused tests for tri-state configuration, provider support gating, route scoping, team aliases, and marker neutralization. |
| tests/test_litellm/proxy/test_proxy_server.py | Adds predicate and generator-level coverage proving marked usage artifacts are removed while content and finish chunks remain visible. |
Reviews (4): Last reviewed commit: "fix(proxy): resolve team-alias models in..." | Re-trigger Greptile
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
…nd neutralize client-sent strip marker Bytez and OCI param maps raise on stream_options when drop_params is unset, so the default injection would have broken every streamed chat completion routed to them. Injection now only happens when every router deployment behind the requested model (wildcards and aliases included) declares stream_options in its supported OpenAI params; providers that do not declare it either reject the param or already stream usage natively, so skipping them keeps old behavior instead of erroring. _litellm_strip_stream_usage arriving in the client request body is now overwritten at ingress (and popped in the experimental queue endpoint), so a client can no longer suppress the usage chunk it explicitly requested by planting the internal marker.
Contributor
Author
Contributor
Author
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Team models skip usage injection
- Threaded team_id through _model_deployments_support_stream_options and passed user_api_key_dict.team_id at the call site so team-scoped models resolve via team_public_model_name and include_usage is injected as intended.
Or push these changes by commenting:
@cursor push b9bbf248e5
Preview (b9bbf248e5)
diff --git a/litellm/proxy/common_request_processing.py b/litellm/proxy/common_request_processing.py
--- a/litellm/proxy/common_request_processing.py
+++ b/litellm/proxy/common_request_processing.py
@@ -270,10 +270,13 @@
def _model_deployments_support_stream_options(
model: object,
llm_router: Optional[Router],
+ team_id: Optional[str] = None,
) -> bool:
if not isinstance(model, str):
return False
- deployments = llm_router.get_model_list(model_name=model) if llm_router is not None else None
+ deployments = (
+ llm_router.get_model_list(model_name=model, team_id=team_id) if llm_router is not None else None
+ )
deployment_models = tuple(
litellm_model
for deployment in deployments or ()
@@ -1310,6 +1313,7 @@
supports_stream_options=lambda: _model_deployments_support_stream_options(
model=self.data.get("model"),
llm_router=llm_router,
+ team_id=user_api_key_dict.team_id,
),
)
)
diff --git a/tests/test_litellm/proxy/test_common_request_processing.py b/tests/test_litellm/proxy/test_common_request_processing.py
--- a/tests/test_litellm/proxy/test_common_request_processing.py
+++ b/tests/test_litellm/proxy/test_common_request_processing.py
@@ -5265,12 +5265,14 @@
class TestModelDeploymentsSupportStreamOptions:
- def _support(self, model, llm_router=None) -> bool:
+ def _support(self, model, llm_router=None, team_id=None) -> bool:
from litellm.proxy.common_request_processing import (
_model_deployments_support_stream_options,
)
- return _model_deployments_support_stream_options(model=model, llm_router=llm_router)
+ return _model_deployments_support_stream_options(
+ model=model, llm_router=llm_router, team_id=team_id
+ )
def test_openai_compatible_deployment_supports_stream_options(self):
router = litellm.Router(
@@ -5337,3 +5339,23 @@
def test_non_string_model_is_not_injected(self):
assert self._support(None, None) is False
+
+ def test_team_public_model_name_resolves_via_team_id(self):
+ # Regression: team-scoped models keyed by (team_id, team_public_model_name)
+ # were previously unresolved because the support gate omitted team_id,
+ # so include_usage injection was skipped for common multi-tenant setups.
+ router = litellm.Router(
+ model_list=[
+ {
+ "model_name": "internal-team-model-uuid",
+ "litellm_params": {"model": "openai/gpt-4o", "api_key": "fake"},
+ "model_info": {
+ "team_id": "team-a",
+ "team_public_model_name": "team-a-gpt",
+ },
+ }
+ ]
+ )
+
+ assert self._support("team-a-gpt", router, team_id="team-a") is True
+ assert self._support("team-a-gpt", router, team_id=None) is FalseYou can send follow-ups to the cloud agent here.
The single-element tuple loop that bound the extracted deployment model inside the comprehension read poorly; an assignment expression in the filter clause does the same call-once-and-filter in one line.
Contributor
Author
ryan-crabbe-berri
approved these changes
Jul 31, 2026
Team-scoped models store an internal model_name_{team_id}_{uuid} name
with the public alias only in team_public_model_name, so resolving them
through get_model_list without team_id returned no deployments and the
gate skipped injection, leaving those streams on tiktoken estimates.
Thread user_api_key_dict.team_id through the gate.
Contributor
Author
Contributor
Author
|
bugbot run |
Contributor
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit c0de87d. Configure here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
Problem this solves:
stream_options.include_usageget tiktoken-estimated usageHow it solves it:
include_usageupstream on/v1/chat/completionsstreamsstream_optionsin its supported OpenAI params, so providers that reject the param (Bytez, OCI) or that stream usage natively (Anthropic, Gemini, Bedrock) are left untouchedalways_include_stream_usage: falseremains a kill switch,truekeeps forwarding usageRelevant issues
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
All runs hit a live proxy backed by real Azure OpenAI and Anthropic APIs, with SpendLogs in Postgres. Before runs were captured at
abd239f903(current litellm_internal_staging), after runs at this PR's headc0de87d08d. Config mirrors an affected deployment, with noalways_include_stream_usageset:The client request never sends
stream_options:SpendLogs rows across the QA matrix (
psql -d llnl_fix_qa -c 'SELECT model, prompt_tokens, completion_tokens, spend FROM "LiteLLM_SpendLogs" ORDER BY "startTime"'):The before rows are tiktoken estimates: the reasoning request produced a 2-token visible answer ("2472") while Azure billed 40 output tokens, so 38 reasoning tokens and 87% of the spend were silently dropped. The after rows match provider-billed usage exactly. Anthropic streams usage natively and its route is untouched by this PR (the support gate skips it entirely), so its numbers were correct before and stay identical
Client-visible SSE frames for the same azure request, before vs after, both with zero usage frames and identical shapes (verified with
diffover the captured streams):Clients that do opt in still get the usage chunk, at both commits:
always_include_stream_usage: truestill forwards the usage chunk to clients that never asked (existing behavior, verified live), and an explicitfalsedisables the upstream injection entirely, reproducing the old estimated row (23 pt, 2 ct) for deployments whose OpenAI-compatible backend rejectsstream_optionsProviders whose param map rejects
stream_optionsare skipped by the support gate instead of erroring. With a Bytez model in the same config (fake key, so the request dies at the provider's door either way), streaming at43efd02d35failed before ever reaching Bytez:while at
c0de87d08dthe identical request passes param mapping and reaches the real Bytez API:Team-scoped models resolve through the gate too. With a team model created via
/team/new+/model/new(internal namemodel_name_{team_id}_{uuid}, public aliasteam-gpt) and streamed through a team key, the reasoning request at0732122536logged the tiktoken estimate (23 pt, 2 ct, $0.00000710) because the gate resolved the alias withoutteam_idand skipped injection; atc0de87d08dthe identical request logs provider-billed usage (22 pt, 38 ct, $0.00005190) with client-visible frames unchangedType
🐛 Bug Fix
Changes
_stream_usage_tracking_updatesincommon_request_processing.pyreplaces the old flag-gated injection block. Withalways_include_stream_usageunset it injectsstream_options.include_usage: trueinto/v1/chat/completionsstreams (merging with any client-sentstream_options) and marks the request with an internal_litellm_strip_stream_usagekey;truekeeps the exact previous inject-and-forward semantics, andfalsenow opts out of injection entirely. The route scoping matters because other streaming surfaces (Anthropic/v1/messages, GooglegenerateContent) already receive provider usage without asking and their generators do not stripDefault injection is additionally gated on provider support:
_model_deployments_support_stream_optionsresolves the requested model through the router (deployments, aliases, and wildcards viaget_model_list) and only allows injection when every backing deployment listsstream_optionsinget_supported_openai_params.stream_optionsis exempt from the genericUnsupportedParamsErrorcheck, so the only providers that can break are ones whosemap_openai_paramsexplicitly raises (Bytez, OCI); those, and providers that stream usage natively and never needed the flag (Anthropic, Gemini, Bedrock), now keep their exact pre-PR behavior. The gate resolves with the request'steam_id, since team-scoped models store an internalmodel_name_{team_id}_{uuid}name and are only reachable through theirteam_public_model_namewhen the team is known. Unresolvable models conservatively skip injection, falling back to today's estimates rather than risking a provider error_litellm_strip_stream_usageis internal, so a value arriving in the client request body is neutralized:_stream_usage_tracking_updatesoverwrites any inbound value (only the proxy's own injection path can set it to true), and the experimental/queue/chat/completionsendpoint, which bypasses that pre-call logic, pops it from the parsed body. A client can no longer plant the marker to make the proxy swallow a usage chunk it explicitly requestedasync_data_generatorinproxy_server.pydrops injection artifacts when the marker is set: chunks whose choices are all empty (no delta content, finish_reason, or logprobs) and that carry noprovider_specific_fields. That covers both the provider's final usage chunk and the empty prompt-filter chunk Azure only emits wheninclude_usageis on, which live testing showed would otherwise leak. Chunks with real payload are never touched, even when a provider attaches usage to them_litellm_strip_stream_usageis registered inall_litellm_paramssoget_non_default_completion_paramstreats it as LiteLLM-internal instead of sweeping it into the provider request bodyTests pin the tri-state config semantics, the artifact predicate against real chunk shapes (synthetic usage chunk, empty-choices usage chunk, prompt-filter chunk, content and finish chunks with usage attached), and the generator end to end: with the marker the usage chunk disappears from client SSE, without it the chunk is forwarded. The support gate is covered against real Router instances (Azure deployment, Bytez deployment, mixed-provider model group, wildcard route, unmapped alias, and a team-alias model that only resolves when
team_idis passed), and the marker neutralization against client-planted values on the default, flag-true, and non-streaming pathsFinal Attestation
Note
Medium Risk
Changes the default proxy streaming request/response path and spend-logging inputs for chat completions, but provider support gating and conservative skip behavior limit breakage; extensive tests cover injection, stripping, and router/team resolution.
Overview
Default
/v1/chat/completionsstreaming now asks the provider for real usage by injectingstream_options.include_usagewhenalways_include_stream_usageis unset, while keeping the client stream unchanged by marking the request with internal_litellm_strip_stream_usageand dropping empty usage-only SSE chunks inasync_data_generator.Injection is scoped to
acompletionand gated on router-resolved deployments (including team aliases): every backing model must advertisestream_optionsinget_supported_openai_params, so providers that reject the param or already stream usage natively are skipped.always_include_stream_usage: truestill injects and forwards usage;falsedisables injection entirely.Client-supplied
_litellm_strip_stream_usageis neutralized on the normal pre-call path and stripped on the queued chat-completions endpoint; the flag is registered inall_litellm_paramsso it is not forwarded upstream.Reviewed by Cursor Bugbot for commit c0de87d. Bugbot is set up for automated code reviews on this repo. Configure here.