fix(passthrough,streaming): close Anthropic pass-through + streaming cost-capture gaps

[billyang-scale]

Summary

Streaming and pass-through requests could be logged with $0 cost, model="unknown", or dropped from SpendLogs entirely — while the upstream provider still billed every token. This consolidates fixes for the distinct leak paths into one change, plus a key-identification improvement for budget errors.

Scenarios fixed (and why each is needed)

Each item is a concrete path where spend was silently lost, which breaks cost attribution, budget enforcement, and chargeback.

1. Pass-through request logged with model="unknown" → priced at $0.
   /anthropic/v1/messages, /openai/... etc. reach logging before the model is resolved, so completion_cost() cannot price the request.
   → Recover <provider>/<model> from the inferred model_group in the central cost calculator and the per-provider handlers, and record the resolved model on the logging object up-front so a later failure can't leave model="unknown".

2. stream_chunk_builder raises on large / agentic streams → the whole request is dropped.
   Tool-use / thinking / web-search streams can make assembly re-raise (as APIError) from inside the success handler / except StopIteration, so it escapes __next__ / __anext__ and the Anthropic pass-through handler. The request never reaches SpendLogs even though every token was billed.
   → Catch the raise in the core CustomStreamWrapper (sync and async) and in the Anthropic handler, and treat it the same as a None result.

3. Usage-only fallback undercount.
   When full assembly fails, rebuild usage from the message_start / message_delta SSE events via AnthropicConfig.calculate_usage — the same path the success case uses — so prompt tokens are cache-inclusive and cache_read / cache_write / web_search / geo tokens are priced correctly. The previous fallback (Usage(...) + setattr) left those fields unset, so they were priced at $0.

4. Stream dies mid-flight → zero usage logged.
   Client disconnect / request timeout / mid-stream provider error: success-only logging records nothing, though the provider billed the tokens generated before the failure.
   → _log_partial_usage_on_stream_error() assembles partial usage from the chunks already received and logs it via the normal success path, called from the failure/timeout handlers. Best-effort: it never raises and never masks the original streaming error.

5. A stream cut mid-multibyte-sequence dropped the whole request.
   b"".join(raw_bytes).decode("utf-8") raised, discarding all usage events already received.
   → decode with errors="replace".

6. Streaming pass-through spend not persisted.
   The pass-through success path reads spend from model_call_details["response_cost"], not from kwargs, so streaming pass-through logged $0 even when the cost was computed.
   → record response_cost (and the resolved model) into model_call_details.

7. Opaque virtual-key budget errors.
   BudgetExceededError for a virtual key didn't name the key, forcing operators to reverse-map a spend figure back to a key during an incident.
   → include the key alias + masked key (which carries the last 4 chars) in the error message.

Repro (headline case — items 1–3)

1. Configure an Anthropic pass-through route on the proxy.
2. Send a streaming POST /anthropic/v1/messages request that exercises tool-use or a long agentic turn (enough that stream_chunk_builder can't reassemble it) — or disconnect the client mid-stream.
3. Inspect SpendLogs for the request:
   • Before: recorded with $0 cost / model="unknown", or missing entirely.
   • After: token usage and cost are recorded from the SSE usage events.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 79.06977% with 36 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...vider_handlers/base_passthrough_logging_handler.py	31.57%	13 Missing ⚠️
..._handlers/anthropic_passthrough_logging_handler.py	86.90%	11 Missing ⚠️
litellm/litellm_core_utils/litellm_logging.py	66.66%	7 Missing ⚠️
litellm/litellm_core_utils/streaming_handler.py	88.88%	5 Missing ⚠️

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR closes several cost-attribution gaps in Anthropic pass-through and streaming paths: model=\"unknown\" caused $0 spend records, stream_chunk_builder raising on agentic streams silently dropped requests from SpendLogs, mid-stream disconnects logged zero usage, a truncated multibyte decode discarded received events, and pass-through response_cost wasn't written to model_call_details where the spend path reads it. BudgetExceededError for virtual keys is also enriched with the key alias/name.

• Core streaming handler (streaming_handler.py): wraps stream_chunk_builder in try/except for both sync and async paths, falls back to calculate_total_usage, and adds _log_partial_usage_on_stream_error called from timeout/exception handlers to capture tokens billed before a failure.
• Pass-through logging (anthropic_passthrough_logging_handler.py, base_passthrough_logging_handler.py): recovers model via get_model_from_passthrough_model_group, writes it and response_cost into model_call_details up-front, and adds _build_usage_only_response_from_chunks as a fallback when full stream assembly fails.
• UTF-8 safety (pass_through_endpoints/streaming_handler.py): adds errors=\"replace\" to the raw-bytes decode so a mid-multibyte-sequence cut no longer drops the log entry.

Confidence Score: 4/5

The change is generally safe — all new error handling is defensive (try/except with fallbacks, never masking the original error), and the core streaming and pass-through logging paths are not restructured, only augmented with recovery logic.

The fixes are well-scoped and the error-handling additions are best-effort (they never raise or disrupt the original flow). The main open question is the absence of any test coverage or before/after evidence for the seven described fix paths; without that it's hard to be certain the recovery logic fires correctly and doesn't introduce subtle double-logging for partial-usage + failure events on the same request.

anthropic_passthrough_logging_handler.py and streaming_handler.py carry the most new logic (usage-only fallback path, _log_partial_usage_on_stream_error) and have no accompanying tests in this PR.

Important Files Changed

Filename	Overview
litellm/litellm_core_utils/litellm_logging.py	Adds get_model_from_passthrough_model_group to recover provider/model from the model_group metadata for pass-through requests priced at $0 due to model=unknown; integrates it into the central cost calculator.
litellm/litellm_core_utils/streaming_handler.py	Adds _log_partial_usage_on_stream_error for mid-flight stream failures and wraps stream_chunk_builder in try/except (sync and async) to fall back to calculate_total_usage when assembly raises; calls partial-usage logging from timeout and exception handlers.
litellm/proxy/auth/auth_checks.py	Enriches BudgetExceededError with key_alias and masked key_name in the error message; BudgetExceededError.init already accepts a message kwarg, so the change is well-formed.
litellm/proxy/pass_through_endpoints/llm_provider_handlers/anthropic_passthrough_logging_handler.py	Wraps _build_complete_streaming_response in try/except and adds _build_usage_only_response_from_chunks to recover cost from raw SSE events when full assembly fails; records resolved model and response_cost into model_call_details up-front.
litellm/proxy/pass_through_endpoints/llm_provider_handlers/base_passthrough_logging_handler.py	Adds model recovery and up-front model_call_details recording to the base handler's cost-calculation and streaming-assembly paths; mirrors the Anthropic-specific changes for all pass-through providers.
litellm/proxy/pass_through_endpoints/streaming_handler.py	Single-line fix: adds errors=replace to the UTF-8 decode of raw SSE bytes so a stream cut mid-multibyte sequence doesn't discard all usage events already received.

_{Reviews (2): Last reviewed commit: "fix(passthrough,streaming): close Anthro..." | Re-trigger Greptile}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Sameerlite]

Thanks for the contribution! A couple of things to help get this ready:

• The CI checks are currently failing (e.g. lint). Could you take a look and let us know if the failures are related to your change, or resolve them if so?
• There appear to be merge conflicts with the base branch — could you rebase/merge to resolve them?
• Could you add proof that the change works as expected? Screenshots, a sample request/response, test output, or a before/after comparison really help speed up review.

Also kicking off a fresh Greptile review on this.

@greptileai