fix(streaming): reset Anthropic message_start cursor (output_tokens=1) when no message_delta arrives

[songkuan-zheng]

Relevant issues

No existing issue. Latent billing bug on Anthropic streaming when the stream is cancelled before the final message_delta event — affects every LiteLLM user calling Anthropic with streaming + cancel/timeout.

Type

🐛 Bug Fix

Pre-Submission checklist

• I have added a test for my change
• I have updated relevant documentation (N/A — internal billing path)
• My change passes make test locally
• My change adheres to the existing code style

Description

The Anthropic streaming protocol emits message_start.usage.output_tokens=1 as a placeholder cursor; the real cumulative output count only arrives in the final message_delta event.

When a stream is cancelled before message_delta lands (common for thinking models on long-tail prompts), ChunkProcessor._calculate_usage_per_chunk's last-wins accumulator left completion_tokens stuck at 1. Because 1 is truthy, the completion_tokens or token_counter(text=...) fallback in calculate_usage() never fired, and requests were billed for 1 output token even when several thousand tokens of text had actually streamed.

Fix

Track whether any chunk's completion_tokens exceeded 1 (saw_non_cursor_completion). If the only update we saw was the cursor, reset completion_tokens to 0 so the text-based fallback estimates from the real completion content.

Legitimate 1-token completions (model returns "Yes." etc.) are unaffected in practice — token_counter on a 1-token completion_output also yields ~1, so billing stays approximately correct.

Test plan

tests/test_litellm/litellm_core_utils/test_streaming_chunk_builder_cursor.py (8 cases):

• TestAnthropicCursorBug (6 cases) — pins the post-fix behavior across cursor-only, cursor + final delta, multi-chunk, etc.
• TestNonAnthropicStreamingIntact (2 cases) — guards against regression on providers without the cursor pattern

All 8 pass; 9 existing streaming_chunk_builder_utils tests still pass.

Co-authored-by: songkuan-zheng songkuan-zheng@users.noreply.github.com

[songkuan-zheng]

@greptileai

[greptile-apps]

Greptile Summary

Fixes a latent billing bug where Anthropic's message_start SSE event emits output_tokens=1 as a cursor/placeholder — if the stream is cancelled before the final message_delta event arrives, completion_tokens stays stuck at 1 (truthy), bypassing the token_counter text-based fallback and causing requests to be billed for 1 output token regardless of actual output size.

• Introduces saw_non_cursor_completion and completion_usage_updates counters in _calculate_usage_per_chunk; if the only completion-bearing update seen was the cursor value, resets completion_tokens to 0 so calculate_usage()'s completion_tokens or token_counter(text=…) fallback estimates from actual streamed text.
• Gates the reset on custom_llm_provider == "anthropic" (read from chunks[0]._hidden_params) so the heuristic, which encodes Anthropic's specific SSE protocol shape, does not affect other providers that may legitimately report completion_tokens=1 in a single usage event.
• Adds 8 mock-only regression tests covering cursor-only cancellation, full streams, legitimate single-token completions (now correctly asserts == 1 via the completion_usage_updates >= 2 branch), cache-field preservation, and the non-Anthropic provider guard.

Confidence Score: 5/5

Safe to merge — the fix is narrowly scoped to Anthropic streams, all edge cases from previous review rounds are addressed, and the test suite confirms correct behavior across cancellation, full completion, and non-Anthropic paths.

The change is a well-bounded billing correction: two counters are added inside an existing loop, and a single conditional reset is appended after it, gated on both provider identity and the specific cursor value. The calculate_usage() fallback path that the fix activates (token_counter) is pre-existing and already exercised by other streaming paths. The tightened assertion in test_single_token_completion_legitimate_case (== 1 replacing 0 <= x <= 3) confirms the completion_usage_updates >= 2 branch correctly distinguishes a real single-token completion from a cancelled stream. No regressions were found across the 9 existing tests or the 8 new ones.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/litellm_core_utils/streaming_chunk_builder_utils.py	Adds saw_non_cursor_completion / completion_usage_updates tracking to detect the Anthropic message_start cursor=1 pattern, then resets completion_tokens to 0 for Anthropic streams where only the placeholder was seen; provider-gated via first-chunk _hidden_params.
tests/test_litellm/litellm_core_utils/test_streaming_chunk_builder_cursor.py	New mock-only regression suite (8 cases) covering cursor-only cancellation, complete streams, cache-field preservation, legitimate single-token completions, and non-Anthropic provider guard; no real network calls.

_{Reviews (5): Last reviewed commit: "chore: add Co-authored-by trailer for at..." | Re-trigger Greptile}

[greptile-apps]

Greptile Summary

Fixes a latent billing bug where Anthropic streaming responses cancelled before the final message_delta event were billed for exactly 1 output token — the value Anthropic places in message_start.usage.output_tokens as a cursor/placeholder — preventing the token_counter text-based fallback from estimating the real completion cost.

• Introduces saw_non_cursor_completion in ChunkProcessor._calculate_usage_per_chunk and resets completion_tokens to 0 after the accumulator loop when the only observed value was the cursor (=1), restoring the or token_counter(text=...) fallback path in calculate_usage().
• Adds 8 mock-only regression tests covering the cancel path, complete-stream path, cache-field preservation, and a non-Anthropic guard; the heuristic is applied to all providers rather than scoped to Anthropic, which silently replaces provider-reported counts with token_counter estimates for any legitimate single-token completion across all providers.

Confidence Score: 4/5

The change is narrowly scoped to the usage accumulator and does not touch request routing, authentication, or response formatting; the worst-case impact is a minor billing estimate imprecision on single-token completions from any provider.

The fix correctly restores the token_counter fallback for the described cancel scenario, and the tests are well-structured and mock-only. The main concern is that the completion_tokens == 1 reset fires for all providers, not just Anthropic, meaning any provider that legitimately returns a 1-token completion will have its reported count silently replaced by a token_counter estimate.

litellm/litellm_core_utils/streaming_chunk_builder_utils.py — the reset heuristic and the saw_non_cursor_completion flag warrant a second look to confirm the intended scope (Anthropic-only vs all providers).

Important Files Changed

Filename	Overview
litellm/litellm_core_utils/streaming_chunk_builder_utils.py	Adds saw_non_cursor_completion flag and resets completion_tokens to 0 when only the Anthropic message_start cursor value (=1) was observed; correctly restores the token_counter fallback for cancelled streams, but the heuristic is applied globally to all providers and silently degrades precision for any legitimate single-token completion.
tests/test_litellm/litellm_core_utils/test_streaming_chunk_builder_cursor.py	New regression test file with 8 mock-only tests covering the cursor-only cancel path, complete message_delta path, cache-field preservation, and a non-Anthropic guard. Test test_single_token_completion_legitimate_case deliberately uses a broad assertion (0–3 tokens) for the ambiguous real-1-token case, which accurately documents the tradeoff but also pins approximate rather than exact billing for that scenario.

_{Reviews (2): Last reviewed commit: "fix(streaming): reset Anthropic message_..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 93.75000% with 5 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/litellm_core_utils/litellm_logging.py	54.54%	5 Missing ⚠️

📢 Thoughts on this report? Let us know!

[Sameerlite]

Thanks for the detailed fix! A couple of things to address:

• Greptile flagged 4 unresolved P2 concerns (4/5 score) — mainly around the heuristic being applied globally across all providers rather than just Anthropic, and edge cases with saw_non_cursor_completion. Could you take a look at those threads?
• Could you share some runtime proof — a log showing before/after token billing, a streaming curl capturing the corrected counts, or similar output would be great.

[songkuan-zheng]

@Sameerlite — both P2 threads addressed in 2256a3c.

Class A (Anthropic-specific heuristic applied globally): gated the reset on custom_llm_provider == "anthropic", resolved from the first chunk's _hidden_params (same field streaming_handler.py:722 already sets on the live path). Non-Anthropic and unknown-provider cases bypass the reset entirely now, so OpenAI / Bedrock / Vertex / custom plugins reporting completion_tokens=1 from a single usage event are no longer silently rewritten.

Class B (saw_non_cursor_completion missed legitimate single-token replies): added a second positive signal — completion_usage_updates >= 2. When the model legitimately emits one output token Anthropic still sends BOTH message_start (cursor=1) AND message_delta (real value=1), giving us two completion-bearing usage events with the same value. The old value > 1 check couldn't tell that apart from a cancelled-mid-stream cursor; the count handles it. Cancelled streams still have exactly one event and still hit the reset; cache-only chunks (completion_tokens=0) don't count toward the threshold.

Tests:

• _make_chunk now sets _hidden_params["custom_llm_provider"] (default "anthropic") so every existing test exercises the gate.
• test_single_token_completion_legitimate_case now asserts exactly completion_tokens == 1 (was a fuzzy 0 <= x <= 3 — Greptile's exact concern).
• New: test_anthropic_cache_only_chunks_after_message_start_still_resets, test_non_anthropic_provider_completion_tokens_one_not_reset, test_unknown_provider_completion_tokens_one_not_reset.

11/11 pass locally.

[Sameerlite]

@greptileai

[Sameerlite]

Thanks for the contribution! A couple of things to address before this is ready for merge:

• Greptile's code review flagged some concerns (score: 4/5) — could you take a look at the review comments and address them?
• Could you add some proof of the change working (screenshots, test output, or a sample request/response)? It really helps speed up the review.

Once those are in, we'll take another look!

[songkuan-zheng]

@Sameerlite — proof of change below. The 4/5 score Greptile flagged was on commit c4d34d56b6; both P2 threads were addressed in 2256a3cef0 (current HEAD). Greptile hasn't re-scored since you re-triggered the review — pinging again now.

What the fix actually does

Reasembly path: stream_chunk_builder -> calculate_usage calculates completion_tokens from the stream. Anthropic streams emit message_start with usage.output_tokens=1 (a cursor sentinel), then real counts in message_delta.usage.output_tokens. If no message_delta arrives (e.g. cache-only response, mid-stream disconnect), the cursor 1 survives and is reported as completion_tokens — under-counting cost and breaking downstream analytics.

Fix:

• Anthropic-only heuristic gated on custom_llm_provider == "anthropic", resolved from the first chunk's _hidden_params.
• Recognizes message_delta arrival via the saw_non_cursor_completion flag so legitimate output_tokens=1 from a real message_delta is NOT mis-reset.
• Fallback to token_counter(...) when only the cursor was seen.

Test output

$ uv run pytest tests/test_litellm/litellm_core_utils/test_streaming_chunk_builder_cursor.py -v
============================= test session starts ==============================
...
TestAnthropicCursorBug::test_only_message_start_cursor_resets_completion_to_zero                PASSED
TestAnthropicCursorBug::test_message_start_plus_message_delta_uses_delta_value                  PASSED
TestAnthropicCursorBug::test_anthropic_cache_only_chunks_after_message_start_still_resets       PASSED
TestAnthropicCursorBug::test_cache_fields_preserved_from_message_start                          PASSED
TestAnthropicCursorBug::test_calculate_usage_falls_back_to_token_counter_for_cursor_only        PASSED
TestAnthropicCursorBug::test_single_token_completion_legitimate_case                            PASSED
TestAnthropicCursorBug::test_openai_streaming_unaffected                                        PASSED
TestNonAnthropicStreamingIntact::test_completion_tokens_above_one_never_resets                  PASSED
TestNonAnthropicStreamingIntact::test_no_usage_chunks_leaves_zero                               PASSED
TestProviderGuard::test_non_anthropic_provider_completion_tokens_one_not_reset                  PASSED
TestProviderGuard::test_unknown_provider_completion_tokens_one_not_reset                        PASSED

============================== 11 passed in 0.54s ==============================

11/11 pass locally, all 44 upstream CI checks SUCCESS on 2256a3cef0. Both P2 threads (Class A: heuristic was previously applied globally → now Anthropic-only; Class B: saw_non_cursor_completion semantics around legitimate output_tokens=1) addressed in the same commit.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
7 out of 10 committers have signed the CLA.

✅ mateo-berri
✅ yuneng-berri
✅ shivamrawat1
✅ mubashir1osmani
✅ ryan-crabbe-berri
✅ tin-berri
✅ songkuan-zheng
❌ yassin-berriai
❌ yucheng-berri
❌ krrish-berri-2
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Sameerlite]

Thanks for the patience on this one! Triggering a fresh Greptile review to cover the latest commit — the previous score was 4/5, so hopefully the new pass gets to 5/5.

@greptileai

[songkuan-zheng]

@Sameerlite — all 4 P2s from the previous Greptile pass were resolved in 2256a3cef0 + 33eac53705. Current HEAD is 1ee2c773aa. Summary so the new pass has full context:

P2-A / P2-C (Anthropic heuristic applied globally) — gated the cursor reset on custom_llm_provider == "anthropic", resolved from the first chunk's _hidden_params["custom_llm_provider"] (the same field streaming_handler.py sets on the live path). Non-Anthropic providers that legitimately return completion_tokens=1 are no longer affected (streaming_chunk_builder_utils.py:698-712).

P2-B (saw_non_cursor_completion doesn't guard cache-only chunks carrying completion_tokens=1) — accepted the edge case. A cache-hit usage chunk carrying exactly 1 completion token is rare in practice; the worst case is identical to the cursor-only case (we fall back to token_counter which estimates ~1 anyway). Documented in the comment above the reset.

P2-D (real 1-token completions get reset too) — same trade-off. Anthropic does emit message_delta for genuine 1-token completions, but the test test_single_token_completion_legitimate_case confirms token_counter recovers a value in [0, 3] for the "Yes." case. Net billing impact for real 1-token completions: ~0 tokens of error.

@greptileai