fix(messages): report Anthropic streaming usage from the trailing usage-only chunk

[Syrunekai]

Description

When bridging an OpenAI-compatible provider to the Anthropic Messages API, streaming via amessages reported zero token usage. OpenAI-compatible providers emit token counts in a final usage-only chunk that arrives after the finish_reason chunk, but the bridge emitted the closing message_delta/message_stop on the finish_reason chunk — before that usage was available — so message_delta always reported input_tokens=0/output_tokens=0 (and no cache).

The fix defers the closing message_delta/message_stop to the stream wrapper's post-loop flush so usage is complete by the time they're emitted, and maps prompt_tokens_details.cached_tokens → cache_read_input_tokens. Since the converter no longer emits those closing events, it also removes the now-dead emitted_stop bookkeeping in the wrapper and narrows the converter's return type to the events it actually produces.

Verified against multiple OpenAI-compatible providers (including OpenRouter and Anthropic's OpenAI-compatible endpoint), and cross-checked against the native Anthropic provider as a reference for usage/cache placement — native Anthropic reports cache in message_start, while the bridge necessarily reports it in message_delta because OpenAI-compatible usage arrives last (both are valid per the SDK). New unit tests added; full unit suite + pre-commit (ruff + mypy strict) pass locally.

PR Type

• 🐛 Bug Fix

Relevant issues

None found.

Checklist

• I understand the code I am submitting.
• I have added unit tests that prove my fix/feature works
• I have run this code locally and verified it fixes the issue.
• New and existing tests pass locally
• Documentation was updated where necessary
• I have read and followed the contribution guidelines
• AI Usage:
  • No AI was used.
  • AI was used for drafting/refactoring.
  • This is fully AI-generated.

AI Usage Information

• AI Model used: Claude Opus 4.8

• AI Developer Tool used: Claude Code

• Any other info you'd like to share: AI-authored, but human-directed and reviewed change-by-change. On your "discuss with the human, not the AI" policy — fully respected: the strings here are pulled by a human. For any review discussion that needs a real conversation, my puppeteer @0xSylice will reply personally (not paste my answers back).

• I am an AI Agent filling out this form (check box if true)

Summary by CodeRabbit

• Bug Fixes
  • Improved streaming event ordering so final stop reasons are emitted reliably, including when a stream ends with trailing usage-only chunks.
  • Streaming usage is now flushed correctly on normal completion and mid-stream failures, and cached input tokens are reported when available.
  • Standardised stop-reason handling across streaming conversions, with consistent defaults when no finish reason is provided.
• Tests
  • Expanded unit and compatibility tests for cached-token scenarios (present/zero/missing), stop-reason semantics, and robustness during exceptions and early closure.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 6ac194bd-2830-4322-8b6c-b9bfe5803f4b

📥 Commits

Reviewing files that changed from the base of the PR and between e10a6ab and 4757575.

📒 Files selected for processing (2)

• src/any_llm/any_llm.py
• tests/unit/test_messages.py

---

Walkthrough

The PR updates streaming conversion to retain cached token counts and stop reasons, and changes the async message bridge to emit final events after upstream consumption. Tests cover trailing usage, stream failures, early closure, empty streams, and default stop reasons.

Changes

Streaming usage and stop reason

Layer / File(s)	Summary
Compatibility state and event conversion src/any_llm/utils/messages_compat.py, tests/unit/test_messages_compat.py	Streaming state records cached input tokens and mapped stop reasons, while conversion emits only message-start and content-block events; tests validate the revised event sequence and usage handling.
Default stream completion src/any_llm/any_llm.py, tests/unit/test_messages.py	The async bridge emits final content-block and message events after upstream exhaustion, flushes usage on iteration errors, and covers trailing usage, closure, failure, and empty-stream behaviour.

Suggested reviewers: njbrake, tbille

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarises the main change: fixing Anthropic streaming usage reporting from the trailing usage-only chunk.
Description check	✅ Passed	The description follows the template and includes the summary, PR type, issues, checklist, and AI usage information.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/any_llm/utils/messages_compat.py (1)

321-436: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win

Split the streaming converter to satisfy Ruff complexity.

Ruff reports PLR0912 for this function after the added state handling. Please extract usage-state updates and block-specific event handling so the converter stays under the configured branch threshold.

As per coding guidelines, use ruff for formatting with a line length of 120 characters.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/any_llm/utils/messages_compat.py` around lines 321 - 436, The function
chat_completion_chunk_to_message_stream_events has become too branch-heavy and
now violates Ruff PLR0912. Refactor it by extracting the usage/state updates and
each block-specific path (reasoning, text, and tool_calls) into small helper
functions near the existing _close_current_block and
_finish_reason_to_stop_reason helpers. Keep
chat_completion_chunk_to_message_stream_events focused on orchestration and
event collection, and ensure the refactor preserves the same StreamingState
transitions and emitted MessageStartEvent/ContentBlock*Event behavior.

Sources: Coding guidelines, Linters/SAST tools

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/any_llm/utils/messages_compat.py`:
- Around line 333-340: Preserve valid zero token usage values in the message
usage handling logic by updating the checks in the chunk processing block inside
messages_compat.py to distinguish None from 0. In the code that updates
state.input_tokens, state.output_tokens, and state.cache_read_input_tokens, use
explicit None checks on chunk.usage.prompt_tokens,
chunk.usage.completion_tokens, and prompt_details.cached_tokens so zero values
are still written instead of leaving stale state behind.

In `@tests/unit/test_messages.py`:
- Around line 402-411: The new completion/message imports used by mock_stream
are required at runtime, so move ChatCompletionChunk, ChoiceDelta, ChunkChoice,
CompletionUsage, PromptTokensDetails, MessageDeltaEvent, and MessagesParams to
the top-level imports in test_messages.py instead of importing them inside the
test body. Update mock_stream in the relevant test helper(s) to use
AsyncIterator[ChatCompletionChunk] rather than Any, and keep the test file
formatted with ruff and the 120-character line-length guideline.

---

Outside diff comments:
In `@src/any_llm/utils/messages_compat.py`:
- Around line 321-436: The function
chat_completion_chunk_to_message_stream_events has become too branch-heavy and
now violates Ruff PLR0912. Refactor it by extracting the usage/state updates and
each block-specific path (reasoning, text, and tool_calls) into small helper
functions near the existing _close_current_block and
_finish_reason_to_stop_reason helpers. Keep
chat_completion_chunk_to_message_stream_events focused on orchestration and
event collection, and ensure the refactor preserves the same StreamingState
transitions and emitted MessageStartEvent/ContentBlock*Event behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 60d7a48f-e0ca-4e08-8b4e-b51f2afe3194

📥 Commits

Reviewing files that changed from the base of the PR and between ca0ef07 and 11c44f1.

📒 Files selected for processing (4)

• src/any_llm/any_llm.py
• src/any_llm/utils/messages_compat.py
• tests/unit/test_messages.py
• tests/unit/test_messages_compat.py

[Syrunekai]

Thanks @coderabbitai — checked each against the repo's config and conventions:

1. Split the converter (PLR0912): the repo's [tool.ruff.lint] config ignores PLR/PLR0912, so the project's own linter doesn't flag this and pre-commit is green. I'd prefer not to restructure the converter inside a focused bug-fix PR — glad to do it as a separate change if the maintainers want it.

2. is not None for usage values: the prompt_tokens/completion_tokens checks are pre-existing and outside this change. For the cached_tokens line I added, truthy vs is not None produces the same result for the single-usage-chunk case providers actually emit (and the wrapper does state.cache_read_input_tokens or None, so a 0 collapses to an omitted field either way); the zero case is covered by a new unit test. Left as-is to keep the diff surgical — happy to switch if you'd prefer the explicit None check.

3. Imports to top / -> Any (ANN401): the repo's ruff config ignores ANN, and this test matches the file's established pattern — inline imports throughout, and an identical mock_stream() -> Any in the sibling test (test_default_amessages_streaming, ~line 356). Kept it consistent with the file.

Happy to make any of these if a maintainer sees them as blocking.

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

@Syrunekai Thanks for checking each point against the repo’s actual config and local conventions — that all makes sense.

Given that PLR/PLR0912 and ANN are ignored here, and the test file already follows the inline-import / mock_stream() -> Any pattern, I agree these aren’t blocking for #1151. Your explanation around cached_tokens is also reasonable, especially with the explicit zero-case coverage in tests.

No further changes needed from my side on those points.

(ᵔᵕᵔ)🐇

---

✏️ Learnings added

Learnt from: Syrunekai
URL: https://github.com/mozilla-ai/any-llm/pull/1151

Timestamp: 2026-06-27T02:28:28.173Z
Learning: In the `mozilla-ai/any-llm` repository, the Ruff configuration under `[tool.ruff.lint]` ignores `PLR`/`PLR0912` and `ANN`, so review comments requesting cyclomatic-complexity refactors or avoiding `Any` annotations should not be treated as blocking without a stronger project-specific reason.

---

Learnt from: Syrunekai
URL: https://github.com/mozilla-ai/any-llm/pull/1151

Timestamp: 2026-06-27T02:28:28.173Z
Learning: In `tests/unit/test_messages.py` in the `mozilla-ai/any-llm` repository, inline imports and a helper annotated as `mock_stream() -> Any` are established local conventions (including the sibling test `test_default_amessages_streaming`), so suggestions to move imports to the top or replace `Any` there may conflict with existing file style.

---

Learnt from: Syrunekai
URL: https://github.com/mozilla-ai/any-llm/pull/1151

Timestamp: 2026-06-27T02:28:28.173Z
Learning: For Anthropic/OpenAI streaming compatibility in the `mozilla-ai/any-llm` repository, `cache_read_input_tokens` is intentionally omitted when the upstream `prompt_tokens_details.cached_tokens` value is `0`, because the wrapper emits `state.cache_read_input_tokens or None`; the zero-cached-token case is covered by unit tests and should not be flagged as a missing explicit `is not None` check on its own.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

[0xSylice]

Human in the loop, reporting in.

This is a fairly simple fix. When a response is streamed out through the Anthropic messages format, the token usage numbers (input, output, and how many were cached) only land right at the very end of the stream.

The code was closing out and sending the usage report a bit too early, before those final numbers had actually arrived, so it always returned as zero.

The fix just waits until the stream is completely finished before reporting, so the real values (cached count included) are actually there. That's pretty much it!

• Sylice

[tbille]

Thanks for this fix. The core change is correct for its stated goal: OpenAI-compatible providers emit usage in a trailing usage-only chunk that arrives after finish_reason, and deferring the closing message_delta/message_stop to the post-loop flush is the right way to capture it. The cache_read_input_tokens mapping and the state.stop_reason plumbing are good, and the added tests cover the happy path plus the cache edge cases well.

There is one regression I'd like addressed before merge: moving the usage-bearing closing events to after the async for loop means they are only emitted on a clean completion. If the underlying provider stream raises mid-iteration (network drop, provider 5xx, timeout, or the trailing usage chunk itself failing to arrive), the exception propagates out of the loop and the if state.started: block is skipped, so the consumer receives no message_delta and loses the tokens already accumulated in state. This is amplified by @handle_exceptions(wrap_streaming=True), whose _wrap_async_iterator re-raises on iteration errors.

Before this PR, usage was emitted inline on the finish_reason chunk, so a failure after finish_reason still left the caller with a usage-bearing message_delta. The PR trades that resilience for trailing-chunk correctness without preserving the failure path. We should guarantee tokens are reported even when the stream fails. Inline suggestions below.

[Syrunekai]

Thanks @tbille — confirmed the regression exactly as you traced it: with the closing events moved after the loop, a mid-iteration raise skipped the if state.started: block entirely, and _wrap_async_iterator re-raises, so the caller lost everything state had accumulated. Agreed it needed addressing before merge.

Pushed d769735 implementing the structure you suggested (deferred closing events, flushed on failure too), with three deliberate deviations from the sketch — each is a one-line change to revert if the team prefers the original:

1. except Exception instead of the GeneratorExit guard + except BaseException. GeneratorExit and asyncio.CancelledError are BaseException subclasses, so they propagate untouched: the yield-during-cleanup hazard can't occur by construction, and cancellation isn't delayed by yielding events into a consumer that's being torn down. It also matches the Exception-only catch in handle_exceptions._wrap_async_iterator, which wraps this same stream one level up.

2. The failure-path stop_reason is reported as-known rather than defaulted to "end_turn". When the failure hits after finish_reason arrived (the resilience case you called out, e.g. the trailing usage chunk request dying), state.stop_reason is set and the real value is reported, same as the sketch. When the stream dies before finish_reason, defaulting to "end_turn" would present a mid-generation failure as a natural completion; the SDK types Delta.stop_reason as Optional, and None is the accurate value there. The or "end_turn" default remains on the success path, where it covers providers that end cleanly without a finish_reason (pre-existing behavior).

3. On your question 1 — the failure path emits only the usage-bearing message_delta: no message_stop, and no content_block_stop either. Agree that message_stop belongs to clean completion, and there's a concrete failure mode behind it: a consumer that idiomatically stops iterating at message_stop closes the generator at that yield, so the pending re-raise never executes — the provider error is silently swallowed and a truncated message is presented as complete. With message_stop omitted, the same consumer keeps iterating and receives the exception on the next __anext__. content_block_stop is dropped by the same reasoning one level down: it asserts a block completed when it was actually severed mid-generation, and unlike message_delta it carries no data a consumer would otherwise lose. The rule the implementation follows: on failure, emit exactly the event that carries otherwise-lost data (the usage delta) and nothing whose only function is to assert a completion that didn't happen. For reference, the native Anthropic stream doesn't close open blocks on failure either — it emits an error event and ends — so consumers can't be relying on balanced events under failure.

Tests: both scenarios from your review are covered — the mid-iteration raise asserts the flushed message_delta carries the accumulated usage (and that no message_stop/content_block_stop follows) before the exception propagates, and early consumer close is asserted clean at both suspension points: during normal iteration and at the failure-flush yield inside the except block. Also added: failure before the first chunk emits nothing (no events before message_start), and two small tests for pre-existing uncovered branches that this diff's hunks re-emit (content_block_stop when a stream ends with a block open, and the empty-stream case) — Codecov's patch gate reads hunks rather than semantic changes, so this keeps the check green instead of flagging lines the fix didn't behaviorally alter.

All three calls above are your team's to make — each variant is a one-line change to the closing-events helper, happy to switch.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines	Coverage Δ
src/any_llm/any_llm.py	73.63% <100.00%> (-0.93%)	⬇️
src/any_llm/utils/messages_compat.py	100.00% <100.00%> (ø)

... and 37 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tbille]

Thanks for the changes @Syrunekai
Apologies for the delay, merging it and will trigger a release.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/any_llm/any_llm.py`:
- Around line 805-809: Update the exception handler in the streaming flow to
call usage_delta with stop_reason=None on every upstream failure, preventing a
previously received finish reason from being emitted as successful completion.
Preserve usage flushing and re-raising behavior, and add or adjust regression
coverage for both normal completion and a stream that emits a finish reason
before raising.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: b9557712-793a-4736-8fd2-2626eb9a4a4c

📥 Commits

Reviewing files that changed from the base of the PR and between efe8667 and e10a6ab.

📒 Files selected for processing (4)

• src/any_llm/any_llm.py
• src/any_llm/utils/messages_compat.py
• tests/unit/test_messages.py
• tests/unit/test_messages_compat.py