Repository navigation
fix(gateway): stream v1/messages usage tokens - #2380
Merged
smakosh merged 2 commits intoMay 23, 2026
Merged
Conversation
The upstream /v1/chat/completions endpoint emits finish_reason and usage in separate chunks. The Anthropic translator only read usage when it saw finish_reason, so the streamed message_delta carried zeros for input_tokens/output_tokens. Defer message_delta until [DONE] (or stream end) and extract usage from any chunk that carries it. Add e2e regression tests for anthropic + bedrock. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
WalkthroughRefactors Anthropic streaming SSE event flow to deduplicate and order terminal events ( ChangesAnthropic Streaming Event Centralization
Possibly related PRs
Suggested reviewers
🎯 3 (Moderate) | ⏱️ ~25 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/gateway/src/native-anthropic-cache.e2e.ts`:
- Around line 724-744: These two tests ("streaming /v1/messages surfaces
non-zero usage tokens for anthropic" and "streaming /v1/messages surfaces
non-zero usage tokens for bedrock") exercise external provider streams and
cache-sensitive behavior and must not run inside the concurrent suite using
getConcurrentTestOptions(); fix by running them serially — either move these two
test(...) calls out of the describe that uses getConcurrentTestOptions() into a
serial describe (using getTestOptions()) or mark the individual tests as serial
(e.g., replace test with test.serial when invoking the tests that call
assertStreamingUsage("anthropic/claude-haiku-4-5", ...) and
assertStreamingUsage("aws-bedrock/claude-haiku-4-5", ...)) so they never execute
in parallel with other concurrent tests.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: b3499a3b-326b-4aac-87b8-53e5cc454c1d
📒 Files selected for processing (2)
apps/gateway/src/anthropic/anthropic.tsapps/gateway/src/native-anthropic-cache.e2e.ts
Comment on lines
+724
to
+744
| (hasAnthropicKey ? test : test.skip)( | ||
| "streaming /v1/messages surfaces non-zero usage tokens for anthropic", | ||
| getTestOptions(), | ||
| async () => { | ||
| await assertStreamingUsage( | ||
| "anthropic/claude-haiku-4-5", | ||
| "streaming /v1/messages anthropic usage", | ||
| ); | ||
| }, | ||
| ); | ||
|
|
||
| (hasBedrockKey ? test : test.skip)( | ||
| "streaming /v1/messages surfaces non-zero usage tokens for bedrock", | ||
| getTestOptions(), | ||
| async () => { | ||
| await assertStreamingUsage( | ||
| "aws-bedrock/claude-haiku-4-5", | ||
| "streaming /v1/messages bedrock usage", | ||
| ); | ||
| }, | ||
| ); |
Contributor
There was a problem hiding this comment.
Run these new e2e cases serially.
These tests are being added under the concurrent describe(..., getConcurrentTestOptions(), ...) suite on Line 114, but they exercise external provider streams and cache-sensitive behavior. That is a flake-prone combination, and it conflicts with the repo rule for *.e2e.ts suites.
As per coding guidelines, "**/*.{spec.ts,e2e.ts}: Do not run test files or suites in parallel unless the repository instructions for that exact suite explicitly require it".
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/gateway/src/native-anthropic-cache.e2e.ts` around lines 724 - 744, These
two tests ("streaming /v1/messages surfaces non-zero usage tokens for anthropic"
and "streaming /v1/messages surfaces non-zero usage tokens for bedrock")
exercise external provider streams and cache-sensitive behavior and must not run
inside the concurrent suite using getConcurrentTestOptions(); fix by running
them serially — either move these two test(...) calls out of the describe that
uses getConcurrentTestOptions() into a serial describe (using getTestOptions())
or mark the individual tests as serial (e.g., replace test with test.serial when
invoking the tests that call assertStreamingUsage("anthropic/claude-haiku-4-5",
...) and assertStreamingUsage("aws-bedrock/claude-haiku-4-5", ...)) so they
never execute in parallel with other concurrent tests.
smakosh
approved these changes
May 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Anthropic Messages translator at
/v1/messageswas reportinginput_tokens: 0, output_tokens: 0in the streamedmessage_deltaevent. The upstream/v1/chat/completionsendpoint emitsfinish_reasonandusagein separate chunks (the usage chunk hasfinish_reason: null), so gating usage extraction onfinish_reasonalways missed it. The fix defersmessage_deltauntil[DONE](or stream end), extracts usage from any chunk that carries it, and makes the terminator events idempotent. Added paired Anthropic/Bedrock e2e tests innative-anthropic-cache.e2e.tsthat fail on the old code (zero usage on bedrock, duplicatemessage_deltaon anthropic) and pass with the fix.Test plan
api.llmgateway.io) after deploy🤖 Generated with Claude Code
Summary by CodeRabbit
Release Notes
Bug Fixes
Tests