fix: openrouter claude cache usage - #2811
Conversation
WalkthroughOne file modified in the Claude relay handler to enhance token usage reporting. The FormatClaudeResponseInfo function now conditionally updates cache-related token fields (CachedTokens, CachedCreationTokens, cache creation time windows) and validates OutputTokens before setting CompletionTokens, with TotalTokens recalculated accordingly. Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@relay/channel/claude/relay-claude.go`:
- Around line 633-648: The code currently uses >0 checks on claudeResponse.Usage
fields (e.g., CacheReadInputTokens, CacheCreationInputTokens,
GetCacheCreation5mTokens/GetCacheCreation1hTokens, OutputTokens) which treats
omitted JSON fields and explicit zero values the same and can leave stale values
in claudeInfo.Usage; change the usage struct to use pointer types (e.g., *int)
or implement custom JSON unmarshaling for the Usage type so you can detect nil
(omitted) vs. explicitly zero, then update the assignment logic in the block
referencing claudeResponse.Usage and claudeInfo.Usage to check for nil (or
presence from Unmarshal) and assign the value (including zero) to
claudeInfo.Usage fields so explicit zeros overwrite prior values.
| } | ||
| claudeInfo.Usage.CompletionTokens = claudeResponse.Usage.OutputTokens | ||
| if claudeResponse.Usage.CacheReadInputTokens > 0 { | ||
| claudeInfo.Usage.PromptTokensDetails.CachedTokens = claudeResponse.Usage.CacheReadInputTokens | ||
| } | ||
| if claudeResponse.Usage.CacheCreationInputTokens > 0 { | ||
| claudeInfo.Usage.PromptTokensDetails.CachedCreationTokens = claudeResponse.Usage.CacheCreationInputTokens | ||
| } | ||
| if cacheCreation5m := claudeResponse.Usage.GetCacheCreation5mTokens(); cacheCreation5m > 0 { | ||
| claudeInfo.Usage.ClaudeCacheCreation5mTokens = cacheCreation5m | ||
| } | ||
| if cacheCreation1h := claudeResponse.Usage.GetCacheCreation1hTokens(); cacheCreation1h > 0 { | ||
| claudeInfo.Usage.ClaudeCacheCreation1hTokens = cacheCreation1h | ||
| } | ||
| if claudeResponse.Usage.OutputTokens > 0 { | ||
| claudeInfo.Usage.CompletionTokens = claudeResponse.Usage.OutputTokens | ||
| } |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# Search for Anthropic/Claude API documentation or comments about streaming behavior
rg -i "message_delta|cache.*token|streaming" relay/channel/claude/relay-claude.go -B 3 -A 3 | head -100Repository: QuantumNous/new-api
Length of output: 3206
🏁 Script executed:
# Check if there's any documentation or constants related to how tokens are handled
rg -i "usage|cache.*token" relay/channel/claude/ --type go | head -50Repository: QuantumNous/new-api
Length of output: 5591
🏁 Script executed:
# Look at the function context to understand how values are initialized and accumulated
sed -n '600,700p' relay/channel/claude/relay-claude.goRepository: QuantumNous/new-api
Length of output: 4368
🌐 Web query:
Anthropic Claude API message_delta usage fields streaming: are zero values omitted or explicitly sent?
💡 Result:
In Anthropic streaming, the message_delta.usage object is sparse: it may include only the usage fields that were added/updated in that delta (often just output_tokens), so you cannot rely on “zero” fields being explicitly sent—they may be omitted entirely. This is reflected in Anthropic’s own streaming examples where message_delta.usage contains only {"output_tokens": ...}. [1]
When a field is included, it can legitimately be 0 (e.g., server responses have been observed sending cache_creation_input_tokens: 0). [2]
Also, depending on the client/SDK/proxy, you may see fields present but set to null (an SDK serialization behavior), so robust code should treat missing or null as “not provided” rather than assuming explicit zeros. [3]
Practical takeaway: treat message_delta.usage as a partial patch; maintain your own cumulative usage state and update only the fields that appear. [1]
Sources: [1] Anthropic streaming docs example shows message_delta with usage: {"output_tokens": ...} only. [2] Captured server streaming message_delta shows explicit cache_creation_input_tokens: 0. [3] SDK/proxy issue report shows message_delta.usage fields sometimes emitted as null.
🌐 Web query:
Anthropic Claude streaming API documentation message_delta cache_read_input_tokens cache_creation_input_tokens behavior
💡 Result:
In Claude’s streaming Messages API, you’ll see one or more message_delta SSE events near the end of the stream. These events apply top-level updates to the in-progress Message object (not per-text chunks—that’s content_block_delta). The usage object on message_delta is cumulative across the whole request/stream, so you should treat the latest message_delta.usage you receive as the final totals. [2]
Within that cumulative usage:
cache_creation_input_tokens: how many input tokens were written into the prompt cache (i.e., used to create/extend a cache entry) for this request. These are counted as part of total input usage. [1]cache_read_input_tokens: how many input tokens were served from an existing prompt cache hit (i.e., read from cache) for this request. Also counted as part of total input usage. [1]- Total input tokens for billing/rate limits are:
input_tokens + cache_creation_input_tokens + cache_read_input_tokens. [1]
So, behaviorally: if caching is enabled and you get a cache hit, you’ll see non-zero cache_read_input_tokens; if your request creates a new cache entry, you’ll see non-zero cache_creation_input_tokens—and both show up in the cumulative message_delta.usage during streaming. [1][2]
Consider using pointer fields or custom JSON unmarshaling to distinguish omitted from explicit zero values in message_delta.usage.
The > 0 checks assume omitted fields are unmarshaled as zero, which aligns with Anthropic's sparse message_delta.usage API (fields not sent are treated as "not provided"). However, Anthropic can legitimately send fields with value 0 (e.g., cache_read_input_tokens: 0), and the current code will skip those updates, preserving stale values from message_start instead of accepting the authoritative cumulative values. While unlikely to manifest in practice for cache fields (rarely present in message_delta), this is a subtle defect. To be fully correct, consider using *int pointers or custom JSON unmarshaling so explicit zeros overwrite prior values while truly omitted fields don't.
🤖 Prompt for AI Agents
In `@relay/channel/claude/relay-claude.go` around lines 633 - 648, The code
currently uses >0 checks on claudeResponse.Usage fields (e.g.,
CacheReadInputTokens, CacheCreationInputTokens,
GetCacheCreation5mTokens/GetCacheCreation1hTokens, OutputTokens) which treats
omitted JSON fields and explicit zero values the same and can leave stale values
in claudeInfo.Usage; change the usage struct to use pointer types (e.g., *int)
or implement custom JSON unmarshaling for the Usage type so you can detect nil
(omitted) vs. explicitly zero, then update the assignment logic in the block
referencing claudeResponse.Usage and claudeInfo.Usage to check for nil (or
presence from Unmarshal) and assign the value (including zero) to
claudeInfo.Usage fields so explicit zeros overwrite prior values.
…de-cache-usage fix: openrouter claude cache usage
2 participants
fix #2791
Summary by CodeRabbit
Release Notes