fix: recognize non-standard cache tokens from OpenAI-compatible channels - #3371
Closed
majiayu000 wants to merge 2 commits into
Closed
fix: recognize non-standard cache tokens from OpenAI-compatible channels#3371majiayu000 wants to merge 2 commits into
majiayu000 wants to merge 2 commits into
Conversation
- Change CachedCreationTokens json tag from "-" to "cache_creation_input_tokens" so providers like Qwen can deserialize this field naturally from prompt_tokens_details. - Add default case in applyUsagePostProcessing to handle generic OpenAI-compatible channels: extract cached_tokens (StepFun) and cache_creation_input_tokens from response body when not already populated via standard fields. - Add extractCacheCreationTokensFromBody helper following the existing extractCachedTokensFromBody pattern. Closes QuantumNous#3309 Signed-off-by: majiayu000 <1835304752@qq.com>
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
✅ Files skipped from review due to trivial changes (1)
WalkthroughAdds JSON deserialization for cache-creation tokens and a default fallback in OpenAI relay usage post-processing that extracts cached read and cached-creation token values from upstream response bodies; includes unit tests for the new extraction behavior. Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant Relay as Relay (applyUsagePostProcessing)
participant Upstream as Upstream Response Body
participant DTO as Usage DTO
Client->>Relay: Send request
Relay->>Upstream: Forward request to upstream
Upstream-->>Relay: Response (body with usage fields)
Relay->>Relay: Unmarshal usage into DTO
alt PromptTokensDetails.CachedTokens == 0
alt PromptCacheHitTokens > 0
Relay->>DTO: Set CachedTokens = PromptCacheHitTokens
else PromptCacheHitTokens == 0
Relay->>Upstream: extractCachedTokensFromBody(body)
Relay->>DTO: Set CachedTokens = cached_tokens (if present)
end
end
alt PromptTokensDetails.CachedCreationTokens == 0
Relay->>Upstream: extractCacheCreationTokensFromBody(body)
Relay->>DTO: Set CachedCreationTokens = cache_creation_input_tokens (if present)
end
Relay->>Client: Return processed response (DTO populated)
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment Tip You can disable the changed files summary in the walkthrough.Disable the |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
relay/channel/openai/relay_openai_test.go (1)
28-36: Consider adding a test for priority verification.When both
PromptCacheHitTokensand bodycached_tokensare present, the default case should preferPromptCacheHitTokens. A test confirming this priority would strengthen the coverage.🧪 Optional test for priority verification
func TestApplyUsagePostProcessing_DefaultCase_PrefersPromptCacheHitTokens(t *testing.T) { body := []byte(`{"usage":{"cached_tokens":100}}`) info := newRelayInfo(0) usage := &dto.Usage{ PromptCacheHitTokens: 200, } applyUsagePostProcessing(info, usage, body) if usage.PromptTokensDetails.CachedTokens != 200 { t.Errorf("CachedTokens = %d, want 200 (should prefer PromptCacheHitTokens over body)", usage.PromptTokensDetails.CachedTokens) } }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@relay/channel/openai/relay_openai_test.go` around lines 28 - 36, Add a new unit test to verify priority when both PromptCacheHitTokens and body.cached_tokens exist: create a test named TestApplyUsagePostProcessing_DefaultCase_PrefersPromptCacheHitTokens that constructs body := []byte(`{"usage":{"cached_tokens":100}}`), info := newRelayInfo(0), and usage := &dto.Usage{PromptCacheHitTokens: 200}, call applyUsagePostProcessing(info, usage, body), and assert that usage.PromptTokensDetails.CachedTokens == 200 (fail the test if it equals 100) to ensure PromptCacheHitTokens takes precedence over the body value.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@relay/channel/openai/relay_openai_test.go`:
- Around line 28-36: Add a new unit test to verify priority when both
PromptCacheHitTokens and body.cached_tokens exist: create a test named
TestApplyUsagePostProcessing_DefaultCase_PrefersPromptCacheHitTokens that
constructs body := []byte(`{"usage":{"cached_tokens":100}}`), info :=
newRelayInfo(0), and usage := &dto.Usage{PromptCacheHitTokens: 200}, call
applyUsagePostProcessing(info, usage, body), and assert that
usage.PromptTokensDetails.CachedTokens == 200 (fail the test if it equals 100)
to ensure PromptCacheHitTokens takes precedence over the body value.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 48b28f54-3062-4505-b450-f48e3b7b2e43
📒 Files selected for processing (3)
dto/openai_response.gorelay/channel/openai/relay-openai.gorelay/channel/openai/relay_openai_test.go
majiayu000
marked this pull request as ready for review
March 21, 2026 13:13
…dy cached_tokens Signed-off-by: majiayu000 <1835304752@qq.com>
This comment was marked as spam.
This comment was marked as spam.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3309
Summary
Three changes to fix cache token recognition for OpenAI-compatible providers (Qwen, DeepSeek, StepFun, etc.):
dto/openai_response.go — Change
CachedCreationTokensjson tag fromjson:"-"tojson:"cache_creation_input_tokens". The-tag prevented deserialization from any OpenAI-compatible JSON response. This field was only populated manually in the Claude adapter. The tag change allows natural deserialization from providers like Qwen. Note: this field is NOT serialized back to clients (the Usage struct usesInputTokenDetailswhich controls client-facing serialization).relay/channel/openai/relay-openai.go — Add a
defaultcase inapplyUsagePostProcessing()to handle all other OpenAI-compatible channels:CachedTokens == 0: checkPromptCacheHitTokens > 0first, then fallback toextractCachedTokensFromBody()(covers StepFun's top-levelcached_tokens)CachedCreationTokens == 0: call newextractCacheCreationTokensFromBody()relay/channel/openai/relay-openai.go — Add
extractCacheCreationTokensFromBody()function following the same pattern asextractCachedTokensFromBody(). Parsesusage.prompt_tokens_details.cache_creation_input_tokensfrom raw JSON.Test Plan
relay/channel/openai/relay_openai_test.gowith unit tests:cached_tokensfrom body (StepFun scenario)PromptCacheHitTokenswhen availablecache_creation_input_tokens(Qwen scenario)extractCacheCreationTokensFromBodyhandles valid and empty/invalid inputSummary by CodeRabbit
Bug Fixes
Tests