feat: use audio token usage if return - #1721
Conversation
WalkthroughThe STT handler in Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant Client
participant Relay as Relay (OpenaiSTTHandler)
participant Upstream as OpenAI Upstream
Client->>Relay: STT request (audio)
Relay->>Upstream: Forward request (stream response)
Upstream-->>Relay: Streamed response body
rect rgba(220,240,255,0.4)
note over Relay: Read upstream body chunk(s)
Relay->>Relay: Try JSON unmarshal { usage }
alt usage present and tokens > 0
Relay->>Relay: Normalize Prompt/Completion from Input/Output if needed
note over Relay: Use upstream usage (early)
else usage missing/zero
Relay->>Relay: Fallback: countAudioTokens(audio)
alt count ok
Relay->>Relay: Use counted tokens
else error
Relay->>Relay: Preserve prior error handling
end
end
end
Relay-->>Client: Stream response body (unchanged)
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
relay/channel/openai/relay-openai.go (1)
5-5: Minor consistency: prefer common.Unmarshal to avoid mixed JSON decodersElsewhere this file uses common.Unmarshal; using a single decoder keeps behavior uniform (e.g., jsoniter, number handling) and lets you drop the extra import.
Apply:
-import "encoding/json" +// no need for encoding/json if using common.UnmarshalAnd:
-if err := json.Unmarshal(responseBody, &responseData); err == nil && responseData.Usage != nil { +if err := common.Unmarshal(responseBody, &responseData); err == nil && responseData.Usage != nil {Also applies to: 294-294
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
relay/channel/openai/relay-openai.go(2 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
relay/channel/openai/relay-openai.go (2)
dto/openai_response.go (1)
Usage(217-230)types/error.go (2)
NewError(187-199)ErrorCodeCountTokenFailed(42-42)
| var responseData struct { | ||
| Usage *dto.Usage `json:"usage"` | ||
| } | ||
| if err := json.Unmarshal(responseBody, &responseData); err == nil && responseData.Usage != nil { | ||
| if responseData.Usage.TotalTokens > 0 { | ||
| usage := responseData.Usage | ||
| if usage.PromptTokens == 0 { | ||
| usage.PromptTokens = usage.InputTokens | ||
| } | ||
| if usage.CompletionTokens == 0 { | ||
| usage.CompletionTokens = usage.OutputTokens | ||
| } | ||
| return nil, usage | ||
| } | ||
| } |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Normalize upstream usage even when total_tokens is missing; also propagate token details
Many providers return only input/output tokens (and set total_tokens to 0). Your current gate requires TotalTokens > 0, which will incorrectly fall back to local estimation and under/over bill. Normalize as soon as any of Input/Output/Prompt/Completion is present, compute TotalTokens if absent, and map InputTokensDetails into PromptTokensDetails (mirrors OpenaiHandlerWithUsage).
Apply:
var responseData struct {
Usage *dto.Usage `json:"usage"`
}
-if err := json.Unmarshal(responseBody, &responseData); err == nil && responseData.Usage != nil {
- if responseData.Usage.TotalTokens > 0 {
- usage := responseData.Usage
- if usage.PromptTokens == 0 {
- usage.PromptTokens = usage.InputTokens
- }
- if usage.CompletionTokens == 0 {
- usage.CompletionTokens = usage.OutputTokens
- }
- return nil, usage
- }
-}
+if err := json.Unmarshal(responseBody, &responseData); err == nil && responseData.Usage != nil {
+ u := responseData.Usage
+ // Fill prompt/completion from input/output if missing
+ if u.PromptTokens == 0 && u.InputTokens > 0 {
+ u.PromptTokens = u.InputTokens
+ }
+ if u.CompletionTokens == 0 && u.OutputTokens > 0 {
+ u.CompletionTokens = u.OutputTokens
+ }
+ // Compute total if missing but components exist
+ if u.TotalTokens == 0 && (u.PromptTokens > 0 || u.CompletionTokens > 0) {
+ u.TotalTokens = u.PromptTokens + u.CompletionTokens
+ }
+ // Propagate input token details to prompt details if provided
+ if u.InputTokensDetails != nil {
+ u.PromptTokensDetails.TextTokens += u.InputTokensDetails.TextTokens
+ u.PromptTokensDetails.ImageTokens += u.InputTokensDetails.ImageTokens
+ u.PromptTokensDetails.AudioTokens += u.InputTokensDetails.AudioTokens
+ }
+ if u.TotalTokens > 0 || u.PromptTokens > 0 || u.CompletionTokens > 0 {
+ return nil, u
+ }
+}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| var responseData struct { | |
| Usage *dto.Usage `json:"usage"` | |
| } | |
| if err := json.Unmarshal(responseBody, &responseData); err == nil && responseData.Usage != nil { | |
| if responseData.Usage.TotalTokens > 0 { | |
| usage := responseData.Usage | |
| if usage.PromptTokens == 0 { | |
| usage.PromptTokens = usage.InputTokens | |
| } | |
| if usage.CompletionTokens == 0 { | |
| usage.CompletionTokens = usage.OutputTokens | |
| } | |
| return nil, usage | |
| } | |
| } | |
| var responseData struct { | |
| Usage *dto.Usage `json:"usage"` | |
| } | |
| if err := json.Unmarshal(responseBody, &responseData); err == nil && responseData.Usage != nil { | |
| u := responseData.Usage | |
| // Fill prompt/completion from input/output if missing | |
| if u.PromptTokens == 0 && u.InputTokens > 0 { | |
| u.PromptTokens = u.InputTokens | |
| } | |
| if u.CompletionTokens == 0 && u.OutputTokens > 0 { | |
| u.CompletionTokens = u.OutputTokens | |
| } | |
| // Compute total if missing but components exist | |
| if u.TotalTokens == 0 && (u.PromptTokens > 0 || u.CompletionTokens > 0) { | |
| u.TotalTokens = u.PromptTokens + u.CompletionTokens | |
| } | |
| // Propagate input token details to prompt details if provided | |
| if u.InputTokensDetails != nil { | |
| u.PromptTokensDetails.TextTokens += u.InputTokensDetails.TextTokens | |
| u.PromptTokensDetails.ImageTokens += u.InputTokensDetails.ImageTokens | |
| u.PromptTokensDetails.AudioTokens += u.InputTokensDetails.AudioTokens | |
| } | |
| // Return as long as any token data is present | |
| if u.TotalTokens > 0 || u.PromptTokens > 0 || u.CompletionTokens > 0 { | |
| return nil, u | |
| } | |
| } |
| audioTokens, err := countAudioTokens(c) | ||
| if err != nil { | ||
| return types.NewError(err, types.ErrorCodeCountTokenFailed), nil | ||
| } |
There was a problem hiding this comment.
Avoid returning an error after writing the upstream body to the client
By Line 289 you’ve already flushed the upstream response. Returning a non-nil error here risks double-send/error paths and inconsistent retries. Align with the TTS handler: log and return a zero-usage fallback instead of surfacing an error.
Apply:
audioTokens, err := countAudioTokens(c)
if err != nil {
- return types.NewError(err, types.ErrorCodeCountTokenFailed), nil
+ logger.LogError(c, fmt.Sprintf("count audio tokens failed: %v", err))
+ // After body is sent, do not bubble errors; return zero-usage fallback.
+ return nil, &dto.Usage{}
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| audioTokens, err := countAudioTokens(c) | |
| if err != nil { | |
| return types.NewError(err, types.ErrorCodeCountTokenFailed), nil | |
| } | |
| audioTokens, err := countAudioTokens(c) | |
| if err != nil { | |
| logger.LogError(c, fmt.Sprintf("count audio tokens failed: %v", err)) | |
| // After body is sent, do not bubble errors; return zero-usage fallback. | |
| return nil, &dto.Usage{} | |
| } |
🤖 Prompt for AI Agents
In relay/channel/openai/relay-openai.go around lines 307–310, do not return a
non-nil error after the upstream response has already been flushed; instead,
catch the countAudioTokens error, log the failure with context, set audioTokens
(or the usage result) to a zero-usage fallback value, and continue execution
returning nil error so we avoid double-send/retry paths (mirror the TTS handler
behavior).
|
{ 返回是0，难道是az版本的问题么？ |
3 participants
openai旧模型whisper-1由于没有返回usage,所以使用本地声音长度预估, 比较麻烦还得安装ffmpeg
新的模型gpt-4o-transcribe已经可以返回usage, 所以改用返回的usage会更精确
使用日志也会记录更准确
Summary by CodeRabbit