feat: enhance OpenAI responses handling and compatibility - #2817
Closed
mrhuangyong wants to merge 5073 commits into
Closed
feat: enhance OpenAI responses handling and compatibility#2817mrhuangyong wants to merge 5073 commits into
mrhuangyong wants to merge 5073 commits into
Conversation
… and streaming support
…-tts feat: support gpt tts series model quota calculate
…in GetAudioDuration" This reverts commit e293be0.
…itelist-cidr feat(auth): enhance IP restriction handling with CIDR support
… meta when token count is disabled Clamp request body size (including post-decompression) to avoid memory exhaustion caused by huge payloads/zip bombs, especially with large-context Claude requests. Add a configurable `MAX_REQUEST_BODY_MB` (default `32`) and document it. - Enforce max request body size after gzip/br decompression via `http.MaxBytesReader` - Add a secondary size guard in `common.GetRequestBody` and cache-safe handling - Return **413 Request Entity Too Large** on oversized bodies in relay entry - Avoid building large `TokenCountMeta.CombineText` when both token counting and sensitive check are disabled (use lightweight meta for pricing) - Update READMEs (CN/EN/FR/JA) with `MAX_REQUEST_BODY_MB` - Fix a handful of vet/formatting issues encountered during the change - `go test ./...` passes
Tighten oversized request handling across relay paths and make error matching reliable. - Align `MAX_REQUEST_BODY_MB` fallback to `32` in request body reader and decompression middleware - Stop ignoring `GetRequestBody` errors in relay retry paths; return consistent **413** on oversized bodies (400 for other read errors) - Add `Unwrap()` to `types.NewAPIError` so `errors.Is/As` can match wrapped underlying errors - `go test ./...` passes
Updated web/src/i18n/locales/fr.json to improve French translations for the user interface. Removed verbose prefixes like 'Gestion des...' and 'Paramètres de...' to prevent truncation in sidebars and menus. Harmonized terms for consistency (e.g., 'Tâches', 'Journaux', 'Dessins'). Renamed 'Place du marché' to 'Marché des modèles'.
- Add minimal case to clampThinkingBudgetByEffort to avoid defaulting to full thinking budget
…t-price fix: /v1/responses/compact default billing
* feat: 引入通用 HTTP BodyStorage/DiskCache 缓存配置与管理 - 新增 common/body_storage.go 提供 HTTP 请求体存储抽象和文件缓存能力 - 增加 common/disk_cache_config.go 支持全局磁盘缓存配置 - main.go 挂载缓存初始化流程 - 新增和补充 controller/performance.go (及 unix/windows) 用于缓存性能监控接口 - middleware/body_cleanup.go 自动清理缓存文件 - router 挂载相关接口 - 前端 settings 页面新增性能监控设置 PerformanceSetting - 优化缓存开关状态和模块热插拔能力 - 其他相关文件同步适配缓存扩展 * fix: 修复 BodyStorage 并发安全和错误处理问题 - 修复 diskStorage.Close() 竞态条件，先获取锁再执行 CAS - 为 memoryStorage 添加互斥锁和 closed 状态检查 - 修复 CreateBodyStorageFromReader 在磁盘存储失败时的回退逻辑 - 添加缓存命中统计调用 (IncrementDiskCacheHits/IncrementMemoryCacheHits) - 修复 gin.go 中 Seek 错误被忽略的问题 - 在 api-router 添加 BodyStorageCleanup 中间件 - 修复前端 formatBytes 对异常值的处理 Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…uantumNous#2793) Explicitly cast Blocks, Bavail, and Bfree to uint64 for cross-platform compatibility, as these fields are int64 on FreeBSD but uint64 on Linux.
feat: Support customizing the success and cancel url of Stripe.
…65bf72572ff8684dd7ef068e576 feat: doubao add first and last image to video
…d7331d0987d96dc78bae181e331 feat: task pre consume modelPrice default use setting value
…3b7b97dbdbd98ade9b372bd6f63 feat: CodeViewer click link and auto wrap
…ling feat(gemini): support cached token billing
fix(ui): use distinct color palette for group tags
* fix: channel affinity log styles * fix: Issue with incorrect data storage when switching key sources * feat: support not retrying after a single rule configuration fails * fix: render channel affinity tooltip as multiline content * feat: channel affinity cache hit * fix: prevent ChannelAffinityUsageCacheModal infinite loading and hide data before fetch * chore: format backend with gofmt and frontend with prettier/eslint autofix
…st-override-take-effect fix: make channel Host override take effect
feat: /v1/responses qwen3 max && perplexity
…5ce3426cd8ea0d13f38e2f4eb81 feat: auto-adapt video modal
…de-cache-usage fix: openrouter claude cache usage
* fix: vertex maas api addr
- Updated ResponsesStreamResponse structure to include new fields: ResponseID, Text, Arguments, and ContentIndex. - Implemented conversion functions for OpenAI responses in the adaptor, enabling compatibility with chat completions requests. - Enhanced stream response handling to support OpenAI responses format, ensuring proper state management and event handling. - Added new service functions for converting between OpenAI and Claude response formats.
Contributor
WalkthroughAdd OpenAI-compatible streaming support for Claude by introducing bidirectional converters between OpenAI Responses and Chat Completions formats, implementing stateful stream processing to emit OpenAI-style events from streaming deltas, and extending Claude's relay pathway to support the Responses API. Changes
Sequence DiagramsequenceDiagram
participant Client
participant Claude Relay
participant Converter Service
participant Stream State
participant OpenAI Compat
Client->>Claude Relay: Responses API Request
Claude Relay->>Converter Service: ResponsesRequestToChatCompletionsRequest()
Converter Service->>OpenAI Compat: ResponsesRequestToChatCompletionsRequest()
OpenAI Compat-->>Converter Service: GeneralOpenAIRequest
Converter Service-->>Claude Relay: Chat Completions Request
Claude Relay->>Stream State: NewChatToResponsesStreamState()
Stream State-->>Claude Relay: StreamState initialized
loop For each streaming chunk
Claude Relay->>Stream State: HandleChatChunk(delta)
Stream State->>Stream State: Process delta, track indices
Stream State-->>Claude Relay: ResponsesStreamResponse events
Claude Relay-->>Client: Stream chunk (created/in_progress/output_item/etc)
end
Claude Relay->>Stream State: FinalEvents(usage)
Stream State-->>Claude Relay: Final response events
Claude Relay-->>Client: Completed response
Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Fix all issues with AI agents
In `@service/openaicompat/chat_stream_to_responses_stream.go`:
- Around line 48-117: HandleChatChunk currently assumes incoming
call.Function.Arguments are always cumulative (startsWith previous) and discards
true delta fragments; change the accumulation logic so when args is non-empty
you compute deltaArgs as the new bytes to append: if prev == "" then deltaArgs =
args; else if strings.HasPrefix(args, prev) then deltaArgs = args[len(prev):];
else /* upstream sent a pure delta/fragment */ deltaArgs = args and treat it as
an appended fragment; then update s.ToolCallArgs[callID] = prev + deltaArgs (not
simply args) and emit the same response with Delta = deltaArgs; keep existing
behavior that skips when deltaArgs == "". Apply this change inside
HandleChatChunk where ToolCallArgs is processed (use the existing callID,
s.ToolCallArgs, and outputIndex logic).
In `@service/openaicompat/chat_to_responses_response.go`:
- Around line 13-51: The ChatCompletionsResponseToResponsesResponse function
currently overwrites existing prompt/completion token counts when
usage.TotalTokens == 0; change the logic so you only fill missing values: if
usage.PromptTokens is zero or nil then set it from
info.GetEstimatePromptTokens() (or estimate), and if usage.CompletionTokens is
zero then set it using estimateTokenFallback(extractChatText(chat)); compute
usage.TotalTokens as the sum of the (possibly-original, possibly-filled)
PromptTokens and CompletionTokens and preserve existing PromptTokensDetails and
CompletionTokenDetails when present. Ensure the referenced symbols are adjusted:
ChatCompletionsResponseToResponsesResponse, usage, chat.Usage.*,
info.GetEstimatePromptTokens, estimateTokenFallback, and extractChatText.
In `@service/openaicompat/responses_to_chat_request.go`:
- Around line 252-281: The function responsesInputItemToMessageContent currently
ignores "input_video" items; update it to handle the "input_video" case by
calling responsesContentItemToMediaContent(item), checking for error, and if
media.Type != "" returning []dto.MediaContent{media}, nil (same pattern used for
"input_audio" and "input_file") so top-level video inputs are not dropped.
- Around line 193-303: The code lets empty/absent image URLs slip through
because parseResponsesImageURL returns a MessageImageUrl with an empty Url;
update validation so we fail fast: in responsesContentItemToMediaContent's
"input_image" case call parseResponsesImageURL(part["image_url"],
part["detail"]) and if the returned MessageImageUrl.Url == "" return a
descriptive error (instead of producing a MediaContent with an empty URL), and
also change responsesInputItemToMessageContent's "input_image" branch to use
responsesContentItemToMediaContent (or perform the same Url non-empty check) and
propagate the error so callers receive a failure rather than silent empty data;
reference functions: parseResponsesImageURL, responsesContentItemToMediaContent,
responsesInputItemToMessageContent.
Comment on lines
+48
to
+117
| func (s *ChatToResponsesStreamState) HandleChatChunk(chunk *dto.ChatCompletionsStreamResponse) []dto.ResponsesStreamResponse { | ||
| if chunk == nil || len(chunk.Choices) == 0 { | ||
| return nil | ||
| } | ||
|
|
||
| if chunk.Model != "" { | ||
| s.Model = chunk.Model | ||
| } | ||
| if s.CreatedAt == 0 && chunk.Created != 0 { | ||
| s.CreatedAt = chunk.Created | ||
| } | ||
|
|
||
| events := s.baseEvents() | ||
|
|
||
| delta := chunk.Choices[0].Delta | ||
| if delta.Content != nil { | ||
| content := *delta.Content | ||
| if content != "" { | ||
| events = append(events, s.ensureMessageItemEvents()...) | ||
| events = append(events, s.ensureContentPartEvents()...) | ||
| s.OutputText.WriteString(content) | ||
| events = append(events, s.outputTextDeltaEvent(content)) | ||
| } | ||
| } | ||
|
|
||
| if len(delta.ToolCalls) > 0 { | ||
| for idx, call := range delta.ToolCalls { | ||
| callID := strings.TrimSpace(call.ID) | ||
| if callID == "" { | ||
| if len(s.ToolCallOrder) > 0 { | ||
| callID = s.ToolCallOrder[len(s.ToolCallOrder)-1] | ||
| } else { | ||
| callID = fmt.Sprintf("call_%d", idx) | ||
| } | ||
| } | ||
| if call.Function.Name != "" { | ||
| s.ToolCallName[callID] = call.Function.Name | ||
| } | ||
| if !s.ToolCallSent[callID] { | ||
| s.ToolCallSent[callID] = true | ||
| s.ToolCallOrder = append(s.ToolCallOrder, callID) | ||
| outIndex := s.allocOutputIndex(callID) | ||
| events = append(events, s.toolItemAddedEvent(callID, outIndex)) | ||
| } | ||
|
|
||
| args := call.Function.Arguments | ||
| if args == "" { | ||
| continue | ||
| } | ||
| prev := s.ToolCallArgs[callID] | ||
| deltaArgs := args | ||
| if prev != "" && strings.HasPrefix(args, prev) { | ||
| deltaArgs = args[len(prev):] | ||
| } | ||
| if deltaArgs == "" { | ||
| continue | ||
| } | ||
| s.ToolCallArgs[callID] = args | ||
| events = append(events, dto.ResponsesStreamResponse{ | ||
| Type: "response.function_call_arguments.delta", | ||
| ResponseID: s.ResponseID, | ||
| ItemID: callID, | ||
| OutputIndex: s.outputIndexPtr(callID), | ||
| Delta: deltaArgs, | ||
| }) | ||
| } | ||
| } | ||
|
|
||
| return events | ||
| } |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
OpenAI streaming API tool_calls function arguments cumulative or delta
💡 Result:
They’re deltas (incremental), not cumulative—you should accumulate/concatenate tool/function arguments across stream chunks/events until you receive the “done”/finalization signal.
Responses API (recommended streaming model)
- During streaming you’ll receive
response.function_call_arguments.deltaevents whosedeltais only the next fragment of the JSON-arguments string. (platform.openai.com) - When complete, you’ll receive
response.function_call_arguments.done, which contains the final fullargumentsstring. (platform.openai.com)
So: append eachdeltain order (keyed by the tool-call item), or just wait for the.doneevent if you don’t need partial args.
Chat Completions streaming
- Chunks contain a
choices[].deltaobject; tool calls appear underchoices[].delta.tool_calls. (platform.openai.com) - In practice, the tool call’s
function.argumentsoften arrives fragmented across multiple chunks, so you must concatenate those fragments per tool call (commonly bytool_calls[].index, and/oridwhen present). (python.langchain.com)
If you tell me which endpoint/SDK you’re using (Responses vs Chat Completions; Python/JS), I can show the exact accumulator pattern for that client.
Citations:
- 1: https://platform.openai.com/docs/api-reference/responses-streaming/response/reasoning
- 2: https://platform.openai.com/docs/api-reference/responses-streaming/response/reasoning
- 3: https://platform.openai.com/docs/api-reference/chat-streaming/streaming
- 4: https://python.langchain.com/docs/how_to/function_calling/?utm_source=openai
🌐 Web query:
Anthropic streaming API tool calls arguments behavior
💡 Result:
When you use Anthropic’s Messages API with stream: true and the model decides to call a tool, tool arguments (“input”) arrive incrementally:
- The tool’s arguments are streamed via
content_block_deltaevents wheredelta.type = "input_json_delta"andpartial_jsonis a fragment of a JSON string, not necessarily valid JSON on its own. You typically concatenate the fragments and only parse once you receivecontent_block_stopfor that tool block; the finaltool_use.inputis an object. [1] - Current model behavior: Claude generally emits one complete key/value at a time for tool
input, which can create gaps/delays between streaming events while it “works” on the next parameter. Once a key/value is ready, it may still be split across multipleinput_json_deltachunks. [1]
If you enable Fine-grained tool streaming (beta):
- Add header
anthropic-beta: fine-grained-tool-streaming-2025-05-14(and stream=true). This streams tool parameters without buffering/JSON validation, so you may receive invalid or incomplete JSON, especially if generation stops due tomax_tokens. Chunking also tends to be faster and in larger pieces. [2]
Sources: Anthropic Streaming Messages docs [1] and Fine-grained tool streaming docs [2].
🏁 Script executed:
# Let's also check if there are tests or other files in the repository that might clarify how this should work
fd -t f -e go | xargs grep -l "ToolCall\|tool_call" | head -20Repository: QuantumNous/new-api
Length of output: 682
🏁 Script executed:
# Check the specific file mentioned in the review comment
cat -n service/openaicompat/chat_stream_to_responses_stream.go | sed -n '48,117p'Repository: QuantumNous/new-api
Length of output: 2408
🏁 Script executed:
# Look for how other parts of the codebase handle tool call accumulation
rg -A 5 -B 5 "ToolCallArgs" service/openaicompat/chat_stream_to_responses_stream.goRepository: QuantumNous/new-api
Length of output: 1856
🏁 Script executed:
# Check the relay layer to see how they handle tool calls
cat -n relay/channel/openai/relay-openai.go | head -200Repository: QuantumNous/new-api
Length of output: 8335
Fix tool call arguments accumulation to handle both cumulative and delta-based streaming.
OpenAI and Anthropic streaming APIs send tool call arguments as fragmented chunks that must be accumulated across multiple events. The current code assumes arguments always arrive as cumulative strings and will truncate them if the upstream sends pure deltas (or fragments that don't start with the previous value).
🛠️ Suggested fix
prev := s.ToolCallArgs[callID]
deltaArgs := args
if prev != "" && strings.HasPrefix(args, prev) {
deltaArgs = args[len(prev):]
-}
-if deltaArgs == "" {
- continue
-}
-s.ToolCallArgs[callID] = args
+ s.ToolCallArgs[callID] = args
+} else {
+ // Treat as delta and append
+ s.ToolCallArgs[callID] = prev + args
+}
+if deltaArgs == "" {
+ continue
+}
events = append(events, dto.ResponsesStreamResponse{🤖 Prompt for AI Agents
In `@service/openaicompat/chat_stream_to_responses_stream.go` around lines 48 -
117, HandleChatChunk currently assumes incoming call.Function.Arguments are
always cumulative (startsWith previous) and discards true delta fragments;
change the accumulation logic so when args is non-empty you compute deltaArgs as
the new bytes to append: if prev == "" then deltaArgs = args; else if
strings.HasPrefix(args, prev) then deltaArgs = args[len(prev):]; else /*
upstream sent a pure delta/fragment */ deltaArgs = args and treat it as an
appended fragment; then update s.ToolCallArgs[callID] = prev + deltaArgs (not
simply args) and emit the same response with Delta = deltaArgs; keep existing
behavior that skips when deltaArgs == "". Apply this change inside
HandleChatChunk where ToolCallArgs is processed (use the existing callID,
s.ToolCallArgs, and outputIndex logic).
Comment on lines
+13
to
+51
| func ChatCompletionsResponseToResponsesResponse(chat *dto.OpenAITextResponse, info *relaycommon.RelayInfo, responseID string) (*dto.OpenAIResponsesResponse, *dto.Usage, error) { | ||
| if chat == nil { | ||
| return nil, nil, errors.New("response is nil") | ||
| } | ||
|
|
||
| responseID = normalizeResponsesID(responseID) | ||
| createdAt := coerceCreatedAt(chat.Created) | ||
|
|
||
| output := chatMessageToResponsesOutput(chat) | ||
|
|
||
| usage := &dto.Usage{ | ||
| PromptTokens: chat.Usage.PromptTokens, | ||
| CompletionTokens: chat.Usage.CompletionTokens, | ||
| TotalTokens: chat.Usage.TotalTokens, | ||
| } | ||
| usage.PromptTokensDetails = chat.Usage.PromptTokensDetails | ||
| usage.CompletionTokenDetails = chat.Usage.CompletionTokenDetails | ||
|
|
||
| if usage.TotalTokens == 0 { | ||
| text := extractChatText(chat) | ||
| if info != nil { | ||
| usage.PromptTokens = info.GetEstimatePromptTokens() | ||
| } | ||
| usage.CompletionTokens = estimateTokenFallback(text) | ||
| usage.TotalTokens = usage.PromptTokens + usage.CompletionTokens | ||
| } | ||
|
|
||
| resp := &dto.OpenAIResponsesResponse{ | ||
| ID: responseID, | ||
| Object: "response", | ||
| CreatedAt: createdAt, | ||
| Status: "completed", | ||
| Model: chat.Model, | ||
| Output: output, | ||
| Usage: usage, | ||
| } | ||
|
|
||
| return resp, usage, nil | ||
| } |
Contributor
There was a problem hiding this comment.
Don’t overwrite real usage values when only total_tokens is missing.
If prompt/completion tokens are already present, the current fallback replaces them with estimates. That can skew billing and analytics.
🛠️ Suggested fix
if usage.TotalTokens == 0 {
- text := extractChatText(chat)
- if info != nil {
- usage.PromptTokens = info.GetEstimatePromptTokens()
- }
- usage.CompletionTokens = estimateTokenFallback(text)
- usage.TotalTokens = usage.PromptTokens + usage.CompletionTokens
+ if usage.PromptTokens == 0 && info != nil {
+ usage.PromptTokens = info.GetEstimatePromptTokens()
+ }
+ if usage.CompletionTokens == 0 {
+ text := extractChatText(chat)
+ usage.CompletionTokens = estimateTokenFallback(text)
+ }
+ if usage.PromptTokens > 0 || usage.CompletionTokens > 0 {
+ usage.TotalTokens = usage.PromptTokens + usage.CompletionTokens
+ }
}🤖 Prompt for AI Agents
In `@service/openaicompat/chat_to_responses_response.go` around lines 13 - 51, The
ChatCompletionsResponseToResponsesResponse function currently overwrites
existing prompt/completion token counts when usage.TotalTokens == 0; change the
logic so you only fill missing values: if usage.PromptTokens is zero or nil then
set it from info.GetEstimatePromptTokens() (or estimate), and if
usage.CompletionTokens is zero then set it using
estimateTokenFallback(extractChatText(chat)); compute usage.TotalTokens as the
sum of the (possibly-original, possibly-filled) PromptTokens and
CompletionTokens and preserve existing PromptTokensDetails and
CompletionTokenDetails when present. Ensure the referenced symbols are adjusted:
ChatCompletionsResponseToResponsesResponse, usage, chat.Usage.*,
info.GetEstimatePromptTokens, estimateTokenFallback, and extractChatText.
Comment on lines
+193
to
+303
| func responsesContentItemToMediaContent(part map[string]any) (dto.MediaContent, error) { | ||
| t, _ := part["type"].(string) | ||
| switch t { | ||
| case "input_text": | ||
| text, _ := part["text"].(string) | ||
| return dto.MediaContent{ | ||
| Type: dto.ContentTypeText, | ||
| Text: text, | ||
| }, nil | ||
| case "input_image": | ||
| return dto.MediaContent{ | ||
| Type: dto.ContentTypeImageURL, | ||
| ImageUrl: parseResponsesImageURL(part["image_url"], part["detail"]), | ||
| }, nil | ||
| case "input_audio": | ||
| if audio, ok := part["input_audio"].(map[string]any); ok { | ||
| data, _ := audio["data"].(string) | ||
| format, _ := audio["format"].(string) | ||
| return dto.MediaContent{ | ||
| Type: dto.ContentTypeInputAudio, | ||
| InputAudio: &dto.MessageInputAudio{ | ||
| Data: data, | ||
| Format: format, | ||
| }, | ||
| }, nil | ||
| } | ||
| case "input_file": | ||
| if file, ok := part["file"].(map[string]any); ok { | ||
| msgFile := &dto.MessageFile{} | ||
| if fileID, ok := file["file_id"].(string); ok { | ||
| msgFile.FileId = fileID | ||
| } | ||
| if fileName, ok := file["filename"].(string); ok { | ||
| msgFile.FileName = fileName | ||
| } | ||
| if fileData, ok := file["file_data"].(string); ok { | ||
| msgFile.FileData = fileData | ||
| } | ||
| if fileURL, ok := file["file_url"].(string); ok && msgFile.FileData == "" { | ||
| msgFile.FileData = fileURL | ||
| } | ||
| return dto.MediaContent{ | ||
| Type: dto.ContentTypeFile, | ||
| File: msgFile, | ||
| }, nil | ||
| } | ||
| case "input_video": | ||
| if url, ok := part["video_url"].(string); ok { | ||
| return dto.MediaContent{ | ||
| Type: dto.ContentTypeVideoUrl, | ||
| VideoUrl: &dto.MessageVideoUrl{ | ||
| Url: url, | ||
| }, | ||
| }, nil | ||
| } | ||
| } | ||
| return dto.MediaContent{}, nil | ||
| } | ||
|
|
||
| func responsesInputItemToMessageContent(itemType string, item map[string]any) (any, error) { | ||
| switch itemType { | ||
| case "input_text": | ||
| if text, ok := item["text"].(string); ok { | ||
| return text, nil | ||
| } | ||
| case "input_image": | ||
| media := dto.MediaContent{ | ||
| Type: dto.ContentTypeImageURL, | ||
| ImageUrl: parseResponsesImageURL(item["image_url"], item["detail"]), | ||
| } | ||
| return []dto.MediaContent{media}, nil | ||
| case "input_audio": | ||
| media, err := responsesContentItemToMediaContent(item) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| if media.Type != "" { | ||
| return []dto.MediaContent{media}, nil | ||
| } | ||
| case "input_file": | ||
| media, err := responsesContentItemToMediaContent(item) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| if media.Type != "" { | ||
| return []dto.MediaContent{media}, nil | ||
| } | ||
| } | ||
| return nil, nil | ||
| } | ||
|
|
||
| func parseResponsesImageURL(image any, detail any) *dto.MessageImageUrl { | ||
| msg := &dto.MessageImageUrl{} | ||
| switch v := image.(type) { | ||
| case string: | ||
| msg.Url = v | ||
| case map[string]any: | ||
| if url, ok := v["url"].(string); ok { | ||
| msg.Url = url | ||
| } | ||
| if det, ok := v["detail"].(string); ok { | ||
| msg.Detail = det | ||
| } | ||
| } | ||
| if msg.Detail == "" { | ||
| if det, ok := detail.(string); ok { | ||
| msg.Detail = det | ||
| } | ||
| } | ||
| return msg | ||
| } |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, let's find and examine the file referenced in the review
fd -t f responses_to_chat_request.go | head -5Repository: QuantumNous/new-api
Length of output: 112
🏁 Script executed:
# Check the learning's reference file
fd -t f task_video.go | head -5Repository: QuantumNous/new-api
Length of output: 87
🏁 Script executed:
# Let's search for patterns related to data: URL validation
rg "data:" --type go -B 2 -A 2 | head -50Repository: QuantumNous/new-api
Length of output: 2846
🏁 Script executed:
# Search for image_url handling and validation patterns
rg "image_url" --type go -B 1 -A 1 | head -50Repository: QuantumNous/new-api
Length of output: 2518
🏁 Script executed:
# Let's examine controller/task_video.go to understand the data: URL validation pattern
cat -n controller/task_video.go | head -100Repository: QuantumNous/new-api
Length of output: 4108
🏁 Script executed:
# Search for FailReason assignment to see the pattern
rg "FailReason" --type go -B 3 -A 3 | head -80Repository: QuantumNous/new-api
Length of output: 4214
🏁 Script executed:
# Check what happens downstream with the returned MediaContent
rg "MediaContent" --type go -B 2 -A 2 | grep -A 5 -B 5 "ImageUrl\|input_image" | head -60Repository: QuantumNous/new-api
Length of output: 3517
🏁 Script executed:
# Look at the actual function context and imports to understand the data flow
sed -n '1,50p' service/openaicompat/responses_to_chat_request.goRepository: QuantumNous/new-api
Length of output: 1353
🏁 Script executed:
# Check if there are any validations on image URLs elsewhere in the codebase
rg "TrimSpace\|Url.*==" --type go | grep -i image | head -20Repository: QuantumNous/new-api
Length of output: 45
🏁 Script executed:
# Verify if parseResponsesImageURL is called with null/nil and what it returns
sed -n '280,320p' service/openaicompat/responses_to_chat_request.goRepository: QuantumNous/new-api
Length of output: 952
🏁 Script executed:
# Check the MessageImageUrl struct to understand what fields are required
rg "MessageImageUrl" --type go -A 5 | head -30Repository: QuantumNous/new-api
Length of output: 1523
Validate input_image URLs before emitting media content.
When image_url is missing or empty, parseResponsesImageURL returns a MessageImageUrl with an empty Url field. Downstream code checks if vv.Url != "" and silently skips processing, allowing invalid data to propagate silently. Failing fast here makes input validation explicit and prevents silent failures.
🛠️ Suggested fix
case "input_image":
- return dto.MediaContent{
- Type: dto.ContentTypeImageURL,
- ImageUrl: parseResponsesImageURL(part["image_url"], part["detail"]),
- }, nil
+ img := parseResponsesImageURL(part["image_url"], part["detail"])
+ if img == nil || strings.TrimSpace(img.Url) == "" {
+ return dto.MediaContent{}, fmt.Errorf("input_image requires image_url")
+ }
+ return dto.MediaContent{
+ Type: dto.ContentTypeImageURL,
+ ImageUrl: img,
+ }, nil🤖 Prompt for AI Agents
In `@service/openaicompat/responses_to_chat_request.go` around lines 193 - 303,
The code lets empty/absent image URLs slip through because
parseResponsesImageURL returns a MessageImageUrl with an empty Url; update
validation so we fail fast: in responsesContentItemToMediaContent's
"input_image" case call parseResponsesImageURL(part["image_url"],
part["detail"]) and if the returned MessageImageUrl.Url == "" return a
descriptive error (instead of producing a MediaContent with an empty URL), and
also change responsesInputItemToMessageContent's "input_image" branch to use
responsesContentItemToMediaContent (or perform the same Url non-empty check) and
propagate the error so callers receive a failure rather than silent empty data;
reference functions: parseResponsesImageURL, responsesContentItemToMediaContent,
responsesInputItemToMessageContent.
Comment on lines
+252
to
+281
| func responsesInputItemToMessageContent(itemType string, item map[string]any) (any, error) { | ||
| switch itemType { | ||
| case "input_text": | ||
| if text, ok := item["text"].(string); ok { | ||
| return text, nil | ||
| } | ||
| case "input_image": | ||
| media := dto.MediaContent{ | ||
| Type: dto.ContentTypeImageURL, | ||
| ImageUrl: parseResponsesImageURL(item["image_url"], item["detail"]), | ||
| } | ||
| return []dto.MediaContent{media}, nil | ||
| case "input_audio": | ||
| media, err := responsesContentItemToMediaContent(item) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| if media.Type != "" { | ||
| return []dto.MediaContent{media}, nil | ||
| } | ||
| case "input_file": | ||
| media, err := responsesContentItemToMediaContent(item) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| if media.Type != "" { | ||
| return []dto.MediaContent{media}, nil | ||
| } | ||
| } | ||
| return nil, nil |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# Find the file and examine the switch statement
fd "responses_to_chat_request.go" --type fRepository: QuantumNous/new-api
Length of output: 112
🏁 Script executed:
# Search for responsesContentItemToMediaContent definition
rg "func responsesContentItemToMediaContent" -A 20Repository: QuantumNous/new-api
Length of output: 1812
🏁 Script executed:
# Search for input_video references in the codebase
rg "input_video" -C 3Repository: QuantumNous/new-api
Length of output: 1056
🏁 Script executed:
# Check the responsesInputItemToMessageContent function more completely
rg "func responsesInputItemToMessageContent" -A 50Repository: QuantumNous/new-api
Length of output: 3829
🏁 Script executed:
# Get the complete responsesContentItemToMediaContent function
rg "func responsesContentItemToMediaContent" -A 100 | head -120Repository: QuantumNous/new-api
Length of output: 7786
Add input_video handling in item-to-content conversion.
responsesContentItemToMediaContent supports input_video, but top‑level input_video items are silently ignored here, causing video inputs to be dropped.
🛠️ Suggested fix
case "input_file":
media, err := responsesContentItemToMediaContent(item)
if err != nil {
return nil, err
}
if media.Type != "" {
return []dto.MediaContent{media}, nil
}
+case "input_video":
+ media, err := responsesContentItemToMediaContent(item)
+ if err != nil {
+ return nil, err
+ }
+ if media.Type != "" {
+ return []dto.MediaContent{media}, nil
+ }
}🤖 Prompt for AI Agents
In `@service/openaicompat/responses_to_chat_request.go` around lines 252 - 281,
The function responsesInputItemToMessageContent currently ignores "input_video"
items; update it to handle the "input_video" case by calling
responsesContentItemToMediaContent(item), checking for error, and if media.Type
!= "" returning []dto.MediaContent{media}, nil (same pattern used for
"input_audio" and "input_file") so top-level video inputs are not dropped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by CodeRabbit
Release Notes