Add WebSocket relay support for /v1/responses - #3370
Conversation
WalkthroughThe changes introduce a new WebSocket relay flow for OpenAI's Responses API. A new route handler processes Responses WebSocket upgrades by parsing the first message for Changes
Sequence DiagramsequenceDiagram
participant Client as Client (WebSocket)
participant Router as Router/Controller
participant Middleware as Middleware
participant Relay as Relay Handler
participant Channel as Channel/Adaptor
participant Target as Target (WebSocket)
Client->>Router: GET /v1/responses?model=gpt-4
Router->>Middleware: Extract model from query
Middleware->>Router: Return model name
Router->>Router: Read first WebSocket message
Router->>Router: Parse & validate (model, input)
Router->>Relay: Setup Responses channel with model
Relay->>Channel: Initialize adaptor
Relay->>Target: Establish WebSocket connection<br/>(URL scheme: https→wss)
Relay->>Target: Send initial request (type: response.create)
Target->>Relay: Acknowledge connection
loop Bidirectional Message Proxying
alt Message from Client
Client->>Relay: Forward message
Relay->>Target: Proxy to target
Target->>Relay: Response
Relay->>Client: Proxy back to client
else Message from Target
Target->>Relay: Stream response
Relay->>Client: Proxy to client
end
end
Client->>Relay: Close connection
Relay->>Relay: Settle billing
Relay->>Target: Send close frame
Target->>Relay: Acknowledge close
Relay->>Client: Connection closed
Estimated Code Review Effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly Related PRs
Suggested Reviewers
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment Tip CodeRabbit can scan for known vulnerabilities in your dependencies using OSV Scanner.OSV Scanner will automatically detect and report security vulnerabilities in your project's dependencies. No additional configuration is required. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@controller/relay.go`:
- Line 4: Replace direct use of encoding/json in the WebSocket request parser by
removing the encoding/json import and swapping json.Unmarshal calls with
common.Unmarshal; specifically, update the code that currently calls
json.Unmarshal(...) in the relay WebSocket request parsing logic to call
common.Unmarshal(...) and adjust error handling accordingly, and ensure the
import list references the package providing common.Unmarshal instead of
"encoding/json".
In `@middleware/distributor.go`:
- Around line 181-188: For WebSocket GET upgrades to /v1/responses, defer model
auth instead of letting the outer Distribute middleware apply token model-limit
checks early: add an explicit flag (e.g., DeferModelAuth bool) to the
modelRequest struct, set modelRequest.DeferModelAuth = true in the websocket
upgrade branch (the block that handles websocket.IsWebSocketUpgrade and returns
&modelRequest), and update the Distribute flow to check
modelRequest.DeferModelAuth and skip the middleware model-limit branch when true
so setupResponsesWSChannel can validate the real model from the first WS frame.
In `@relay/websocket.go`:
- Around line 79-80: The billing settle call currently uses
info.FinalPreConsumedQuota (service.SettleBilling and
info.FinalPreConsumedQuota) which never gets updated because the frame-relay
code that forwards frames (the relay/forwarding loop handling streamed frames)
doesn't extract a terminal usage event; modify the relay/frames-handling logic
to detect the terminal usage/usage-summary frame (or track streamed tokens/bytes
usage as frames are processed), populate or compute the actual final usage value
(update info.FinalPreConsumedQuota or create a new finalUsage variable) before
calling service.SettleBilling, and then call service.SettleBilling with that
actual final usage so the final billing reflects real streamed consumption
rather than the pre-consumed estimate.
- Around line 91-93: The current logic only defaults initialRequest.Type to
"response.create" when empty, allowing a client-supplied non-empty Type to slip
through; change the behavior in the websocket handler so the first upstream
event always has Type "response.create" by unconditionally setting
initialRequest.Type = "response.create" (instead of only when empty) or, if you
prefer validation, explicitly reject any initialRequest.Type !=
"response.create"; update the code that inspects/forwards initialRequest (the
variable named initialRequest in websocket.go) so it no longer forwards
client-provided Types for the first frame.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 3e94198c-9756-4f2c-b1a3-d0d090fbc257
📒 Files selected for processing (6)
controller/relay.godto/openai_request.gomiddleware/distributor.gorelay/channel/api_request.gorelay/websocket.gorouter/relay-router.go
| package controller | ||
|
|
||
| import ( | ||
| "encoding/json" |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# Check if the file exists and examine the import section
head -20 controller/relay.go | cat -nRepository: QuantumNous/new-api
Length of output: 609
🏁 Script executed:
# Check lines 290-300 to see the json.Unmarshal usage
sed -n '290,300p' controller/relay.go | cat -nRepository: QuantumNous/new-api
Length of output: 511
🏁 Script executed:
# Verify that common.Unmarshal exists in common/json.go
grep -n "func.*Unmarshal" common/json.go | head -5Repository: QuantumNous/new-api
Length of output: 160
Use common.Unmarshal instead of direct encoding/json in the WebSocket request parser.
Line 4 imports encoding/json and line 294 uses json.Unmarshal directly in business code. Replace with common.Unmarshal to comply with the repo's JSON wrapper contract.
Fix
- "encoding/json"
"errors"
@@
- if err := json.Unmarshal(payload, request); err != nil {
+ if err := common.Unmarshal(payload, request); err != nil {
return nil, fmt.Errorf("parse first websocket message failed: %w", err)
}Per coding guidelines: All JSON marshal/unmarshal operations MUST use wrapper functions from common/json.go (common.Marshal, common.Unmarshal, common.UnmarshalJsonStr, common.DecodeJson, common.GetJsonType). Do NOT directly call encoding/json in business code.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@controller/relay.go` at line 4, Replace direct use of encoding/json in the
WebSocket request parser by removing the encoding/json import and swapping
json.Unmarshal calls with common.Unmarshal; specifically, update the code that
currently calls json.Unmarshal(...) in the relay WebSocket request parsing logic
to call common.Unmarshal(...) and adjust error handling accordingly, and ensure
the import list references the package providing common.Unmarshal instead of
"encoding/json".
| if c.Request.Method == http.MethodGet && | ||
| websocket.IsWebSocketUpgrade(c.Request) && | ||
| strings.HasPrefix(c.Request.URL.Path, "/v1/responses") { | ||
| modelRequest.Model = c.Query("model") | ||
| if strings.HasPrefix(c.Request.URL.Path, "/v1/responses/compact") && modelRequest.Model != "" { | ||
| modelRequest.Model = ratio_setting.WithCompactModelSuffix(modelRequest.Model) | ||
| } | ||
| return &modelRequest, false, nil |
There was a problem hiding this comment.
Defer token model checks until the first Responses WS frame is parsed.
Line 188 returns with shouldSelectChannel=false, but the outer Distribute flow still applies token model-limit checks before it looks at that flag. For restricted tokens, a client that sends model only in the first WebSocket frame will be rejected before setupResponsesWSChannel can validate the real model. Please skip the middleware model-limit branch for this path, or carry an explicit “defer model auth” flag into the post-upgrade flow.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@middleware/distributor.go` around lines 181 - 188, For WebSocket GET upgrades
to /v1/responses, defer model auth instead of letting the outer Distribute
middleware apply token model-limit checks early: add an explicit flag (e.g.,
DeferModelAuth bool) to the modelRequest struct, set modelRequest.DeferModelAuth
= true in the websocket upgrade branch (the block that handles
websocket.IsWebSocketUpgrade and returns &modelRequest), and update the
Distribute flow to check modelRequest.DeferModelAuth and skip the middleware
model-limit branch when true so setupResponsesWSChannel can validate the real
model from the first WS frame.
| if err := service.SettleBilling(c, info, info.FinalPreConsumedQuota); err != nil { | ||
| logger.LogError(c, "responses websocket settle billing failed: "+err.Error()) |
There was a problem hiding this comment.
This path never settles against actual streamed usage.
Line 79 passes info.FinalPreConsumedQuota into service.SettleBilling, and Lines 108-151 only relay frames; they never extract a terminal usage event to update that value. On successful sessions, the pre-consumed estimate therefore becomes the final bill, so any delta between reserved quota and actual usage is lost.
Also applies to: 108-151
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@relay/websocket.go` around lines 79 - 80, The billing settle call currently
uses info.FinalPreConsumedQuota (service.SettleBilling and
info.FinalPreConsumedQuota) which never gets updated because the frame-relay
code that forwards frames (the relay/forwarding loop handling streamed frames)
doesn't extract a terminal usage event; modify the relay/frames-handling logic
to detect the terminal usage/usage-summary frame (or track streamed tokens/bytes
usage as frames are processed), populate or compute the actual final usage value
(update info.FinalPreConsumedQuota or create a new finalUsage variable) before
calling service.SettleBilling, and then call service.SettleBilling with that
actual final usage so the final billing reflects real streamed consumption
rather than the pre-consumed estimate.
| if initialRequest.Type == "" { | ||
| initialRequest.Type = "response.create" | ||
| } |
There was a problem hiding this comment.
Always force the first upstream event to response.create.
Lines 91-93 only default Type when it is empty. If the client sends any other non-empty value, this helper will still forward that as the first upstream frame, which breaks the exact protocol guarantee this PR is trying to add. Reject non-response.create values or overwrite the field unconditionally here.
Minimal fix
initialRequest := *request
- if initialRequest.Type == "" {
- initialRequest.Type = "response.create"
- }
+ if initialRequest.Type != "" && initialRequest.Type != "response.create" {
+ return fmt.Errorf("first responses websocket message must have type %q", "response.create")
+ }
+ initialRequest.Type = "response.create"
converted, err := adaptor.ConvertOpenAIResponsesRequest(c, info, initialRequest)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if initialRequest.Type == "" { | |
| initialRequest.Type = "response.create" | |
| } | |
| initialRequest := *request | |
| if initialRequest.Type != "" && initialRequest.Type != "response.create" { | |
| return fmt.Errorf("first responses websocket message must have type %q", "response.create") | |
| } | |
| initialRequest.Type = "response.create" | |
| converted, err := adaptor.ConvertOpenAIResponsesRequest(c, info, initialRequest) |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@relay/websocket.go` around lines 91 - 93, The current logic only defaults
initialRequest.Type to "response.create" when empty, allowing a client-supplied
non-empty Type to slip through; change the behavior in the websocket handler so
the first upstream event always has Type "response.create" by unconditionally
setting initialRequest.Type = "response.create" (instead of only when empty) or,
if you prefer validation, explicitly reject any initialRequest.Type !=
"response.create"; update the code that inspects/forwards initialRequest (the
variable named initialRequest in websocket.go) so it no longer forwards
client-provided Types for the first frame.
|
赶紧加啊 |
|
粗略看了下，缺少了对usage的解析，没有计费逻辑；没有选择渠道类型为OpenAI和Codex的逻辑；这两个都是有必要的。 |
This comment was marked as spam.
This comment was marked as spam.
|
伟大，无需多言，管理员辛苦快点加一下，真的是个很有帮助的更新 |
4 participants
Summary
GET /v1/responsestype: "response.create"Problem
new-apihandled OpenAI Realtime over WebSocket, but Responses WS was incomplete:/v1/responseshad no WS routetype: "response.create"at the top levelThis caused the first request to downgrade or fail instead of staying on WebSocket when the upstream supported Responses WS.
Verification
go test ./controller ./middleware ./relay/... ./dto/v1/responsesover WS returns standard Responses events end-to-end against a CPA upstreamSummary by CodeRabbit
Typefield in Responses API requests.