feat(relay): add OpenAI Batch API passthrough - #6730
Conversation
WalkthroughAdds native OpenAI Files and Batches passthrough. The change adds resource persistence, pinned channel and key routing, request validation, relay routes, cleanup, and frontend configuration. ChangesNative OpenAI resource passthrough
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant PrepareOpenAIUpstreamResource
participant Distribute
participant RelayOpenAIUpstreamResource
participant OpenAIUpstream
Client->>PrepareOpenAIUpstreamResource: Submit file or batch request
PrepareOpenAIUpstreamResource->>Distribute: Pass model and resource routing metadata
Distribute->>RelayOpenAIUpstreamResource: Select channel and pinned key
RelayOpenAIUpstreamResource->>OpenAIUpstream: Forward request
OpenAIUpstream-->>RelayOpenAIUpstreamResource: Return response and resource IDs
RelayOpenAIUpstreamResource-->>Client: Forward filtered response
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment Warning |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (3)
middleware/distributor.go (1)
49-52: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUse an i18n key for the new abort message.
Every other abort in
Distributeusesi18n.T(c, ...). This message is a hardcoded English string. Add a key ini18n/keys.goand translate it for consistency with the surrounding error responses.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@middleware/distributor.go` around lines 49 - 52, The new abort message in Distribute is hardcoded instead of using localization. Add a dedicated key to i18n/keys.go, add its translations through the existing i18n mechanism, and pass i18n.T(c, ...) to abortWithOpenAiMessage while preserving the current forbidden response and message meaning.model/channel_cache.go (1)
231-236: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winPrecompute native batch support like the Advanced Custom config.
SupportsNativeOpenAIBatch()callsGetOtherSettings(), which unmarshals the channelsettingsJSON on every call. This runs for every candidate channel on every File/Batch request. The Advanced Custom branch avoids this by reading the precomputedchannel2advancedCustomConfigmap that the cache build populates.
GetOtherSettings()also writes tochannel.OtherSettingsand callschannel.Save()when the JSON is malformed. The*Channelvalues inchannelsIDMare shared across requests, so that fallback mutates shared cache state from concurrent request goroutines.Add a
channel2nativeOpenAIBatch map[int]boolpopulated during cache initialization and read it here.Run the following script to confirm where the channel caches are built:
#!/bin/bash rg -n -C5 'channel2advancedCustomConfig' model/🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@model/channel_cache.go` around lines 231 - 236, Add a channel2nativeOpenAIBatch map alongside channel2advancedCustomConfig, populate it during channel cache initialization using each channel’s native batch capability, and read this precomputed map in the IsOpenAIUpstreamResourcePath branch instead of calling SupportsNativeOpenAIBatch(). Ensure the map is initialized and refreshed with the existing cache lifecycle.middleware/openai_upstream_resource_test.go (1)
364-400: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd a case for the token-specific channel conflict.
Distributenow returns 403 when a token pins a channel that does not own the requested resource (middleware/distributor.golines 49-52). No test covers that branch. Add a case that setsconstant.ContextKeyTokenSpecificChannelIdto a different channel id and asserts 403.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@middleware/openai_upstream_resource_test.go` around lines 364 - 400, Extend TestPrepareOpenAIUpstreamResourceRejectsMissingPinnedKey with a token-specific channel conflict case: set constant.ContextKeyTokenSpecificChannelId to a different channel ID in the request context, invoke the same middleware route, and assert that Distribute returns HTTP 403 without reaching the handled callback.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@controller/openai_upstream_resource.go`:
- Around line 192-206: Update the bindOpenAIUpstreamResourceResponse failure
branch to log the upstream resource ID and bindErr before calling
openAIUpstreamResourceError and returning. Reuse the existing upstream resource
identifier from the handler’s response context, preserving the current 502
behavior.
In `@middleware/distributor.go`:
- Around line 181-193: Guard both the channel request-path validation and
SetupContextForSelectedChannel error handling in Distribute so they run only
when channel is non-nil. Preserve the existing behavior for selected channels
while allowing fetch-style routes with shouldSelectChannel == false and no
channel to continue without either abort.
---
Nitpick comments:
In `@middleware/distributor.go`:
- Around line 49-52: The new abort message in Distribute is hardcoded instead of
using localization. Add a dedicated key to i18n/keys.go, add its translations
through the existing i18n mechanism, and pass i18n.T(c, ...) to
abortWithOpenAiMessage while preserving the current forbidden response and
message meaning.
In `@middleware/openai_upstream_resource_test.go`:
- Around line 364-400: Extend
TestPrepareOpenAIUpstreamResourceRejectsMissingPinnedKey with a token-specific
channel conflict case: set constant.ContextKeyTokenSpecificChannelId to a
different channel ID in the request context, invoke the same middleware route,
and assert that Distribute returns HTTP 403 without reaching the handled
callback.
In `@model/channel_cache.go`:
- Around line 231-236: Add a channel2nativeOpenAIBatch map alongside
channel2advancedCustomConfig, populate it during channel cache initialization
using each channel’s native batch capability, and read this precomputed map in
the IsOpenAIUpstreamResourcePath branch instead of calling
SupportsNativeOpenAIBatch(). Ensure the map is initialized and refreshed with
the existing cache lifecycle.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 9e0e6099-3717-465d-94d5-bc67e1209df5
📒 Files selected for processing (34)
constant/context_key.gocontroller/openai_upstream_resource.gocontroller/openai_upstream_resource_test.gomiddleware/distributor.gomiddleware/openai_upstream_resource.gomiddleware/openai_upstream_resource_test.gomodel/ability.gomodel/channel.gomodel/channel_cache.gomodel/main.gomodel/openai_upstream_resource.gomodel/openai_upstream_resource_test.gorelaykit/dto/channel_settings.gorouter/openai_upstream_resource_test.gorouter/relay-router.gosetting/operation_setting/openai_batch_setting.goweb/src/features/channels/components/drawers/channel-mutate-drawer.tsxweb/src/features/channels/lib/__tests__/openai-batch-settings.test.tsweb/src/features/channels/lib/channel-form-errors.tsweb/src/features/channels/lib/channel-form.tsweb/src/features/channels/types.tsweb/src/features/system-settings/general/__tests__/system-behavior-settings.test.tsweb/src/features/system-settings/general/system-behavior-section.tsxweb/src/features/system-settings/general/system-behavior-settings.tsweb/src/features/system-settings/operations/index.tsxweb/src/features/system-settings/operations/section-registry.tsxweb/src/features/system-settings/types.tsweb/src/i18n/locales/en.jsonweb/src/i18n/locales/fr.jsonweb/src/i18n/locales/ja.jsonweb/src/i18n/locales/ru.jsonweb/src/i18n/locales/vi.jsonweb/src/i18n/locales/zh-TW.jsonweb/src/i18n/locales/zh.json
| if shouldBind { | ||
| body, readErr := io.ReadAll(response.Body) | ||
| if readErr != nil { | ||
| openAIUpstreamResourceError(c, http.StatusBadGateway, "failed to read upstream response") | ||
| return | ||
| } | ||
| if bindErr := bindOpenAIUpstreamResourceResponse(c, body); bindErr != nil { | ||
| openAIUpstreamResourceError(c, http.StatusBadGateway, "failed to persist upstream resource binding") | ||
| return | ||
| } | ||
| copyOpenAIUpstreamResponseHeaders(c.Writer.Header(), response.Header) | ||
| c.Status(response.StatusCode) | ||
| _, _ = c.Writer.Write(body) | ||
| return | ||
| } |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Log the upstream resource id when binding fails.
The upstream file or batch already exists when bindOpenAIUpstreamResourceResponse fails. The handler returns 502 and discards the body, so the client never learns the upstream id. The resource then stays upstream with no way to reference or delete it through the gateway.
Log the upstream id and the binding error before returning, so operators can reconcile orphaned resources.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@controller/openai_upstream_resource.go` around lines 192 - 206, Update the
bindOpenAIUpstreamResourceResponse failure branch to log the upstream resource
ID and bindErr before calling openAIUpstreamResourceError and returning. Reuse
the existing upstream resource identifier from the handler’s response context,
preserving the current 502 behavior.
| if !channelSupportsRequestPath(channel, c.Request.URL.Path, modelRequest.Model) { | ||
| abortWithOpenAiMessage(c, http.StatusServiceUnavailable, "selected channel does not support this request path") | ||
| return | ||
| } | ||
| common.SetContextKey(c, constant.ContextKeyRequestStartTime, time.Now()) | ||
| SetupContextForSelectedChannel(c, channel, modelRequest.Model) | ||
| if setupErr := SetupContextForSelectedChannel(c, channel, modelRequest.Model); setupErr != nil { | ||
| statusCode := setupErr.StatusCode | ||
| if statusCode < http.StatusBadRequest || setupErr.GetErrorCode() == types.ErrorCodeChannelNoAvailableKey { | ||
| statusCode = http.StatusServiceUnavailable | ||
| } | ||
| abortWithOpenAiMessage(c, statusCode, setupErr.Error(), setupErr.GetErrorCode()) | ||
| return | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🔴 Critical | 🏗️ Heavy lift
Guard the new checks when no channel was selected.
getModelRequest returns shouldSelectChannel == false for fetch-style routes (Midjourney task fetch, Suno fetch, video fetch by id, video remix). For those routes channel stays nil.
Line 181 now calls channelSupportsRequestPath(channel, ...), which returns false for a nil channel, so the request aborts with 503. Line 186 also now propagates the channel is nil error from SetupContextForSelectedChannel. Both aborts are new; the previous code ignored the setup error and continued. This breaks all task-fetch endpoints that route through Distribute().
Skip both checks when channel == nil.
🐛 Proposed fix
- if !channelSupportsRequestPath(channel, c.Request.URL.Path, modelRequest.Model) {
- abortWithOpenAiMessage(c, http.StatusServiceUnavailable, "selected channel does not support this request path")
- return
- }
- common.SetContextKey(c, constant.ContextKeyRequestStartTime, time.Now())
- if setupErr := SetupContextForSelectedChannel(c, channel, modelRequest.Model); setupErr != nil {
- statusCode := setupErr.StatusCode
- if statusCode < http.StatusBadRequest || setupErr.GetErrorCode() == types.ErrorCodeChannelNoAvailableKey {
- statusCode = http.StatusServiceUnavailable
- }
- abortWithOpenAiMessage(c, statusCode, setupErr.Error(), setupErr.GetErrorCode())
- return
- }
+ if channel != nil {
+ if !channelSupportsRequestPath(channel, c.Request.URL.Path, modelRequest.Model) {
+ abortWithOpenAiMessage(c, http.StatusServiceUnavailable, "selected channel does not support this request path")
+ return
+ }
+ }
+ common.SetContextKey(c, constant.ContextKeyRequestStartTime, time.Now())
+ if channel != nil {
+ if setupErr := SetupContextForSelectedChannel(c, channel, modelRequest.Model); setupErr != nil {
+ statusCode := setupErr.StatusCode
+ if statusCode < http.StatusBadRequest || setupErr.GetErrorCode() == types.ErrorCodeChannelNoAvailableKey {
+ statusCode = http.StatusServiceUnavailable
+ }
+ abortWithOpenAiMessage(c, statusCode, setupErr.Error(), setupErr.GetErrorCode())
+ return
+ }
+ }Run the following script to confirm which routes reach Distribute() without selecting a channel:
#!/bin/bash
# List routes that use Distribute and the shouldSelectChannel=false branches.
rg -n -C3 'shouldSelectChannel = false' middleware/distributor.go
rg -n 'middleware.Distribute\(\)' router/🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@middleware/distributor.go` around lines 181 - 193, Guard both the channel
request-path validation and SetupContextForSelectedChannel error handling in
Distribute so they run only when channel is non-nil. Preserve the existing
behavior for selected channels while allowing fetch-style routes with
shouldSelectChannel == false and no channel to continue without either abort.
1 participant
Important
AI-assisted: the implementation, tests, review, and this description were prepared with Codex. The "人工确认" item remains unchecked intentionally.
📝 变更描述 / Description
Add native OpenAI-compatible File and Batch API passthrough so image models such as
gpt-image-2can be used through Batch JSONL requests to/v1/images/generationsand/v1/images/edits.This adds the core File/Batch workflow:
POST /v1/files,GET /v1/files/:id,GET /v1/files/:id/content,DELETE /v1/files/:idPOST /v1/batches,GET /v1/batches/:id,POST /v1/batches/:id/cancelcustom_id, a single non-empty model, supported POST endpoints, and a 50,000-request limit🔒 Rollout and limitations
openai_batch_setting.enabledand the OpenAI channel'snative_openai_batchopt-in.GET /v1/filesandGET /v1/batchesremain unsupported.🚀 变更类型 / Type of change
🔗 关联任务 / Related Issue
Related to #6535 (OpenAI scope only). Gemini Batch API is outside this PR.
✅ 提交前检查项 / Checklist
🧪 验证 / Testing
Passed:
go test ./model ./middleware ./controller ./router -count=1go vet ./model ./middleware ./controller ./routerbun test src/features/system-settings/general/__tests__/system-behavior-settings.test.ts src/features/channels/lib/__tests__/openai-batch-settings.test.ts(2 pass, 0 fail)bun run typecheckbun run build:checkoxlintfor every changed TypeScript/TSX filegit diff --checkFull-suite baseline notes:
go test ./... -count=1still reproduces three failures on a cleanorigin/mainworktree: the Windows HTTP/2 GOAWAY retry test and twoservicechannel-affinity metric-count tests.bun run format:checkstill reports five pre-existing files; none are modified by this PR.📸 运行证明 / Proof of Work
Summary by CodeRabbit
New Features
Configuration