feat: jimeng use openai sdk input_reference i2v - #2029
Conversation
WalkthroughImplements multipart form handling for input_reference files in jimeng adaptor. Reads uploaded files, base64-encodes contents into req.Images, and sets Action based on image count. Updates request key transformation and payload construction logic to use req.Images instead of ImageUrls. Validates request type and returns errors on invalid types. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant C as Client
participant A as JimengAdaptor
participant M as Multipart Parser
participant R as Req Transformer
participant B as Payload Builder
C->>A: BuildRequestBody(request, multipart form)
A->>A: Validate request type
A->>M: Read input_reference files
M-->>A: File streams
A->>A: Base64-encode files -> req.Images
alt Image count == 1
A->>R: Set Action=Generate
else Image count == 2
A->>R: Set Action=FirstTailGenerate
else Image count > 2
A->>R: Set Action=ReferenceGenerate
else No images
A->>R: Use existing fields
end
A->>B: convertToRequestPayload using req.Images
B-->>A: Payload
A-->>C: Payload / error
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested reviewers
Poem
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
relay/channel/task/jimeng/adaptor.go(3 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
relay/channel/task/jimeng/adaptor.go (2)
relay/common/relay_info.go (1)
TaskSubmitReq(485-494)constant/task.go (3)
TaskActionGenerate(14-14)TaskActionFirstTailGenerate(16-16)TaskActionReferenceGenerate(17-17)
🔇 Additional comments (3)
relay/channel/task/jimeng/adaptor.go (3)
7-7: LGTM!The base64 import is correctly added to support file encoding in the multipart form handler.
121-124: LGTM!The type assertion with error handling is a good defensive practice that ensures type safety before processing the request.
399-402: LGTM!The ReqKey transformation correctly uses
req.Imagesinstead ofImageUrls, which is consistent with the new multipart form handling that populatesreq.Images.
| if mf, err := c.MultipartForm(); err == nil { | ||
| if files, exists := mf.File["input_reference"]; exists && len(files) > 0 { | ||
| if len(files) == 1 { | ||
| info.Action = constant.TaskActionGenerate | ||
| } else if len(files) == 2 { | ||
| info.Action = constant.TaskActionFirstTailGenerate | ||
| } else if len(files) > 2 { | ||
| info.Action = constant.TaskActionReferenceGenerate | ||
| } | ||
|
|
||
| // 将上传的文件转换为base64格式 | ||
| var images []string | ||
| for _, fileHeader := range files { | ||
| file, err := fileHeader.Open() | ||
| if err != nil { | ||
| continue | ||
| } | ||
| fileBytes, err := io.ReadAll(file) | ||
| file.Close() | ||
| if err != nil { | ||
| continue | ||
| } | ||
| // 将文件内容转换为base64 | ||
| base64Str := base64.StdEncoding.EncodeToString(fileBytes) | ||
| images = append(images, base64Str) | ||
| } | ||
| req.Images = images | ||
| } | ||
| } |
There was a problem hiding this comment.
Critical: Silent error handling causes Action/Images mismatch.
The current implementation silently skips files that fail to open or read (lines 140-141, 145-147), which creates a critical data integrity issue. If 2 files are uploaded but one fails to process, Action will be set to TaskActionFirstTailGenerate based on the file count, but req.Images will contain only 1 image. This mismatch will cause incorrect downstream behavior.
Apply this diff to fail fast on file processing errors:
- // 将上传的文件转换为base64格式
var images []string
for _, fileHeader := range files {
file, err := fileHeader.Open()
if err != nil {
- continue
+ return nil, fmt.Errorf("failed to open uploaded file %s: %w", fileHeader.Filename, err)
}
fileBytes, err := io.ReadAll(file)
file.Close()
if err != nil {
- continue
+ return nil, fmt.Errorf("failed to read uploaded file %s: %w", fileHeader.Filename, err)
}
- // 将文件内容转换为base64
base64Str := base64.StdEncoding.EncodeToString(fileBytes)
images = append(images, base64Str)
}
+ if len(images) == 0 {
+ return nil, fmt.Errorf("no valid images processed from input_reference files")
+ }
req.Images = images
}
}Additionally, consider adding file size validation to prevent memory exhaustion:
const maxFileSize = 10 * 1024 * 1024 // 10MB
for _, fileHeader := range files {
if fileHeader.Size > maxFileSize {
return nil, fmt.Errorf("file %s exceeds maximum size of %d bytes", fileHeader.Filename, maxFileSize)
}
// ... rest of processing
}🤖 Prompt for AI Agents
In relay/channel/task/jimeng/adaptor.go around lines 126 to 154, the code
currently sets info.Action based on the raw uploaded file count and then
silently skips files that fail to open/read, causing Action vs req.Images
mismatches; change the logic to validate file sizes up front (e.g. reject
>10MB), attempt to open/read each file and immediately return an error if any
file fails to open or read (fail-fast), collect only after successful reads, and
finally set info.Action based on the number of successfully processed images
(not the original file slice length) so downstream consumers see consistent
Action and req.Images.
…t-oai-files feat: jimeng use openai sdk input_reference i2v
2 participants
增加即梦支持openai格式图生视频和首尾帧视频
openai上传本地图像文件, 会转化为base64编码以符合即梦格式
注意:
由于openai sdk限制只支持上传一个文件, 因此只支持图生视频
如果需要首尾帧生视频需要自己构造两个input_reference请求
Summary by CodeRabbit