feat(gemini): implement video generation configuration - #3059
Merged
Conversation
…timation - Added Gemini video generation configuration structures and payloads. - Introduced functions for parsing and resolving video duration and resolution from metadata. - Enhanced the Vertex adaptor to support Gemini video generation requests and billing estimation based on duration and resolution. - Updated model pricing settings for new Gemini video models.
Contributor
WalkthroughRefactors Gemini and Vertex video generation adaptors with new request/response payload structures, introduces image handling utilities and billing estimation logic, expands model support, and updates pricing configuration. Changes
Sequence Diagram(s)sequenceDiagram
actor Client
participant Adaptor as Gemini Adaptor
participant ImgHandler as Image Handler
participant Billing as Billing Utils
participant GeminiAPI as Gemini API
Client->>Adaptor: Video Generation Request
alt Image Provided
Adaptor->>ImgHandler: ExtractMultipartImage()
ImgHandler->>ImgHandler: Read & Encode (20MB limit)
ImgHandler-->>Adaptor: VeoImageInput
end
Adaptor->>Billing: ResolveVeoDuration()
Billing-->>Adaptor: Duration (seconds)
Adaptor->>Billing: ResolveVeoResolution()
Billing-->>Adaptor: Resolution (normalized)
Adaptor->>Adaptor: BuildRequestBody()
Adaptor->>Adaptor: Create payload with Config + optional Image
Adaptor->>GeminiAPI: POST /generateVideos
GeminiAPI-->>Adaptor: Operation Response (GeneratedVideos)
Adaptor->>Adaptor: FetchTask parsing
Adaptor->>Adaptor: Extract RemoteUrl from first GeneratedVideos
Adaptor-->>Client: Video Task Result
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches
🧪 Generate unit tests (beta)
Tip Try Coding Plans. Let us write the prompt for your AI agent so you can ship faster (with fewer bugs). Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
relay/channel/task/vertex/adaptor.go (1)
181-197:⚠️ Potential issue | 🟡 MinorMissing type assertion check can cause panic.
At line 187, the type assertion
v.(relaycommon.TaskSubmitReq)will panic ifvis not of typeTaskSubmitReq. This is inconsistent with the safe assertion pattern used in the Gemini adaptor at line 74-77.🛡️ Proposed fix
func (a *TaskAdaptor) BuildRequestBody(c *gin.Context, info *relaycommon.RelayInfo) (io.Reader, error) { v, ok := c.Get("task_request") if !ok { return nil, fmt.Errorf("request not found in context") } - req := v.(relaycommon.TaskSubmitReq) + req, ok := v.(relaycommon.TaskSubmitReq) + if !ok { + return nil, fmt.Errorf("unexpected task_request type") + } instance := veoInstance{Prompt: req.Prompt}🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@relay/channel/task/vertex/adaptor.go` around lines 181 - 197, The BuildRequestBody method currently does an unsafe type assertion v.(relaycommon.TaskSubmitReq) which can panic; update the code in BuildRequestBody to use a safe comma-ok type assertion (e.g., req, ok := v.(relaycommon.TaskSubmitReq)) and return a descriptive error if ok is false, mirroring the safe pattern used in the Gemini adaptor; ensure references to req, instance := veoInstance{Prompt: req.Prompt}, and subsequent logic (image extraction via geminitask.ExtractMultipartImage and geminitask.ParseImageInput) only run after the successful assertion.
🧹 Nitpick comments (1)
relay/channel/task/gemini/image.go (1)
77-99: Consider validating the base64 payload inparseDataURI.Unlike
ParseImageInput(which decodes and validates base64 at line 67-70),parseDataURIdoesn't validate that the extracted payload is valid base64. This could lead to invalid data being passed to the API.🔧 Proposed validation
func parseDataURI(uri string) *VeoImageInput { // data:image/png;base64,iVBOR... rest := uri[len("data:"):] idx := strings.Index(rest, ",") if idx < 0 { return nil } meta := rest[:idx] b64 := rest[idx+1:] if b64 == "" { return nil } + // Validate base64 payload + if _, err := base64.StdEncoding.DecodeString(b64); err != nil { + return nil + } mimeType := "application/octet-stream" parts := strings.SplitN(meta, ";", 2) if len(parts) >= 1 && parts[0] != "" { mimeType = parts[0] } return &VeoImageInput{ BytesBase64Encoded: b64, MimeType: mimeType, } }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@relay/channel/task/gemini/image.go` around lines 77 - 99, The parseDataURI function currently extracts the base64 payload into BytesBase64Encoded without validating it; update parseDataURI to attempt a base64.StdEncoding (or base64.RawStdEncoding as appropriate) DecodeString of b64 and return nil if decoding fails (mirroring ParseImageInput behavior), and keep BytesBase64Encoded only for valid payloads (or populate MimeType and BytesBase64Encoded only after successful decode) so invalid base64 data is rejected before creating a VeoImageInput.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@relay/channel/task/vertex/adaptor.go`:
- Around line 163-179: In TaskAdaptor.EstimateBilling, avoid the unsafe type
assertion v.(relaycommon.TaskSubmitReq) — first perform a checked assertion
(e.g., req, ok := v.(relaycommon.TaskSubmitReq)) and handle the case where ok is
false by returning nil (or an appropriate default) instead of letting it panic;
update references to req in ResolveVeoDuration/ResolveVeoResolution accordingly
so the method safely handles missing or wrong-typed "task_request" values.
---
Outside diff comments:
In `@relay/channel/task/vertex/adaptor.go`:
- Around line 181-197: The BuildRequestBody method currently does an unsafe type
assertion v.(relaycommon.TaskSubmitReq) which can panic; update the code in
BuildRequestBody to use a safe comma-ok type assertion (e.g., req, ok :=
v.(relaycommon.TaskSubmitReq)) and return a descriptive error if ok is false,
mirroring the safe pattern used in the Gemini adaptor; ensure references to req,
instance := veoInstance{Prompt: req.Prompt}, and subsequent logic (image
extraction via geminitask.ExtractMultipartImage and geminitask.ParseImageInput)
only run after the successful assertion.
---
Nitpick comments:
In `@relay/channel/task/gemini/image.go`:
- Around line 77-99: The parseDataURI function currently extracts the base64
payload into BytesBase64Encoded without validating it; update parseDataURI to
attempt a base64.StdEncoding (or base64.RawStdEncoding as appropriate)
DecodeString of b64 and return nil if decoding fails (mirroring ParseImageInput
behavior), and keep BytesBase64Encoded only for valid payloads (or populate
MimeType and BytesBase64Encoded only after successful decode) so invalid base64
data is rejected before creating a VeoImageInput.
ℹ️ Review info
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (7)
dto/openai_video.gorelay/channel/task/gemini/adaptor.gorelay/channel/task/gemini/billing.gorelay/channel/task/gemini/dto.gorelay/channel/task/gemini/image.gorelay/channel/task/vertex/adaptor.gosetting/ratio_setting/model_ratio.go
Comment on lines
+163
to
179
| // EstimateBilling returns OtherRatios based on durationSeconds and resolution. | ||
| func (a *TaskAdaptor) EstimateBilling(c *gin.Context, info *relaycommon.RelayInfo) map[string]float64 { | ||
| v, ok := c.Get("task_request") | ||
| if ok { | ||
| req := v.(relaycommon.TaskSubmitReq) | ||
| if req.Metadata != nil { | ||
| if sc, exists := req.Metadata["sampleCount"]; exists { | ||
| if i, ok := sc.(int); ok && i > 0 { | ||
| sampleCount = i | ||
| } | ||
| if f, ok := sc.(float64); ok && int(f) > 0 { | ||
| sampleCount = int(f) | ||
| } | ||
| } | ||
| } | ||
| if !ok { | ||
| return nil | ||
| } | ||
| req := v.(relaycommon.TaskSubmitReq) | ||
|
|
||
| seconds := geminitask.ResolveVeoDuration(req.Metadata, req.Duration, req.Seconds) | ||
| resolution := geminitask.ResolveVeoResolution(req.Metadata, req.Size) | ||
| resRatio := geminitask.VeoResolutionRatio(info.UpstreamModelName, resolution) | ||
|
|
||
| return map[string]float64{ | ||
| "sampleCount": float64(sampleCount), | ||
| "seconds": float64(seconds), | ||
| "resolution": resRatio, | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
Missing type assertion check can cause panic.
At line 169, the type assertion v.(relaycommon.TaskSubmitReq) will panic if v is not of type TaskSubmitReq. While line 166 checks if the key exists, the value type is not verified.
🛡️ Proposed fix
func (a *TaskAdaptor) EstimateBilling(c *gin.Context, info *relaycommon.RelayInfo) map[string]float64 {
v, ok := c.Get("task_request")
if !ok {
return nil
}
- req := v.(relaycommon.TaskSubmitReq)
+ req, ok := v.(relaycommon.TaskSubmitReq)
+ if !ok {
+ return nil
+ }
seconds := geminitask.ResolveVeoDuration(req.Metadata, req.Duration, req.Seconds)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| // EstimateBilling returns OtherRatios based on durationSeconds and resolution. | |
| func (a *TaskAdaptor) EstimateBilling(c *gin.Context, info *relaycommon.RelayInfo) map[string]float64 { | |
| v, ok := c.Get("task_request") | |
| if ok { | |
| req := v.(relaycommon.TaskSubmitReq) | |
| if req.Metadata != nil { | |
| if sc, exists := req.Metadata["sampleCount"]; exists { | |
| if i, ok := sc.(int); ok && i > 0 { | |
| sampleCount = i | |
| } | |
| if f, ok := sc.(float64); ok && int(f) > 0 { | |
| sampleCount = int(f) | |
| } | |
| } | |
| } | |
| if !ok { | |
| return nil | |
| } | |
| req := v.(relaycommon.TaskSubmitReq) | |
| seconds := geminitask.ResolveVeoDuration(req.Metadata, req.Duration, req.Seconds) | |
| resolution := geminitask.ResolveVeoResolution(req.Metadata, req.Size) | |
| resRatio := geminitask.VeoResolutionRatio(info.UpstreamModelName, resolution) | |
| return map[string]float64{ | |
| "sampleCount": float64(sampleCount), | |
| "seconds": float64(seconds), | |
| "resolution": resRatio, | |
| } | |
| } | |
| // EstimateBilling returns OtherRatios based on durationSeconds and resolution. | |
| func (a *TaskAdaptor) EstimateBilling(c *gin.Context, info *relaycommon.RelayInfo) map[string]float64 { | |
| v, ok := c.Get("task_request") | |
| if !ok { | |
| return nil | |
| } | |
| req, ok := v.(relaycommon.TaskSubmitReq) | |
| if !ok { | |
| return nil | |
| } | |
| seconds := geminitask.ResolveVeoDuration(req.Metadata, req.Duration, req.Seconds) | |
| resolution := geminitask.ResolveVeoResolution(req.Metadata, req.Size) | |
| resRatio := geminitask.VeoResolutionRatio(info.UpstreamModelName, resolution) | |
| return map[string]float64{ | |
| "seconds": float64(seconds), | |
| "resolution": resRatio, | |
| } | |
| } |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@relay/channel/task/vertex/adaptor.go` around lines 163 - 179, In
TaskAdaptor.EstimateBilling, avoid the unsafe type assertion
v.(relaycommon.TaskSubmitReq) — first perform a checked assertion (e.g., req, ok
:= v.(relaycommon.TaskSubmitReq)) and handle the case where ok is false by
returning nil (or an appropriate default) instead of letting it panic; update
references to req in ResolveVeoDuration/ResolveVeoResolution accordingly so the
method safely handles missing or wrong-typed "task_request" values.
ennnnny
pushed a commit
to ennnnny/new-api
that referenced
this pull request
Mar 17, 2026
feat(gemini): implement video generation configuration
11 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by CodeRabbit
New Features
Chores