feat: gemini-3-pro - #2243
Conversation
WalkthroughAdded new JSON payload fields to Gemini DTOs (four fields to Changes
Sequence Diagram(s)(omitted — changes are schema/constant updates without control-flow modifications) Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes
Possibly related PRs
Poem
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Tip 📝 Customizable high-level summaries are now available in beta!You can now customize how CodeRabbit generates the high-level summary in your pull requests — including its content, structure, tone, and formatting.
Example instruction:
Note: This feature is currently in beta for Pro-tier users, and pricing will be announced later. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
dto/gemini.go (1)
208-218: ChangeThoughtSignaturefromjson.RawMessagetostringThe Gemini schema documents
thoughtSignatureas a string type, not arbitrary JSON. Update the field definition indto/gemini.goline 213 from:ThoughtSignature json.RawMessage `json:"thoughtSignature,omitempty"`to:
ThoughtSignature string `json:"thoughtSignature,omitempty"`
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
dto/gemini.go(2 hunks)relay/channel/gemini/constant.go(1 hunks)
🧰 Additional context used
🧠 Learnings (2)
📚 Learning: 2025-06-21T03:37:41.726Z
Learnt from: 9Ninety
Repo: QuantumNous/new-api PR: 1273
File: relay/channel/gemini/relay-gemini.go:97-116
Timestamp: 2025-06-21T03:37:41.726Z
Learning: In relay/channel/gemini/relay-gemini.go, the thinking budget calculation logic (including the MaxOutputTokens multiplication) was introduced in PR #1247. PR #1273 focused specifically on decoupling the thoughts summary feature from thinking budget settings and did not modify the existing thinking budget behavior.
Applied to files:
relay/channel/gemini/constant.godto/gemini.go
📚 Learning: 2025-08-08T17:12:43.157Z
Learnt from: RedwindA
Repo: QuantumNous/new-api PR: 1537
File: relay/gemini_handler.go:330-342
Timestamp: 2025-08-08T17:12:43.157Z
Learning: In the new-api repository, the `GeminiEmbeddingHandler` function in `relay/gemini_handler.go` is designed specifically for native Gemini embedding requests and therefore does not require the `ConvertGeminiRequest` step that is used in the chat handler. The embedding requests are already in the native Gemini format and don't need conversion.
Applied to files:
relay/channel/gemini/constant.godto/gemini.go
🔇 Additional comments (1)
relay/channel/gemini/constant.go (1)
3-28: Newgemini-3-pro-previewmodel entry looks consistentThe added
"gemini-3-pro-preview"entry is correctly grouped under the preview models and follows the existing naming convention. No issues from this file’s perspective.
| type GeminiFunctionResponse struct { | ||
| Name string `json:"name"` | ||
| Response map[string]interface{} `json:"response"` | ||
| Name string `json:"name"` | ||
| Response map[string]interface{} `json:"response"` | ||
| WillContinue json.RawMessage `json:"willContinue,omitempty"` | ||
| Scheduling json.RawMessage `json:"scheduling,omitempty"` | ||
| Parts json.RawMessage `json:"parts,omitempty"` | ||
| ID json.RawMessage `json:"id,omitempty"` | ||
| } |
There was a problem hiding this comment.
🧩 Analysis chain
Additions to GeminiFunctionResponse are structurally sound; verify field names vs Gemini spec
Using json.RawMessage for WillContinue, Scheduling, Parts, and ID is consistent with the existing pattern of treating provider‑specific metadata as opaque JSON. Tags with omitempty avoid emitting unused fields.
Please double‑check that the JSON keys (willContinue, scheduling, parts, id) exactly match what Gemini 3 returns, since any mismatch would silently drop these fields during unmarshalling.
🌐 Web query:
Confirm the exact JSON field names and shapes for Gemini 3 function/tool response metadata (willContinue, scheduling, parts, id) in the latest Gemini API documentation.
💡 Result:
Summary of exact JSON field names and shapes (Gemini / Generative Language API):
-
id — string. Optional. Unique id of the function response. [1]
-
name — string. Required (function name). [1]
-
response — object (JSON object / Struct). Required. Arbitrary JSON containing the function's output. [1]
-
parts — array of FunctionResponsePart objects. Optional. Each part is an object with a data union (e.g., inlineData: { mimeType: string, data: string }) or other part types. [1]
-
willContinue — boolean. Optional. Signals the function call continues (generator behavior) for NON_BLOCKING functions; false ends the generator. (Docs also show it may be nullable in some SDK schemas.) [1][2]
-
scheduling — enum (Scheduling). Optional. Allowed enum values: SCHEDULING_UNSPECIFIED, SILENT, WHEN_IDLE, INTERRUPT. Controls how the response is scheduled for NON_BLOCKING calls. [1]
References
- Google Generative Language / Gemini API — FunctionResponse and related types. [1]
- Google ADK / SDK schema excerpt (shows willContinue nullable / schema shape). [2]
Struct field types misaligned with Gemini spec—willContinue and scheduling should be concrete types, not json.RawMessage
The JSON field names (willContinue, scheduling, parts, id) correctly match the Gemini API. However, the types are problematic:
willContinueshould bebool, notjson.RawMessage(spec: boolean)schedulingshould bestring, notjson.RawMessage(spec: enum)partsshould be[]FunctionResponsePartor similar, notjson.RawMessage(spec: array of objects)idcan remainjson.RawMessageif intentionally opaque, but spec shows it as simplestring
Using json.RawMessage for well-defined fields defers type checking to runtime and requires manual unmarshalling downstream, increasing error risk.
🤖 Prompt for AI Agents
In dto/gemini.go around lines 184-191, several fields use json.RawMessage but
should use concrete types per the Gemini spec: change WillContinue from
json.RawMessage to bool, Scheduling from json.RawMessage to string (or a defined
enum type), Parts from json.RawMessage to a slice of the appropriate struct
(e.g., []FunctionResponsePart) so it is unmarshalled automatically, and change
ID to string (or keep as json.RawMessage only if intentionally opaque); update
any imports/usages and adjust downstream code that relied on manual
json.RawMessage unmarshalling accordingly.
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
dto/gemini.go (1)
189-192: Field types remain misaligned with Gemini API specification.These fields still use
json.RawMessageinstead of concrete types, which was flagged in a previous review. Per the Gemini API specification:
WillContinueshould bebool(notjson.RawMessage)Schedulingshould bestringor enum type (notjson.RawMessage)Partsshould be a typed slice (notjson.RawMessage)IDshould bestring(notjson.RawMessage)Using
json.RawMessagefor well-defined fields defers type checking to runtime and requires manual unmarshalling downstream, increasing error risk and reducing type safety.Apply this diff to use concrete types:
type GeminiFunctionResponse struct { Name string `json:"name"` Response map[string]interface{} `json:"response"` - WillContinue json.RawMessage `json:"willContinue,omitempty"` - Scheduling json.RawMessage `json:"scheduling,omitempty"` - Parts json.RawMessage `json:"parts,omitempty"` - ID json.RawMessage `json:"id,omitempty"` + WillContinue bool `json:"willContinue,omitempty"` + Scheduling string `json:"scheduling,omitempty"` + Parts []GeminiPart `json:"parts,omitempty"` + ID string `json:"id,omitempty"` }
🧹 Nitpick comments (1)
dto/gemini.go (1)
144-145: Remove or clarify the vague TODO comment—ThinkingLevel was never used.Script verification shows
ThinkingLevelhas no references anywhere in the codebase. This means commenting it out is not a breaking change. However, the TODO comment remains unclear: either remove the dead field entirely or explain what the conflict withthinkingBudgetactually is.All existing usages of
GeminiThinkingConfig(inrelay/channel/gemini/relay-gemini.go) only setThinkingBudgetandIncludeThoughts—neverThinkingLevel.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
dto/gemini.go(3 hunks)
🧰 Additional context used
🧠 Learnings (2)
📚 Learning: 2025-06-21T03:37:41.726Z
Learnt from: 9Ninety
Repo: QuantumNous/new-api PR: 1273
File: relay/channel/gemini/relay-gemini.go:97-116
Timestamp: 2025-06-21T03:37:41.726Z
Learning: In relay/channel/gemini/relay-gemini.go, the thinking budget calculation logic (including the MaxOutputTokens multiplication) was introduced in PR #1247. PR #1273 focused specifically on decoupling the thoughts summary feature from thinking budget settings and did not modify the existing thinking budget behavior.
Applied to files:
dto/gemini.go
📚 Learning: 2025-08-08T17:12:43.157Z
Learnt from: RedwindA
Repo: QuantumNous/new-api PR: 1537
File: relay/gemini_handler.go:330-342
Timestamp: 2025-08-08T17:12:43.157Z
Learning: In the new-api repository, the `GeminiEmbeddingHandler` function in `relay/gemini_handler.go` is designed specifically for native Gemini embedding requests and therefore does not require the `ConvertGeminiRequest` step that is used in the chat handler. The embedding requests are already in the native Gemini format and don't need conversion.
Applied to files:
dto/gemini.go
🧬 Code graph analysis (1)
dto/gemini.go (1)
dto/openai_response.go (1)
FunctionResponse(133-139)
| ThoughtSignature json.RawMessage `json:"thoughtSignature,omitempty"` | ||
| FunctionResponse *GeminiFunctionResponse `json:"functionResponse,omitempty"` | ||
| // Optional. Media resolution for the input media. | ||
| MediaResolution json.RawMessage `json:"mediaResolution,omitempty"` | ||
| VideoMetadata json.RawMessage `json:"videoMetadata,omitempty"` |
There was a problem hiding this comment.
🧩 Analysis chain
Verify field types against Gemini API spec—MediaResolution type is inconsistent.
Three new fields use json.RawMessage, following the same pattern flagged in GeminiFunctionResponse. More critically, MediaResolution at line 218 conflicts with the existing MediaResolution type definition used in GeminiChatGenerationConfig at line 284 (defined as MediaResolution string at line 292).
Using json.RawMessage for potentially well-defined fields reduces type safety and requires manual unmarshalling.
Please verify the correct types for these fields in the Gemini 3 API specification:
If MediaResolution should use the existing MediaResolution type, apply this change:
- MediaResolution json.RawMessage `json:"mediaResolution,omitempty"`
+ MediaResolution MediaResolution `json:"mediaResolution,omitempty"`🌐 Web query:
What are the exact field types for thoughtSignature, mediaResolution, and videoMetadata in Gemini 3 API GeminiPart response structure?
💡 Result:
- thoughtSignature — binary (byte array); in JSON responses it's base64-encoded (Go: []byte). [1][2]
- mediaResolution — enum / structured enum field (MediaResolution / PartMediaResolution with levels like MEDIA_RESOLUTION_LOW / MEDIUM / HIGH). In JSON it's the enum name (string). [3][1]
- videoMetadata — object (VideoMetadata structure) containing video-specific fields (timestamps, fps, mime, duration, etc.). [2][4]
Sources: Go schema showing Part and thoughtSignature/videoMetadata. [1][2]; Gemini API docs for media_resolution enum/details. [3]; video understanding docs for VideoMetadata details. [4]
Replace json.RawMessage with concrete types per Gemini 3 API spec.
The three new fields are defined with incorrect types. According to the Gemini 3 API specification:
thoughtSignatureshould be[]byte(binary, base64-encoded in JSON)mediaResolutionshould be the enum typeMediaResolution(as already defined at line 292)videoMetadatashould be theVideoMetadatastruct
Using json.RawMessage sacrifices type safety and requires manual unmarshalling. Replace all three:
- ThoughtSignature json.RawMessage `json:"thoughtSignature,omitempty"`
+ ThoughtSignature []byte `json:"thoughtSignature,omitempty"`
FunctionResponse *GeminiFunctionResponse `json:"functionResponse,omitempty"`
// Optional. Media resolution for the input media.
- MediaResolution json.RawMessage `json:"mediaResolution,omitempty"`
- VideoMetadata json.RawMessage `json:"videoMetadata,omitempty"`
+ MediaResolution MediaResolution `json:"mediaResolution,omitempty"`
+ VideoMetadata *VideoMetadata `json:"videoMetadata,omitempty"`📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| ThoughtSignature json.RawMessage `json:"thoughtSignature,omitempty"` | |
| FunctionResponse *GeminiFunctionResponse `json:"functionResponse,omitempty"` | |
| // Optional. Media resolution for the input media. | |
| MediaResolution json.RawMessage `json:"mediaResolution,omitempty"` | |
| VideoMetadata json.RawMessage `json:"videoMetadata,omitempty"` | |
| ThoughtSignature []byte `json:"thoughtSignature,omitempty"` | |
| FunctionResponse *GeminiFunctionResponse `json:"functionResponse,omitempty"` | |
| // Optional. Media resolution for the input media. | |
| MediaResolution MediaResolution `json:"mediaResolution,omitempty"` | |
| VideoMetadata *VideoMetadata `json:"videoMetadata,omitempty"` |
🤖 Prompt for AI Agents
In dto/gemini.go around lines 215 to 219, the fields thoughtSignature,
mediaResolution and videoMetadata are typed as json.RawMessage but per Gemini 3
API they must be concrete types: change thoughtSignature to []byte (JSON
base64-encoded), mediaResolution to the existing MediaResolution enum type, and
videoMetadata to the VideoMetadata struct; update struct tags as needed to keep
",omitempty", remove manual unmarshalling for these fields and adjust any
callers/tests that rely on RawMessage to use the concrete types instead.
feat: gemini-3-pro
2 participants
Summary by CodeRabbit
New Features
Chores