feat(gemini): implement markdown image handling in text processing - #2346
Merged
Conversation
Contributor
WalkthroughThe Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes
Poem
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
relay/channel/gemini/relay-gemini.go(1 hunks)
🧰 Additional context used
🧠 Learnings (3)
📓 Common learnings
Learnt from: RedwindA
Repo: QuantumNous/new-api PR: 1537
File: relay/gemini_handler.go:330-342
Timestamp: 2025-08-08T17:12:43.157Z
Learning: In the new-api repository, the `GeminiEmbeddingHandler` function in `relay/gemini_handler.go` is designed specifically for native Gemini embedding requests and therefore does not require the `ConvertGeminiRequest` step that is used in the chat handler. The embedding requests are already in the native Gemini format and don't need conversion.
📚 Learning: 2025-08-08T17:12:43.157Z
Learnt from: RedwindA
Repo: QuantumNous/new-api PR: 1537
File: relay/gemini_handler.go:330-342
Timestamp: 2025-08-08T17:12:43.157Z
Learning: In the new-api repository, the `GeminiEmbeddingHandler` function in `relay/gemini_handler.go` is designed specifically for native Gemini embedding requests and therefore does not require the `ConvertGeminiRequest` step that is used in the chat handler. The embedding requests are already in the native Gemini format and don't need conversion.
Applied to files:
relay/channel/gemini/relay-gemini.go
📚 Learning: 2025-06-21T03:37:41.726Z
Learnt from: 9Ninety
Repo: QuantumNous/new-api PR: 1273
File: relay/channel/gemini/relay-gemini.go:97-116
Timestamp: 2025-06-21T03:37:41.726Z
Learning: In relay/channel/gemini/relay-gemini.go, the thinking budget calculation logic (including the MaxOutputTokens multiplication) was introduced in PR #1247. PR #1273 focused specifically on decoupling the thoughts summary feature from thinking budget settings and did not modify the existing thinking budget behavior.
Applied to files:
relay/channel/gemini/relay-gemini.go
🧬 Code graph analysis (1)
relay/channel/gemini/relay-gemini.go (3)
dto/gemini.go (2)
GeminiPart(241-254)GeminiInlineData(183-186)constant/env.go (1)
GeminiVisionMaxImageNum(13-13)service/image.go (1)
DecodeBase64FileData(43-66)
Comment on lines
+449
to
+510
| // check markdown image  | ||
| // 使用字符串查找而非正则，避免大文本性能问题 | ||
| text := part.Text | ||
| hasMarkdownImage := false | ||
| for { | ||
| // 快速检查是否包含 markdown 图片标记 | ||
| startIdx := strings.Index(text, " | ||
| if bracketIdx == -1 { | ||
| break | ||
| } | ||
| bracketIdx += startIdx | ||
| // 找到闭合的 ) | ||
| closeIdx := strings.Index(text[bracketIdx+2:], ")") | ||
| if closeIdx == -1 { | ||
| break | ||
| } | ||
| closeIdx += bracketIdx + 2 | ||
|
|
||
| hasMarkdownImage = true | ||
| // 添加图片前的文本 | ||
| if startIdx > 0 { | ||
| textBefore := text[:startIdx] | ||
| if textBefore != "" { | ||
| parts = append(parts, dto.GeminiPart{ | ||
| Text: textBefore, | ||
| }) | ||
| } | ||
| } | ||
| // 提取 data URL (从 "](" 后面开始，到 ")" 之前) | ||
| dataUrl := text[bracketIdx+2 : closeIdx] | ||
| imageNum += 1 | ||
| if constant.GeminiVisionMaxImageNum != -1 && imageNum > constant.GeminiVisionMaxImageNum { | ||
| return nil, fmt.Errorf("too many images in the message, max allowed is %d", constant.GeminiVisionMaxImageNum) | ||
| } | ||
| format, base64String, err := service.DecodeBase64FileData(dataUrl) | ||
| if err != nil { | ||
| return nil, fmt.Errorf("decode markdown base64 image data failed: %s", err.Error()) | ||
| } | ||
| imgPart := dto.GeminiPart{ | ||
| InlineData: &dto.GeminiInlineData{ | ||
| MimeType: format, | ||
| Data: base64String, | ||
| }, | ||
| } | ||
| if shouldAttachThoughtSignature { | ||
| imgPart.ThoughtSignature = json.RawMessage(strconv.Quote(thoughtSignatureBypassValue)) | ||
| } | ||
| parts = append(parts, imgPart) | ||
| // 继续处理剩余文本 | ||
| text = text[closeIdx+1:] | ||
| } | ||
| // 添加剩余文本或原始文本（如果没有找到 markdown 图片） | ||
| if !hasMarkdownImage { | ||
| parts = append(parts, dto.GeminiPart{ | ||
| Text: part.Text, | ||
| }) | ||
| } |
Contributor
There was a problem hiding this comment.
Trailing text after the last markdown image is dropped
When part.Text contains one or more markdown data‑URL images plus trailing text (e.g. foo  bar), the trailing segment (" bar") is never appended:
- Inside the loop you consume
textand reassigntext = text[closeIdx+1:]. - After the loop, if
hasMarkdownImageistrue, you skip the fallback that appendspart.Textas a whole. - There is no code that appends the remaining
textvalue when at least one image was parsed.
This loses user content and is a functional correctness bug.
You can fix it by appending the leftover text after the loop when hasMarkdownImage is true:
- // 添加剩余文本或原始文本（如果没有找到 markdown 图片）
- if !hasMarkdownImage {
- parts = append(parts, dto.GeminiPart{
- Text: part.Text,
- })
- }
+ // 添加剩余文本或原始文本（如果没有找到 markdown 图片）
+ if hasMarkdownImage {
+ if text != "" {
+ parts = append(parts, dto.GeminiPart{
+ Text: text,
+ })
+ }
+ } else {
+ parts = append(parts, dto.GeminiPart{
+ Text: part.Text,
+ })
+ }This keeps existing behavior when no markdown image is present, while preserving any tail text after the last parsed image.
🤖 Prompt for AI Agents
In relay/channel/gemini/relay-gemini.go around lines 449 to 510 the loop that
extracts markdown data‑URL images consumes and reassigns text but never appends
the trailing remainder when hasMarkdownImage is true, causing tail text to be
lost; after the image-parsing loop, if hasMarkdownImage is true and the
remaining text (variable text) is non-empty, append a dto.GeminiPart with Text:
text to parts (instead of or in addition to the existing fallback that only runs
when hasMarkdownImage is false), so you preserve any trailing text after the
last parsed image.
ennnnny
pushed a commit
to ennnnny/new-api
that referenced
this pull request
Mar 17, 2026
…i-turn feat(gemini): implement markdown image handling in text processing
6 tasks
11 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by CodeRabbit
✏️ Tip: You can customize this high-level summary in your review settings.