Skip to content

feat(gemini): implement markdown image handling in text processing #2346

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Calcium-Ion merged 1 commit into from
Dec 1, 2025
Merged

feat(gemini): implement markdown image handling in text processing #2346

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 62 additions & 3 deletions relay/channel/gemini/relay-gemini.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -446,9 +446,68 @@ func CovertOpenAI2Gemini(c *gin.Context, textRequest dto.GeneralOpenAIRequest, i
if part.Text == "" {
continue
}
parts = append(parts, dto.GeminiPart{
Text: part.Text,
})
// check markdown image ![image](data:image/jpeg;base64,xxxxxxxxxxxx)
// 使用字符串查找而非正则，避免大文本性能问题
text := part.Text
hasMarkdownImage := false
for {
// 快速检查是否包含 markdown 图片标记
startIdx := strings.Index(text, "![")
if startIdx == -1 {
break
}
// 找到 ](
bracketIdx := strings.Index(text[startIdx:], "](data:")
if bracketIdx == -1 {
break
}
bracketIdx += startIdx
// 找到闭合的 )
closeIdx := strings.Index(text[bracketIdx+2:], ")")
if closeIdx == -1 {
break
}
closeIdx += bracketIdx + 2

hasMarkdownImage = true
// 添加图片前的文本
if startIdx > 0 {
textBefore := text[:startIdx]
if textBefore != "" {
parts = append(parts, dto.GeminiPart{
Text: textBefore,
})
}
}
// 提取 data URL (从 "](" 后面开始，到 ")" 之前)
dataUrl := text[bracketIdx+2 : closeIdx]
imageNum += 1
if constant.GeminiVisionMaxImageNum != -1 && imageNum > constant.GeminiVisionMaxImageNum {
return nil, fmt.Errorf("too many images in the message, max allowed is %d", constant.GeminiVisionMaxImageNum)
}
format, base64String, err := service.DecodeBase64FileData(dataUrl)
if err != nil {
return nil, fmt.Errorf("decode markdown base64 image data failed: %s", err.Error())
}
imgPart := dto.GeminiPart{
InlineData: &dto.GeminiInlineData{
MimeType: format,
Data: base64String,
},
}
if shouldAttachThoughtSignature {
imgPart.ThoughtSignature = json.RawMessage(strconv.Quote(thoughtSignatureBypassValue))
}
parts = append(parts, imgPart)
// 继续处理剩余文本
text = text[closeIdx+1:]
}
// 添加剩余文本或原始文本（如果没有找到 markdown 图片）
if !hasMarkdownImage {
parts = append(parts, dto.GeminiPart{
Text: part.Text,
})
}
Comment on lines +449 to +510

@coderabbitai coderabbitai Bot Dec 1, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Trailing text after the last markdown image is dropped

When part.Text contains one or more markdown data‑URL images plus trailing text (e.g. foo ![img](data:...) bar), the trailing segment (" bar") is never appended:

  • Inside the loop you consume text and reassign text = text[closeIdx+1:].
  • After the loop, if hasMarkdownImage is true, you skip the fallback that appends part.Text as a whole.
  • There is no code that appends the remaining text value when at least one image was parsed.

This loses user content and is a functional correctness bug.

You can fix it by appending the leftover text after the loop when hasMarkdownImage is true:

-				// 添加剩余文本或原始文本（如果没有找到 markdown 图片）
-				if !hasMarkdownImage {
-					parts = append(parts, dto.GeminiPart{
-						Text: part.Text,
-					})
-				}
+				// 添加剩余文本或原始文本（如果没有找到 markdown 图片）
+				if hasMarkdownImage {
+					if text != "" {
+						parts = append(parts, dto.GeminiPart{
+							Text: text,
+						})
+					}
+				} else {
+					parts = append(parts, dto.GeminiPart{
+						Text: part.Text,
+					})
+				}

This keeps existing behavior when no markdown image is present, while preserving any tail text after the last parsed image.

🤖 Prompt for AI Agents 
In relay/channel/gemini/relay-gemini.go around lines 449 to 510 the loop that
extracts markdown data‑URL images consumes and reassigns text but never appends
the trailing remainder when hasMarkdownImage is true, causing tail text to be
lost; after the image-parsing loop, if hasMarkdownImage is true and the
remaining text (variable text) is non-empty, append a dto.GeminiPart with Text:
text to parts (instead of or in addition to the existing fallback that only runs
when hasMarkdownImage is false), so you preserve any trailing text after the
last parsed image.

} else if part.Type == dto.ContentTypeImageURL {
imageNum += 1

Expand Down