fix: pass through gs:// image URLs on Vertex Gemini closes #4402 - #4568
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
🚧 Files skipped from review as they are similar to previous changes (3)
📝 WalkthroughSummary by CodeRabbit
WalkthroughRefactors image URL sanitization to use scheme allowlists, threads them through Gemini request conversion paths, and wires Vertex Gemini-family conversions to allow ChangesGCS image URL scheme allowlist
Sequence Diagram(s)sequenceDiagram
participant Caller
participant vertex.go
participant gemini converter
participant sanitization
Caller->>vertex.go: Gemini request with gs:// image URL
vertex.go->>gemini converter: WithImageURLSchemes(..., [http,https,gs])
gemini converter->>sanitization: sanitize image URL with allowlist
sanitization-->>gemini converter: FileData for gs:// URL
gemini converter-->>vertex.go: Gemini request payload
vertex.go-->>Caller: forwards image block intact
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 golangci-lint (2.12.2)level=error msg="[linters_context] typechecking error: pattern ./...: directory prefix . does not contain main module or its selected dependencies" Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
The merge-base changed after approval.
|
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@core/providers/vertex/vertex.go`:
- Around line 439-443: The issue is that when Content-Type is missing, the code
attempts to call SanitizeImageURL on raw base64-encoded bytes (the encoded
variable), but SanitizeImageURL expects a URL string, not raw bytes. To fix this
fallback media-type branch, instead of sanitizing the encoded bytes as a URL,
construct a proper data URL by prepending a data URL scheme with a default media
type (such as "data:image/jpeg;base64,") to the base64-encoded data before
assigning it to img.URL, which will inline the image correctly without requiring
sanitization.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: f57fa475-4686-4890-b06f-92840d04e75c
📒 Files selected for processing (11)
core/providers/gemini/batch.gocore/providers/gemini/chat.gocore/providers/gemini/gemini_test.gocore/providers/gemini/responses.gocore/providers/gemini/utils.gocore/providers/gemini/videos.gocore/providers/vertex/utils_test.gocore/providers/vertex/vertex.gocore/providers/vertex/vertex_test.gocore/schemas/utils.gocore/schemas/utils_test.go
🚧 Files skipped from review as they are similar to previous changes (8)
- core/providers/gemini/videos.go
- core/providers/gemini/batch.go
- core/providers/vertex/utils_test.go
- core/providers/gemini/utils.go
- core/schemas/utils_test.go
- core/schemas/utils.go
- core/providers/vertex/vertex_test.go
- core/providers/gemini/responses.go
There was a problem hiding this comment.
Caution
Inline review comments failed to post. This is likely due to GitHub's internal server error or limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@core/providers/vertex/vertex.go`:
- Around line 439-443: The issue is that when Content-Type is missing, the code
attempts to call SanitizeImageURL on raw base64-encoded bytes (the encoded
variable), but SanitizeImageURL expects a URL string, not raw bytes. To fix this
fallback media-type branch, instead of sanitizing the encoded bytes as a URL,
construct a proper data URL by prepending a data URL scheme with a default media
type (such as "data:image/jpeg;base64,") to the base64-encoded data before
assigning it to img.URL, which will inline the image correctly without requiring
sanitization.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: f57fa475-4686-4890-b06f-92840d04e75c
📒 Files selected for processing (11)
core/providers/gemini/batch.gocore/providers/gemini/chat.gocore/providers/gemini/gemini_test.gocore/providers/gemini/responses.gocore/providers/gemini/utils.gocore/providers/gemini/videos.gocore/providers/vertex/utils_test.gocore/providers/vertex/vertex.gocore/providers/vertex/vertex_test.gocore/schemas/utils.gocore/schemas/utils_test.go
🚧 Files skipped from review as they are similar to previous changes (8)
- core/providers/gemini/videos.go
- core/providers/gemini/batch.go
- core/providers/vertex/utils_test.go
- core/providers/gemini/utils.go
- core/schemas/utils_test.go
- core/schemas/utils.go
- core/providers/vertex/vertex_test.go
- core/providers/gemini/responses.go
🛑 Comments failed to post (1)
core/providers/vertex/vertex.go (1)
439-443:
⚠️ Potential issue | 🟠 Major | ⚡ Quick winFallback media-type branch currently rejects valid remote images.
At Line 439,
schemas.SanitizeImageURL(encoded)is called with raw base64 bytes (not a URL), so this path errors whenContent-Typeis missing instead of inlining the image.Suggested fix
@@ - } else { - // Content-Type header absent; sniff the media type from the - // fetched bytes so we never emit a malformed "data:;base64,..." - // URI, which Anthropic-on-Vertex rejects. - sanitized, sErr := schemas.SanitizeImageURL(encoded) - if sErr != nil { - return sErr - } - img.URL = sanitized - } + } else { + // Content-Type header absent; sniff media type from bytes and + // still emit a valid data URI. + decoded, decodeErr := base64.StdEncoding.DecodeString(encoded) + if decodeErr != nil { + return fmt.Errorf("failed to decode fetched image for data URI: %w", decodeErr) + } + detectedType := http.DetectContentType(decoded) + img.URL = "data:" + detectedType + ";base64," + encoded + }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@core/providers/vertex/vertex.go` around lines 439 - 443, The issue is that when Content-Type is missing, the code attempts to call SanitizeImageURL on raw base64-encoded bytes (the encoded variable), but SanitizeImageURL expects a URL string, not raw bytes. To fix this fallback media-type branch, instead of sanitizing the encoded bytes as a URL, construct a proper data URL by prepending a data URL scheme with a default media type (such as "data:image/jpeg;base64,") to the base64-encoded data before assigning it to img.URL, which will inline the image correctly without requiring sanitization.
Thanks @coderabbitai, but I don't think this is correct on two counts:
So no fix needed here. |
|
Tip For best results, initiate chat on the files or code changes.
No fix needed here. Thanks for the detailed explanation. 🐇 ✏️ Learnings added
|
Signed-off-by: Akshay Deo <akshay@akshaydeo.com>
… (maximhq#4568) Signed-off-by: Akshay Deo <akshay@akshaydeo.com> Co-authored-by: Akshay Deo <akshay@akshaydeo.com>
3 participants
Summary
Fixes Vertex Gemini support for
gs://image URLs.Vertex AI accepts Google Cloud Storage URIs in Gemini
FileData.fileUri, but Bifrost’s Vertex Geminipath reused Gemini converters that only allowed
http/httpsimage URLs. This causedgs://image blocksto be rejected or dropped before reaching Vertex.
This PR adds explicit provider-level image URL scheme handling so Vertex can opt into
gs://whileGemini keeps its default
http/httpsbehavior.Changes
schemas.SanitizeImageURLWithAllowedSchemesto support provider-specific image URL schemeallowlists.
schemas.SanitizeImageURLrestricted to the defaulthttp/httpsbehavior.ctx-based canonical model resolution in Gemini converters.httphttpsgsskipping image blocks. (This intentionally surfaces invalid image URL inputs instead of silently sending a request with missing image content.)
gs://opt-ings://gs://asFileData.fileUrifile://Type of change
Affected areas
How to test
Expected outcome: all tests pass.
Screenshots/Recordings
N/A. No UI changes.
Breaking changes
Invalid or unsupported image URL schemes in Gemini chat/responses inputs now return an explicit conversion error instead of silently dropping the image block. This may affect callers that relied on the previous partial-success behavior.
Related issues
Fixes #4402
Security considerations
The default image URL sanitizer remains restricted to http/https. Support for additional schemes such as
gs://is explicit and scoped to the Vertex provider path, avoiding accidental broadening of URL handlingbehavior across other providers.
Checklist
docs/contributing/README.mdand followed the guidelines