fix: gemini parts in tool response - #4202
Merged
Merged
Conversation
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (4)
💤 Files with no reviewable changes (1)
📝 WalkthroughWalkthroughThis PR adds model/provider-aware conversion and type support to preserve multimodal Gemini FunctionResponse parts through Gemini↔Bifrost conversions, and includes unit and integration tests validating image preservation, Vertex $ref behavior, and round-trip integrity. ChangesGemini multimodal function-response support
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 golangci-lint (2.12.2)level=error msg="[linters_context] typechecking error: pattern ./...: directory prefix . does not contain main module or its selected dependencies" Comment |
TejasGhatte
marked this pull request as ready for review
June 9, 2026 10:39
|
tejas ghatte seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
Contributor
Confidence Score: 5/5Core conversion logic is correct, round-trips are well-tested at both unit and integration level, and the only open issue is a documentation inconsistency in an integration test. The forward and reverse conversions handle all documented cases (Gemini 3 image preserved, older model drop, Vertex $ref, media-only empty-output placeholder) correctly, and the json.Valid guard ensures structured JSON payloads survive the round-trip without double-encoding. The snake_case UnmarshalJSON additions are additive and do not change existing serialisation. No regressions were found in the provider converter logic. tests/integrations/python/tests/test_google.py — test_30's parametrize list does not include gemini-2.5-flash despite the docstring claiming the gating is validated there. Important Files Changed
Reviews (2): Last reviewed commit: "fix: gemini parts in tool response" | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@core/providers/gemini/responses.go`:
- Around line 2030-2064: When part.FunctionResponse.Parts is non-empty the code
currently only builds media blocks and omits the original full JSON response
(responseStr), dropping other fields; ensure you always preserve the full
serialized function response by assigning output.ResponsesToolCallOutputStr =
&responseStr even when you also set
output.ResponsesFunctionToolCallOutputBlocks, so update the branch handling
part.FunctionResponse.Parts in convertResponsesMessagesToResponsesToolMessage
(the block creating output := &schemas.ResponsesToolMessageOutputStruct{}) to
always set ResponsesToolCallOutputStr alongside the blocks generated by
convertGeminiInlineDataToContentBlock/convertGeminiFileDataToContentBlock.
In `@tests/integrations/python/tests/test_google.py`:
- Line 1771: The test function signature for
test_30b_multimodal_function_response_full_workflow includes an unused fixture
parameter test_config; remove test_config from the parameter list of the
function definition so the signature becomes (self, model,
expect_image_understood), ensuring no other references to test_config exist
inside the function or its assertions.
- Line 1767: The pytest.mark.parametrize decorator usage is incorrect: change
the first positional argument from a single string to a tuple of parameter names
so the decorator reads pytest.mark.parametrize(("model",
"expect_image_understood"), [...]) (locate the decorator invocation using
pytest.mark.parametrize in this test file and update similarly to the other
tests like test_30 that use the correct tuple form).
- Line 1667: The test function test_30_multimodal_function_response_image
currently declares an unused pytest fixture parameter test_config; remove
test_config from the function signature so the fixture is not injected (update
def test_30_multimodal_function_response_image(self, model,
expect_image_understood):) and run tests to ensure no other references rely on
that parameter.
- Line 1664: The pytest.mark.parametrize decorator is using a comma-separated
string for parameter names which is non-standard; update the decorator on the
test (pytest.mark.parametrize) to pass a tuple of parameter names (e.g.
("model", "expect_image_understood")) as the first argument instead of the
single string "model,expect_image_understood" so linters stop complaining and
tests remain unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 35a04376-103b-4582-b636-78491961e3de
📒 Files selected for processing (4)
core/providers/gemini/gemini_test.gocore/providers/gemini/responses.gocore/providers/gemini/types.gotests/integrations/python/tests/test_google.py
TejasGhatte
force-pushed
the
06-09-fix_gemini_parts_in_tool_response
branch
from
June 9, 2026 12:09
c096655 to
f7dd93e
Compare
Pratham-Mishra04
approved these changes
Jun 9, 2026
Collaborator
Merge activity
|
akshaydeo
pushed a commit
that referenced
this pull request
Jun 12, 2026
## Summary
Adds support for multimodal function responses (images and files returned by tools) in the Gemini provider. Previously, when a tool returned image or file content alongside text, the media was either dropped or serialized as a raw JSON fallback. This PR preserves those media blocks by routing them through `FunctionResponse.Parts` (a Gemini 3+ feature), with correct provider-specific behavior for the Gemini Developer API vs. Vertex AI.
## Changes
- **`FunctionResponse.Parts` field added** to the `FunctionResponse` type so images/files returned by tools can be attached as sibling parts alongside the structured response object.
- **Forward conversion (`convertResponsesMessagesToGeminiContents`)** now accepts `model` and `provider` arguments. For Gemini 3+ models, image/file content blocks from `ResponsesFunctionToolCallOutputBlocks` are converted to `inlineData`/`fileData` parts and attached to `FunctionResponse.Parts`. For older models (e.g. `gemini-2.5-flash`), media is silently dropped to avoid a hard upstream 400. Vertex AI emits a `{"$ref": "<displayName>"}` entry in the response object (as documented); the Gemini Developer API does not (the `$ref` form triggers an upstream bug).
- **Reverse conversion (`convertGeminiContentsToResponsesMessages`)** reconstructs multimodal function responses back into `ResponsesFunctionToolCallOutputBlocks` (text + image blocks), preserving media on the Bifrost side instead of collapsing everything to a plain string.
- **`Part.UnmarshalJSON`** now handles snake_case fallbacks (`inline_data`, `file_data`) emitted by the google-genai SDK inside `functionResponse.parts`.
- **`Blob.UnmarshalJSON`** now handles snake_case fallbacks (`mime_type`, `display_name`) from `FunctionResponseBlob`.
- **`FileData.UnmarshalJSON`** added with snake_case fallbacks (`mime_type`, `file_uri`, `display_name`) from `FunctionResponseFileData`.
- **Unit tests** added for: image preserved on Gemini 3 (Developer API, no `$ref`), image dropped on older models, Vertex emitting `$ref`, and a full round-trip (`GeminiGenerationRequest` → `BifrostResponsesRequest` → `GeminiGenerationRequest`).
- **Integration tests** added (`test_30`, `test_30b`) covering a fabricated multimodal tool history and a real two-turn workflow, parameterized across `gemini-3-flash-preview` (image understood) and `gemini-2.5-flash` (image dropped, request still succeeds).
## Type of change
- [x] Bug fix
- [x] Feature
## Affected areas
- [x] Core (Go)
- [x] Providers/Integrations
## How to test
```sh
# Unit tests
go test ./core/providers/gemini/... -run TestResponsesAPIParallelFunctionCalling
go test ./core/providers/gemini/... -run TestMultimodalFunctionResponse_RoundTrip
# Integration tests (requires GEMINI_API_KEY)
cd tests/integrations/python
pytest tests/test_google.py::TestGoogleProvider::test_30_multimodal_function_response_image
pytest tests/test_google.py::TestGoogleProvider::test_30b_multimodal_function_response_full_workflow
```
Expected outcomes:
- `gemini-3-flash-preview`: model identifies the tool-returned image color as "red".
- `gemini-2.5-flash`: request succeeds without a 400; model produces a text reply (image was dropped by gating).
## Breaking changes
- [x] No
## Security considerations
No new auth, secrets, or PII surface. Base64 image data passes through in-memory only and is not logged or persisted.
## Checklist
- [ ] I read `docs/contributing/README.md` and followed the guidelines
- [x] I added/updated tests where appropriate
- [ ] I updated documentation where needed
- [x] I verified builds succeed (Go and UI)
- [ ] I verified the CI pipeline passes locally if applicable
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
* **New Features**
* Improved multimodal function-response support: tools can return images alongside text for Gemini 3 models; other model types gracefully fall back to text-only or reference-style handling.
* **Compatibility**
* Better handling of provider/model variations so image-containing tool outputs are preserved where supported and safely downgraded otherwise.
* **Tests**
* Added end-to-end and regression tests validating multimodal function-response round‑trips and field preservation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This was referenced Jun 12, 2026
Merged
18 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds support for multimodal function responses (images and files returned by tools) in the Gemini provider. Previously, when a tool returned image or file content alongside text, the media was either dropped or serialized as a raw JSON fallback. This PR preserves those media blocks by routing them through
FunctionResponse.Parts(a Gemini 3+ feature), with correct provider-specific behavior for the Gemini Developer API vs. Vertex AI.Changes
FunctionResponse.Partsfield added to theFunctionResponsetype so images/files returned by tools can be attached as sibling parts alongside the structured response object.convertResponsesMessagesToGeminiContents) now acceptsmodelandproviderarguments. For Gemini 3+ models, image/file content blocks fromResponsesFunctionToolCallOutputBlocksare converted toinlineData/fileDataparts and attached toFunctionResponse.Parts. For older models (e.g.gemini-2.5-flash), media is silently dropped to avoid a hard upstream 400. Vertex AI emits a{"$ref": "<displayName>"}entry in the response object (as documented); the Gemini Developer API does not (the$refform triggers an upstream bug).convertGeminiContentsToResponsesMessages) reconstructs multimodal function responses back intoResponsesFunctionToolCallOutputBlocks(text + image blocks), preserving media on the Bifrost side instead of collapsing everything to a plain string.Part.UnmarshalJSONnow handles snake_case fallbacks (inline_data,file_data) emitted by the google-genai SDK insidefunctionResponse.parts.Blob.UnmarshalJSONnow handles snake_case fallbacks (mime_type,display_name) fromFunctionResponseBlob.FileData.UnmarshalJSONadded with snake_case fallbacks (mime_type,file_uri,display_name) fromFunctionResponseFileData.$ref), image dropped on older models, Vertex emitting$ref, and a full round-trip (GeminiGenerationRequest→BifrostResponsesRequest→GeminiGenerationRequest).test_30,test_30b) covering a fabricated multimodal tool history and a real two-turn workflow, parameterized acrossgemini-3-flash-preview(image understood) andgemini-2.5-flash(image dropped, request still succeeds).Type of change
Affected areas
How to test
Expected outcomes:
gemini-3-flash-preview: model identifies the tool-returned image color as "red".gemini-2.5-flash: request succeeds without a 400; model produces a text reply (image was dropped by gating).Breaking changes
Security considerations
No new auth, secrets, or PII surface. Base64 image data passes through in-memory only and is not logged or persisted.
Checklist
docs/contributing/README.mdand followed the guidelinesSummary by CodeRabbit
New Features
Compatibility
Tests