Fix: Preserve Gemini tool-output media parts

[vypxl]

Summary

At the moment, image tool call results are dropped when using Gemini. This makes the model effectively blind when using something akin to a screenshot tool, etc.

This is critical for us, so we've been running a modified version of bifrost. Would love to see this fixed in the next release so we can switch back to upstream.

Changes

Preserve Gemini functionResponse.parts media for tool outputs across Chat, Responses, streaming, and native GenAI paths.

Type of change

• Bug fix
• Feature
• Refactor
• Documentation
• Chore/CI

Affected areas

• Core (Go)
• Transports (HTTP)
• Providers/Integrations
• Plugins
• UI (React)
• Docs

How to test

  cd core
  go test ./providers/gemini -run 'TestGemini(ResponsesToolOutputMediaParts|ResponsesToolOutputMediaOnlyDoesNotDuplicateRawBlocks|ChatToolOutputMediaParts|NativeFunctionResponsePartsPreserved|NativeFunctionResponsePartsPreservesWholeNonOutputResponse|NativeFunctionResponseMediaOnlyDoesNotEmitEmptyResponseText)'

I wrote a standalone test script for an e2e test (current upstream version does not pass this):

Details

const BASE = process.env.BIFROST_BASE_URL;
const KEY = process.env.BIFROST_API_KEY;
const MODEL = process.env.GEMINI_MODEL || "gemini-3.1-flash-lite";
const RED_PNG =
  "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAADElEQVR4nGP4z8AAAAMBAQDJ/pLvAAAAAElFTkSuQmCC";

if (!KEY) throw Error("missing BIFROST_API_KEY");

const tool = {
  type: "function",
  name: "capture_screen",
  description: "Capture current screen.",
  parameters: { type: "object", properties: {}, additionalProperties: false },
};

const user = {
  role: "user",
  content: [
    {
      type: "input_text",
      text: "Call capture_screen now. After tool result, inspect attached screenshot and return exactly its single dominant color as one lowercase word.",
    },
  ],
};

const headers = {
  Authorization: `Bearer ${KEY}`,
  "x-model-provider": "gemini",
};

const preflight = await post("/v1/responses", {
  model: `gemini/${MODEL}`,
  input: [user],
  tools: [tool],
  tool_choice: { type: "function", name: "capture_screen" },
  max_output_tokens: 64,
  temperature: 0,
  store: false,
});

const call = preflight.output?.find(
  (x) => x.type === "function_call" && x.name === "capture_screen",
);
if (!call?.call_id)
  throw Error(`no function_call: ${JSON.stringify(preflight).slice(0, 1000)}`);

const response = await post("/v1/responses", {
  model: `gemini/${MODEL}`,
  input: [
    user,
    call,
    {
      type: "function_call_output",
      call_id: call.call_id,
      name: "capture_screen",
      output: [
        {
          type: "input_text",
          text: "Screenshot captured. Inspect attached image.",
        },
        { type: "input_image", image_url: `data:image/png;base64,${RED_PNG}` },
      ],
    },
  ],
  tools: [tool],
  max_output_tokens: 64,
  temperature: 0,
  store: false,
});

const text = JSON.stringify(response).toLowerCase();
if (response.error) throw Error(`error: ${JSON.stringify(response.error)}`);
if (!Array.isArray(response.output))
  throw Error(`no output array: ${JSON.stringify(response).slice(0, 1000)}`);
if (!text.includes("red"))
  throw Error(`expected red, got: ${text.slice(0, 1000)}`);

console.log("PASS gemini tool output media blocks");

async function post(path, body) {
  const r = await fetch(`${BASE}${path}`, {
    method: "POST",
    headers: { "content-type": "application/json", ...headers },
    body: JSON.stringify(body),
  });
  const json = await r.json().catch(async () => ({ raw: await r.text() }));
  if (!r.ok)
    throw Error(`HTTP ${r.status}: ${JSON.stringify(json).slice(0, 1000)}`);
  return json;
}

Screenshots/Recordings

N/A

Breaking changes

• Yes
• No

Related issues

N/A

Security considerations

N/A

Checklist

• I read contributing PR guidelines and followed them
• I added/updated tests where appropriate
• I updated documentation where needed
• I verified builds succeed (Go and UI)
• I verified the CI pipeline passes locally if applicable

Summary by CodeRabbit

• New Features

  • Enhanced Gemini integration to properly handle tool/function outputs containing both text and image content blocks, enabling better multimodal response support.

• Tests

  • Added comprehensive unit tests validating conversion of mixed text and image tool outputs.

• Chores

  • Updated build dependencies.

[CLAassistant]

All committers have signed the CLA.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 22825015-67ff-4bcf-a718-94b9d62ebaa0

📥 Commits

Reviewing files that changed from the base of the PR and between 995bd28 and e6d62fe.

📒 Files selected for processing (4)

• core/providers/gemini/gemini_test.go
• core/providers/gemini/responses.go
• core/providers/gemini/types.go
• core/providers/gemini/utils.go

💤 Files with no reviewable changes (4)

• core/providers/gemini/utils.go
• core/providers/gemini/types.go
• core/providers/gemini/gemini_test.go
• core/providers/gemini/responses.go

---

📝 Walkthrough

Walkthrough

Adds structured Gemini FunctionResponse parts support: new types and unmarshalling for function_response/parts, utilities to collect/convert mixed text+media blocks, bidirectional conversion preserving Parts, tests for mixed/media-only/native flows, and two Nix hash updates.

Changes

Gemini provider media support

Layer / File(s)	Summary
Type system and unmarshaling foundation core/providers/gemini/types.go	Part, Blob, and FileData accept camelCase and snake_case variants; FunctionResponse now holds Parts []*FunctionResponsePart; new FunctionResponsePart parses inlineData/fileData.
Content block conversion utilities core/providers/gemini/utils.go	Helpers convert schemas.ChatContentBlock/ResponsesMessageContentBlock ↔ Gemini Part/FunctionResponsePart, collect text vs media parts, and build response payload maps.
Bidirectional response conversion implementation core/providers/gemini/responses.go	Adds convertGeminiFunctionResponseToToolOutput and functionResponseHasEmptyResponse; streaming, non-stream, candidate-level, and Bifrost→Gemini conversions now preserve FunctionResponse.Parts and build structured payloads.
Media handling test coverage core/providers/gemini/gemini_test.go	Seven tests validate mixed text+image, media-only, Chat API, native Gemini→Bifrost reconstructions, and regression for JSON-object text payload preservation.

Build dependency updates

Layer / File(s)	Summary
Nix vendoring and package hashes nix/packages/bifrost-http.nix, nix/packages/bifrost-ui.nix	Updated vendorHash and npmDepsHash values to repin Go vendoring and npm dependency sets.

Sequence Diagram(s)

sequenceDiagram
  participant Gemini as Gemini.FunctionResponse
  participant Converter as convertGeminiFunctionResponseToToolOutput
  participant Bifrost as ResponsesToolMessage.Output
  participant Builder as convertResponsesMessagesToGeminiContents
  Gemini->>Converter: provide FunctionResponse (Response + Parts)
  Converter->>Bifrost: produce ResponsesFunctionToolCallOutputBlocks or ResponsesToolCallOutputStr
  Bifrost->>Builder: when sending → include FunctionResponse.Parts in emitted Gemini FunctionResponse

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related issues

• [Bug]: incorrect npm-deps hash in nix package for bifrost-ui #2192 — The updated npmDepsHash in nix/packages/bifrost-ui.nix matches the issue about an incorrect npm-deps hash.

Possibly related PRs

• maximhq/bifrost#3761 — Overlapping Gemini test/assertion adjustments and output shaping for function responses.
• maximhq/bifrost#3630 — Related changes to Gemini tool/function-output conversion paths and utilities.

Suggested reviewers

• akshaydeo
• danpiths

Poem

🐰 Soft paws tap keys at break of dawn,

Images and JSON tucked where they belong,
Parts and payloads dance in tidy rows,
Tests nibble crumbs to check each flow,
A rabbit cheers: conversions hum along.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 34.48% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely identifies the main change: preserving Gemini tool-output media parts, which aligns with the bug fix objective described in the PR.
Description check	✅ Passed	The description includes all key sections: a clear summary of the problem (image tool results being dropped), the changes made, type of change (bug fix), affected areas, testing instructions with specific test commands, and a comprehensive e2e test script.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[greptile-apps]

Confidence Score: 5/5

Safe to merge — the change is a pure provider converter fix, all new helpers are transformation-only functions, and the bug being fixed (dropped media in tool outputs) is well-exercised by the new tests.

The logic in convertGeminiFunctionResponseToToolOutput and buildResponsesFunctionResponsePayloadFromBlocks is correct across all traced scenarios: empty response with media, non-empty response with media, text+media, and media-only. The functionResponseHasEmptyResponse guard correctly suppresses spurious empty text blocks. The decodeTextJSON=true flag used in ToGeminiResponsesResponse matches the other conversion path, resolving the JSON double-encoding concern flagged in a previous review. The streaming path shares a single outputStruct pointer between the in-progress and completed events, which is consistent with the previous pattern of sharing a string pointer. All converters remain pure functions with no side effects.

No files require special attention.

Important Files Changed

Filename	Overview
core/providers/gemini/types.go	Adds FunctionResponse.Parts field, new FunctionResponsePart type, and snake_case unmarshalers for Blob, FileData, and FunctionResponsePart to support native GenAI tool-output media in round-trips; also adds function_response snake_case alias to Part.UnmarshalJSON.
core/providers/gemini/responses.go	Refactors function-response-to-tool-output conversion across Chat, Responses, streaming, and native GenAI paths to use the new convertGeminiFunctionResponseToToolOutput helper and buildResponsesFunctionResponsePayloadFromBlocks, preserving media parts where they exist.
core/providers/gemini/utils.go	Replaces the text-only block extraction loop with collectFunctionResponseBlockOutput (generic), adds convertChatContentBlockToGeminiPart, convertGeminiPartToFunctionResponsePart, and chatContentBlock* adapters; media parts in Chat tool responses are now threaded through to FunctionResponse.Parts.
core/providers/gemini/gemini_test.go	Adds six new unit tests covering Responses API, Chat API, and native GenAI paths for tool-output media parts, including edge cases for image-only outputs and JSON-object preservation.

_{Reviews (4): Last reviewed commit: "fix: gemini tool call media outputs" | Re-trigger Greptile}

The merge-base changed after approval.

[vypxl]

Hi,
I've rebased this again. Is there something preventing this from being merged that I could fix?
We would love this to ship soon so that we can go back to using upstream instead of a fork.

[vypxl]

Seems like this was superceeded by #4202 ?
Please confirm @TejasGhatte